OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding

🎯 Pitch

OpenVoxel achieves state-of-the-art 3D referring expression segmentation without any training, outperforming approaches that require human-annotated captions and learned embeddings by over 13 mIoU points. It accomplishes this by transforming open-vocabulary queries into a pure text-to-text search over explicit object captions generated by a VLM, bypassing the limitations of CLIP-style embedding alignment entirely.

1. Executive Summary

This paper proposes OpenVoxel, a training-free algorithm for open-vocabulary 3D scene understanding that operates on sparse voxel representations obtained from multi-view images. Rather than learning language embeddings aligned to CLIP or BERT, OpenVoxel constructs an informative scene map by grouping voxels into object-level instances—using a Training-Free Sparse Voxel Grouping process that lifts and matches per-frame SAM2 segmentation masks to 3D via spatial centroid voting (e.g., projecting 2D instance masks across views and accumulating 3D position votes per voxel)—and then captioning each group through a Canonical Scene Map Construction pipeline that leverages Vision Language Models and Multi-modal Large Language Models to produce structured, human-readable descriptions. On the Ref-LeRF referring expression segmentation benchmark, OpenVoxel achieves 42.4 mIoU, outperforming ReferSplat by 13.2 percentage points (29.2 → 42.4) while requiring no human-annotated description–mask pairs and completing in ~3 minutes per scene versus over one hour for training-based methods, establishing that text-to-text retrieval over explicit captions can surpass learned embedding alignment for complex language queries, but only when the underlying voxel grouping correctly partitions objects into semantically coherent instances that a VLM can describe.

2. Context and Motivation

The fundamental challenge this paper addresses is deceptively simple: given a 3D reconstruction of a scene built from multi-view images, how do you make that representation respond to arbitrary natural language queries without any additional training? A user should be able to ask "which object can be used for cutting paper?" or "find the funny toy with spindly legs" and receive a precise 3D segmentation mask pointing to the correct object—even though the original 3D reconstruction was built purely for rendering, not for understanding.

This matters because the field of neural rendering has made extraordinary progress in the past five years. We can now reconstruct photorealistic 3D scenes from casual phone captures (NeRF in 2021, 3D Gaussian Splatting in 2023, Sparse Voxel Rasterization in 2025), but these representations are fundamentally geometric and photometric—they know where surfaces are and what color they reflect, but they have no concept of what objects those surfaces belong to or what semantic properties they possess. A reconstructed apple is just a collection of voxels with red-ish colors at some 3D coordinates; the system has no internal representation that says "this cluster of voxels is an apple."

Bridging this gap between 3D geometry and semantic understanding is not merely an academic exercise. Several downstream applications depend directly on it:

3D scene editing: If you want to remove the green chair from a reconstructed living room, you need to identify which primitives (voxels, Gaussians, or neural samples) belong to that chair. Without semantic grouping, editing requires laborious manual selection or painting in 3D space.
Augmented reality and robotics: An AR headset or robot navigating a reconstructed environment needs to answer queries like "where is the nearest exit sign?" or "hand me the blue mug on the counter." These require both object identification and spatial reasoning over natural language.
Accessibility: Describing 3D environments to visually impaired users requires the system to understand what objects are present and where they are relative to each other—exactly the kind of scene-level semantic understanding that pure geometry lacks.

The theoretical significance is equally compelling: it probes the question of whether explicit, symbolic scene understanding (grouping, naming, describing) can outperform implicit, embedded understanding (distilling CLIP features into per-primitive vectors). This is a live debate in the vision-language community, and 3D serves as an especially demanding testbed because objects must be understood from multiple viewpoints with consistent identity.

The problem is further sharpened by the Sparse Voxel Rasterization (SVR) representation that the paper builds on. SVR produces a discrete set of voxels—sparse 3D grid cells with color and opacity attributes—that are inherently well-suited for assigning auxiliary per-primitive information. Unlike NeRF's continuous fields or 3DGS's overlapping anisotropic Gaussians, voxels have well-defined spatial extents and a fixed topology, making them a natural substrate for segmentation and tagging. The paper exploits this property, but the underlying challenge—making any 3D primitive understand language—is representation-agnostic.

Prior Approaches: Embedding Distillation and Its Discontents

The dominant paradigm for giving 3D representations language capabilities has been embedding distillation. The recipe, introduced by works like LangSplat (Qin et al., 2024), proceeds as follows:

Reconstruct the 3D scene using your chosen primitive (e.g., 3DGS, NeRF).
Obtain 2D language-aligned features for each training image. This typically means running a CLIP or DINO model over the images to produce a per-pixel feature vector embedded in a semantically meaningful space (e.g., the CLIP vision-language joint embedding).
Distill these 2D features into the 3D primitives through rendering-based training. For each 3D primitive (Gaussian, voxel, etc.), you add trainable feature parameters and optimize them so that, when rendered from the training viewpoints, the projected feature maps match the 2D CLIP/DINO features. This is essentially training a mini neural field on top of the geometry.
At inference, a text query is encoded into the same embedding space (via CLIP text encoder), and primitives whose learned features are closest to the query embedding are selected.

This approach is elegant in its conceptual simplicity—it leverages the alignment between vision and language that CLIP learned from 400M image-text pairs, and it extends that alignment into 3D through differentiable rendering. LangSplat, OpenGaussian, LEGaussians, 3DVLGS, and many others (the paper's Section 2 lists over a dozen) follow variants of this recipe.

However, this paradigm suffers from several structural limitations that the paper identifies as motivating its alternative approach.

Limitation 1: The Embedding Bottleneck Constrains Query Expressivity

CLIP embeddings—and their successors like DINOv2 or SigLIP—are trained to align images with their short textual descriptions, typically single sentences or even just category labels. The embedding space is excellent at capturing coarse semantic similarity: "apple" and "red fruit" will be close together. But it struggles with compositional, relational, or nuanced queries.

Consider the query "the chair that the sheep is sitting on, nearest to the sheep and the camera." This requires:

Identifying multiple objects (sheep, chair)
Understanding a spatial relationship (sitting on)
Performing relative distance reasoning (nearest to)
Combining these into a single target selection

A CLIP embedding of this sentence collapses all of this into a single 512-dimensional vector. The compositional semantics—who is sitting on what, which one is closest—are lost in the embedding. The paper cites Kamath et al. (2024), who showed that vision-language models struggle with compositional queries precisely because the embedding loses structural information.

This is why, as the paper notes in Section 1, prior methods "are limited by learning short words or tags, which hinders advanced 3D scene understanding with complex sentence queries such as the referring segmentation task." The Referring Expression Segmentation (RES) task—where queries are full natural language sentences with attributes, affordances, and spatial relations—directly exposes the weakness of the embedding paradigm. Simple open-vocabulary segmentation (OVS), where queries are single category labels like "apple" or "chair," is much more forgiving because the CLIP embedding of "apple" is quite reliable.

Limitation 2: Training Requirements Create a Per-Scene Burden

Every embedding-based method requires training on each new scene. The per-primitive feature vectors are not transferable—they must be distilled from the specific images of the specific scene being reconstructed. This means:

Time cost: The paper reports that ReferSplat training takes "at least 2 hours per scene" in their reproductions (Table 5), and ObjectGS requires approximately 40 minutes. These are not one-time costs; they recur for every new environment.
Compute cost: Training requires GPU resources and gradient-based optimization, making it impractical for real-time or on-device applications where a user captures a scene and expects immediate semantic understanding.
No generalization: A model trained on one living room cannot be applied to a different living room without retraining. Each scene requires its own optimization loop.

This per-scene training creates a fundamental mismatch with the deployment scenarios that 3D understanding aims to enable. If the goal is to let anyone with a smartphone capture their environment and immediately query it, a process that takes 40+ minutes of GPU training is a non-starter.

Limitation 3: Human Annotation Dependency (The ReferSplat Case)

The most ambitious prior work, ReferSplat (He et al., 2025), attempts to tackle the RES task head-on—aligning 3D representations with full-sentence queries. But to do so, it requires human-annotated description–mask pairs for each training scene. Specifically, as described in Section 1 and Figure 1, ReferSplat needs:

2D object masks identifying each observable object in the training views
Natural language sentence descriptions for each object, written by humans

This annotation pipeline is expensive and non-scalable. For every new scene, a human must:

Identify all observable objects
Draw precise per-object segmentation masks
Write detailed natural language descriptions

The paper's Figure 1 visually contrasts this: ReferSplat's pipeline shows "2D masks + human annotated caption" as inputs, while OpenVoxel's pipeline shows only "2D masks." This annotation requirement makes ReferSplat closer to a supervised learning method than an open-vocabulary system—it only works well on objects it was explicitly trained to describe, a limitation the paper demonstrates in Figure 10 where ReferSplat fails on queries for objects not in its training annotations.

Limitation 4: The Inherent Ambiguity of Per-Primitive Embeddings

A subtler but fundamental issue with the embedding distillation approach is that it operates at the primitive level, not the object level. In 3DGS, for example, each Gaussian has its own language feature vector, and during inference, Gaussians whose features match the query are selected. But a single object might be represented by thousands of Gaussians, and there is no mechanism to enforce that all Gaussians belonging to the same object have consistent features. Works like Gaussian Grouping (Ye et al., 2024) and ObjectGS (Zhu et al., 2025) attempt to address this by jointly training object assignment and language features, but this adds yet another training objective and increases optimization time.

More critically, a per-primitive embedding for a complex object can only capture a local view of that object. A Gaussian on the back of a chair might "see" a different visual context than one on the chair's seat, and the distilled features may diverge. This leads to fragmented segmentation—some primitives match the query while others on the same object do not. Qualitative results from prior methods often show "speckle" patterns where only part of a target object is segmented because individual primitives failed to develop consistent features.

How This Paper Positions Itself: Text-to-Text Over Embedding-to-Embedding

OpenVoxel's central positioning move is a paradigm shift from embedding-space matching to text-space matching. Rather than train 3D primitives to align with a fixed embedding manifold (CLIP, BERT), the paper asks: what if we just describe each object in plain English and then use an LLM to match the user's query against those descriptions?

This reframing has several conceptual consequences:

It eliminates training entirely. The pipeline is "training-free" and "gradient-descent-free" (Section 4). The SVR model is pre-trained purely for rendering, and all subsequent steps—grouping, captioning, retrieval—use frozen, off-the-shelf models (SAM2, DAM, Qwen3-VL) without any fine-tuning or optimization on the target scene.

It replaces latent-space proximity with explicit reasoning. Rather than computing cosine similarity between a query embedding and a primitive feature vector, OpenVoxel feeds the query and a structured scene map (list of objects with captions and 3D positions) into an MLLM and asks it to select the best match. The MLLM can reason about attributes ("yellow with cartoonish looking"), affordances ("can be used for cutting"), spatial relations ("next to the apple"), and even resolve ambiguities by cross-referencing the scene map. This is a symbolic, interpretable operation—the MLLM's selection is explicitly linked to the captions it compared, not a number emerging from a black-box dot product.

It produces human-readable scene understanding. The intermediate output is a scene map that a person can read and verify: a list of objects with captions like "Apple, light green with fluffy material and a prominent, dark green leaf attached to its stem, placed on table." This is valuable in its own right for accessibility, debugging, and explainability. If the system makes an error, you can look at the caption and see whether the description was wrong (captioning failure), whether the grouping merged two objects (grouping failure), or whether the MLLM selected the wrong match (retrieval failure). In embedding-based methods, errors manifest as mysterious vector-space distances with no interpretable explanation.

It naturally handles compound and compositional queries. Because retrieval is delegated to an MLLM, the system inherits the MLLM's capacity for compositional understanding. A query like "the drinking utensil with a smooth surface near a sake bottle" requires understanding that "drinking utensil" could refer to a cup or glass, that "near" means spatial proximity, and that "sake bottle" must be identified as a reference object. The MLLM, having been trained on massive text corpora, can decompose this query and match it against the structured scene descriptions—something a single CLIP embedding vector simply cannot do.

The paper explicitly contrasts these advantages with the "limiting" nature of learned embeddings, quoting the observation that embedding-based methods are "limited by learning short words or tags" and that their approach "bypasses the speed and accuracy limitations of learning text embedding space" (Section 1).

The Enabling Assumptions and Why They Work Now

OpenVoxel's approach depends critically on recent advances in three foundation model families:

SAM2 (Segment Anything Model 2) provides high-quality, per-frame 2D instance segmentation masks without any per-scene training or annotation. The paper uses SAM2 in two ways: first, to generate initial per-view segmentation masks that are lifted to 3D; second, to re-prompt and merge fragmented masks during the progressive grouping process (described in detail in Appendix A, Figure 7). This is what makes training-free grouping possible—the 2D "grounding" comes from a pre-trained model, not from scene-specific optimization.

DAM (Describe Anything Model) provides detailed, free-form captions for masked image regions. This model is specifically designed for localized image and video captioning, outputting sentences like "A green round object, possibly an apple, with a dark stem and leaf attached, sitting on a wooden surface." The paper uses DAM as a starting point, then refines its output through an MLLM canonicalization step.

Qwen3-VL-8B-Instruct serves as the multi-modal reasoning engine. The paper uses it for three distinct purposes: (1) canonicalizing DAM's free-form captions into a structured template, (2) refining user queries into the same structured format, and (3) performing the actual text-to-text retrieval by matching the refined query against the scene map. The fact that a single model can handle all three tasks—image-conditioned caption refinement, text-only query reformulation, and structured information retrieval—is a testament to the versatility of modern instruction-tuned VLMs.

These models did not exist when LangSplat was published in 2024. SAM2 and DAM both appeared in 2024-2025, and the Qwen3-VL family emerged in 2025. OpenVoxel is, in this sense, a product of its moment: it combines foundation models that are individually powerful but were not designed to work together, creating a pipeline whose components are stronger than any single end-to-end trained system.

The Specific Gaps OpenVoxel Targets

Reading the paper's introduction and related work against the backdrop of the field, four concrete gaps emerge that OpenVoxel explicitly fills:

Gap 1: No training-free method for 3D referring expression segmentation. Prior work either handled simple category queries (OVS) with embedding methods, or required substantial human annotation and per-scene training for sentence queries (RES). OpenVoxel is the first to achieve strong RES performance without either.

Gap 2: No scene-level captioning for 3D primitive representations. While 2D image captioning is mature, extending it to 3D—where objects must be consistently identified across views and described in view-independent terms—had not been demonstrated with sparse voxels. The Canonical Scene Map Construction pipeline provides this capability as an intermediate product, not just as a means to retrieval.

Gap 3: No text-to-text retrieval pipeline for 3D queries. Prior work universally used embedding-space nearest-neighbor search for 3D open-vocabulary tasks. The paper's Referring Query Inference stage is, to the authors' knowledge, the first to use an MLLM performing explicit text comparison over a structured scene map for 3D segmentation.

Gap 4: No framework that separates grouping from captioning from retrieval as modular, model-agnostic stages. Existing methods tightly couple these functions: the object grouping is trained jointly with the language features, making it impossible to swap out the segmentation model (e.g., upgrading from SAM to SAM2) or the language model (e.g., from CLIP to SigLIP to an MLLM) without retraining the entire pipeline. OpenVoxel's three-stage architecture—Training-Free Sparse Voxel Grouping → Canonical Scene Map Construction → Referring Query Inference—is deliberately modular. The ablation studies in Appendix C explicitly test different segmentation models (SAM vs. SAM2, Table 7), different captioning models (Osprey, Qwen3-VL, DAM, Table 8), and different MLLMs (Qwen2.5-VL-7B, Qwen3-VL-2B/4B/8B, Table 9), demonstrating that each component can be upgraded independently.

This modularity is philosophically significant: it treats 3D scene understanding not as a monolithic optimization problem but as a composition of capabilities—segment, describe, retrieve—each of which can be addressed by the best available foundation model for that subtask.

Reconciling the Paradigm Shift: Why Hasn't This Been Done Before?

A natural question is: if text-to-text retrieval over captions is so effective, why did the field spend years building embedding distillation pipelines? The paper doesn't explicitly answer this, but the implicit answer lies in the timing of foundation model availability:

Before 2023: CLIP was the gold standard for vision-language alignment. There were no open-source MLLMs capable of reliable image-conditioned captioning or structured retrieval. The only path to language-aware 3D was to piggyback on CLIP's embedding space.
2023-2024: 3DGS emerged as the preferred 3D primitive, and LangSplat established the embedding distillation recipe. SAM (v1) enabled some training-free segmentation, but its per-frame masks were not temporally consistent across views, making cross-view matching unreliable.
2024-2025: SAM2 introduced video-level segmentation with persistent object IDs, solving the cross-view consistency problem. DAM provided mask-conditioned captioning at a level of detail previously unavailable. Qwen3-VL and similar models reached the capability threshold where they could reliably canonicalize and compare structured text descriptions.

OpenVoxel is therefore not a method that could have been proposed in 2023—it requires the foundation model ecosystem to have matured to a point where each link in the pipeline is reliable enough that the chain doesn't break. The paper's contribution is recognizing that this threshold has been crossed and engineering the pipeline that connects these models into a coherent 3D understanding system.

The Central Research Question

With this context established, the paper's central research question can be stated precisely: Can a training-free pipeline—composed entirely of frozen foundation models for 2D segmentation, mask-conditioned captioning, and multi-modal reasoning—produce open-vocabulary 3D scene understanding that matches or exceeds training-based methods that learn per-scene language embeddings, particularly on complex referring expressions that require compositional reasoning?

The remainder of the paper—the grouping algorithm, the canonical captioning procedure, the retrieval mechanism, and the extensive evaluations—is structured to answer this question affirmatively and to characterize the conditions under which the answer holds.

3. Technical Approach

3.1 Reader Orientation

The paper builds a training-free pipeline that takes a pre-constructed 3D scene (represented as sparse voxels from multi-view images) and produces a structured, queryable map of the objects in that scene—identifying each object, describing it in rich natural language, and enabling retrieval of objects via arbitrary text queries ranging from single words to complex referring expressions. The system solves the problem of making a purely geometric 3D reconstruction "understand" language without learning any per-scene language embeddings, instead relying entirely on frozen foundation models for segmentation, captioning, and multi-modal reasoning.

3.2 Big-Picture Architecture (Diagram in Words)

The system has three sequentially connected major components, each feeding into the next:

Training-Free Sparse Voxel Grouping (Section 4.1) — ingests the pre-trained SVR voxel model ( $V_{1:N}$ ) and per-view SAM2 segmentation masks ( $M_{1:K}$ ) across $K$ camera viewpoints, and outputs a group field ( $\mathcal{F}_{1:N}$ ) that assigns each of the $N$ voxels to an object instance ID by having voxels "vote" for their instance centroid via accumulated 3D spatial features.
Canonical Scene Map Construction (Section 4.2) — takes the group field and the original images, renders per-group masks, feeds masked images to the Describe Anything Model (DAM) for initial captions, canonicalizes those captions into a fixed template using Qwen3-VL, and produces a scene map ( $S$ ) that stores each group's ID, 3D centroid position, and structured natural-language caption.
Referring Query Inference (Section 4.3) — takes a user's natural language query $D$ and an optional query image, canonicalizes the query into the same structured template, and performs text-to-text retrieval using Qwen3-VL to match the refined query against the captions stored in $S$ , returning the matching group ID(s) and rendering the corresponding binary segmentation mask for the target viewpoint.

Information flows strictly forward: camera images and poses → SVR reconstruction → SAM2 masks → group field → per-group renders → DAM captions → canonical captions → scene map → query refinement → MLLM retrieval → output mask. There is no feedback loop, no gradient-based optimization, and no training step anywhere in the pipeline.

3.3 Roadmap for the Deep Dive

First, the Training-Free Sparse Voxel Grouping mechanism (Section 4.1), because it is the foundation that transforms raw voxels into object instances—the entire downstream pipeline depends on its quality. I will explain the group representation, how 2D masks are lifted to 3D, how views are progressively matched and merged, and the key design choices (centroid voting, IoU matching, SAM2 re-prompting).
Second, the Canonical Scene Map Construction (Section 4.2), because it builds on the group field to produce the structured text descriptions that enable text-to-text retrieval. I will explain the two-stage captioning process (DAM free-form → Qwen3-VL canonicalization), the visual prompting strategy, and the canonical template format.
Third, the Referring Query Inference (Section 4.3), because it completes the pipeline by showing how queries are refined and matched against the scene map. I will explain the query canonicalization step, the text-to-text retrieval mechanism, and how spatial relations in queries are handled.
Fourth, a synthesis of design choices and hyperparameter rationale that cuts across all three sections, explaining why specific decisions (e.g., 3D centroid features vs. high-dimensional embeddings, progressive matching vs. joint optimization, canonical template format) were made and what alternatives they avoid.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and pipeline paper whose core idea is that explicit object grouping, captioning, and text-to-text retrieval can replace learned embedding alignment for 3D open-vocabulary understanding, provided that the foundation models for segmentation, captioning, and multi-modal reasoning have reached sufficient capability.

Training-Free Sparse Voxel Grouping: From Voxels to Object Instances

The Core Insight: Voxels Vote for Their Instance Centroid

The grouping algorithm is built on a single geometric intuition: voxels belonging to the same object instance should, when viewed from multiple camera viewpoints, project to 2D pixels that belong to the same SAM2 segmentation mask. If a voxel consistently contributes to pixels labeled as "object 3" across many views, then that voxel is part of object 3. The key insight is that this consistency can be captured by having each voxel accumulate 3D position votes toward the centroid of the instance it projects to, rather than learning abstract feature embeddings.

This is explicitly inspired by Deep Hough Voting (Qi et al., 2019) and spatial embedding approaches (Neven et al., 2019), as acknowledged in Section 4.1. The paper adapts the voting concept to the sparse voxel setting by making the "vote" a simple 3D coordinate—the predicted centroid of the instance—rather than a high-dimensional feature vector. This is the critical simplification that makes the approach training-free and efficient: a 3-dimensional feature can be accumulated by direct weighted averaging without any learned parameters, whereas high-dimensional features require gradient-based optimization to converge to meaningful representations.

Group Representation: Three Tensors Track Instance Membership

The algorithm maintains three data structures that evolve as each new camera view is processed:

**Group Feature $\mathcal{F}_{1:N} \in \mathbb{R}^{N \times 3} $:** A tensor storing, for each of the `$ N$` voxels, a 3D position vector that represents the predicted centroid of the object instance that voxel belongs to. The initial value is zero, and it accumulates weighted votes from each processed view. The fact that this is only 3-dimensional (as opposed to the 32+ dimensional features in Gaussian Grouping or ObjectGS) is what enables the training-free, single-pass update.

**Feature Weight $W_{1:N} \in \mathbb{R}^{N} $:** A scalar per voxel indicating the **cumulative blending weight** that has contributed to the feature. This serves as a confidence measure—voxels that contribute heavily to many views accumulate higher weights, and their normalized features (`$ \mathcal{F}_i / W_i$`) are more reliable.

Group Dictionary $G$ : A running record mapping each unique instance ID to its centroid position in 3D world space. This dictionary grows as new instances are discovered in new views and is used to match incoming 2D masks to existing instances via nearest-centroid lookup.

The paper notes (Section 4.1) that "even if the voxels are assigned to an incorrect group in some view, the accumulated $\mathcal{F}_{1:N}$ and $W_{1:N}$ still force the voxel to 'vote' the most confident group it belongs, reducing the possible miss-grouping." This is a crucial property: the voting mechanism is inherently robust to occasional view-level errors because the cumulative weighted average converges to the true centroid as more views are processed, assuming errors are not systematically biased.

Lifting 2D Masks to 3D: Computing and Accumulating Instance Centroids

For each camera view with pose $\xi_t$ , the system first obtains a 2D instance segmentation mask $M_t$ from SAM2, which assigns an integer instance ID to each pixel (0 for background). The mask contains $m$ unique instance IDs. The goal is to transfer this 2D instance information into the 3D group field.

The first step is to compute the 3D world-space centroid of each instance in the current view. This requires knowing where each pixel projects into the 3D scene. The paper uses the SVR renderer to produce a point map $\mathbf{f}^{\text{pts}} \in \mathbb{R}^{HW \times 3}$ , which records the expected 3D position that each camera ray hits when tracing through the voxel field. This is the same concept as the "expected depth" in volume rendering, but expressed in world coordinates rather than along the ray.

The centroid $\mathbf{f}^{\text{center}}_k \in \mathbb{R}^3$ for instance $k$ is computed by masked average pooling over the point map:

$\mathbf{f}^{\text{center}}_k = \frac{\sum_{j \in HW} \mathbb{1}(M_j = k) \cdot \mathbf{f}^{\text{pts}}_j}{\sum_{j \in HW} \mathbb{1}(M_j = k)}$

where $\mathbb{1}(M_j = k)$ is an indicator function that equals 1 if pixel $j$ belongs to instance $k$ and 0 otherwise, $\mathbf{f}^{\text{pts}}_j$ is the 3D world position of the ray-surface intersection at pixel $j$ , and the sums run over all $HW$ pixels in the image.

What it computes: For each 2D instance mask, the numerator sums the 3D positions of all pixels belonging to that instance (weighted equally), and the denominator counts how many pixels belong to that instance. The result is the arithmetic mean 3D position of the visible surface of the instance in the current view—a rough estimate of the instance's spatial centroid.

Why this form: Using the masked average of the point map is a natural, parameter-free way to estimate "where in 3D space is this 2D mask?" It assumes that the visible surface of an object is roughly centered around the object's true 3D center, which holds for most everyday objects viewed from typical angles. The alternative—using only the center pixel of the 2D mask—would be viewpoint-dependent and fail when the object is partially occluded or viewed from an angle. The masked average is viewpoint-robust because it integrates information from all visible surface points of the instance.

Once the instance centroids are computed, they are accumulated into the voxel group field using the volume rendering weights from the SVR model:

$\mathcal{F}^{t+1}_i = \mathcal{F}^t_i + \sum_{j \in HW} w_{ij} \mathbf{f}^{\text{center}}_{M_j}$

$W^{t+1}_i = W^t_i + \sum_{j \in HW} w_{ij}$

where $w_{ij}$ is the blending weight from the volume rendering equation (Equation 1 in the paper): $w_{ij} = \alpha_i(\mathbf{r}_j) \cdot \prod_{l < i} (1 - \alpha_l(\mathbf{r}_j))$ . This is the contribution of voxel $i$ to the rendered color of pixel $j$ along ray $\mathbf{r}_j$ , and it encodes both the opacity of the voxel and the accumulated transmittance along the ray up to that voxel.

What it computes: For each voxel $i$ , the updated feature $\mathcal{F}^{t+1}_i$ is the old feature plus a weighted sum of the centroids of all instances whose 2D masks cover pixels that voxel $i$ contributed to. The weight $w_{ij}$ is exactly the rendering contribution—a voxel that heavily influences a pixel's color receives a large weight when voting for the instance assigned to that pixel. The updated weight $W^{t+1}_i$ accumulates the total rendering contribution of the voxel across all pixels in the current view.

Why this form: This formulation elegantly reuses the existing volume rendering machinery—the blending weights that SVR already computes for color rendering—to distribute 2D instance information into 3D. A voxel that is occluded or contributes negligibly to a pixel (low $w_{ij}$ ) casts a weak vote for that pixel's instance ID. A voxel that is directly visible and opaque at a pixel (high $w_{ij}$ ) casts a strong vote. This is more principled than, say, assigning the mask ID only to the first-hit voxel, because it correctly handles semi-transparent regions, fuzzy boundaries, and multi-view consistency. The paper notes that this update is implemented by customizing the sparse voxel renderer to perform the accumulation "in one rendering pass," which is an engineering detail that ensures the grouping process is efficient despite processing potentially millions of voxels.

Start Condition: Initializing from the First View

At the very first view ( $\xi_1$ ), there is no existing group field to update. The initialization is simple: the group feature $\mathcal{F}^1$ and weight $W^1$ are directly set to the accumulated values from Equation 3 (i.e., starting from zero, they equal exactly the weighted sum of centroids and the accumulated weights, respectively). The group dictionary $G^1$ is initialized with the $m$ instance IDs from $M_1$ and their corresponding centroids $\mathbf{f}^{\text{center}}$ .

This bootstrap is sufficient because SAM2 provides reasonable segmentation on the first frame, and subsequent views will refine and correct any initial errors through the matching and accumulation process.

Progressive Matching and Merging: Building Cross-View Consistency

When processing view $t+1$ (for $t \geq 1$ ), the system cannot simply add the new SAM2 masks as new instances—it must determine which masks correspond to already-seen objects and which are truly new. This is the matching and merging step, and it is the most algorithmically involved part of the grouping process.

Step 1: Assign current voxel IDs. First, each voxel's current best-guess instance ID is computed by finding the nearest centroid in the group dictionary:

$\text{ID}^t_i = \arg\min_j \left\| \frac{\mathcal{F}^t_i}{W^t_i} - G^t_j \right\|_2$

where $\mathcal{F}^t_i / W^t_i$ is the normalized voxel vote—the weighted-average centroid that voxel $i$ currently predicts—and $G^t_j$ is the stored centroid of instance $j$ in the group dictionary. The $\arg\min$ selects the instance whose centroid is closest in Euclidean distance to the voxel's predicted centroid.

What it computes: For each voxel, find which existing instance (as of the previous view) the voxel most likely belongs to, based on how close the voxel's accumulated centroid vote is to each instance's stored centroid.

Why this form: Using Euclidean distance to the nearest centroid is a nearest-neighbor classifier in 3D space—it assumes that voxels belonging to the same object cluster around that object's 3D center. This is the same inductive bias as k-means clustering and is reasonable for compact, non-overlapping objects. The alternative—learning a high-dimensional feature per voxel and using cosine similarity—would require training and is unnecessary when the only information needed is spatial grouping.

Step 2: Render the current grouping to the new view. The voxel IDs are rendered to view $t+1$ using the SVR rasterizer, producing a 2D projected mask $M^{\text{proj}}_{t+1}$ . This shows, for each pixel in the new view, which existing instance ID is predicted to be visible there (based on the voxel with the highest blending weight along the ray). If a pixel's ray hits no voxel (background), it receives ID 0. If it hits voxels from multiple instances, the one with the largest accumulated weight dominates.

Step 3: Match by IoU. For each instance present in $M^{\text{proj}}_{t+1}$ , find the SAM2 mask in $M_{t+1}$ that has the highest Intersection-over-Union (IoU) with it. If the IoU exceeds a threshold (the paper does not specify the exact value, but the concept is standard), the two are considered the same object. The SAM2 mask's ID is replaced with the existing instance ID from the group dictionary.

Step 4: Merge fragmented masks. The paper identifies a common failure mode: SAM2 sometimes oversegments a single object into multiple small masks (e.g., a "hand sculpture" might be split into one mask for the main hand and another for a fingertip, as shown in Appendix A, Figure 7). To correct this, the system re-prompts SAM2 using the existing grouping as a guide. Specifically, for each observable group under the current view, the system:

Samples several pixels from the group's region in $M^{\text{proj}}_{t+1}$ and provides them as positive point prompts to SAM2.
Samples pixels from other groups and provides them as negative point prompts.
Provides the entire group mask in $M^{\text{proj}}_{t+1}$ as a mask prompt (mask value 20 for positive, -20 for negative, 0 for unknown).

SAM2 then produces a refined mask for this group. If the refined mask of a smaller group is almost entirely contained within (over 90% inside) a larger group's mask, the two groups are merged—the smaller group's ID is replaced with the larger group's ID, and the group dictionary is updated.

This merging step is executed periodically (every 1–5 views, as stated in Appendix A) rather than at every view to balance grouping quality against inference time. The paper reports that this merging contributes a 3.7% mIoU improvement in the ablation study (Table 4, going from model A to model B).

Step 5: Add new instances. Masks in $M_{t+1}$ that could not be matched to any existing instance (IoU below threshold with all projected masks) are assigned new unique IDs and added to the group dictionary with their centroids computed from the point map.

Step 6: Update the group field. Using the now-matched masks (with IDs unified across views), the group feature and weight are updated via Equation 3, the same accumulation as in the first view but now with consistent IDs.

Final Group Assignment

After processing all $K$ views, each voxel's final instance ID is determined by Equation 4 using the fully accumulated group feature $\mathcal{F}^K$ and weight $W^K$ , and the final group dictionary $G^K$ . This produces a per-voxel instance label that is view-consistent and training-free.

The paper emphasizes that this progressive matching approach is done "in one training-free pass through all $K$ views without the lengthy gradient descent," contrasting it explicitly with Gaussian Grouping, which "relies on gradient descent to update per-primitive high-dimensional features with sequential training processes." The key engineering observation is that because the feature space is only 3-dimensional (spatial coordinates), the convergence to a consistent grouping does not require iterative optimization—a single pass with weighted averaging suffices, assuming the SAM2 masks are reasonably consistent across views.

Canonical Scene Map Construction: From Groups to Structured Text

Motivation: Embedding-Free Language Assignment

Once voxels are grouped into object-level instances, the next challenge is to associate language descriptions with those instances. The paper's key design decision is to generate explicit text captions rather than learn per-primitive or per-group language embeddings. This is the departure point from all prior work: instead of training a 3D feature to align with CLIP's embedding space (which constrains queries to the representational capacity of that space), the paper produces human-readable captions and performs text-to-text matching at inference time.

The output of this stage is a Scene Map $S$ , which the paper describes as a JSON-like structure containing, for each group: its unique ID, its 3D center position (in world coordinates), and a canonical caption in a fixed structured template. This scene map is simultaneously:

A product in its own right—a human-readable description of the entire scene.
An intermediate representation for the query inference stage.
An interpretable artifact for debugging and verification.

Stage 1: Free-Form Captioning with DAM

The first step generates initial captions for each object group using the Describe Anything Model (DAM) (Lian et al., 2025). DAM is a model specifically designed for detailed localized image and video captioning—given an image or video and a binary mask, it produces a descriptive sentence about the contents of the masked region.

For each group, the system:

Renders the group's binary mask from each of the $K$ camera views using the SVR rasterizer (only rendering voxels belonging to that group).
Pairs each rendered mask with the corresponding original image $I_k$ .
Feeds these mask-image pairs (as a short video clip, effectively) to DAM.

The paper notes a practical optimization: rather than using all $K$ views (which could be hundreds), it samples 8 frame-mask pairs per group (padding to 8 if fewer are available) "to reduce the visual tokens for the model, making sure that the inference is fast" (Appendix A). This is important because each additional frame adds tokens to the MLLM input, linearly increasing inference time and memory.

DAM's output is a free-form sentence, e.g., "A green round object, possibly an apple, with a dark stem and leaf attached, sitting on a wooden surface." This is descriptive but suffers from two problems:

Subject ambiguity: DAM frequently uses the generic word "object" as the subject noun, which is not discriminating enough for retrieval (many groups would be described as "object").
Inconsistent formatting: Free-form sentences vary in structure (some start with color, some with material, some with "A"), making them difficult to compare automatically. The canonicalization step addresses both issues.

Stage 2: Canonical Captioning with Qwen3-VL

The second step refines DAM's free-form captions into a fixed structured template using Qwen3-VL-8B-Instruct, an instruction-tuned multimodal large language model. The canonical template is:

<category noun>, <appearance details> <function/affordance or part-of> <placement/relation>

Each component has defined semantics:

Category noun: The most specific, concrete class name for the object (e.g., "apple" not "fruit," "toy dog" not "toy"). The system prompt (List 1 in Appendix A) explicitly forbids generic words: "Replace vague words ('object/thing/item/surface') with a concrete class or fine-grained subtype."
Appearance details: Distinctive visual attributes chosen from color, material, texture/pattern, shape/geometry, subtype/model, visible text/logo/printed character, state/condition. The prompt requires "at least FOUR distinct cues."
Function/affordance or part-of: Either what the object does or is used for, or—under strict conditions—that it is "part of" a larger object. The prompt specifies that "part of" should be used only if three conditions are met simultaneously: visible physical continuity within the mask, the region being an intrinsic component, and the larger object's category being visibly identifiable. Otherwise, placement is preferred over "part of."
Placement/relation: View-independent spatial relations such as "on table," "in bowl," "against wall." The prompt explicitly forbids view-dependent terms like left/right/front/behind.

Visual prompting strategy. Because general MLLMs are not specifically trained for mask-conditioned captioning (they expect full images with natural context), the paper uses a visual prompting technique inspired by Wu et al. (2024):

The regions outside the mask are darkened (reducing their visual salience).
A small red dot is placed on the masked object to "focus the model's attention."
The system prompt instructs: "Describe ONLY what lies INSIDE the masked region across frames. Never name or infer unmasked neighbors as the subject."

This is a clever workaround for the lack of native mask-conditioning in Qwen3-VL. By reducing the visual information outside the mask to near-zero (darkened pixels), the model's attention is forced onto the highlighted object. The red dot provides an explicit visual cue that this region is the target, which the model can learn to associate with the instruction to describe "the region highlighted by red dot" (as stated in Figure 4's caption).

Caption format enforcement. The system prompt (List 1) is extensive, containing explicit rules about:

Printed content: if text/logos are visible, they must be transcribed or described.
View-independent wording only: "no left/right/front/top; no camera terms."
Output format: exactly one line, 12–20 words, comma-separated phrases, no period, starting with the subject noun.
Strict phrase ordering: category noun → appearance details → function/affordance/part-of → placement/relation.

The resulting canonical caption from the example might be: "apple, light green with fluffy material and a prominent, dark green leaf attached to its stem, placed on table." This is structured, comparable across objects, and semantically rich.

Why canonicalization matters. The ablation study (Table 4) shows that adding canonical captioning (model C vs. model B) improves mIoU by 8.4 percentage points (28.0 → 36.4). This is a substantial gain, confirming that (a) reducing subject ambiguity (replacing "object" with "apple") and (b) enforcing a consistent template make retrieval significantly more reliable. Without canonicalization, the MLLM at retrieval time must compare free-form sentences that may describe the same object type in completely different syntactic structures—a harder matching problem.

Significant design detail. The paper states (Appendix A) that "we 'DO NOT' provide any visual example for the MLLM as in-context examples since the inference time would be slowed down by doing so." This is an explicit efficiency-performance tradeoff: in-context examples would likely improve canonicalization quality (by showing the model exactly what format is expected) but would add substantial tokens to each MLLM call, increasing both latency and compute cost. The authors leave exploring this balance as future work.

Scene Map Assembly

For each group (identified by its integer ID), the system stores:

ID: A unique integer identifier.
Position: The 3D world-space center of the group. The paper does not specify exactly how this is computed, but implicitly it is the centroid of the group's voxels or the stored centroid from the group dictionary $G^K$ .
Caption: The canonical caption string.

The scene map $S$ is the collection of all such entries, forming a structured JSON-like representation. This is the artifact that bridges grouping and retrieval: it captures both the spatial layout (through positions) and the semantic content (through captions) of every object in the scene.

Referring Query Inference: Text-to-Text Retrieval over the Scene Map

When a user provides a natural language query $D$ —which could be a single word ("chair"), a referring expression ("the funny toy with spindly legs that looks very interesting in the sunlight"), or anything in between—the system first canonicalizes it into the same structured template used for the scene map captions. This is done using Qwen3-VL, with an optional query image for context.

The query refinement system prompt (List 2 in Appendix A) instructs the MLLM to rewrite the query into the canonical form:

<class noun>, <appearance>, <placement?>

The output format is specified as: a JSON object with a single field "canonical" containing "2 to 6 words, lowercase, spaces only, no punctuation." The constraints are carefully designed:

No spatial relations unless in query: "You must NOT include any spatial relations in the output if not explicitly mentioned in the query." This prevents the model from hallucinating placements.
No generic nouns: The prompt explicitly forbids "object, thing, item, stuff, part, area, region, section, portion, background, foreground, surface, area of interest."
Conservative material vocabulary: The prompt provides specific guidance for materials: "plastic bag" becomes "plastic-like bag," "glass cup" becomes "transparent cup," "porcelain" becomes "ceramic." These are chosen to be visually descriptive without requiring the model to guess material composition.
Character name decomposition: Named characters like "Pikachu" are mapped to descriptions: "yellow character with long ears and possibly red cheek." This is critical because the scene captions also use visual descriptions, not proper names, so matching requires consistent vocabulary.

The refined query for the spindly toy example becomes something like: "toy, yellow, slim legs." This canonicalization step achieves format alignment between query and scene map, which is what enables deterministic text-to-text matching rather than ambiguous semantic similarity computation.

The ablation study (Table 4) shows that adding canonical query refinement (model "Ours" vs. model C) improves mIoU by 6.0 percentage points (36.4 → 42.4). This is intuitive: even with canonical captions, if the query remains in free-form, the MLLM must implicitly canonicalize it during retrieval, which is less reliable than doing it as an explicit preprocessing step.

Stage 2: Text-to-Text Retrieval with MLLM Reasoning

The core retrieval step is remarkably simple in concept: the entire scene map $S$ (list of IDs, positions, and canonical captions) plus the canonicalized query are fed into Qwen3-VL, and the model is asked to return the ID(s) of the best-matching group(s). The system prompt for retrieval (List 3 in Appendix A) specifies:

Caption-first matching: "Primary signal: scene_map CAPTIONS (semantic match to the canonical phrase; allow common synonyms/hypernyms)."
Spatial reasoning using world coordinates: "Ignore view-dependent relations (left/right/front/behind). Use WORLD coords ONLY for near/far/between/closest/farthest if such words appear."
Multi-group merging: "One real object may be split across multiple voxel groups (ids) that are spatially adjacent and semantically consistent. If so, RETURN ALL ids for that instance."
Optional image-based veto: "Secondary signal: view_image (if provided) only to veto obvious mismatches when visible; do NOT penalize occlusion."

The output format is a JSON object: {"ids": [<int>, ...], "captions": ["<EXACT caption>", ...]}. The captions are required to be copied exactly from the scene map (not paraphrased), ensuring traceability.

What the MLLM actually does. Internally, the model:

Reads all scene map entries (each with ID, centroid position, and canonical caption).
Compares the canonical query against each caption, considering class nouns, appearance attributes, functions, and placements.
If the query contains spatial relations like "near the apple" or "between the bottle and the cup," uses the stored 3D centroid positions to compute relative distances and filter or rank candidates.
If a query image is provided, optionally checks that the selected group is visually consistent (not vetoing if the object might be occluded).
Returns the best-matching ID(s).

This is fundamentally a reasoning-based retrieval, not an embedding-distance retrieval. The MLLM can handle:

Synonymy: "drinking utensil" matches "cup" or "glass."
Attribute disambiguation: "yellow with cartoonish looking" distinguishes a specific toy from other toys.
Affordance-based queries: "can be used for cutting paper" maps to "scissors" or "knife" based on functional understanding.
Spatial relations: "the chair that the sheep is sitting on" requires identifying the sheep, finding objects spatially below/under it, and selecting the chair.
Negation/implicit exclusion: "the apple, but not the green one" requires comparing colors across multiple apple candidates.

No single CLIP embedding could perform these operations because they require structured reasoning over multiple attributes and relations, which is precisely what an LLM's autoregressive attention mechanism enables.

Stage 3: Rendering the Output

Once the MLLM returns the selected group ID(s), the system renders the binary mask for the target viewpoint by rasterizing only the voxels belonging to those group(s). The output is:

A binary mask highlighting the referred object(s).
The canonical caption of the selected object(s) from the scene map (for user verification).

The entire inference pipeline—query refinement + retrieval—takes "less than 1 sec" (Section 5.4, Runtime Estimation), which is fast enough for interactive use.

Design Choices and Their Justifications

Why 3D centroid voting instead of high-dimensional learned features? The paper explicitly contrasts with Gaussian Grouping and ObjectGS, which learn per-primitive feature vectors (typically 16–32 dimensions) through gradient-based optimization. The 3D centroid approach has several advantages: it requires zero training, it converges in a single pass through the views, it is interpretable (you can visualize the accumulated centroids), and it implicitly enforces spatial coherence (voxels near each other that project to the same 2D masks will accumulate similar centroid votes). The tradeoff is that it cannot capture non-spatial grouping cues (e.g., two visually distinct parts of the same object that are spatially separated), but the SAM2 masks already handle this at the 2D level.

Why progressive matching instead of joint optimization? An alternative approach would be to collect all 2D masks across all views and solve a global assignment problem (which masks belong to which objects). The progressive approach is simpler to implement, requires no global optimization solver, and naturally handles the discovery of new objects in later views. The matching-by-IoU step (Step 3) is greedy but works well because SAM2 masks are generally consistent within a scene. The occasional errors are corrected by the merging step (re-prompting SAM2) and by the cumulative voting, which averages out inconsistent assignments.

Why DAM + Qwen3-VL two-stage captioning instead of end-to-end MLLM captioning? Appendix C, Table 8 shows that using Qwen3-VL alone (without DAM) achieves 33.3 mIoU vs. 42.4 with DAM. The paper attributes this to Qwen3-VL not being "trained specially for captioning masked region." DAM is purpose-built for this task and produces more detailed, accurate initial descriptions. Qwen3-VL's role is to structure and canonicalize, not to generate from scratch. This division of labor—specialized model for raw perception, general model for formatting—is a recurring theme in the paper's design.

Why the specific canonical template structure? The template <category noun>, <appearance details> <function/affordance or part-of> <placement/relation> encodes a specific theory of what makes captions useful for retrieval: (1) the category noun enables class-level matching, (2) appearance details enable within-class disambiguation, (3) function/affordance enables querying by use-case, and (4) placement enables spatial reasoning. The strict ordering ensures that the MLLM at retrieval time knows where to find each type of information. A free-form caption would require the MLLM to parse the structure on the fly for every comparison, which is less reliable.

Why text-to-text retrieval instead of embedding similarity? This is the paper's central conceptual contribution. The embedding approach (used by LangSplat, OpenGaussian, etc.) has three failure modes that text-to-text retrieval avoids: (1) Compositionality: A CLIP embedding of "the chair near the apple" is a single vector that loses the relational structure; an MLLM can parse the phrase into target=chair, relation=near, reference=apple and verify each part. (2) Open-endedness: Embedding spaces have fixed dimensionality and are optimized for their training distribution; text can express arbitrary concepts. (3) Explainability: When retrieval fails, you can examine the captions and the MLLM's output to diagnose whether the caption was wrong or the matching was wrong; embedding distances offer no such insight.

Why 3-minute processing time? The paper reports ~3 minutes for the full pipeline on an RTX 5090 (Table 5), broken down roughly as: ~1 minute for voxel grouping (processing up to 150 views with periodic merging), ~2 minutes for captioning (generating DAM captions for all groups and canonicalizing with Qwen3-VL). The key efficiency decisions are: (a) processing only a subset of available views (limited to 150, uniformly sampled), (b) sampling only 8 frames per group for DAM, (c) not using visual in-context examples for Qwen3-VL, (d) using a 3D feature space (not high-dimensional) to make the rendering pass fast. The contrast with ReferSplat's >2 hours per scene is stark and is a primary practical advantage.

Why SVR instead of 3DGS or NeRF? The paper doesn't extensively justify this choice, but the implicit reasons are: (1) SVR produces discrete voxels with well-defined spatial extents, which are natural for assigning group IDs (there's no ambiguity about whether a point in space "belongs" to a voxel); (2) SVR supports efficient rasterization, which is essential for the repeated rendering passes during grouping; (3) the voxel structure allows direct per-voxel attribute storage (the group feature). 3DGS would require similar machinery but with overlapping Gaussians, making group assignment fuzzier. NeRF would require expensive ray-marching for every rendering pass, making the ~150-view processing prohibitively slow.

4. Key Insights and Innovations

Innovation 1: Text-to-Text Retrieval as a Paradigm Replacement for Embedding Alignment in 3D Understanding

The paper's most fundamental intellectual move is not algorithmic but paradigmatic: it argues that the dominant approach to open-vocabulary 3D understanding—distilling CLIP-like embeddings into per-primitive features and performing retrieval via cosine similarity in a learned latent space—is not just suboptimal in practice but conceptually wrong for complex language queries, and it replaces it wholesale with a text-to-text reasoning paradigm.

This is a fundamental shift, not an incremental improvement. Since LangSplat (Qin et al., 2024) established the embedding-distillation recipe, the field has converged on a shared assumption: to make 3D representations respond to language, you must embed both the 3D primitives and the query into a shared vector space where proximity equals semantic similarity. This assumption was so deeply embedded that subsequent work—OpenGaussian, LEGaussians, 3DVLGS, ReferSplat, CCL-LGS—all competed on how to learn better embeddings (contrastive losses, codebooks, object-aware training) rather than questioning whether embeddings were the right abstraction at all. The paper's Figure 1 visually captures this: the prior pipeline shows "Bert embedding" as the bridge between language and 3D; OpenVoxel shows no embeddings anywhere.

What makes this a genuine insight rather than an obvious substitution is that it identifies compositionality as the critical failure mode of embedding-based methods. A CLIP embedding of "the chair that the sheep is sitting on, nearest to the sheep" is a single 512-dimensional vector. The compositional structure—identify the sheep, find objects with "sitting on" spatial relation, select the nearest—is irrecoverably lost in this compression. The paper's central diagnostic move is recognizing that this is not a training-data or model-architecture problem that can be fixed with more data or better contrastive losses; it is a representational capacity problem inherent to the embedding abstraction itself. No fixed-dimensional vector can encode arbitrary relational compositions, because the number of possible relations grows combinatorially while the vector dimension remains constant.

The evidence for this claim is embedded in the experimental design rather than in a single ablation. The paper deliberately evaluates on two tasks that differ in compositional complexity: OVS (queries are single class labels like "apple") and RES (queries are full sentences with attributes, relations, and affordances). The embedding-based methods perform competently on OVS—LangSplat achieves 53.7 mIoU on LeRF-OVS, and CCL-LGS reaches 65.1—because single-word queries map cleanly to CLIP embeddings. But on the RES task, which demands compositional reasoning, the gap between OpenVoxel (42.4 mIoU) and ReferSplat (29.2 mIoU reported, 24.5 mIoU reproduced) is 13–18 points—a margin that dwarfs the typical 1–3 point gains from improved embedding training recipes. This asymmetric performance gap is the empirical signature of a representational bottleneck: embedding methods plateau on compositional tasks regardless of how carefully they are trained.

The significance extends beyond 3D understanding. This paper is part of a broader intellectual current—visible also in work on visual programming, LLM-based planners, and neuro-symbolic systems—that argues for explicit symbolic intermediate representations over end-to-end learned embeddings when tasks require composition, relation, or reasoning. The 3D domain is a particularly compelling testbed because the geometric structure naturally supports symbolic grouping (objects are spatially coherent clusters), making the symbolic route more viable than in, say, abstract visual reasoning. The paper's success on RES suggests that the embedding paradigm may be approaching a fundamental limit for compositional language tasks, and that the path forward lies in making 3D representations "speak natural language" rather than "live in embedding space."

Innovation 2: The Modular Decomposition of 3D Understanding into Separable, Swappable Stages

A second conceptual contribution—less flashy than the paradigm shift but equally significant for the research community—is the modular decomposition of the 3D understanding pipeline into three independent stages: grouping (which object is which), captioning (what is each object), and retrieval (which object matches the query). Each stage is implemented by a different frozen foundation model (SAM2, DAM, Qwen3-VL), and each can be upgraded independently without retraining the others.

This is distinctive because prior work tightly coupled these functions. In LangSplat, the segmentation masks from SAM are used to regularize language feature training, but the grouping and the language assignment are learned jointly through gradient-based optimization. In Gaussian Grouping and ObjectGS, the object assignment and the per-object language features are trained simultaneously in a single optimization loop. This coupling means that improving one component—say, upgrading from SAM to SAM2 for better segmentation—requires retraining the entire pipeline, because the language features were trained conditioned on the specific segmentation quality and mask distribution of the old model.

OpenVoxel's modularity is demonstrated empirically through the ablation studies in Appendix C, which are arguably the paper's most intellectually honest experiments. Table 7 shows that swapping SAM for SAM2 in the grouping stage changes RES mIoU from 30.5 to 42.4—a 12-point gain purely from a better segmenter, with no other changes to the pipeline. Table 8 shows that swapping the captioning model (DAM vs. Osprey vs. Qwen3-VL alone) yields 42.4 vs. 29.3 vs. 33.3 mIoU—a 13-point range driven entirely by captioning quality, not grouping or retrieval quality. Table 9 shows that swapping the MLLM for retrieval (Qwen3-VL-8B vs. 4B vs. 2B vs. Qwen2.5-VL-7B) produces 42.4 vs. 35.6 vs. 10.0 vs. 23.4 mIoU—a staggering 32-point range that isolates the MLLM's reasoning capability as the dominant factor in retrieval quality.

These ablations are not just parameter sweeps; they constitute a sensitivity analysis that decomposes the pipeline's performance into attributable components. The fact that the MLLM size alone accounts for a 32-point swing tells us that retrieval reasoning is the current bottleneck, not grouping quality or caption detail. The fact that the captioning model accounts for 13 points tells us that DAM's specialization for mask-conditioned captioning provides genuine value over general-purpose VLMs. The fact that SAM2 provides 12 points over SAM tells us that better 2D segmentation directly translates to better 3D grouping with no other changes needed.

This modularity has a second-order implication that the paper doesn't explicitly state but that is clear from the design: it future-proofs the pipeline. When SAM3 or DAM2 or Qwen4-VL are released, they can be slotted into the appropriate stage without redesigning the system. When a new captioning model specialized for 3D-consistent descriptions appears, it replaces DAM. When a faster MLLM with better reasoning appears, it replaces Qwen3-VL. This is in stark contrast to ReferSplat, where upgrading any component would require re-annotating training data and retraining the per-scene language field. In a field where foundation models improve at an exponential pace, this modularity is not just an engineering convenience—it is a strategic advantage that makes the system's performance track the frontier of foundation model capability without additional investment.

The paper's runtime numbers (Table 5) further underscore the practical significance: the modular, training-free approach processes a scene in ~3 minutes versus >1 hour for training-based methods. But the deeper point is that this speed comes not from hardware optimization or clever engineering but from eliminating the coupling that necessitates gradient-based optimization. The grouping converges in one pass because 3D centroid voting doesn't require iterative refinement. The captioning uses a single forward pass per group because DAM and Qwen3-VL are frozen. The retrieval is a single MLLM call. There are no training loops anywhere, because there are no learned parameters to optimize.

Innovation 3: Difficulty-Free Object Grouping via Spatial Centroid Voting as an Alternative to Learned Feature Clustering

The paper's grouping mechanism—3D centroid voting accumulated across views and resolved by nearest-centroid lookup—is a conceptually elegant simplification that merits recognition as a distinct innovation, separate from the modularity argument. Prior work on 3D instance segmentation from 2D masks (Gaussian Grouping, ObjectGS, Gaga) universally used learned per-primitive feature vectors optimized via gradient descent, typically with contrastive losses that push primitives belonging to the same object together and different objects apart in feature space.

The paper's key insight is that for the specific problem of lifting 2D instance masks to 3D object groupings, a 3-dimensional spatial feature (the predicted object centroid) is not merely a cheaper alternative to high-dimensional learned features—it is arguably a better representation because it directly encodes the geometric structure that defines an "object" in 3D space. An object is, by definition, a spatially coherent cluster of matter. Its most fundamental property is its location in 3D space. A 3D centroid feature captures this directly; a 32-dimensional learned feature must rediscover this geometric structure through optimization.

The paper explicitly connects this to Deep Hough Voting (Qi et al., 2019), but the adaptation to the sparse-voxel-multi-view setting involves a non-obvious design choice: the voting space is the geometric space. In classical Hough voting for 3D object detection, points vote for object centers in a parameter space (e.g., translation, rotation, scale). Here, the parameter space and the geometric space are the same—voxels vote for where in 3D world coordinates their object's center is. This collapses the voting problem from a high-dimensional parameter estimation to a simple weighted-average estimation, which is why it converges in a single pass without iterative refinement.

The progressive matching and merging strategy (Section 4.1, Steps 1–6) is the algorithmic contribution that makes this spatial voting robust to SAM2's per-view inconsistencies. The key operations—IoU matching between projected and SAM masks, SAM2 re-prompting for merge detection, nearest-centroid instance assignment—are individually simple but collectively form a self-correcting system: errors in one view are overridden by consistent evidence from other views through the cumulative weighted average of centroid votes. The paper's claim that "even if the voxels are assigned to an incorrect group in some view, the accumulated features still force the voxel to 'vote' the most confident group" captures this robustness property.

This is a fundamental simplification, not an incremental one, because it demonstrates that for the specific task of 3D instance grouping from 2D segmentation masks, geometry is sufficient—you do not need learned semantic features, contrastive training, or joint optimization. The ablation in Appendix C, Table 7 (SAM vs. SAM2) confirms that the grouping quality is bottlenecked by the 2D segmentation quality, not by the 3D aggregation mechanism—a strong signal that the spatial voting approach is not losing information relative to learned alternatives.

Innovation 4: Canonical Caption Templates as a Formatting Interface Between Perception and Reasoning

A subtler but practically crucial innovation is the canonical caption template and its dual use for both scene description and query refinement. On the surface, this looks like prompt engineering—just specifying a format for the MLLM's output. But it represents a deeper design principle: enforcing structural alignment between the representation of the scene and the representation of the query eliminates an entire class of matching failures.

Prior work handled the representation-mismatch problem implicitly through embedding spaces: both the scene features and the query are projected into the same CLIP space, so they are aligned by construction of the embedding. OpenVoxel cannot rely on this because there is no embedding space. Instead, it creates alignment by convention: every object caption and every query is rewritten into the same structured template with the same phrase ordering, the same vocabulary constraints, and the same level of detail.

The template <category noun>, <appearance details> <function/affordance or part-of> <placement/relation> encodes a theory of what information is necessary and sufficient for object disambiguation:

The category noun handles class-level matching.
Appearance details handle within-class disambiguation (multiple apples, multiple chairs).
Function/affordance handles querying by use-case rather than by name.
Placement handles spatial reasoning.

The ablation in Table 4 quantifies the value of this structural alignment. Starting from a baseline where neither captions nor queries are canonicalized (model A, 24.3 mIoU), adding mask merging (model B) yields +3.7 points, adding canonical captions (model C) yields an additional +8.4 points, and adding canonical queries (full model) yields a further +6.0 points. The cumulative effect of canonicalization (from model B to full) is +14.4 mIoU—over one-third of the total performance—establishing that format alignment is not a minor engineering detail but a major performance driver.

The paper's system prompts (Appendix A, Listings 1–3) reveal the sophistication behind this apparent simplicity. The canonical captioning prompt contains detailed rules about when to use "part of" vs. placement, how to handle printed text, what words are forbidden, and what information types are required. The query refinement prompt contains explicit vocabulary mappings ("plastic bag → plastic-like bag," "porcelain → ceramic," "Pikachu → yellow character with long ears"). These rules encode domain knowledge about what causes retrieval failures—subject ambiguity, material vocabulary mismatch, proper name vs. description mismatch—and systematically prevent them.

This is a diagnostic contribution: it identifies that the bottleneck in text-to-text retrieval for 3D understanding is not the MLLM's reasoning capacity (which is strong) but the consistency of the textual representations being compared. By forcing both sides of the comparison into the same structured format, the system makes the retrieval problem dramatically easier for the MLLM, effectively offloading the alignment work from the reasoning stage to a preprocessing stage. This is analogous to database query optimization, where putting data into a normalized schema makes queries simpler and more reliable—except here the "schema" is a natural language template enforced through LLM prompting rather than a rigid database constraint.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on subsets of the LeRF dataset (Kerr et al., 2023), which contains iPhone Polycam-captured indoor scenes reconstructed from multi-view images. Three subsets are used: Ref-LeRF (He et al., 2025) for referring expression segmentation (RES), containing 4 scenes (ramen, figurines, teatime, kitchen) with 13, 17, 12, and 11 queryable objects respectively, each paired with sentence-level natural language referring expressions; LeRF-OVS (Qin et al., 2024) for open-vocabulary segmentation, containing 3 scenes (ramen, figurines, teatime) with 13, 17, and 12 annotated objects; and LeRF-Mask (Ye et al., 2024; Zhu et al., 2025) for OVS with boundary IoU evaluation, containing the same 3 scenes with 6, 7, and 10 queryable objects respectively. Additionally, semantic segmentation is evaluated on ScanNet (Dai et al., 2017), specifically 10 scenes with 19 semantic classes, using ground truth point clouds and depth maps.
Base model(s). The 3D scene representation uses Sparse Voxel Rasterization (SVR) (Sun et al., 2025), which reconstructs scenes as sparse voxel fields from multi-view images and supports real-time rasterization-based rendering. The foundation models used as frozen components are SAM2 (Ravi et al., 2024) for per-view 2D instance segmentation, Describe Anything Model (DAM) (Lian et al., 2025) for mask-conditioned captioning, and Qwen3-VL-8B-Instruct (Bai et al., 2023, 2025; Wang et al., 2024) for canonical captioning, query refinement, and text-to-text retrieval. These are chosen because they are state-of-the-art open-source models that collectively enable the training-free pipeline: SAM2 provides temporally consistent segmentation, DAM provides detailed localized captions, and Qwen3-VL provides multi-modal reasoning capacity.
Metrics. For referring expression segmentation (RES) on Ref-LeRF, the paper reports mean Intersection-over-Union (mIoU) between predicted and ground-truth masks, following He et al. (2025). For open-vocabulary segmentation (OVS) on LeRF-OVS, mIoU is reported following He et al. (2025) and Qin et al. (2024). For OVS on LeRF-Mask, both mIoU and mean Boundary IoU (mBIoU) are reported, following Ye et al. (2024) and Zhu et al. (2025). For semantic segmentation on ScanNet, both mIoU and mean Accuracy (mAcc) are reported, following OpenGaussian (Wu et al., 2024). IoU measures the overlap between predicted and ground-truth masks; BIoU additionally evaluates the accuracy of mask boundaries.
Baselines. For RES, the paper compares against Grounded SAM (Ren et al., 2024), LangSplat (Qin et al., 2024), SPIn-NeRF (Mirzaei et al., 2023), GS-Grouping (Ye et al., 2024), GOI (Qu et al., 2024), and ReferSplat (He et al., 2025), with results for ReferSplat reproduced from the official implementation (denoted ReferSplat*). For OVS on LeRF-OVS, baselines include Feature-3DGS (Zhou et al., 2024), LEGaussians (Shi et al., 2024), LangSplat, GS-Grouping, GOI, 3DVLGS (Peng et al., 2025), ReferSplat, and CCL-LGS (Tian et al., 2025). For OVS on LeRF-Mask, baselines include DEVA (Cheng et al., 2023), LERF (Kerr et al., 2023), SA3D (Cen et al., 2023), LangSplat, GS Grouping, Gaga (Lyu et al., 2024), and ObjectGS (Zhu et al., 2025). Note that only ReferSplat requires ground truth annotation of description–mask pairs for training; all other baselines (including OpenVoxel) do not.
Generation budget / compute accounting. The paper does not measure compute in FLOPs or generation tokens; instead, it reports wall-clock run time for the full pipeline on a single RTX 5090 GPU (Table 5). The total processing time includes voxel grouping (processing up to 150 uniformly sampled views per scene, with periodic merging every 1–5 views), canonical captioning (DAM on 8 sampled frame-mask pairs per group, Qwen3-VL canonicalization), and inference (query refinement + retrieval, "less than 1 sec" per query). This is compared against training-based methods (ReferSplat: >1 hour; ObjectGS: ~40 minutes). For the grouping stage specifically, the paper notes that processing is limited to 150 views per scene (uniformly sampled from all available training views) to control inference time.
Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. Results are reported as single-run values on the test subsets. The LeRF subsets (Ref-LeRF, LeRF-OVS, LeRF-Mask) are existing benchmarks with pre-defined test scenes; the paper reports results on these fixed test sets. For the ReferSplat reproduction, the paper notes that the official pre-trained models and data do not cover all four scenes, so results are reproduced "following the original implementation and configurations." No confidence intervals, standard deviations, or multiple-run averaging is reported.

Main Quantitative Results

Referring Expression Segmentation (RES) on Ref-LeRF

Headline result: OpenVoxel achieves 42.4 mIoU on the Ref-LeRF RES benchmark, substantially outperforming all baselines including the prior state-of-the-art ReferSplat, which achieves 29.2 mIoU as reported in the original paper and 24.5 mIoU as reproduced by the authors (Table 1). This represents a +13.2 percentage point improvement over the published ReferSplat result and a +17.9 point improvement over the reproduced result.

The performance advantage is not uniform across scenes. On ramen, OpenVoxel achieves 52.5 mIoU vs. ReferSplat's 35.2 (+17.3 points). On figurines, 43.5 vs. 25.7 (+17.8 points). On teatime, 48.4 vs. 31.3 (+17.1 points). On kitchen, 25.1 vs. 24.4 (+0.7 points). The kitchen scene is the notable exception where the gap nearly disappears; the paper does not analyze this scene-specific failure but the consistently large margins on the other three scenes drive the average.

The gap between OpenVoxel and the next-best training-free baseline (GOI, 20.5 mIoU) is even larger at +21.9 points, confirming that the training-free baselines using embedding-based approaches are fundamentally limited on the RES task. GS-Grouping achieves only 14.4 mIoU, and LangSplat achieves 13.9 mIoU, both substantially below OpenVoxel.

Reproducibility note: The paper explicitly reports the reproduced ReferSplat results (ReferSplat*, 24.5 mIoU) separately from the published ones (29.2 mIoU), noting that "during reproduction, we observe that ReferSplat's strategy of learning sentence-level embeddings for 3D representations tends to overfit the seen descriptions, resulting in unstable evaluation performance." This suggests that ReferSplat's published results may not be representative of what a user would obtain when running the method on new scenes, and that OpenVoxel's training-free approach avoids this overfitting problem.

Qualitative evidence (Figures 5 and 6): The RES qualitative results reveal specific failure modes of ReferSplat that OpenVoxel avoids. For the query "A minimalist style toy with natural grooves next to a red apple" in the figurines scene (Figure 5), ReferSplat "focuses on the region near the apple... without localizing the correct object," while OpenVoxel correctly retrieves the toy pumpkin. For the query "A drinking utensils with a smooth surface near a sake bottle" in the ramen scene (Figure 6), ReferSplat "only focuses on the term 'drinking utensils' and incorrectly segments both the glass of water and the sake bottle," while OpenVoxel identifies only the sake cup. These examples demonstrate the paper's claim that embedding-based methods struggle with compositional queries (distinguishing "drinking utensil near sake bottle" from "drinking utensil" in general), while text-to-text retrieval over explicit captions can handle the relational reasoning.

Supplemental qualitative results (Figure 10): The paper further tests RES queries that are not in the Ref-LeRF annotations, demonstrating generalization beyond the benchmark distribution. For queries like "a tag that writes the word 'dalle' and in front of the sheep" or "the only table that is nearest to the current view," ReferSplat "struggles to recognize unseen target objects even in the same scene it is optimized, showing that it tends to overfit on annotated objects from the dataset." OpenVoxel, being training-free and not dependent on per-scene annotations, generalizes to these unseen queries. For the last two columns with view-dependent descriptions, the paper notes that OpenVoxel "retrieve two targets instead of the only one matched" but "the results are still including the correct target, showing the potential capability of solving view-specific tasks."

Open-Vocabulary Segmentation (OVS) on LeRF-OVS

Headline result: OpenVoxel achieves 66.2 mIoU on LeRF-OVS, marginally outperforming the previous best method CCL-LGS (65.1 mIoU) by +1.1 points (Table 2). The performance is not uniform across scenes: on teatime, OpenVoxel achieves 75.4 mIoU vs. CCL-LGS's 71.8 (+3.6 points); on ramen, 62.5 vs. 62.3 (+0.2 points); on figurines, 60.7 vs. 61.2 (−0.5 points). The figurines scene is the only one where OpenVoxel slightly underperforms CCL-LGS.

The margin over embedding-based methods that do not use additional training objectives is larger: vs. ReferSplat (57.6 mIoU), +8.6 points; vs. LangSplat (53.7 mIoU), +12.5 points; vs. Feature-3DGS (47.7 mIoU), +18.5 points. This shows that even on the simpler OVS task where embedding methods are strongest, OpenVoxel's text-to-text approach remains competitive or superior, without any of the per-scene training costs.

Important context: The paper notes that "the OVS task involves relatively simpler queries, leading to higher overall performance across methods—even when evaluated on the same target objects as in Ref-LeRF." The fact that all methods score 47–66 mIoU on OVS compared to 10–42 mIoU on RES confirms that RES is the harder task where compositional understanding is tested, and it is on RES that OpenVoxel's advantages are most pronounced.

Open-Vocabulary Segmentation (OVS) on LeRF-Mask

Headline result: OpenVoxel achieves 87.2 mIoU and 81.4 mBIoU on LeRF-Mask (Table 3). This is slightly below ObjectGS (88.3 mIoU, 84.4 mBIoU) by −1.1 mIoU and −3.0 mBIoU, but above all other baselines. GS Grouping achieves 72.8/67.6, Gaga achieves 78.5/74.2, and LangSplat achieves 61.2/56.1.

The paper's analysis (Section 5.2) notes that "where the queries are fewer and less ambiguous, all current state-of-the-art methods (including ours) achieve over 70% mIoU." This observation—that OVS on LeRF-Mask is approaching performance saturation—suggests that the task may not adequately differentiate methods anymore. OpenVoxel's slight underperformance relative to ObjectGS on this benchmark is noted but not analyzed in depth; possible explanations include that ObjectGS jointly optimizes object assignment and 3DGS training, producing more precise object boundaries than the training-free grouping, or that the LeRF-Mask subset's query distribution favors the embedding approach.

The mBIoU gap (−3.0 points vs. ObjectGS) is larger than the mIoU gap (−1.1 points), suggesting that while OpenVoxel's retrieved masks cover the correct objects well, their boundaries are slightly less precise than ObjectGS's jointly optimized masks. This is consistent with the paper's acknowledgment (Section E, Limitations) that "the grouping process of OpenVoxel is relatively sensitive to parameters" and that SAM2 parameters need careful adjustment per scene—a training-free grouping approach may produce slightly coarser boundaries than a method that optimizes object masks jointly with the 3D reconstruction.

Semantic Segmentation on ScanNet

Headline result: OpenVoxel achieves 30.0–31.6 mIoU and 41.1–42.3 mAcc on ScanNet semantic segmentation (Table 6), depending on the evaluation protocol (Nearest, Majority of 25-NN, or Majority of 50-NN). This outperforms all embedding-based baselines, including OpenGaussian (24.7 mIoU, 41.5 mAcc), which uses ground truth point clouds as initialization. LangSplat and LEGaussian achieve only 3.8 mIoU each.

Evaluation protocol nuance: The paper notes a significant setup difference: OpenGaussian uses ground truth point clouds directly as initialization for 3DGS, deactivating merging and splitting so that "perfect geometry alignment is naturally obtained for evaluation." SVR does not support this initialization, so the paper uses depth maps to guide SVR pre-training. Additionally, SVR produces far more voxels (5M–10M) than ground truth points (50K–350K), necessitating the three evaluation protocols that map voxel-level predictions to point-level ground truth. The Nearest protocol assigns each ground truth point the label of the spatially nearest voxel; Majority of 25-NN and Majority of 50-NN take the majority label among the 25 or 50 nearest voxels.

The results are encouraging but should be interpreted cautiously given the evaluation-protocol mismatch with baselines. The paper reports that OpenVoxel "outperforms all baselines in terms of mIoU, and is comparable in mAcc," but the comparison is not fully apples-to-apples because baselines used ground-truth-initialized geometry while OpenVoxel did not.

Runtime Analysis

Headline result: OpenVoxel completes the full pipeline in ~3 minutes per scene on a single RTX 5090 GPU, compared to >1 hour for ReferSplat and ~40 minutes for ObjectGS (Table 5). Per-query inference takes "less than 1 sec." This represents at least a 10× speedup over training-based methods, and more critically, the runtime advantage is structural rather than hardware-dependent—training-based methods require multiple GPU-hours of gradient-based optimization for every new scene, while OpenVoxel's runtime is dominated by frozen model forward passes.

The paper notes that the runtime estimate for ReferSplat may understate the actual cost: "in all our reproductions, we find that following the official implementation configuration takes at least 2 hours per scene to obtain the best results," compared to the 58 minutes reported in the original ReferSplat paper on an A6000 GPU. This discrepancy highlights the practical challenge of training-based methods—their runtime depends sensitively on convergence behavior and hyperparameter tuning, while OpenVoxel's deterministic, training-free pipeline has predictable runtime.

Ablation Studies and Robustness Checks

Pipeline component ablation (Table 4): The paper ablates the four key components of OpenVoxel in a cumulative fashion on the Ref-LeRF RES benchmark. Starting from model A (no mask merging, no canonical captioning, no canonical query), which achieves 24.3 mIoU, each component is added:

+Mask merging (model B): Adding the SAM2 re-prompting procedure for merging fragmented masks during grouping improves mIoU from 24.3 to 28.0 (+3.7 points). The paper attributes this to "less noisy groups"—without merging, SAM2's occasional oversegmentation creates multiple groups for a single object, making it harder for the retrieval MLLM to match queries to the correct full object.
+Canonical captioning (model C): Enforcing the structured caption template during scene map construction improves mIoU from 28.0 to 36.4 (+8.4 points). This is the single largest component-level gain, confirming that standardizing captions (reducing subject ambiguity, enforcing consistent formatting) substantially improves retrieval reliability. The paper notes this is due to "more specific captioning (i.e., standard template for each caption and reducing the word 'object' as subject noun)."
+Canonical query refinement (Full model): Canonicalizing user queries into the same template format as captions improves mIoU from 36.4 to 42.4 (+6.0 points). This demonstrates that aligning the query format to the caption format is nearly as important as standardizing the captions themselves—even with well-structured captions, free-form queries create a matching gap that degrades retrieval.

The cumulative improvement from baseline to full model is +18.1 mIoU (24.3 → 42.4), with the canonicalization components (caption + query) contributing +14.4 of those points. This validates the paper's central architectural claim that format alignment between scene representation and query representation is a critical performance driver.

Segmentation model (Table 7): Replacing SAM2 with the original SAM (Kirillov et al., 2023) for 2D mask generation in the grouping stage reduces RES mIoU from 42.4 to 30.5 (−11.9 points). The paper observes that "SAM tends to segment small fragments that are over-detailed, and therefore, the grouping results are slightly noisier than our original version using SAM2." This large drop confirms that 2D segmentation quality is a primary bottleneck for the grouping stage, and that SAM2's improved temporal consistency and reduced oversegmentation directly translate to better 3D instance grouping. It also demonstrates OpenVoxel's modularity—the segmentation model can be swapped without changing any other component, and the performance delta isolates the segmentation model's contribution.

Captioning model (Table 8): Three captioning configurations are tested for generating initial (pre-canonicalization) captions:

DAM (the default): 42.4 mIoU.
Qwen3-VL-8B-Instruct (bypassing DAM, using Qwen3-VL directly with darkened-background visual prompting): 33.3 mIoU (−9.1 points).
Osprey (Yuan et al., 2024, a captioning-specialized model, with per-frame captioning summarized by Qwen3-VL): 29.3 mIoU (−13.1 points).

The DAM advantage over Qwen3-VL (−9.1 points) demonstrates that specialized mask-conditioned captioning models provide genuine value over general-purpose VLMs, even when the VLM is prompted with visual cues (red dot, darkened background). The paper's explanation is that Qwen3-VL is "not trained specially for captioning masked region." The Osprey result (−13.1 points vs. DAM) demonstrates that per-frame captioning followed by summarization produces "inconsistent caption for the same group from different views," confusing the canonicalization step. However, the paper notes that "using either captioning model achieves better mIoU than ReferSplat," indicating that the overall pipeline is robust to caption quality variations and that even imperfect captions, when canonicalized, outperform learned embeddings for RES.

MLLM model (Table 9): Four MLLM configurations are tested for canonical captioning, query refinement, and retrieval (all three stages use the same model):

Qwen3-VL-8B-Instruct: 42.4 mIoU.
Qwen3-VL-4B-Instruct: 35.6 mIoU (−6.8 points).
Qwen2.5-VL-7B-Instruct: 23.4 mIoU (−19.0 points).
Qwen3-VL-2B-Instruct: 10.0 mIoU (−32.4 points).

The 32.4-point range is the largest ablation swing in the paper, demonstrating that MLLM capability is the dominant factor in retrieval quality. The Qwen3-VL-2B model's catastrophic 10.0 mIoU is attributed to its inability to canonicalize captions: "it tends to repeat some of the words in the original caption as the refined caption... the incorrect refined captions are hard for [the model] to locate the correct target object." This confirms that the canonicalization step is not merely cosmetic—it requires non-trivial language understanding that small MLLMs lack. The Qwen2.5-VL-7B result (23.4 mIoU vs. Qwen3-VL-8B's 42.4) suggests that the Qwen3-VL family has specific instruction-following capabilities (likely related to the structured output format) that the Qwen2.5-VL family lacks, despite similar parameter counts.

The paper does not report ablations where different MLLMs are used for different stages (e.g., 8B for canonicalization, 4B for retrieval), which would test whether the retrieval stage is more or less sensitive to model capacity than the canonicalization stage.

Evaluation protocol on ScanNet (Table 6): The three protocols for mapping voxel-level predictions to point-level ground truth (Nearest, Majority of 25-NN, Majority of 50-NN) show small but consistent improvements as the neighborhood size increases: 30.0 → 31.3 → 31.6 mIoU and 41.1 → 42.1 → 42.3 mAcc. This suggests that OpenVoxel's voxel-level predictions are reasonably consistent within local neighborhoods (majority voting improves over single-nearest-neighbor), but the gains are modest (~1.6 mIoU, ~1.2 mAcc), indicating that errors are not randomly scattered but spatially correlated—voxels near semantic boundaries may be systematically misclassified.

MLLM prompt design: The paper explicitly notes that it "do[es] not spend much effort exploring different kinds of system prompt design," instead using ChatGPT to generate prompts from task descriptions. This means the reported results may not represent the upper bound of what the pipeline can achieve with optimized prompts—a potentially significant unquantified factor given that the MLLM ablation shows 32-point sensitivity to model choice, which implies high sensitivity to prompt quality as well.

No ablation on view sampling rate or merging frequency: The paper states (Appendix A) that it "uniformly sample[s] the processed views to make sure the total processed view of each scene does not exceed 150" and conducts merging "per 1 to 5 steps to speed up the inference." However, no ablation quantifies how performance varies with the number of processed views or the merging frequency. This is a notable gap because the paper acknowledges (Section E) that "the sampling rate of frames and merging frequency are customized for each scene," implying sensitivity to these parameters without providing evidence for the degree of sensitivity.

No ablation on the number of DAM frames per group: The paper uses 8 frame-mask pairs per group for DAM captioning (Appendix A), but does not ablate this number. Using fewer frames would reduce inference time; using more frames might improve caption quality. The absence of this ablation makes it unclear whether 8 is near-optimal or an arbitrary choice.

No failure case analysis by query type: While qualitative results show specific failure modes of ReferSplat, the paper does not provide a systematic breakdown of OpenVoxel's failures by query type (e.g., attribute-only queries vs. relational queries vs. affordance queries). Such an analysis would reveal which aspects of the pipeline are weakest and guide future improvements.

No comparison to an "oracle" retrieval upper bound: The paper does not report what mIoU would be achieved if retrieval were replaced with ground-truth matching (i.e., using the known correct group ID for each query). This oracle bound would decompose the total error into grouping error + captioning error + retrieval error, isolating which stage is the primary bottleneck. The current ablation only decomposes the improvement from adding components, not the absolute error sources.

Critical Assessment

Claim 1: "OpenVoxel achieves superior performance compared to recent studies, particularly in complex referring expression segmentation tasks."

The experiments support this claim strongly for RES, conditionally for OVS. On Ref-LeRF RES (Table 1), the 42.4 mIoU result is unambiguously state-of-the-art, with margins of +13.2 points over published ReferSplat and +17.9 points over the reproduced version. The supplementary qualitative results (Figure 10) further demonstrate generalization to queries outside the benchmark annotations, which no other method can handle.

However, on OVS (Tables 2 and 3), the claim is more nuanced. On LeRF-OVS, OpenVoxel achieves 66.2 mIoU vs. CCL-LGS's 65.1—a narrow +1.1 point margin that may not be statistically significant (no confidence intervals reported). On LeRF-Mask, OpenVoxel's 87.2 mIoU is slightly below ObjectGS's 88.3—a −1.1 point deficit. The paper's framing emphasizes the RES advantage while downplaying that OVS performance is essentially tied with the best training-based methods. This is consistent with the paper's thesis—text-to-text retrieval benefits most from compositional queries, and OVS queries lack the compositionality that exposes embedding methods' weaknesses—but the claim of "superior performance" should be qualified as RES-specific, not universal.

The ScanNet results (Table 6) are harder to interpret due to the evaluation-protocol mismatch with baselines. OpenVoxel's 30.0–31.6 mIoU vs. OpenGaussian's 24.7 mIoU is nominally superior, but OpenGaussian benefits from ground-truth point cloud initialization while OpenVoxel does not—comparing them directly may understate OpenGaussian's capability in a fairer setting. The paper acknowledges this asymmetry but does not attempt to equalize the comparison, making the ScanNet claim weaker than the LeRF claims.

Claim 2: "OpenVoxel is training-free and does not introduce embeddings from a CLIP/BERT text encoder."

This claim is fully supported by the method description and runtime results. The paper demonstrates that all components use frozen models (SAM2, DAM, Qwen3-VL) with zero gradient-based optimization. Table 5's runtime comparison (3 minutes vs. >1 hour) provides empirical evidence that the training-free claim translates to practical speed advantages.

However, the claim elides an important distinction: while OpenVoxel does not train on the target scene, it relies on foundation models that were themselves trained on massive datasets (SAM2 on video segmentation data, DAM on image-caption pairs, Qwen3-VL on multimodal instruction data). The "training-free" label refers to per-scene training, not to the absence of training in the system overall. This is a reasonable and conventional use of the term in the domain, but readers unfamiliar with the foundation-model paradigm might misinterpret it as implying no learned components whatsoever.

Additionally, the paper does not address the storage and memory cost of the foundation models. SAM2 (multiple GB), DAM (size unspecified), and Qwen3-VL-8B (16 GB in FP16) must all be loaded simultaneously or sequentially during the pipeline. The per-scene runtime of 3 minutes on an RTX 5090 (32 GB VRAM) suggests that all models fit in GPU memory, but the memory footprint is not reported and could be a practical barrier for deployment on consumer hardware.

Claim 3: "OpenVoxel produces meaningful groups that describe different objects in the scene" via Training-Free Sparse Voxel Grouping.

The grouping quality is indirectly supported by downstream task performance but not directly evaluated. The ablation in Table 7 shows that switching from SAM to SAM2 changes RES mIoU by 11.9 points, confirming that grouping quality affects final performance. However, no direct grouping metric is reported—no 3D instance segmentation mIoU, no precision/recall on object detection, no comparison of group boundaries to ground truth instance annotations (if available). The qualitative figures show grouped objects, but these are cherry-picked examples.

This is a significant gap because the grouping stage is claimed as a contribution (the first bullet in the paper's contribution list) and is the foundation for all downstream processing. A grouping failure (merging two objects into one group, or splitting one object into multiple groups) directly limits the maximum achievable RES and OVS performance—the retrieval MLLM cannot select a correct group that was never formed. Without direct grouping metrics, it is impossible to determine whether the performance ceiling on RES (42.4 mIoU) is limited by grouping quality, captioning quality, or retrieval quality. An oracle-retrieval ablation (using ground-truth group-to-query matching) would partially address this by decomposing grouping error from retrieval error, but it is not reported.

The paper acknowledges in Section E that grouping "is relatively sensitive to parameters comparing to the end-to-end generalizable ones" and that "the SAM2 parameters are needed to be adjust carefully." This sensitivity, combined with the lack of direct evaluation, makes it difficult for a practitioner to determine whether their grouping results are "good enough" before proceeding to captioning and retrieval.

Claim 4: Canonical captioning and query refinement are necessary for effective text-to-text retrieval.

The ablation in Table 4 strongly supports this claim. The +14.4 mIoU cumulative gain from canonicalization components (caption: +8.4, query: +6.0) demonstrates that format alignment is essential, not optional. The MLLM ablation (Table 9) provides converging evidence: Qwen3-VL-2B fails because it cannot canonicalize captions, producing outputs that break the format alignment. The 32-point swing from 2B to 8B is largely attributable to canonicalization capability.

However, the claim is demonstrated for one specific MLLM family (Qwen3-VL) and one specific template format. The paper does not test alternative template designs (e.g., key-value pairs, free-form paragraphs with section headers, JSON schemas) or alternative canonicalization strategies (e.g., using a separate lightweight model for format enforcement vs. relying on the MLLM's instruction following). The sensitivity of results to the specific template and the specific system prompts (which the paper acknowledges were not extensively optimized) suggests that the "+14.4 mIoU from canonicalization" figure is likely prompt-dependent and may not transfer to other MLLMs or other prompt designs.

Missing experiments that would strengthen the paper:

Oracle retrieval to decompose error sources: Running retrieval with ground-truth group-to-query matching would separate grouping+captioning error from retrieval error, identifying the primary bottleneck.
Per-query-type breakdown: Reporting mIoU separately for attribute queries, spatial relation queries, affordance queries, and single-category queries would reveal whether text-to-text retrieval's advantage is uniform or concentrated in specific query types.
Multiple runs or confidence intervals: The LeRF test sets are small (4 scenes for RES, 3 scenes for OVS), and single-run results may be noisy. Reporting mean and standard deviation over multiple runs (at least for the MLLM retrieval step, which has stochastic sampling) would help assess whether the narrow margins on OVS (e.g., 66.2 vs. 65.1) are reliable.
Direct grouping quality metrics: Computing 3D instance segmentation metrics (mIoU, precision, recall) against ground truth instance annotations (if available for LeRF scenes) would provide an independent quality measure for the grouping stage.
View count and merging frequency ablations: Quantifying how performance varies with the number of processed views (50, 100, 150, all) and merging frequency (every 1, 3, 5, 10 views) would provide practical guidance for the parameter sensitivity the paper acknowledges.

Where the claims hold and where they weaken:

RES on LeRF: Claims hold strongly. The +13–18 point margin over ReferSplat is large enough to be robust to evaluation noise.
OVS on LeRF: Claims hold marginally. OpenVoxel is competitive with state-of-the-art but not clearly superior; the +1.1 point margin on LeRF-OVS and −1.1 point deficit on LeRF-Mask are within the range where unmeasured factors (prompt tuning, random seed, scene-specific parameter adjustment) could flip the ranking.
ScanNet: Claims hold weakly. The evaluation-protocol mismatch with baselines and the use of ground-truth point cloud initialization by baselines but not by OpenVoxel make the comparison inequitable.
Runtime advantage: Claims hold strongly. The 10×+ speedup is structural (no training loops) and would persist across hardware generations.
Generalizability to unseen queries: The Figure 10 qualitative results demonstrate this qualitatively, but no quantitative benchmark exists for this claim. The Ref-LeRF test queries are, by construction, drawn from the same distribution as ReferSplat's training annotations—so the benchmark itself does not test generalization to out-of-distribution queries. The supplementary queries in Figure 10 are hand-picked and not evaluated quantitatively.

6. Limitations and Trade-offs

Limitation 1: Grouping Quality Is Not Directly Evaluated, Creating an Unmeasured Ceiling for All Downstream Tasks

The assumption or constraint. The entire downstream pipeline—captioning, scene map construction, and retrieval—depends on the Training-Free Sparse Voxel Grouping correctly partitioning voxels into semantically coherent object instances. If two objects are incorrectly merged into one group, the caption describes a hybrid that matches neither object's true identity. If one object is split into multiple groups, the retrieval MLLM may match only a fragment, producing incomplete segmentation masks. The paper assumes that the centroid-voting mechanism, combined with progressive IoU matching and SAM2 re-prompting, produces groupings of sufficient quality to support accurate retrieval.

The consequence. Without direct grouping metrics, we cannot determine whether the 42.4 mIoU ceiling on RES and the 66.2/87.2 mIoU on OVS are limited by grouping quality, captioning quality, or retrieval quality. A practitioner who obtains poor results has no diagnostic tool to identify which stage failed. More subtly, grouping errors create a hard upper bound on achievable performance that no amount of caption quality or retrieval sophistication can exceed—a merged group can never be correctly retrieved for a query targeting only one of its constituent objects, and a split group can never be retrieved as a single entity if the query expects one object.

What evidence exists in the paper. The paper acknowledges this sensitivity implicitly. Section E (Limitations) states that "the grouping process of OpenVoxel is relatively sensitive to parameters comparing to the end-to-end generalizable ones" and that "SAM2 parameters are needed to be adjust carefully." The ablation in Table 7 (SAM vs. SAM2) shows an 11.9 mIoU swing on RES purely from changing the segmentation model, confirming that grouping quality substantially affects final performance. However, no direct grouping metric is reported—no 3D instance segmentation mIoU, no precision/recall on object detection, no comparison of group boundaries to ground truth instance annotations. The qualitative figures (Figures 5, 6, 8, 9) show grouped objects, but these are cherry-picked successes. The critical missing experiment—an oracle retrieval upper bound using ground-truth group-to-query matching—is not performed, so we cannot decompose total error into grouping error + captioning error + retrieval error.

Mitigation status. The paper does not attempt to measure grouping quality directly. It suggests (Section E) that "construct[ing] a complex scene graph for all the groups to indicate the spatial relations (e.g., on top of, between) or ownership (e.g., belongs to, part of) explicitly" might improve robustness, but this is framed as future work. The parameter sensitivity is acknowledged but not quantified—no ablation on the number of processed views, merging frequency, or SAM2 confidence thresholds is provided. A practitioner must tune these parameters per scene by trial and error, guided only by downstream RES/OVS performance, which conflates grouping quality with captioning and retrieval quality.

Limitation 2: Difficulty Estimation Cost Is Unaccounted for in the Headline Speed Claims

The assumption or constraint. The paper reports that OpenVoxel completes the full pipeline in "~3 minutes" per scene on an RTX 5090 GPU (Table 5), compared to >1 hour for training-based methods. This ~3-minute figure assumes that the SVR model is already pre-trained from multi-view images—a process that is not free. The paper also assumes that SAM2 masks for all views are available, that DAM and Qwen3-VL are loaded and ready for inference, and that the specific hyperparameter values (150 views per scene, merging every 1–5 views, 8 DAM frames per group) have already been determined. None of these setup costs are included in the runtime estimate.

The consequence. The "~3 minutes" figure represents the marginal cost per scene after all setup is complete, not the end-to-end cost from raw images to queryable scene map. For a practitioner, the true end-to-end time includes: (a) capturing multi-view images, (b) running SVR reconstruction (training time not reported), (c) running SAM2 on all views (time not reported, but SAM2 on hundreds of frames is non-trivial), (d) loading and initializing the three foundation models (memory and I/O overhead), and (e) potentially tuning grouping parameters through trial and error. The comparison with ReferSplat's ">1 hour" is not apples-to-apples because ReferSplat's reported time includes both 3DGS training and language field training—the SVR training time for OpenVoxel's reconstruction step is excluded from the comparison.

Additionally, the paper's stated reason for limiting processing to 150 views is "to make sure that the inference is fast" (Appendix A). This means the ~3-minute runtime is achieved by deliberately processing fewer views than are available. The paper does not report how many total views exist per LeRF scene or what mIoU drop occurs when processing fewer than all views—the runtime claim is achieved at an unknown accuracy cost relative to processing all available data.

What evidence exists in the paper. The runtime comparison (Table 5) explicitly states that ReferSplat "Requires Training: yes" and OpenVoxel "Requires Training: no," but the table footnote clarifies that this refers to "gradient-based training to obtain the semantic or language field"—not to all computation. The paper notes that "running a vanilla 3DGS on RTX 5090 only takes about 10 minutes" and that ObjectGS "needs an additional 30 minutes to obtain the object feature," but does not report the SVR pre-training time that OpenVoxel requires. Appendix A mentions engineering optimizations (limiting views to 150, merging every 1–5 views, 8 DAM frames per group) that trade accuracy for speed, but these tradeoffs are not quantified with ablations.

Mitigation status. The paper does not attempt to account for the full end-to-end cost or to normalize the comparison with baselines by including reconstruction time. The speed claim is qualified by the "Requires Training: no" footnote, but the qualification is easily overlooked. No ablation on the speed-accuracy tradeoff of the view sampling or merging frequency parameters is provided. A practitioner evaluating whether to adopt OpenVoxel over a training-based method cannot determine the true end-to-end time advantage from the reported numbers alone.

Limitation 3: Single Dataset, Single Domain, Single Reconstruction Backbone Leaves Generalization Unverified

The assumption or constraint. All primary quantitative evaluations are on the LeRF dataset (iPhone Polycam-captured indoor scenes) with SVR as the 3D reconstruction backbone. The paper assumes that the pipeline's performance transfers to other reconstruction methods (3DGS, NeRF), other capture modalities (DSLR, drone, LiDAR), other scene types (outdoor, dynamic, large-scale), and other query domains (beyond household objects and food items). The ScanNet experiments (Table 6) partially test generalization to a different dataset and domain, but with evaluation-protocol caveats (discussed below).

The consequence. Several aspects of OpenVoxel's design may be LeRF-specific or SVR-specific:

SAM2 segmentation quality: LeRF scenes contain a small number of well-separated, foreground objects on simple backgrounds (tables, countertops). SAM2's per-frame segmentation is likely more reliable in this setting than in cluttered, occluded, or outdoor environments where object boundaries are ambiguous.
Centroid voting assumption: The grouping mechanism assumes that objects are compact enough that their 3D centroids are well-defined and non-overlapping. For large, articulated, or thin structures (furniture, railings, trees), the centroid may be a poor representation of the instance, and voxels from different objects may vote for overlapping centroid regions.
Captioning specificity: DAM and Qwen3-VL were evaluated on everyday objects (apples, toys, utensils, figurines). Their captioning quality may degrade for domain-specific objects (industrial parts, medical instruments, architectural elements) where vocabulary and visual features differ from the training distribution.
SVR-specific voxel attributes: The grouping update equation (Equation 3) relies on SVR's blending weights w_{ij} to distribute 2D instance information into 3D voxels. A 3DGS-based implementation would require adapting this weighting to Gaussian primitives (possibly using the alpha-compositing weights, but 3DGS Gaussians have spatial overlap that voxels do not, potentially introducing cross-object contamination). The paper does not demonstrate or discuss this adaptation.
Query complexity distribution: The Ref-LeRF RES queries are human-written referring expressions, but their linguistic diversity (vocabulary, syntactic complexity, relational depth) is not characterized. It is possible that these queries are simpler than those a user might naturally produce, and that OpenVoxel's advantage over embedding methods diminishes for genuinely open-ended queries that contain concepts outside DAM's or Qwen3-VL's training vocabulary.

What evidence exists in the paper. The paper acknowledges the dataset scope implicitly by evaluating only on LeRF and ScanNet. The ScanNet experiments (Table 6) show that OpenVoxel achieves 30.0–31.6 mIoU on semantic segmentation, but as noted in Section 5, the evaluation protocol differs from baselines (voxel-to-point mapping vs. direct point-based evaluation) and baselines used ground-truth point cloud initialization while OpenVoxel did not. This makes the ScanNet results suggestive rather than conclusive. No experiments test OpenVoxel on 3DGS reconstructions, outdoor scenes, dynamic scenes, or non-LeRF datasets like Replica, ScanNet++, or custom captures.

Mitigation status. The paper does not claim generalization beyond the tested settings and does not discuss domain transfer. The modular design (Section 4, the three-stage pipeline) is structurally amenable to swapping the reconstruction backbone (SVR → 3DGS) or the foundation models (SAM2 → future segmenters), but this amenability is not demonstrated. Section E acknowledges that "how well-separated for different instances largely affects the performance," hinting at the dependence on scene structure, but does not characterize what scene properties lead to failures.

Limitation 4: The MLLM Is the Dominant Performance Bottleneck, Creating Dependence on a Single Model Family That the Paper Does Not Control

The assumption or constraint. OpenVoxel delegates all language understanding—canonical captioning, query refinement, and text-to-text retrieval—to a single MLLM (Qwen3-VL-8B-Instruct). The system's performance is therefore bounded by this model's instruction-following capability, visual understanding, and reasoning capacity. The paper assumes that Qwen3-VL-8B is "good enough" that its errors are not the primary failure mode, and that larger or future MLLMs will linearly improve performance.

The consequence. The ablation in Table 9 reveals that MLLM choice is the single largest performance lever: RES mIoU ranges from 10.0 (Qwen3-VL-2B) to 42.4 (Qwen3-VL-8B), a 32.4-point swing. This means that OpenVoxel's performance is fundamentally an MLLM benchmark, not a 3D understanding benchmark. A practitioner who swaps Qwen3-VL for a different MLLM family (Claude, GPT-4V, Gemini) may obtain dramatically different results—better or worse—with no way to predict the outcome from the paper's experiments.

The dependence creates several specific failure modes:

Canonicalization failure: The Qwen3-VL-2B model "tends to repeat some of the words in the original caption as the refined caption" (Table 9 analysis), producing captions that are not in the canonical format. This breaks the format alignment that the ablation in Table 4 shows is essential (+14.4 mIoU from canonicalization). Any MLLM with poor instruction-following will produce similarly unusable captions.
Retrieval hallucination: The retrieval step requires the MLLM to compare the canonical query against all scene map entries and select the best match. If the MLLM hallucinates a non-existent relationship, misreads a caption, or fails to parse a spatial relation (e.g., "nearest to"), the returned ID is wrong with no fallback. The paper does not report retrieval error rates or analyze retrieval failure cases.
Prompt sensitivity: The paper explicitly states it "do[es] not spend much effort exploring different kinds of system prompt design" (Appendix A), instead using ChatGPT-generated prompts. Given that MLLM behavior is notoriously sensitive to prompt phrasing, the reported performance may not be reproducible with slightly different prompts, and a practitioner attempting to replicate the results may need to re-tune prompts for their specific MLLM version.
Closed-source dependency in practice: While Qwen3-VL is open-source, the paper's approach intrinsically favors the most capable MLLM available. Future state-of-the-art MLLMs may be closed-source (GPT-5V, Claude 4, Gemini Ultra), accessible only via API, at which point the "training-free" advantage is partially offset by API costs, latency, and vendor dependency.

What evidence exists in the paper. Table 9 directly quantifies MLLM sensitivity, showing a 4× performance range (10.0 to 42.4) across the Qwen family. Table 8 shows that captioning model choice (DAM vs. Osprey vs. Qwen3-VL) produces a 13.1-point range (29.3 to 42.4), indicating that MLLM capability in the captioning stage also matters substantially. The paper does not report results with non-Qwen MLLMs, does not ablate prompts, and does not analyze retrieval failure cases by query type.

Mitigation status. The paper acknowledges MLLM dependence in Section E: "if better open-source MLLM appear with faster thinking/reasoning ability, our OpenVoxel can be benefited from them." This reframes the limitation as a feature (performance tracks foundation model progress), but sidesteps the practical concern that current performance is fragile with respect to MLLM choice. The paper does not propose fallback mechanisms for when the MLLM fails (e.g., embedding-based retrieval as a backup), does not provide prompt-tuning guidance, and does not report confidence scores or uncertainty estimates from the retrieval step that a practitioner could use to detect likely failures.

Limitation 5: Instance-Level Grouping Precludes Part-Based and Sub-Object Queries, Creating a Resolution Ceiling

The assumption or constraint. The grouping algorithm is designed to produce instance-level groups—each group corresponds to a whole object (apple, toy, cup, table). This is a deliberate design choice, embedded in the centroid-voting mechanism (which assumes one centroid per instance) and the SAM2 re-prompting merge step (which merges fragments that overlap into a single group). The paper assumes that users will query for whole objects and that part-based queries can be satisfied by retrieving the containing object.

The consequence. Queries that refer to parts of objects cannot be answered at the correct granularity. The paper explicitly acknowledges this in Section E: "if the user gives a query to indicate some part of a larger object (e.g., flash light of the camera in Fig. 5), our OpenVoxel would still segment the whole object (i.e., the whole camera) since the small parts of the same object is bundled together." This is not a rare edge case—many natural referring expressions target parts ("the handle of the mug," "the lens of the camera," "the screen of the laptop," "the drawer of the desk," "the wheel of the bicycle"). In these cases, OpenVoxel returns the entire parent object, which is incorrect for the RES task (ground truth mask covers only the part) and produces a false-positive segmentation that may be substantially larger than the intended target.

This limitation creates a granularity ceiling: OpenVoxel's segmentation resolution is bounded by the instance-level grouping, regardless of how good the captioning or retrieval becomes. Even if DAM produces a perfect caption describing the camera's flash, and Qwen3-VL correctly identifies that the query refers to the flash, the system can only segment the entire camera—because that is the only group that exists. The retrieved mask will have low IoU with the part-level ground truth.

The problem is structural rather than incidental—it arises from the decision to merge fragmented SAM2 masks into instance-level groups. If the merging step were disabled or made less aggressive, OpenVoxel would produce part-level groups, but this would create the opposite problem: parts would be described independently, and queries for whole objects would need to retrieve and merge multiple part groups, which the current retrieval mechanism is not designed to do. There is no single grouping granularity that satisfies both whole-object and part-object queries.

What evidence exists in the paper. The paper gives the camera flash example in Section E, acknowledging the limitation directly. No quantitative evaluation of part-based query performance is provided. The LeRF subsets (Ref-LeRF, LeRF-OVS, LeRF-Mask) primarily contain whole-object annotations; it is unclear how many ground-truth masks correspond to object parts and how this affects the reported mIoU. The qualitative results (Figures 5, 6, 8, 9) all show whole-object segmentation, not part segmentation.

Mitigation status. The paper proposes two possible directions (Section E): (1) "curate the groups as small as possible while still keeping them semantically reasonable (requires 2D segmentation maps that includes those small part)," and (2) "construct a complex scene graph for all the groups to indicate the spatial relations (e.g., on top of, between) or ownership (e.g., belongs to, part of) explicitly." Neither is implemented or evaluated. The first direction essentially advocates for a finer grouping granularity with a hierarchical grouping scheme (part-of relations linking sub-object groups to parent-object groups), which would require significant redesign of both the grouping and retrieval stages. The second direction would add symbolic relations to the scene map but would not solve the segmentation granularity problem—knowing that the flash "belongs to" the camera does not produce a flash-only segmentation mask if the flash voxels are not a separate group.

Limitation 6: No Mechanism for View-Dependent or Deictic Queries, Limiting Interactive Applicability

The assumption or constraint. The scene map S stores only view-independent information: group IDs, 3D centroid positions, and canonical captions. All spatial relations in queries are interpreted using world-coordinate distances between these centroids. The system explicitly forbids view-dependent reasoning: the retrieval prompt (List 3, Appendix A) instructs the MLLM to "ignore view-dependent relations (left/right/front/behind)" and to "use WORLD coords ONLY for near/far/between/closest/farthest." The paper assumes that users will formulate queries in view-independent terms (e.g., "the chair near the table") rather than view-dependent terms (e.g., "the chair to my left").

The consequence. Many natural queries in interactive AR or robotics settings are inherently view-dependent or deictic (pointing-based). A user wearing an AR headset might say "the red cup on the right" (where "right" is from their current perspective) or "that chair" (while pointing). A robot navigating a scene might receive instructions like "the door in front of you." OpenVoxel cannot handle these queries because the scene map lacks any representation of the user's current viewpoint, and the retrieval policy explicitly discards view-dependent spatial language.

The paper's supplementary qualitative results (Figure 10) partially demonstrate this limitation. The last two columns of Figure 10 show queries with view-dependent descriptions ("nearest to the current view," "nearest to the sheep and the camera"). The paper notes that OpenVoxel "retrieve[s] two targets instead of the only one matched, but the results are still including the correct target, showing the potential capability of solving view-specific tasks." This is optimistic framing—retrieving multiple targets when only one is correct means the system cannot disambiguate the query, and in a deployed system, returning multiple masks without knowing which is correct is a failure mode, not a feature. The "potential capability" is not realized.

A subtler consequence is that even view-independent queries may require viewpoint awareness for correct disambiguation. "The chair nearest to the sheep" is technically a view-independent query (world-space distances can be computed from centroids), but if two chairs are at similar distances from the sheep, a human would use visual perspective to disambiguate—something OpenVoxel cannot do because the retrieval MLLM only sees the scene map (a JSON list) and an optional query image, not a 3D rendering with the user's camera pose.

What evidence exists in the paper. Figure 10 shows the two-target retrieval for view-dependent queries with the author's commentary. The retrieval system prompt (List 3) explicitly excludes view-dependent relations. No quantitative evaluation of view-dependent query performance is provided—the Ref-LeRF and LeRF-OVS queries appear to be written from a neutral, view-independent perspective. The paper does not report what fraction of natural referring expressions in common benchmarks (e.g., ReferIt3D, ScanRefer) are view-dependent, so the scope of the limitation is unclear.

Mitigation status. The paper does not propose a solution for view-dependent queries. Section E discusses building "a complex scene graph," but this would improve spatial relation reasoning in world coordinates, not enable viewpoint-relative reasoning. A full solution would require: (a) accepting the user's camera pose as input, (b) projecting the scene map into the user's view to compute "left," "right," "in front," "behind" relations, and (c) potentially rendering the scene from the user's viewpoint and providing this rendering to the retrieval MLLM as additional context. None of these are implemented, and they would add complexity to the pipeline's interface (currently accepting only an optional query image, not a camera pose). The paper's training-free design does not preclude these extensions, but they are non-trivial and would require changes to the retrieval mechanism.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper is best understood not as a new method that competes with embedding distillation, but as a paradigm diagnostic—it demonstrates that for open-vocabulary 3D scene understanding, the embedding paradigm hits a fundamental representational limit on compositional queries, and that crossing that limit requires abandoning learned embeddings entirely in favor of explicit symbolic scene representations with text-to-text reasoning. The paper did not set out to be a diagnostic (it proposes a specific pipeline), but its experimental results inadvertently make the diagnostic case more strongly than any methodological contribution: the 13–18 point mIoU gap between OpenVoxel and ReferSplat on RES, compared to the ~1 point margin on OVS, is the empirical signature of a representational bottleneck that cannot be closed by better training recipes, contrastive losses, or codebook designs.

This changes the landscape in four specific ways:

First, it reframes the primary research question. Before OpenVoxel, the question in the 3D language embedding community was "how can we learn better language-aligned features for 3D primitives?"—a question asked and answered by LangSplat, OpenGaussian, 3DVLGS, CCL-LGS, and a dozen others. After OpenVoxel, the question becomes "should we be learning embeddings at all, or should we be constructing explicit symbolic scene maps and delegating retrieval to multi-modal reasoning models?" This is a higher-stakes question because it forces the field to choose between two incompatible paradigms: embedding distillation (which requires per-scene training but is self-contained) versus symbolic scene construction (which is training-free but depends on external foundation models that the researcher does not control). The paper does not answer this question definitively—it shows superiority on RES but near-parity on OVS—but it makes the question unavoidable for future work.

Second, it redefines what constitutes a "method" in this space. OpenVoxel is not a single algorithm but a composition of frozen foundation models connected by a dataflow graph (SAM2 → grouping → DAM → Qwen3-VL → retrieval). Its performance is not a property of any novel learning objective or architecture but of the orchestration of existing capabilities. This is a methodological shift with practical consequences: future work in this paradigm will advance not by proposing new loss functions or network architectures but by identifying better foundation models for each subtask, designing better interfaces between them (canonical templates, scene maps), and characterizing failure modes to route queries to the appropriate reasoning strategy. The paper's modular ablation studies (Tables 7–9) are arguably more important than the final performance numbers because they establish the decomposability of the problem—each component can be improved independently, and the gains compound.

Third, it reconciles a latent tension in the field between 3D scene representation purists and foundation model pragmatists. The 3D reconstruction community has invested heavily in learning better primitives, better geometric regularizers, and better rendering pipelines. The embedding distillation approach extended this investment by adding trainable language features to the primitives—the representation itself became language-aware. OpenVoxel demonstrates that language awareness can be achieved entirely ex post facto, by processing the reconstructed geometry with external models. This decoupling means that progress in 3D reconstruction (better voxels, better Gaussians, better NeRFs) and progress in language understanding (better VLMs, better MLLMs, better captioners) can proceed independently and be combined at the pipeline level. The implication is that a researcher specializing in neural rendering does not need to also become an expert in vision-language alignment to produce a language-aware 3D system—they can reconstruct the scene, run OpenVoxel, and be done.

Fourth, it establishes RES as the canonical benchmark for compositional 3D understanding, displacing OVS as the default evaluation task. Prior work used OVS (single-category queries like "apple" or "chair") as the primary evaluation, with RES as a secondary, harder task. OpenVoxel's results invert this: OVS performance has begun to saturate (ObjectGS achieves 88.3 mIoU on LeRF-Mask, CCL-LGS achieves 65.1 on LeRF-OVS), and the remaining headroom is in RES, where even the best training-based method scores only 29.2 mIoU. This paper demonstrates that RES is where architectures diverge—embedding methods plateau while symbolic methods continue to improve—making it the more diagnostic benchmark for assessing whether a method genuinely "understands" language in 3D. Future work that reports only OVS results should be viewed with skepticism; RES is the task that distinguishes paradigm-level choices from incremental refinements.

The paper does not resolve the fundamental question of what task complexity embedding methods can eventually handle. It shows they fail on Ref-LeRF RES queries, but it does not characterize the linguistic properties that cause failure (relational depth? attribute count? vocabulary rarity?), so we cannot predict whether embedding methods will remain competitive on, say, robotic instruction following or AR scene description. A negative-results paper that systematically characterized the failure boundary of embedding methods as a function of query complexity would be highly valuable and is enabled by OpenVoxel's existence as a contrasting paradigm.

Follow-Up Research This Work Enables

Decomposing RES errors into grouping, captioning, and retrieval failures using oracle upper bounds. The single most informative experiment this paper does not perform is an oracle retrieval study: replace the MLLM retrieval step with ground-truth group-to-query matching (i.e., always return the correct group ID for each query) and measure the resulting mIoU. This would establish the upper bound achievable with perfect retrieval, decomposing the 42.4 mIoU into: (a) error from incorrect grouping (objects merged or split), (b) error from inadequate captions (correct group but MLLM cannot match because the caption lacks distinguishing detail), and (c) error from retrieval failure (MLLM selects wrong group despite adequate captions). A follow-up study would run this decomposition across all LeRF scenes and query types, producing a stacked bar chart of error attribution. The prediction based on the paper's ablations: retrieval error dominates (~8–12 mIoU loss, from the 32-point MLLM sensitivity in Table 9 and the +6.0 point query canonicalization gain in Table 4), grouping error contributes significantly (~5–8 mIoU loss, from the 11.9-point SAM vs. SAM2 gap in Table 7), and captioning error is modest (~3–5 mIoU loss, from the 13.1-point captioning model gap in Table 8). If retrieval error is indeed the dominant term, the research priority shifts to MLLM selection and prompt optimization rather than grouping algorithms.

Systematic characterization of the embedding-vs-text boundary as a function of query complexity. The paper shows a 13–18 point gap favoring text-to-text on RES but near-parity on OVS, suggesting a crossover point in query complexity where the paradigms diverge. A follow-up study would construct a graded benchmark of 3D queries ordered by linguistic complexity—starting from single nouns (OVS), through adjective-noun phrases ("red cup"), through multi-attribute descriptions ("red ceramic mug with a chip on the rim"), through relational queries ("the cup to the left of the plate"), through compositional queries ("the object that can hold liquid and is closest to the window"), and ending with full referring expressions from Ref-LeRF. Plotting performance of both paradigms (OpenVoxel and the best embedding method) against this complexity axis would reveal the exact linguistic threshold where embeddings fail and text-to-text pulls ahead. The concrete hypothesis: embedding methods match or exceed text-to-text up to adjective-noun phrases, begin to lag at multi-attribute descriptions, and collapse at relational and compositional queries. If this hypothesis holds, it provides a principled basis for hybrid systems that route simple queries to fast embedding-based retrieval and complex queries to slower MLLM-based retrieval.

Applying OpenVoxel to 3D Gaussian Splatting reconstructions to test representation independence. The paper builds on SVR and explicitly leverages voxel-specific properties (discrete grid cells, well-defined spatial extents, the blending weight w_{ij} for updating the group field). A critical test of the paradigm's generality is whether the same training-free pipeline works on 3DGS scenes, which are currently the dominant 3D primitive in the neural rendering community. The adaptation would require: (a) replacing SVR's volume rendering with 3DGS's alpha-blending rasterizer for rendering group masks and point maps, (b) adapting the group field update (Equation 3) to use 3DGS's alpha-compositing weights (which are analogous to w_{ij} but operate on overlapping Gaussians rather than discrete voxels), and (c) determining whether the centroid-voting mechanism remains stable when primitives have overlapping spatial support. A concrete experiment: reconstruct the LeRF scenes with 3DGS, run SAM2 grouping and canonical captioning on the Gaussians, and report RES mIoU. If performance is comparable to the SVR-based results (~42 mIoU), the training-free paradigm is representation-agnostic. If performance drops significantly, it identifies a dependency on discrete, non-overlapping primitives and motivates research on Gaussian-specific grouping methods.

Combining OpenVoxel's scene map with an embedding-based method for hybrid retrieval with graceful degradation. The paper positions text-to-text as an alternative to embedding retrieval, but the two are not mutually exclusive—the scene map S contains structured captions that could be embedded (with a lightweight sentence encoder) to support fast nearest-neighbor lookup for simple queries, with the MLLM invoked only when the embedding-based match is ambiguous (low cosine similarity to the top candidate, or multiple candidates within a narrow similarity band). A follow-up study would implement this hybrid: embed all scene map captions with a sentence transformer, perform embedding-based retrieval as a first pass, compute a confidence score (e.g., the ratio of the top-1 to top-2 similarity), and route queries with low confidence to the MLLM retrieval pipeline. This preserves OpenVoxel's compositional reasoning capability for hard queries while reducing MLLM inference cost (and latency) for the majority of queries that are simple. The concrete research question: what fraction of Ref-LeRF queries can be correctly answered by embedding-based retrieval alone if the embeddings are computed from OpenVoxel's canonical captions rather than from per-primitive distilled features? If the answer is >50%, the hybrid system offers a practical deployment advantage (reduced API costs and latency) with no accuracy degradation on simple queries.

Stress-testing the pipeline with intentionally difficult scenes and queries to find the failure boundary. The paper evaluates on LeRF scenes with well-separated objects and human-written referring expressions of unknown difficulty distribution. A rigorous follow-up would construct a challenge set specifically designed to probe each component's failure modes: (a) scenes with transparent, reflective, or textureless objects (to stress SAM2 segmentation), (b) scenes with articulated or nested objects like "drawer in a desk" or "pillow on a chair" (to stress the instance-level grouping granularity), (c) queries using rare vocabulary, metaphors, or negations ("the object that is NOT cylindrical"), (d) queries requiring counting or set operations ("the third cup from the left," "all the red objects"), and (e) queries with contradictory or underspecified information ("the blue apple" when no blue apple exists). Measuring performance on this challenge set would produce a capability profile that tells practitioners which classes of queries OpenVoxel can reliably handle and which require human intervention or a different approach. The paper's qualitative examples (Figures 5, 6, 8, 9) already hint at some failure modes (view-dependence, over-segmentation) but do not quantify their frequency.

Training a lightweight difficulty estimator and scene-specific router that allocates queries between OpenVoxel and a fast embedding baseline. The paper acknowledges the cost of difficulty estimation (2048 samples in the referenced example) as a key limitation, and OpenVoxel inherits a related problem: the full pipeline with MLLM retrieval takes ~1 second per query (acceptable for interactive use), but the MLLM inference still consumes GPU memory and has non-trivial energy cost. A follow-up study would train a small classifier (e.g., a DistilBERT variant) that takes a query text as input and predicts whether it is "simple" (likely answerable by embedding retrieval of the canonical captions) or "complex" (requiring MLLM reasoning). The classifier would be trained on the Ref-LeRF queries labeled by whether embedding-based retrieval succeeds on them. At deployment, simple queries bypass the MLLM entirely, reducing average per-query cost and latency. The concrete evaluation metric: trade-off curves of accuracy vs. MLLM call frequency, showing that 80% of queries can be answered with embedding retrieval at 95% of the accuracy of full MLLM retrieval. This would directly address the practical concern that MLLM inference is expensive at scale.

Practical Applications and Downstream Use Cases

Rapid post-capture 3D scene understanding for consumer AR. A user captures their living room with a smartphone (30-second video), SVR reconstructs a sparse voxel model in ~10 minutes on-device or in the cloud, and OpenVoxel processes the reconstruction in ~3 additional minutes, producing a scene map that can answer queries like "where are my keys?" or "show me the book I was reading yesterday." The 3-minute processing time (Table 5) makes this viable as a background task that completes shortly after capture, compared to the >1 hour required by ReferSplat, which would be unacceptable in a consumer context. The scene map is human-readable, so the user can verify that the system correctly identified objects. The primary technical risk is that SAM2 segmentation quality on casually captured phone video (with motion blur, varying lighting, occlusions) may be lower than on the LeRF dataset, requiring the grouping robustness analysis called for in the failure-boundary study above.

Batch annotation of 3D scene datasets for embodied AI training. Training embodied AI agents (robotics, navigation, manipulation) requires large datasets of 3D scenes annotated with object identities, attributes, and spatial relations. Current annotation pipelines rely on human labelers drawing 3D bounding boxes and writing descriptions—a slow, expensive process. OpenVoxel can process thousands of reconstructed scenes offline, producing scene maps with object-level groupings and canonical captions, which human annotators can then verify and correct rather than create from scratch. The ~3 minute per-scene processing time means 500 scenes can be annotated in ~25 GPU-hours—a day of compute on a single 8-GPU machine. The 42.4 mIoU on RES suggests that automated object identification is reliable enough to serve as a starting point, with human effort focused on correcting the ~58% of queries where retrieval fails, rather than annotating from scratch. The key unknown is the verification cost: how long does a human take to review and correct a scene map entry compared to creating one from scratch? If verification is >3× faster than creation, the approach is economically compelling.

Accessibility tools for describing environments to visually impaired users. A headset-mounted camera captures the user's surroundings, reconstructs a 3D scene, and OpenVoxel produces a scene map that a text-to-speech system can query in response to natural language questions: "what's on the table in front of me?", "is there an empty chair?", "where's the exit?" The critical advantage of OpenVoxel over embedding-based methods in this setting is that the scene map is interpretable—if the system makes a mistake (e.g., misidentifies a lamp as a person), the error is visible in the caption and can be flagged by a verification step. The <1 second per query inference time enables conversational interaction. The primary deployment challenge is that the current pipeline requires an RTX 5090 GPU and multiple large foundation models (SAM2, DAM, Qwen3-VL), which are infeasible for a wearable device. This could be addressed by offloading computation to a cloud endpoint or by model distillation—training smaller, specialized models that approximate each stage's output.

When to Prefer This Method

The paper does not provide an explicit decision framework comparing OpenVoxel to named alternatives under specific conditions. The experimental results (Tables 1–3) position OpenVoxel as superior on RES tasks, competitive on OVS tasks, and dramatically faster than training-based methods—but these are empirical comparisons, not prescriptive tradeoff rules. The paper does not state conditions like "prefer OpenVoxel when queries are compositional and annotating description-mask pairs is infeasible; prefer ReferSplat when per-scene training time is not a constraint and human annotations are available for every object." Readers should consult Section 6 (Limitations) for the specific failure modes—grouping sensitivity, MLLM dependence, instance-level granularity ceiling, no view-dependent queries—that define the current boundaries of applicability.