WildDet3D: Scaling Promptable 3D Detection in the Wild

🎯 Pitch

A single monocular 3D detector can now accept text, points, or boxes as prompts and see a 20.7 AP boost just by adding depth at test time—no retraining needed. This leap comes from training on 1M human-verified 3D boxes across 13.5K categories, proving that broad data, not just bigger models, unlocks open-world spatial intelligence.

1. Executive Summary

This paper introduces WildDet3D, a unified, geometry-aware architecture for monocular 3D object detection that natively accepts text, point, and box prompts within a single model, and WildDet3D-Data, the largest open 3D detection dataset to date—over 1M human-verified images across 13.5K categories. The core technical contributions are a dual-vision encoder with an optional depth fusion module (allowing the model to gracefully leverage depth cues from LiDAR or stereo at inference time without architectural changes) and a promptable detector that conditioning metric-depth-aware features on flexible user prompts, trained with a deeply-supervised 3D head using unambiguous rotation normalization. On the newly introduced WildDet3D-Bench, WildDet3D achieves 22.6/24.8 AP3D with text and box prompts respectively, while incorporating depth yields a +20.7 AP average gain across settings (e.g., reaching 41.6 AP with text prompts and ground-truth depth). In a FLOPs-matched finding, the model establishes a new state-of-the-art on Omni3D (34.2 AP text, 36.4 AP oracle) using only 12 training epochs versus 80–120 for prior methods, and transfers zero-shot to Argoverse 2 and ScanNet at 40.3 and 48.9 ODS, establishing that test-time depth integration and multi-prompt unification drive substantial generalization gains only when paired with broad open-world training data—on the hardest categories and without any depth signal, monocular ambiguity remains fundamentally limiting.

2. Context and Motivation

The Core Problem: 3D Understanding That Works in the Open World

The paper addresses a fundamental tension in computer vision: 2D perception has become remarkably flexible and open-world, but 3D perception remains brittle and closed-world. Systems like SAM 3 [8] can segment any object from a point or box prompt, vision-language models can detect thousands of categories from text queries, and open-vocabulary 2D detectors can recognize novel objects never seen during training. Yet when we need to understand where objects are in physical space—their metric position, their dimensions in meters, their orientation in 3D—existing systems fall dramatically short of this flexibility.

This gap matters because spatial intelligence requires geometry, not just appearance. An augmented reality headset needs to know exactly where your coffee mug sits on the desk to overlay virtual content anchored to it. A robot arm needs the 3D center and dimensions of an object to generate a viable grasp. An autonomous vehicle needs metric depth and orientation to reason about occlusions and safe trajectories. These applications cannot function with 2D bounding boxes alone—they demand recovery of the full 6-DoF (six degrees of freedom) pose: 3D position, 3D dimensions, and 3D orientation.

The specific technical problem is monocular 3D object detection: given a single RGB image, predict oriented 3D bounding boxes for objects in the scene. This is inherently ill-posed. A single image provides no direct depth measurement, so the same 2D projection can correspond to infinitely many 3D interpretations. A small nearby object is indistinguishable from a large distant one (scale ambiguity). An object rotated by a certain amount can produce nearly identical appearance to one with different dimensions rotated differently (rotation-dimension ambiguity). These ambiguities cannot be resolved from appearance alone—they require learned priors about typical object sizes, shapes, and scene layouts, which are fundamentally statistical and break down for novel objects or unusual viewpoints.

The paper frames what it would take to build a "truly general-purpose monocular 3D detector" through three requirements that are "not well addressed by existing methods" (Section 1):

"First, it should generalize in the wild, where object categories are long-tailed, open-ended, and frequently unseen during training. Second, it should support multiple prompt modalities. [...] Third, real deployments may sometimes provide extra geometric cues, such as sparse LiDAR or partial depth, which should be leveraged to improve 3D localization when available."

These requirements together define an open-world, interactive, geometry-aware 3D perception system. No prior work addresses all three simultaneously.

Why This Problem Matters Now

The timing of this work is not accidental. Several technological trends have converged to make open-world 3D detection both more necessary and more feasible:

1. The 2D perception stack has matured. Foundation models for 2D vision—SAM [40], SAM 3 [8], open-vocabulary detectors like Grounding DINO [30], vision-language models like Molmo [10, 13]—can now reliably ground objects in 2D from diverse prompt types (text, points, boxes, masks). This creates a clear opportunity: rather than building 3D perception from scratch, can we leverage these powerful 2D priors and lift them into 3D? The paper's explicit goal is to "bring the prompt flexibility of modern 2D foundation models into 3D" (Section 6).

2. Depth sensing is becoming ubiquitous but remains patchy. Modern smartphones ship with LiDAR scanners (iPhone Pro models), AR headsets have depth sensors (Meta Quest 3, Apple Vision Pro), and autonomous vehicles combine cameras with LiDAR and radar. But these sensors have dramatically different characteristics: smartphone LiDAR is short-range and sparse, automotive LiDAR is long-range but expensive, and many consumer devices have no depth sensor at all. A practical system must handle this heterogeneity—deploying in monocular mode on a budget phone, leveraging sparse LiDAR on an iPhone Pro, and using dense stereo depth on an AR headset—without requiring different models for each hardware configuration.

3. Robotics and embodied AI demand 3D understanding. The recent surge in robotics research (manipulation, navigation, mobile manipulation) has exposed a critical bottleneck: 2D vision-language models can describe scenes and identify objects, but robots need metric 3D information to act. The paper explicitly demonstrates this with a Franka Emika Panda arm (Section 5, Figure 9c), where WildDet3D provides open-vocabulary 3D bounding boxes directly consumed for grasp planning, "providing a zero-shot alternative to task-specific 3D perception modules that require per-object training or CAD models."

4. AR/VR requires spatial anchoring. Augmented reality applications need to place virtual content in physical space with correct occlusion, scale, and persistence. The paper demonstrates integration with Meta Quest 3 (Figure 9b), where WildDet3D produces metric 3D boxes anchored in the headset's coordinate frame—something that 2D detection alone cannot provide.

Where Prior Approaches Fall Short

The paper identifies three distinct lines of prior work, each addressing a subset of the full problem but leaving critical gaps.

Closed-Set Monocular 3D Detection: Accurate but Inflexible

Early monocular 3D detectors achieved reasonable accuracy within narrow domains. Methods like M3D-RPN [5], SMOKE [31], FCOS3D [53], and MonoDETR [66] focused primarily on autonomous driving, detecting a small fixed set of categories (car, pedestrian, cyclist) from forward-facing cameras. Cube R-CNN [6] and Omni3D [6] took an important step toward unification by training across multiple datasets (KITTI, nuScenes, SUNRGBD, Hypersim, ARKitScenes, Objectron) with a shared model architecture and standardized 98-category label space. UniMODE [28] further improved cross-domain generalization.

However, these methods share a fundamental limitation: they operate in a closed-set vocabulary. The model can only detect categories seen during training, and adding a new category requires retraining or fine-tuning. This makes them unsuitable for open-world applications where users might query arbitrary objects—"pick up the red mug," "find the fire extinguisher," "locate the closest exit sign"—that may not appear in any training dataset. Moreover, they assume a fixed interaction mode: the model detects all objects of all known categories in every image. There is no mechanism for a user to specify which object they care about through a flexible prompt (a text query, a point click, a 2D box, or a visual exemplar).

Open-Vocabulary 3D Detection: Flexible Vocabulary but Narrow Interfaces

A more recent line of work addresses the closed-set limitation by leveraging open-vocabulary 2D detectors. 3D-MOOD [59] runs an open-vocabulary 2D detector (Grounding DINO), extracts region features from detected 2D boxes, and lifts them to 3D using a learned lifting network. Open Vocabulary Monocular 3D Object Detection [60] (denoted OVMono3D-LIFT) follows a similar paradigm. OVM3D-Det [19] explores training without any 3D data by using 2D detection priors and geometric reasoning. LocateAnything3D [33] couples vision-language models with chain-of-sight reasoning for 3D detection.

These methods represent genuine progress toward open-world 3D perception—they can detect novel categories specified by text queries, generalizing beyond their training vocabulary. But the paper identifies two key limitations:

First, they specialize to a single prompt interface. 3D-MOOD and OVMono3D-LIFT are text-prompt detectors: you provide a category name, and the model returns all instances of that category. This works for category-level queries ("find all chairs") but cannot handle instance-level interaction ("this specific chair, the one I'm pointing at"). In practice, different applications demand different prompt modalities: a robot following a language command needs text-prompt detection; an AR user tapping on an object needs point-prompt detection; an upstream 2D object detector providing bounding boxes needs box-prompt lifting. Prior open-vocabulary methods cannot unify these interfaces within a single model.

Second, they lack a mechanism to incorporate additional geometric cues. When depth sensors are available (LiDAR, stereo, structured light), these methods cannot benefit from the extra information. Their architectures are purely monocular—they learn to estimate depth implicitly from RGB features but have no pathway to ingest actual depth measurements at inference time. This means they leave substantial accuracy on the table when sensors are present, and cannot degrade gracefully between sensor-rich and sensor-poor deployment scenarios.

Promptable 3D Box Prediction: Flexible Interfaces but Oracle Assumptions

A complementary line of work focuses on prompt-conditioned 3D box prediction. DetAny3D [64] takes a 2D bounding box as input and predicts the corresponding 3D box—lifting 2D detections into 3D. SAM-3D [50] reconstructs 3D meshes from object masks and depth-derived point clouds, then extracts oriented bounding boxes. These methods support geometric prompts (2D boxes or masks) but lack text-prompt interfaces for category-level open-vocabulary detection. They also typically assume that the 2D prompt is provided by an oracle (e.g., a ground-truth 2D box) rather than by an end-to-end detection pipeline, making them less suitable for fully autonomous open-world perception where the user specifies only a category name.

Crucially, none of these prior methods—closed-set detectors, open-vocabulary text detectors, or promptable box lifters—support all three prompt modalities (text, point, box) within a single unified architecture, nor do they provide a mechanism for optional depth integration at inference time.

The Data Bottleneck: Why 3D Detection Hasn't Scaled Like 2D

Beyond model architecture, the paper identifies a data bottleneck as the primary obstacle to open-world 3D detection. The contrast with 2D detection is stark: COCO [29] provides 118K training images with 80 categories; LVIS [18] annotates the same images with 1,200+ categories; Objects365 [45] offers 609K images across 365 categories; V3Det [51] spans 13K+ fine-grained categories. These datasets enabled the development of open-vocabulary 2D detectors that generalize to thousands of object types.

In 3D, the situation is dramatically poorer. Omni3D [6]—the largest unified 3D detection benchmark—covers only 98 categories across 234K images, most of which come from autonomous driving and indoor scene datasets with limited visual diversity. The reason is straightforward economics:

"Unlike 2D bounding boxes, 3D annotations require metric depth and calibrated camera intrinsics, both of which are costly to obtain at scale." (Section 3)

You cannot simply ask crowdworkers to draw 3D boxes on images the way they draw 2D boxes. 3D annotation requires depth sensors, calibrated multi-view setups, or skilled annotators manipulating 3D cuboids in point clouds—all of which are orders of magnitude more expensive than 2D annotation. As a result, existing 3D detection datasets are (1) small in vocabulary (tens to low hundreds of categories), (2) narrow in domain (mostly driving and furniture), and (3) captured in controlled environments rather than in-the-wild imagery.

Several recent efforts have attempted to address this through automatic or semi-automatic pipelines. LabelAny3D [61] uses analysis-by-synthesis to produce 3D box annotations and builds COCO3D with 80 categories. CA-1M [24] provides class-agnostic cuboid annotations for indoor scenes. But automatic annotation is inherently noisy—especially for object rotation and extent—and existing pipelines rely on a single lifting method, which amplifies the biases and failure modes of that method.

The paper's insight is that scaling 3D detection to the open world requires both a model that can leverage diverse, imperfect supervision and a data pipeline that combines multiple complementary sources with human verification. The WildDet3D-Data pipeline (Section 3) operationalizes this insight by generating candidate 3D boxes from five different lifting methods (3D-MOOD, DetAny3D, SAM-3D, RANSAC-PCA, LabelAny3D), then selecting the best candidates through VLM scoring and human annotation, yielding a dataset of 1M images spanning 13.5K categories—a 138× increase over Omni3D's 98 categories (Table 1).

How This Paper Positions Itself

WildDet3D positions itself at the intersection of three converging trends that have so far remained separate:

First, it unifies prompting flexibility with open-vocabulary detection. Rather than forcing a choice between text-prompt open-vocabulary detectors (3D-MOOD) and box-prompt 3D lifters (DetAny3D), WildDet3D proposes a single architecture that accepts all prompt types—text, point, box, and exemplar—and conditions its predictions on whichever prompt the user provides. This is achieved through a promptable detector (Section 2.2) that borrows prompt encoding from SAM 3 [8] and extends it to 3D, enabling the same model to run open-vocabulary text detection for category-level queries and instance-level geometric prompting for interactive applications.

Second, it treats depth as an optional rather than required input. The dual-vision encoder design (Section 2.1) separates semantic feature extraction (image encoder) from geometric reasoning (RGBD encoder with optional depth input). When depth is available—from LiDAR, stereo, or depth sensors—the depth fusion module injects geometric cues into the visual features via a ControlNet-style [65] residual connection. When depth is absent, the RGBD encoder operates on a zero-filled depth channel, and the model falls back to learned monocular depth priors. This design is deliberately modular: "different depth models can be integrated without modifying the core detection pipeline" (Section 2.1). The paper provides extensive evidence that this optional depth mechanism works in practice: ground-truth depth at test time yields a +20.7 AP average gain across settings (Section 4.2, Table 3), and real stereo depth on Stereo4D produces a 2.8× improvement (Table 6).

Third, it recognizes that architecture alone cannot solve the open-world generalization problem—data scale and diversity are equally critical. The WildDet3D-Data pipeline is not an afterthought but a core contribution, comprising roughly one-third of the paper (Section 3). By combining five complementary candidate generation methods with VLM-based scoring and human verification, the pipeline produces a dataset that is both larger (1M+ images) and more diverse (13.5K categories, 22 scene types, indoor/outdoor/nature) than any prior 3D detection dataset. The paper explicitly validates the pipeline through human rejection rate analysis (Table 2), showing that candidate quality varies by >3× across generation methods and that VLM scoring correlates perfectly with human judgment (Spearman ρ = −1.0).

A crucial design philosophy evident throughout the paper is graceful degradation rather than brittle dependence. The architecture degrades from depth-enhanced to monocular mode; the promptable detector degrades from instance-level box prompts to category-level text prompts; the training pipeline degrades from human-verified annotations to VLM-filtered synthetic annotations to purely monocular Omni3D supervision. This philosophy reflects a realistic assessment of deployment conditions: the model must work when sensors fail, when users provide coarse prompts, and when only weak supervision is available for novel categories.

The paper's ultimate positioning is as a general-purpose 3D perception module—"a universal 3D lifting module that bridges high-level language reasoning with precise spatial localization" (Section 5). The deployment demonstrations (iPhone, Meta Quest 3, Franka robot, VLM integration) are intended not as polished products but as evidence that the model can serve as a plug-and-play component across diverse platforms and interaction paradigms, accepting whatever prompts and sensor data are available and producing metric 3D understanding in return.

3. Technical Approach

This is primarily a systems and data engineering paper whose core ideas are: (1) a unified architecture that makes monocular 3D detection flexible (multi-prompt), geometry-aware (optional depth), and open-vocabulary, and (2) a data pipeline that produces large-scale 3D supervision by combining multiple lifting methods with human verification, enabling generalization to 13.5K categories.

3.1 Reader Orientation

What the system is: WildDet3D is a neural network that takes an RGB image, an optional depth map, and a user prompt (text phrase, 2D point click, or 2D bounding box) and outputs metric 3D bounding boxes—each with a 3D center position (in meters), physical dimensions (width, height, length in meters), 3D orientation (a rotation matrix), and a confidence score—for the objects specified by the prompt.

What problem it solves and the shape of the solution: The problem is that existing monocular 3D detectors are either closed-set (fixed categories), single-prompt (text only or box only), or geometry-oblivious (cannot use depth when available). The solution is an architecture with two parallel vision encoders (one for semantics, one for geometry) whose outputs are fused, fed into a prompt-conditional detector, and decoded by a deeply-supervised 3D head that resolves the inherent rotation-dimension ambiguity through a normalization procedure. This architecture is trained on a new dataset built by generating candidate 3D boxes from five complementary methods, filtering them through geometric rules and VLM scoring, and having humans select the best candidate, yielding 1M+ images with verified 3D annotations.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, arranged as a feed-forward pipeline with multiple input pathways:

Dual-Vision Encoders (Section 2.1): Two parallel backbones process the input image. The Image Encoder (ViT-H from SAM 3, frozen through layer 28 of 32) extracts high-resolution multi-scale semantic features for detection. The RGBD Encoder (DINOv2 ViT-L from LingBot-Depth, frozen through layer 21 of 24) ingests a 4-channel RGBD input—where the depth channel is zero-filled when no external depth is available—and produces a 5-level feature pyramid from which depth latents are extracted.
Depth Fusion Module (Section 2.1, yellow block in Figure 3): A lightweight residual module that injects the RGBD encoder's depth latents into the Image Encoder's feature maps before they enter the transformer decoder. It bilinearly interpolates depth latents to match visual feature resolution, normalizes them via LayerNorm, projects them via a zero-initialized 1×1 convolution, and adds the result element-wise to the visual features. The zero-initialization ensures identity at training start, preserving pretrained feature distributions.
Promptable Detector (Section 2.2, purple block in Figure 3): A transformer-based module that conditions the depth-fused visual features on user-provided prompts. Text prompts are tokenized by a CLIP-style BPE tokenizer and encoded by a 24-layer causal Transformer (width 1024, 16 heads, projected to 256-dimensional embeddings). Box and point prompts are encoded by a geometry encoder that sums coordinate projections, ROI-aligned or grid-sampled image features, and sinusoidal positional encodings. All prompt tokens are concatenated into a single sequence that serves as cross-attention memory for both encoder and decoder transformer stages.
3D Detection Head (Section 2.3, red block in Figure 3): A stack of L Transformer decoder layers, each producing its own 3D predictions (deep supervision). Each decoder layer sequentially enriches its hidden states through two dedicated cross-attention modules: a camera prompt branch that fuses spherical-harmonic-encoded ray directions derived from camera intrinsics, and a depth prompt branch that fuses projected depth latents. The enriched features are passed through a two-layer MLP to predict a 12-dimensional 3D box encoding (center offset, log-depth, log-dimensions, 6D rotation), and a parallel two-layer MLP predicts a 3D confidence score.
Auxiliary Heads (Section 2.4, gray blocks in Figure 3): Two additional heads provide complementary supervision during training and enable broader applications. A 2D detection head predicts 2D bounding boxes with IoU-aware classification scores. A depth estimation head predicts a metric depth map, camera intrinsics, and per-pixel confidence from the depth latents. These heads are used only at training time (their losses contribute to the overall training objective) and during deployment for applications that need depth maps or 2D detections alongside 3D boxes.

Information flow: An RGB image (1008×1008 pixels) and optional depth map enter the system → the Image Encoder produces multi-scale feature maps → the RGBD Encoder produces depth latents → the Depth Fusion Module merges geometry into semantics → the user's prompt (text, point, box, or exemplar) is encoded into a token sequence → the Promptable Detector's transformer layers condition visual features on prompt tokens through cross-attention → the 3D Detection Head lifts the resulting 2D query features into 3D bounding box predictions at each decoder layer → the 3D confidence branch re-ranks predictions by geometric quality → NMS filters overlapping detections → final 3D boxes are output with confidence scores.

3.3 Roadmap for the Deep Dive

First, the dual-vision encoder and depth fusion module (Section 2.1)—how the system separates semantic and geometric feature extraction and then reunifies them, since this is the foundation that enables optional depth integration.
Second, the promptable detector (Section 2.2)—how diverse prompt modalities (text, point, box, exemplar) are encoded and used to condition detection, since this is the mechanism that unifies multiple interfaces within one architecture.
Third, the deeply-supervised 3D detection head (Section 2.3)—how 2D query features are lifted into 3D through multi-source information aggregation, the 3D box parameterization that resolves rotation ambiguity, and the confidence prediction that re-ranks detections by geometric quality, since this is where the actual 3D reasoning occurs.
Fourth, the multi-task training framework (Section 2.4)—how 2D detection, depth estimation, and 3D detection losses are combined, how ignore-region suppression handles non-exhaustive annotations, and how one-to-many matching provides denser supervision, since the training recipe is critical to making the architecture work in practice.
Fifth, the WildDet3D-Data pipeline (Section 3)—how the training dataset is constructed through candidate generation, rule-based filtering, and human/VLM selection, since the model's open-world generalization depends critically on the scale and quality of this data.

3.4 Detailed, Sentence-Based Technical Breakdown

Dual-Vision Encoder and Depth Fusion Module (Section 2.1)

Motivation. Accurate 3D detection from monocular images requires two capabilities that are in tension: rich semantic features for recognizing objects across an open vocabulary, and metrically grounded depth and camera-aware representations for reasoning in 3D. A single encoder trained for both objectives faces a trade-off—optimizing for depth estimation can degrade detection features and vice versa. Moreover, a single fixed encoder cannot accommodate optional depth input: when an external depth sensor (LiDAR, stereo, ToF) is available, the model should use it; when it is absent, the model should fall back to learned monocular priors. The dual-encoder design addresses both issues by decoupling semantics from geometry at the encoder level and reuniting them through a dedicated fusion stage, keeping the architecture modular so that different depth models can be plugged in without modifying the core detection pipeline.

Image Encoder. The image encoder is a ViT-H [14] with a SimpleFPN neck, initialized from a segmentation-pretrained checkpoint from SAM 3 [8] that provides strong dense prediction features. Given an input image resized to H × W pixels (specifically 1008×1008 during training and inference) with patch size p, the ViT generates H/p × W/p spatial tokens. The SimpleFPN projects these to 256-channel feature maps at multiple scales that are fed to the downstream detector. During training, the first 28 of 32 ViT blocks are frozen, with only the last 4 blocks fine-tuned. This partial freezing preserves the strong pretrained representations while allowing the encoder to adapt its highest-level features to the 3D detection task. The authors adopt the architecture and weights directly from SAM 3, which provides high-quality dense features for detection and segmentation—the paper explicitly states this avoids training a detection backbone from scratch and contributes to the model's training efficiency (12 epochs vs. 80–120 for prior methods).

RGBD Encoder. The RGBD encoder is built on a DINOv2 ViT-L/14 [37] pretrained for metric depth estimation as part of LingBot-Depth [49]. It accepts 4-channel RGBD input at 686×686 resolution, producing 49×49 tokens (686/14 = 49). The fourth channel is the depth map, which is zero-filled when no external depth is available—this is the mechanism that makes depth optional. The encoder features are passed through a ConvStack neck that produces a 5-level feature pyramid, from which depth latents Z_d ∈ ℝ^{C_d × 49 × 49} with C_d = 256 are extracted via average pooling over the pyramid levels. During training, the first 21 of 24 DINOv2 blocks are frozen, with the last 3 blocks fine-tuned to allow the depth encoder to adapt to the data distribution while retaining the strong depth estimation priors from LingBot-Depth's large-scale RGBD pretraining.

Why different backbones for image and RGBD encoders? The paper deliberately uses different pretrained backbones: the image encoder is pretrained for segmentation (SAM 3, which excels at dense feature extraction for object boundaries and instance discrimination), while the RGBD encoder is pretrained for depth completion (LingBot-Depth, which excels at metric depth estimation from RGBD input). This specialization means each encoder provides complementary information: semantics from the image encoder, geometry from the RGBD encoder. The alternative—using a single backbone for both tasks—would force a compromise between semantic and geometric feature quality.

Stochastic depth training. To support optional depth input at inference, training uses a stochastic strategy for the depth channel: 70% of training samples use monocular mode (zero-filled depth channel), 20% use patch-masked depth (where random patches of the depth map are masked to simulate sparse or partial depth), and 10% use full depth copy-through (the actual depth map is provided). This mixed training ensures the model learns to (a) operate in purely monocular mode when no depth sensor is available, (b) leverage complete depth maps when available, and (c) handle sparse or partial depth such as from sparse LiDAR point clouds. The 70/20/10 split reflects the assumption that monocular operation is the most common deployment scenario, with depth availability being less frequent but highly valuable when present.

Depth Fusion Module. The depth fusion module (yellow block in Figure 3) injects depth latents into the image encoder's feature maps through a ControlNet-style [65] residual connection. The operation is:

$V' = V + \text{Conv}_{1\times1}\big(\text{LN}(Z^\uparrow_d)\big)$

where V ∈ ℝ^{C × H_v × W_v} are the visual features from the SimpleFPN neck, Z^\uparrow_d denotes the depth latents bilinearly interpolated from their native 49×49 resolution to match the visual feature resolution H_v × W_v, LN is LayerNorm that normalizes the depth latents to unit scale (preventing the depth signal from overwhelming the visual features due to different activation magnitudes), and Conv_{1×1} is a 1×1 convolution projecting from the depth dimension C_d = 256 to the visual dimension C = 256.

What it computes: The depth latents are spatially upsampled to match the visual feature map resolution, normalized to have zero mean and unit variance across the channel dimension, projected to the same dimensional space as the visual features, and added element-wise to the visual features. The result V' replaces V as the input to the subsequent transformer encoder layers.

Why this form: The zero-initialization of the Conv_{1×1} projection means that at the start of training, V' = V (identity)—the depth contribution is zero and the model initially relies purely on visual features, exactly as a monocular-only model would. The depth contribution is gradually learned as training proceeds, allowing the pretrained visual features to remain undisturbed while the model slowly learns to incorporate geometric cues. Only the depth branch passes through trainable layers (LayerNorm and the 1×1 convolution); the visual features are added as-is with a frozen skip connection. This design preserves the pretrained feature distribution—the depth signal acts as a residual correction rather than replacing or distorting the visual features. The ControlNet-style approach was chosen because it enables plug-and-play integration of different depth backbones: any RGBD encoder can be used as long as it produces depth latents that can be projected to 256 channels, without modifying the image encoder or detection head at all.

Promptable Detector (Section 2.2)

Motivation. Different applications require different ways of specifying the target object. A robot following a language command needs text-prompt detection ("pick up the mug"). An augmented reality interface where a user taps on a screen needs point-prompt detection (the 2D pixel coordinate the user touched). An upstream 2D object detector whose outputs need to be lifted to 3D needs box-prompt detection. A system that should find all objects visually similar to a reference needs exemplar-prompt detection. Rather than building separate models for each prompt type—which would duplicate parameters, complicate deployment, and prevent transfer learning across prompt modalities—WildDet3D unifies all four prompt types within a single promptable detector. The key mechanism is prompt encoding: each prompt type is converted into a sequence of 256-dimensional token embeddings, all tokens are concatenated into a single unified prompt sequence, and this sequence serves as cross-attention memory for the transformer decoder layers that produce per-object predictions.

Four prompt types. The detector accepts four complementary prompt modalities:

Text prompt. A category name string (e.g., "car," "red mug," "fire extinguisher"), which selects all instances of that category in the image. This is the standard open-vocabulary detection interface.
Point prompt. One or more 2D pixel coordinates (u, v), each labeled as positive (on the object) or negative (background). Positive points select the single object at that location; negative points exclude regions. This enables interactive instance-level selection—"this object here, not that one."
Box prompt. A 2D bounding box (x1, y1, x2, y2) in pixel coordinates, selecting the single object within that region. This is the interface for lifting an upstream 2D detector's output to 3D or for a user drawing a rough box around a target.
Exemplar prompt. A 2D bounding box used as a visual exemplar, detecting all visually similar objects in the scene regardless of category. This enables "find all objects that look like this one" queries, useful when the category name is unknown or ambiguous.

Why these four? They span the space from category-level (text: "all instances of X") to instance-level (point/box: "this specific instance") to appearance-level (exemplar: "anything resembling this"), covering the interaction modes needed for robotics, AR/VR, and integration with upstream vision systems. During training, all four prompt types are sampled jointly to ensure balanced learning across modalities—the model does not specialize to one prompt type at the expense of others.

Text prompt encoding. Text prompts are tokenized using a CLIP-style [39] byte-pair encoding (BPE) tokenizer—the same tokenizer used in SAM 3 [8]. The resulting token sequence is encoded by a 24-layer causal Transformer (width 1024, 16 attention heads) that processes the text autoregressively (each token attends only to preceding tokens). The output hidden states are then linearly projected from 1024 to the unified embedding dimension d = 256. This text encoder is part of the SAM 3 weights that initialize the image encoder, providing a pretrained vision-language alignment that supports open-vocabulary recognition.

Geometry prompt encoding (box and point). Box and point prompts are encoded by a shared geometry encoder that sums three complementary representations for each prompt:

Direct coordinate projection: The raw pixel coordinates (u, v) for points or (x1, y1, x2, y2) for boxes are linearly projected to a 256-dimensional vector. This provides an explicit positional signal—"where in the image the prompt is located."
ROI-aligned or grid-sampled features: For box prompts, the image backbone's feature maps are pooled using ROI-Align [40] over the box region, producing a 256-dimensional feature vector that captures the visual content inside the box. For point prompts, features are sampled at the exact pixel location using grid sampling. This provides an appearance signal—"what the region looks like."
Sinusoidal positional encoding: The coordinates are encoded using sinusoidal functions at multiple frequencies, producing a 256-dimensional positional encoding that captures both absolute position and relative spatial relationships—the same encoding used in standard Transformers to represent sequence positions, adapted to 2D coordinates.

These three representations are summed element-wise, producing a single 256-dimensional embedding per geometry prompt. A learnable positive/negative label embedding is then added: positive points and box prompts receive a learned "positive" embedding vector, negative points receive a learned "negative" embedding vector. This allows the model to distinguish between prompts that specify where the object IS (positive) versus where it IS NOT (negative). The resulting embedding is refined by a 3-layer Transformer with cross-attention to image features, which allows the prompt embedding to incorporate visual context from the surrounding image region.

Exemplar prompt encoding. Exemplar prompts reuse the same box encoding pipeline as regular box prompts, but are differentiated by two mechanisms: (1) a special text token ("visual") is prepended to the prompt sequence, signaling to the model that this is an exemplar rather than a regular box prompt, and (2) a multi-target matching strategy assigns all instances of the same category as ground truth (rather than a single instance), training the model to detect all visually similar objects rather than just the one inside the box.

Prompt sequence construction. The encoded text tokens and geometry tokens are concatenated into a single prompt sequence (a 2D tensor of shape [num_tokens, 256]). This sequence serves as cross-attention memory in both the encoder and decoder stages of the promptable detector: queries (representing candidate object detections) attend to all prompt tokens simultaneously, allowing the model to flexibly combine information from multiple prompt modalities (e.g., a text prompt specifying the category and a point prompt specifying the location).

Per-prompt batching. A crucial implementation detail: training batches are constructed at the per-prompt level rather than the per-image level. Each unique text category in the batch produces a separate batch entry that aggregates all images containing that category. For example, if "chair" appears in 5 images, those 5 images form one batch entry with the text prompt "chair." This strategy enables fine-grained multi-instance supervision: the model can be trained to detect all instances of a category across many images within a single forward pass, handling an arbitrary number of categories per image without padding or truncation. It also ensures that each training step provides balanced supervision across categories, preventing frequent categories from dominating the gradient.

Deeply-Supervised 3D Detection Head (Section 2.3)

Motivation. The query features produced by the promptable detector are 256-dimensional vectors representing candidate objects in 2D feature space. To produce 3D bounding boxes, these features must be lifted into metric 3D space, which requires integrating three sources of information: (1) the query features themselves, which encode semantic and spatial information about the object; (2) camera geometry, which relates 2D pixel coordinates to 3D rays in space; and (3) depth information, which provides the metric scale needed to resolve the fundamental monocular ambiguity. The 3D detection head performs this lifting through a series of Transformer decoder layers, each of which enriches its hidden state with camera and depth information, then predicts a complete 3D box parameterization. The deep supervision strategy—applying the training loss at every decoder layer—encourages even the earliest layers to develop 3D localization capability, yielding faster convergence and more robust intermediate representations.

Architecture overview. The head comprises L Transformer decoder layers (the paper does not specify the exact value of L, but the deep supervision analysis in Section 4.6 treats it as a hyperparameter). The hidden states at each layer l are denoted H_l ∈ ℝ^{S × d} where S is the number of query tokens (candidate detections) and d = 256 is the feature dimension. Each layer l outputs its own set of 3D predictions, and the training loss (detailed in Section 2.4) is applied at every layer with equal weight—this is the "deep supervision" that gives the head its name.

Multi-source information aggregation. Each decoder layer sequentially enriches its hidden states through two dedicated cross-attention modules. Both modules use single-head attention and each layer has its own independent set of projection parameters (weights are not shared across layers).

Step 1 — Camera prompt branch (ray feature fusion). The first cross-attention module fuses geometric ray features derived from camera intrinsics. Given the camera intrinsics matrix K ∈ ℝ^{3×3} (either provided as input or estimated by the depth head), per-pixel ray directions are computed as:

$r_{i,j} = K^{-1} [u, v, 1]^\top$

where (u, v) are the pixel coordinates and r_{i,j} is the 3D ray direction vector pointing from the camera center through that pixel into the scene. These rays encode the geometric relationship between 2D image positions and 3D directions in space.

The ray directions are encoded using 8th-order real spherical harmonics:

$\phi(r) = \text{RSH}_8\left(\frac{r}{\|r\|}\right) \in \mathbb{R}^{81}$

where RSH_8 denotes the 8th-order real spherical harmonic basis functions evaluated on the normalized ray direction (the unit vector pointing from the camera through the pixel). Spherical harmonics are a set of orthonormal basis functions on the sphere—analogous to Fourier basis functions but on the surface of a sphere rather than a line—that efficiently encode directional information. The 8th-order expansion produces 81 coefficients (the number of basis functions up to order 8 is (8+1)² = 81).

What this does: Each pixel's ray is mapped from a 3D Cartesian vector to an 81-dimensional encoding that captures the full directional information at multiple spatial frequencies. The normalization by ∥r∥ ensures the encoding depends only on direction, not on the unnormalized ray magnitude. This is the standard representation for encoding directions in neural networks, used because Cartesian coordinates (x, y, z) suffer from discontinuity at the poles and wrap-around, while spherical harmonics provide a smooth, continuous encoding over the sphere.

The ray features are then fused via cross-attention:

$\tilde{H}_l = \text{FFN}\big(\text{CrossAttn}(\text{SelfAttn}(H_l), f_r(\phi(r)))\big)$

where SelfAttn(H_l) applies multi-head self-attention over the query tokens (allowing queries to share information about the scene), CrossAttn(·, f_r(ϕ(r))) performs cross-attention from the query tokens to the ray features (each query attends to ray features from relevant image locations), f_r : ℝ^{81} → ℝ^d is a learned linear projection that maps the 81-dimensional spherical harmonic features to the 256-dimensional query space, and FFN is a standard feed-forward network (two linear layers with an activation function in between). The self-attention, cross-attention, and FFN together form one complete Transformer decoder sub-layer.

Why this ordering: Self-attention first (allowing queries to share contextual information), then cross-attention to ray features (injecting geometric information), then FFN (transforming the enriched representations). This is the standard Transformer decoder pattern, adapted to inject geometric rather than linguistic information at the cross-attention step.

Step 2 — Depth prompt branch (depth latent fusion). The second cross-attention module fuses the depth latents from the RGBD encoder. The enriched features from Step 1 are further processed:

$\hat{H}_l = \text{FFN}\big(\text{CrossAttn}(\text{SelfAttn}(\tilde{H}_l), f_d(Z_d))\big)$

where Z_d ∈ ℝ^{C_d × 49 × 49} are the depth latents from the RGBD encoder (at their native 49×49 spatial resolution), f_d : ℝ^{C_d} → ℝ^d is a learned projection that maps the C_d = 256-dimensional depth features at each spatial location to the 256-dimensional query embedding space, and the self-attention, cross-attention, and FFN follow the same pattern as Step 1.

What this does: Each query token attends to depth latents from all spatial locations (49×49 = 2401 positions), learning to extract metric depth information relevant to the object it represents. The cross-attention mechanism allows the query to focus on depth values at the spatial locations where the object appears, rather than averaging over the entire depth map. The separate projection f_d from the ray projection f_r allows the two geometric signals to be integrated independently—rays provide directional geometry, depth latents provide metric distance.

Why dual cross-attention rather than concatenation? Concatenating ray and depth features before attention would force a fixed interaction between them. Cross-attention allows each query to independently weight the ray and depth signals, learning which geometric information is relevant for which object. The sequential ordering (first rays, then depth) reflects the logical dependency: camera geometry establishes the relationship between 2D and 3D space, and depth provides the metric scale within that space.

3D box parameterization. After the multi-source information aggregation, the enriched query features \hat{H}_l at each decoder layer are passed through a two-layer MLP to predict a 12-dimensional 3D box encoding:

$p_{3D} = (\underbrace{\Delta cx, \Delta cy}_{\text{center offset}}, \underbrace{\hat{d}}_{\text{log depth}}, \underbrace{\hat{w}, \hat{h}, \hat{l}}_{\text{log dims}}, \underbrace{r_1, \ldots, r_6}_{\text{rotation}})$

where each component is defined as:

Center offset (Δcx, Δcy): The displacement between the 2D projection of the 3D center and the 2D box center (the center of the 2D region associated with this query), normalized by a scale factor s_c = 10. Specifically, if (c_x^{2D}, c_y^{2D}) is the 2D box center in pixel coordinates, the predicted 3D center projects to (c_x^{2D} + s_c · Δcx, c_y^{2D} + s_c · Δcy) in the image plane. The scale factor prevents the offset from being too small for the network to predict accurately.
Log-depth \hat{d} = s_d · \log(d): The logarithm of the metric depth d (distance from the camera to the 3D object center), scaled by s_d = 2.0. Using log-space makes the prediction task easier because depth varies over orders of magnitude (objects can be 0.5m or 50m away), and log-space converts this multiplicative range to an additive one that a linear output can capture. The scaling factor adjusts the dynamic range of the log-depth to a range suitable for neural network outputs (typical metric depths of 0.1m to 100m map to log-depths of approximately −9.2 to +9.2, which get scaled to approximately −4.6 to +4.6 by s_d = 0.5—wait, the paper states s_d = 2.0, so the scaled log-depth ranges from approximately −4.6 to +9.2 for 0.1m to 100m depths).
Log-dimensions (\hat{w}, \hat{h}, \hat{l}) = s_{dim} · \log(w, h, l): The logarithm of the physical dimensions (width, height, length) in meters, scaled by s_{dim} = 2.0. Same motivation as log-depth: object sizes span orders of magnitude (a coffee cup is ~0.1m, a car is ~5m), and log-space provides a linear prediction target.
6D rotation (r_1, …, r_6): The first two rows of the 3×3 rotation matrix, following the continuous 6D representation of Zhou et al. [67]. The full 3×3 rotation matrix is recovered via Gram–Schmidt orthogonalization: the first row (r_1, r_2, r_3) is normalized to unit length to produce the first basis vector; the second row (r_4, r_5, r_6) is made orthogonal to the first via projection, then normalized to produce the second basis vector; the third basis vector is the cross product of the first two, ensuring a valid rotation matrix in SO(3).

Why the 6D rotation representation? The standard alternatives are Euler angles (3 numbers, but suffer from gimbal lock and discontinuous wrap-around), quaternions (4 numbers, but the mapping to rotation is 2-to-1—q and −q represent the same rotation—causing representational ambiguity), and axis-angle (3 numbers, but discontinuous at 0° and 360°). The 6D representation overparameterizes rotation (6 numbers for 3 degrees of freedom) but is continuous everywhere—small changes in the 6D vector always produce small changes in the resulting rotation matrix—and can represent any rotation without singularities. The Gram–Schmidt process provides a differentiable mapping from 6D vectors to SO(3), enabling end-to-end training. This is the representation used in most modern 3D vision systems that predict rotation, chosen for its empirical training stability.

Recovering the 3D center at inference. At inference time, the metric 3D center c ∈ ℝ^3 is recovered from the predicted parameters and the camera intrinsics:

$c = d · K^{-1} \begin{bmatrix} u_c \\ v_c \\ 1 \end{bmatrix}$

where (u_c, v_c) are the pixel coordinates of the predicted 2D center (computed by adding the predicted offset (Δcx, Δcy) to the 2D box center), d = exp(\hat{d} / s_d) is the predicted metric depth (exponentiating the scaled log-depth to recover actual depth in meters), and K^{-1} back-projects the pixel coordinate to a 3D ray direction, then scales by depth to get the 3D position. This is the standard pinhole camera model: pixel coordinates are rays through the camera center, and depth determines the distance along that ray.

Unambiguous rotation normalization. A fundamental challenge in predicting oriented 3D bounding boxes is rotation ambiguity. A box with dimensions (w, h, l) rotated by yaw θ about the vertical axis is geometrically identical to one with swapped dimensions (l, h, w) rotated by θ + 90°. Additionally, a 180° yaw flip preserves the box geometry for any object with front-back symmetry (which is most man-made objects—a chair looks the same from the back in terms of its bounding box). Without normalization, the model must learn multiple equivalent representations for the same physical box, increasing the complexity of the regression task.

WildDet3D resolves this with a two-step normalization applied to the ground-truth rotation and dimensions before loss computation (the same normalization is applied to predictions at inference before evaluation):

Step 1 — Dimension ordering: If the predicted width w is greater than the predicted length l, swap w and l and apply an additional 90° rotation about the vertical axis (yaw): R ← R · R_y(90°). After this step, w ≤ l always holds—the length is always the longer horizontal dimension, and the width is the shorter one.

Step 2 — Yaw folding: Fold the yaw angle into the range [0, π) by applying a 180° rotation about the vertical axis (R ← R · R_y(180°)) when the yaw angle is less than 0 or greater than or equal to π. This ensures the yaw angle is always in [0, π).

What this achieves: Together, these two steps reduce a 4-fold rotation-dimension ambiguity to a unique unambiguous form. Before normalization, (w, h, l, θ), (l, h, w, θ+90°), (w, h, l, θ+180°), and (l, h, w, θ+270°) all represent the same physical box. After normalization, exactly one of these four is the canonical form, providing a one-to-one mapping between box geometry and the regression target. The model only needs to learn one representation per box rather than four.

Why not use a symmetry-aware loss? An alternative approach would be to keep the multiple representations and compute the loss as the minimum over all equivalent parameterizations—for each prediction, find which of the four equivalent ground-truth representations is closest and compute the loss against that one. The issue with this approach is that the loss landscape becomes non-convex (multiple local minima corresponding to different representations), making training harder and convergence slower. The normalization approach provides a single, well-defined target, simplifying optimization at the cost of a slightly more complex preprocessing step.

3D confidence prediction. In addition to the box regression branch, a parallel branch—a separate two-layer MLP—predicts a scalar 3D detection quality score s_{3D} ∈ [0, 1]. This score is not a classification confidence (how likely the box corresponds to the queried category) but rather a regression quality estimate (how geometrically accurate the predicted 3D box is). During training, the soft target for this branch is defined as:

$q^* = \beta · q_{depth} + (1 - \beta) · \text{IoU}_{3D}$

where q_{depth} = \exp(-|\log \hat{d} - \log d^*|) measures depth prediction quality as a symmetric ratio bounded in [0, 1] (if predicted depth equals ground truth, q_{depth} = 1; if predicted depth is twice or half the ground truth, q_{depth} = 0.5; the exponential of the negative absolute log-ratio is 1 when the ratio is 1, and approaches 0 as the ratio approaches 0 or infinity), IoU_{3D} is the 3D bounding box Intersection-over-Union between the predicted and ground-truth boxes (measuring spatial overlap of the oriented 3D cuboids), and β = 0.7 weights depth accuracy more heavily than 3D IoU.

Why this form: The β = 0.7 weighting emphasizes depth accuracy because depth estimation is the primary bottleneck in monocular 3D detection—if the depth is wrong, the entire 3D box is displaced, and IoU will be low regardless of dimension and rotation accuracy. The log-ratio formulation for depth quality is symmetric (predicting half the depth is penalized the same as predicting twice the depth) and naturally bounded in [0, 1] without requiring clipping. The exponential function converts the absolute log-error (a non-negative value that can be arbitrarily large) into a score between 0 and 1.

At inference, the final detection score combines the 2D objectness score s_{2D} (from the IoU-aware classification head in the 2D detection branch) and the 3D confidence s_{3D}:

$s = s_{2D} + \alpha · s_{3D}$

with α = 0.5. The 2D objectness score s_{2D} is a logit (unbounded, typically in [-10, 10] after sigmoid) that represents how likely the detection corresponds to a real object of the queried category. The 3D confidence s_{3D} ∈ [0, 1] provides a geometric quality adjustment.

Why additive rather than multiplicative? A multiplicative combination s = s_{2D} · s_{3D} would suppress all detections proportionally to s_{3D}, which would be problematic because s_{2D} is typically several orders of magnitude larger than s_{3D} in logit space (sigmoid probabilities near 0 or 1 are extreme logits). The additive formulation with α = 0.5 allows the 3D confidence to re-rank detections that have similar 2D scores but differ in geometric quality—two detections with similar 2D objectness will be ordered by their 3D accuracy—without letting uncertain 3D estimates suppress high-confidence 2D detections entirely. The α = 0.5 scaling ensures s_{3D} acts as a tiebreaker rather than a dominant term.

Deep supervision. The training loss is applied at every decoder layer l ∈ {1, …, L} with equal weight. This means the earliest layer already produces 3D predictions that are supervised against ground truth, encouraging it to develop 3D localization capability from the start of training. The ablation in Table 8 shows that deep supervision contributes a modest −0.3 AP improvement on Omni3D—small but consistent, and the paper argues it provides faster convergence and more robust intermediate representations that benefit downstream fine-tuning.

Multi-Task Training Framework (Section 2.4)

Motivation. Training a system that simultaneously performs 2D detection, 3D detection, and depth estimation requires careful loss design to balance the competing objectives. The framework must handle non-exhaustive annotations (not every visible object has a valid 3D ground truth), support multiple prompt modalities with balanced supervision, and leverage auxiliary signals that improve 3D performance without requiring additional annotation effort.

Dual-branch query construction. During training, each category in an image produces two types of queries:

Multi-target query: Sampled as 50% text-only and 50% exemplar box (with optional category label). This query is supervised against all instances of that category in the image—the model must detect every chair, not just one. This teaches the model category-level open-vocabulary detection.
Single-target geometric query: A box or point prompt (with optional category label) that is directly assigned to one selected instance. This query is supervised against only that specific instance—the model must detect the object at the specified location. This teaches the model instance-level prompt-conditioned detection.

Why this dual design: By sampling both query types in every training batch, the model learns to handle both category-level text queries (which should return all instances) and instance-level geometric queries (which should return the specific object at the prompted location). The 50/50 text/exemplar split ensures exemplar-based detection ("find objects similar to this one") receives as much training signal as text-based detection.

One-to-many (O2M) matching. Rather than matching each ground-truth object to a single predicted box (standard one-to-one matching), WildDet3D follows SAM 3 [8] in employing one-to-many matching: each ground-truth object is paired with its top-k scoring predictions, where k = 4. This means every ground-truth box provides supervision to four predicted boxes (the four with the highest confidence scores that have sufficient IoU with the ground truth), rather than just one.

Why O2M? One-to-one matching provides sparse supervision—each ground-truth object can only supervise one prediction, which slows convergence because many reasonable predictions go unrewarded. O2M provides denser gradient signals: multiple predictions receive positive feedback for being close to a ground-truth object, encouraging the model to produce a diverse set of reasonable proposals rather than collapsing to a single hypothesis. The ablation in Table 8 shows that removing O2M causes the largest performance drop of any training objective (−2.5 AP on Omni3D), with driving datasets hit hardest (KITTI: 27.9 → 23.2, nuScenes: 28.2 → 23.9), because dense, similarly-sized objects benefit most from the richer supervision.

Overall training loss. The total loss aggregates four components:

$\mathcal{L} = \underbrace{\mathcal{L}_{3D} + \mathcal{L}_{conf}}_{\text{3D detection losses}} + \underbrace{\mathcal{L}_{geom} + \mathcal{L}_{2D}}_{\text{auxiliary losses}}$

where L_{3D} is the 3D regression loss on the 12-dimensional box encoding, L_{conf} is the 3D confidence loss, L_{geom} is the geometry estimation loss (depth, point maps, camera intrinsics), and L_{2D} is the 2D detection loss. The paper provides detailed formulations of L_{3D}, L_{conf}, and L_{2D} in Section 2.4, and L_{geom} in Appendix A.

3D regression loss L_{3D}. For each matched prediction–target pair (from O2M matching), the loss computes an L1 distance on the 12-dimensional encoded parameters:

$\mathcal{L}_{3D} = \frac{1}{N_{pos}} \sum_{i \in \mathcal{M}} \sum_{k} w_k \left| p^{(i)}_k - p^{*(i)}_k \right|$

where M is the set of matched prediction–target indices, N_{pos} is the number of matched pairs, p_k and p^*_k are the k-th components of the predicted and target 12-dimensional encodings (center offset, log-depth, log-dimensions, 6D rotation), and w_k are per-component validity weights that are set to zero when the corresponding ground-truth annotation is unavailable (e.g., if the dataset lacks 3D dimensions, the dimension loss components are masked out; if rotation is unavailable, the rotation loss components are masked). This masking allows training on datasets with heterogeneous annotation completeness—Omni3D has full annotations, while some subsets of WildDet3D-Data may lack depth or dimensions.

Why L1 rather than L2 (MSE)? L1 loss is more robust to outliers, which matter in 3D detection because depth and dimension predictions can occasionally be severely wrong (e.g., predicting 50m depth for a 2m object). L2 loss would square these large errors, causing them to dominate the gradient and destabilize training. L1 loss gives equal gradient magnitude to all errors regardless of size, providing more stable optimization.

3D confidence loss L_{conf}. The confidence branch is trained with an IoU-aware focal binary cross-entropy loss. For each matched prediction with raw logit c_i (before sigmoid), an adaptive soft target is constructed:

$t_i = \sigma(c_i)^\alpha · q_i^{*^{1-\alpha}}$

where σ(c_i) is the sigmoid of the predicted logit (the model's current confidence estimate), q_i^* is the regression quality target from Equation 6 (the weighted combination of depth quality and 3D IoU), and α = 0.25.

What this computes: The soft target t_i is a weighted geometric mean between the model's own prediction σ(c_i) and the actual regression quality q_i^*. When α = 0, the target is purely q_i^* (standard BCE with fixed target). When α = 0.25, the target is pulled toward the model's own prediction, creating a form of self-consistency regularization—the model is encouraged to predict confidences that are consistent with its own current beliefs, tempered by the actual quality. This is the same IoU-aware formulation used in SAM 3 [8] and has been shown to produce better-calibrated confidence estimates than fixed targets.

The total confidence loss combines a positive term over matched queries and a focal-weighted negative term over unmatched queries:

$\mathcal{L}_{conf} = \frac{1}{N_{pos}} \sum_{i \in \mathcal{M}} w_+ · \text{BCE}(c_i, t_i) + \frac{1}{N_{neg}} \sum_{j \notin \mathcal{M}} \sigma(c_j)^\gamma · \text{BCE}(c_j, 0)$

where w_+ = 5 upweights positive examples (since there are typically far more negatives than positives), γ = 2 is the focal exponent that down-weights easy negatives (queries that the model is already confident are not objects—their sigmoid is near 0, so σ(c_j)^γ is near 0, reducing their contribution), N_{neg} is the number of unmatched queries, and BCE(c, t) is the standard binary cross-entropy: −[t · log σ(c) + (1−t) · log(1−σ(c))].

Why focal loss for negatives? In detection, the vast majority of queries are negative (don't match any ground truth). Standard BCE would let these dominate the loss, drowning out the signal from the rare positive matches. The focal term σ(c_j)^γ automatically reduces the weight of negatives that the model already classifies correctly (low confidence), allowing training to focus on hard negatives that the model still mistakenly assigns high confidence.

Auxiliary geometry loss L_{geom}. The geometry backend produces depth estimates, 3D point maps, a confidence mask, and camera intrinsics. The auxiliary geometry loss comprises eight terms (detailed in Appendix A), summarized here:

Metric depth L1 (weight w = 1.0): Standard L1 loss between predicted and ground-truth depth at valid pixels (where d^* > 0 and the depth ratio \hat{d}/d^* ∈ [1/3, 3]). The ratio clipping excludes gross outliers where depth estimation completely fails.
Scale-invariant logarithmic depth loss (weight w = 0.5): The SILog loss from Eigen et al. [15]: L_{SILog} = √(Var(g) + 0.15 · Mean(g)²) where g_p = log \hat{d}_p − log d^*_p is the log-depth error at pixel p. This loss penalizes the variance of log-errors (encouraging consistent relative depth ordering) and the mean squared log-error (encouraging correct absolute scale). The 0.15 weight on the mean term relative to the variance term makes the loss more sensitive to relative depth relationships than absolute scale—important because metric scale can be ambiguous from monocular cues alone.
Affine-invariant point-map losses (three terms, each with weight w = 10.0): Back-projected 3D point clouds (from predicted and ground-truth depth) are compared using MoGe2-based [52] geometric losses: (1) global alignment loss aligns the predicted point cloud to ground truth via optimal affine transform at resolution 48², (2) local alignment at two scales (24² patches with 16 samples, 12² patches with 256 samples) captures fine-grained local geometry, and (3) edge loss penalizes depth discontinuity mismatches at object boundaries. These losses are affine-invariant—they allow the predicted point cloud to be scaled, rotated, and translated to match the ground truth—which is crucial because monocular depth estimation has an inherent scale ambiguity that should not be penalized.
Confidence mask BCE (weight w = 0.1): A per-pixel BCE loss supervising the depth validity confidence prediction against a three-state ground-truth mask (finite depth, infinite/invalid depth, unknown). For sparse depth inputs (coverage < 70%), only annotated pixels contribute to the loss, preventing the model from being penalized for predicting depth in unobserved regions.
Camera ray MSE (weight w = 1.0): L2 loss between predicted and ground-truth camera ray directions, derived from the respective intrinsics matrices. This supervises the intrinsic prediction head, teaching the model to estimate focal length and principal point when camera calibration is unavailable.

All geometry losses are scaled by a global factor λ_{geom} = 5.0 and each component is clipped to a maximum of 10 before scaling, preventing gradient explosion from outlier pixels (e.g., a single pixel with extremely wrong depth).

Auxiliary 2D detection loss L_{2D}. The 2D detection losses follow the SAM 3 [8] design:

IoU-aware classification BCE (weight w = 20): For each matched prediction–target pair, the classification target is an IoU-aware soft label t = σ(z)^α · IoU_{2D}^{1-α} with α = 0.25, same formulation as the 3D confidence target. Positive predictions are weighted by w_+ = 5; unmatched predictions receive a focal-weighted negative loss with γ = 2.
Box regression (combined weight 5·L1 + 2·(1−GIoU)): Combines an L1 loss on normalized center-size coordinates (weight 5) and a generalized IoU loss [42] on pixel-space boxes (weight 2). The generalized IoU loss is L_{GIoU} = 1 − GIoU(ˆb, b^*), where GIoU ranges from −1 to 1 (1 for perfect overlap, 0 for non-overlapping boxes, negative for boxes far apart). This formulation is more robust than standard IoU loss when boxes don't overlap (standard IoU is 0 for any non-overlapping pair, providing no gradient).
Per-category presence (weight w = 20): A sigmoid BCE loss (without focal weighting, α = 0.5, γ = 0) predicting whether each queried category has any instance in the image. This teaches the model to output "no detection" when the queried category is absent, preventing false positives.
One-to-many matching loss (scale w_{o2m} = 2.0, clipped at 150): The same classification, box, and 3D losses computed for all O2M-matched pairs. The scaling factor of 2.0 doubles the gradient from O2M pairs relative to one-to-one pairs. The clipping at 150 prevents individual O2M losses from exploding.

Ignore-region suppression. A fundamental challenge in monocular 3D detection is non-exhaustive annotation: not every visible object has a valid 3D ground truth. In both Omni3D [6] and WildDet3D-Data, objects with invalid 3D measurements, heavy truncation, severe occlusion, or placement behind the camera are annotated as IGNORE—they retain their 2D bounding boxes but are excluded from the set of positive 3D targets.

WildDet3D handles this consistently across training and evaluation. During evaluation, a prediction that matches an ignored ground-truth box (via 2D IoU > 0.5) is treated as neutral: it counts as neither a true positive nor a false positive, and ignored ground truths do not contribute to the false-negative count. During training, the ignore-region suppression strategy suppresses the negative classification loss for any prediction whose 2D IoU with an ignore-annotated box exceeds 0.5. Concretely, if a query predicts a box that overlaps substantially with an ignored object, that query is excluded from the negative loss terms—the model is not penalized for "detecting" an object that is present in 2D but lacks 3D annotation.

Why 2D IoU for ignore matching? Ignored objects lack valid 3D ground truth (their depth, dimensions, or rotation may be incorrect), so 3D IoU cannot be computed. 2D IoU is the only reliable matching criterion. The threshold of 0.5 is standard in detection literature, representing substantial spatial overlap.

Why not simply remove ignored objects from training? If ignored objects were simply removed, the model would learn to predict boxes in those regions (since nothing tells it those regions are invalid), and those predictions would become false positives during evaluation. The suppression strategy teaches the model that the region contains a real object (2D annotations are present) but without valid 3D information, so the model should neither be rewarded for detecting it (since 3D evaluation cannot verify the prediction) nor penalized for detecting it (since the object is genuinely present). The ablation in Table 8 shows this contributes a modest −0.2 AP on Omni3D (where ignore annotations are sparse) but the authors expect a larger effect on WildDet3D-Bench where partial 3D annotations are more common.

WildDet3D-Data Pipeline (Section 3)

Motivation. Open-world 3D detection requires training data that spans diverse categories, scenes, and viewing conditions. Existing 3D detection datasets are limited to ~100 categories in narrow domains (driving, indoor rooms). The WildDet3D-Data pipeline addresses this by leveraging existing large-scale 2D detection datasets (which already have rich category vocabularies and scene diversity) and lifting their 2D annotations to 3D through a multi-stage process designed to produce high-quality, human-verified 3D boxes at scale.

How it works, step by step:

Step 1: Acquire 2D annotations and 3D structure. The pipeline starts with 2D bounding box annotations from four large-scale detection datasets: COCO [29] (118K train, 5K val images, 80 categories), LVIS [18] (COCO images with 1,200+ long-tail categories), Objects365 [45] (609K train, 30K val images, 365 categories), and V3Det [51] (183K train, 30K val images, 13K+ fine-grained categories). For each image, the pipeline generates metric 3D structure through a sequence of off-the-shelf models: (1) 4× image super-resolution [63] increases spatial detail, (2) MoGe-2 [52] produces a metric depth map at 1024-long-edge resolution, (3) PerspectiveFields [21] estimates camera roll and pitch, and (4) WildCamera [68] estimates camera intrinsics (f_x, f_y, c_x, c_y). The depth map is reprojected into a 3D point cloud using the estimated camera parameters. This gives every 2D-annotated image a dense 3D point cloud and camera model, which are the geometric foundation needed for lifting 2D boxes to 3D.

Step 2: Generate candidate 3D boxes using five complementary methods. For each 2D annotation, the pipeline runs five different methods that convert the 2D box into a candidate 3D box, each capturing different geometric cues and having different failure modes:

3D-MOOD [59]: Runs open-vocabulary text-based 3D detection on the image and matches its predictions to the ground-truth 2D box via IoU. This provides model-based candidates that leverage learned 3D priors.
DetAny3D [64]: Directly regresses a 3D box from the 2D box region using dense feature extraction from a pretrained backbone. This provides a direct regression approach that doesn't require text matching.
SAM-3D [50]: Reconstructs a 3D mesh from the object's segmentation mask (obtained from SAM 2 [40]) and the depth-derived point map, then extracts an oriented 3D bounding box from the mesh vertices by finding the minimal enclosing oriented box. This is the only method that explicitly uses the object's 3D shape rather than just its 2D appearance.
RANSAC-PCA: A purely geometric method with no learning. It extracts 3D points inside the object's mask from the depth-derived point cloud, applies statistical outlier removal (filtering points whose distance to neighbors exceeds a threshold, indicating depth errors) and HDBSCAN clustering (density-based clustering that separates the object from background points), then fits an oriented box using RANSAC rectangle fitting (randomly sampling minimal subsets of points, fitting a rectangle, and selecting the fit with most inliers) with PCA-based gravity alignment (the vertical axis of the box is aligned with the dominant principal component of the point cloud, assumed to be gravity-aligned). This is the most geometry-driven method and works well when the point cloud is accurate but fails when depth estimation is poor.
LabelAny3D [61]: Single-image 3D reconstruction that lifts a 2D image crop to a 3D mesh using a learned reconstruction model, then aligns the mesh to the scene's depth map. This provides a reconstruction-based approach that leverages learned shape priors.

After initial prediction, each candidate undergoes two refinement steps (Section 3, Paragraph "3D box optimization" and Appendix D): translation optimization aligns the predicted depth to the estimated depth map using percentile-based scaling or anchor-based optimization (iteratively adjusting the 3D center to maximize agreement with the depth-derived point cloud), and rotation optimization corrects orientation using PCA-based gravity alignment and 2D projection constraints (the projected 2D box should match the original annotation). The candidates are merged into a unified 10D format (3 for center, 3 for dimensions, 4 for quaternion rotation), yielding up to five candidate 3D boxes per 2D annotation.

Why five methods? Each method has different strengths and failure modes. SAM-3D excels when the depth point cloud accurately captures the object's 3D shape. RANSAC-PCA works well for objects with clear geometric structure but fails on thin or heavily occluded objects. DetAny3D and 3D-MOOD leverage learned priors that generalize to objects where depth is unreliable. LabelAny3D uses shape reconstruction that works for objects with canonical shapes. By generating candidates from all five, the pipeline maximizes the probability that at least one candidate is good. The pipeline validation (Table 2, top) confirms this: selection shares range from 3.8% (3D-MOOD) to 40.4% (SAM-3D), and rejection rates vary from 12.5% (RANSAC-PCA) to 42.9% (DetAny3D)—no single method dominates, and the best method depends on the specific object and image.

Step 3: Rule-based filtering. Before candidates are presented to human annotators or VLM scoring, three geometric criteria remove obviously implausible boxes:

Edge contact ratio ≥ 3%: The box touches the image boundary (more than 3% of its projected 2D box is at the image edge), indicating a truncated object whose full 3D extent cannot be determined.
Occlusion ratio > 15% (RANSAC-PCA only): For the RANSAC method, more than 15% of the object's 3D points are occluded by other surfaces, making the geometric fit unreliable.
3D-to-2D projection size ratio outside [0.5, 1.5]: The 3D box, when projected into 2D, has an area that differs from the 2D annotation by more than a factor of 1.5, indicating a gross depth or scale error.

Candidates failing any criterion are discarded before entering the selection stage.

Additional pre-selection filters use VLMs (Qwen3.5-9B [2]) to identify and exclude annotations of depicted objects (pictures, posters, reflections, screen displays—objects that appear in 2D but are not real 3D instances) and composite images (images stitched from multiple sub-images, which produce inaccurate depth maps). An LLM-based filter (GPT-4.1-mini [36]) estimates expected physical dimensions for each object category and filters annotations where predicted dimensions fall outside these ranges (using 1.5× tolerance for fixed-size categories like "person" or "car," and 3.0× for variable-size categories like "toy" or "sculpture," relaxed to 2.5×/5.0× for fine-grained datasets).

Why LLM-estimated dimensions? Physical size priors are essential for filtering because monocular depth estimation can produce grossly incorrect scales (e.g., predicting a coffee mug to be 2 meters wide). An LLM can provide reasonable size ranges for arbitrary categories—including rare ones never seen in 3D datasets—without requiring manual specification. The LLM prompt asks for the shortest, middle, and longest axis ranges in meters, plus classification of whether the object is flat (like a poster), elongated (like a pole), or neither, which determines which size checks apply.

Step 4: Candidate selection. The final 3D annotations are obtained through two complementary paths:

Human selection (for ~103K images). Crowdsourced annotators on Prolific [38] evaluate up to five candidates per object. Each candidate is visualized from four viewpoints: a perspective overlay on the original image (showing the 3D box projected into the 2D image), and three orthographic point cloud views (front, side, top) showing the 3D box in the reconstructed scene geometry. Annotators select the best candidate and rate its quality as good_fit (the box accurately captures the object's 3D extent, orientation, and position), acceptable (the box is roughly correct but has minor errors), or unacceptable (the box is wrong). Each batch of 55 annotations includes 5 gold (quality-control) tasks with known-bad annotations; annotators who fail to identify ≥2 of the 5 gold tasks are disqualified, and their batches are discarded and reassigned. Overall pass rates range from 84% to 98% across dataset splits (meaning 2–16% of annotator batches are rejected for failing quality control).

Why gold tasks? Crowdsourced 3D annotation is inherently subjective and error-prone—annotators may lack spatial reasoning skills or may rush through tasks. Gold tasks with known-bad annotations (where the correct answer is that none of the candidates are acceptable) provide an objective quality check that cannot be gamed: an annotator who simply clicks "acceptable" on the first candidate without evaluating would fail the gold tasks. The threshold of 2/5 is calibrated to accept annotators who make occasional mistakes but reject those who are not engaging seriously.

The human-annotated portion provides 229,934 annotations across 102,979 training images, plus 9,256 validation and 5,596 test annotations (Table 1). Quality ratings: 35–48% good_fit, 33–50% acceptable, 24–39% unacceptable (the latter flagged as ignored during training).

VLM selection (for ~896K images). For images without human annotation, an automatic selector—a Molmo2 [10] checkpoint fine-tuned for this task—scores each candidate on six perceptual criteria:

Category correctness (0–1): Does the 3D box correspond to the correct object category?
Scale accuracy (0–2): Are the physical dimensions plausible?
Translation accuracy (0–2): Is the 3D center correctly positioned?
Shape fidelity (0–2): Does the box shape match the object's 3D structure?
Rotation correctness (0–2): Is the orientation accurate?
Vertical tilt alignment (0–2): Is the box aligned with gravity or the surface normal?

The total score ranges from 0 to 11. The VLM is fine-tuned on synthetically generated positive and negative candidate pairs from Omni3D, where the ground-truth 3D box serves as the positive candidate and perturbed versions serve as negatives. The highest-scoring candidate is kept when its total score exceeds 10.

Why Molmo2? Molmo2 is a vision-language model with strong spatial reasoning and grounding capabilities, making it suitable for evaluating 3D box quality from 2D projections and point cloud renderings. The fine-tuning on synthetic Omni3D pairs teaches it to recognize the specific failure modes of the candidate generation pipeline (e.g., depth errors causing scale mistakes, rotation errors causing orientation mistakes). Training on synthetic data rather than human labels avoids the chicken-and-egg problem of needing a trained selector to get human labels.

Pipeline validation. The paper validates the pipeline by analyzing the human-annotated training split (230K accepted annotations) along two axes (Table 2):

Candidate model quality: SAM-3D contributes 40.4% of accepted annotations, followed by RANSAC-PCA (28.2%), DetAny3D (14.5%), LabelAny3D (13.0%), and 3D-MOOD (3.8%). Rejection rates vary from 12.5% (RANSAC-PCA) to 42.9% (DetAny3D), confirming that candidate quality differs substantially and human evaluation is necessary to identify the best candidate.
VLM-human correlation: VLM scores correlate perfectly with human rejection rates (Spearman ρ = −1.0): rejection decreases from 71.9% at score < 7 to 9.2% at score 11. The strongest individual criteria for predicting human acceptance are scale (AUC = 0.60) and shape (AUC = 0.56), confirming that size and geometry fidelity are the primary bottlenecks. However, even at the maximum VLM score of 11, 9.2% of candidates are still rejected by humans, indicating that VLM scoring cannot fully substitute for human judgment—motivating the two-stage design with VLM pre-filtering and human verification.

Val/test sampling strategy. To ensure balanced evaluation, the validation and test sets are constructed using a three-phase sampling algorithm: (1) greedy set cover ensures 100% category coverage (every category appears at least once), (2) multi-objective balanced fill optimizes category rarity, scene diversity, depth distribution, and source balance simultaneously, and (3) targeted patching ensures each category has ≥3 samples. The resulting splits cover 785 validation and 633 test categories, with 99.9% having ≥1 training annotation and ~820 having ≥3.

Dataset statistics (Table 1). The complete WildDet3D-Data comprises 1,003,886 images with 3,728,078 valid 3D annotations across 13,499 unique categories—a 138× increase over Omni3D's 98 categories. The human-annotated portion covers ~103K training images (229,934 annotations), while the VLM-filtered synthetic portion adds ~896K training images (3,483,292 annotations). Scene categories span Indoor (52%, including home/apartment and public spaces), Urban (32%, including streets and buildings), and Nature (15%, including landscapes and wildlife), with maximum depth reaching 81 meters (compared to Omni3D's 67 meters).

The paper explicitly notes that this dataset construction approach is what enables WildDet3D's open-world generalization: without the breadth of categories and scene diversity provided by WildDet3D-Data, the model would overfit to Omni3D's 98 categories and narrow domains, as evidenced by the dramatic performance gap between Omni3D-only training (6.8 AP on WildDet3D-Bench) and training with the full dataset (22.6 AP)—a 3.3× improvement that comes entirely from data scale and diversity.

4. Key Insights and Innovations

Innovation 1: Optional Depth as a First-Class Inference-Time Modality, Not a Training Requirement

The dominant assumption in monocular 3D detection—from Cube R-CNN [6] through 3D-MOOD [59]—has been that monocular means monocular: the model must produce 3D estimates from RGB alone, both at training and inference time. This makes sense as a pure research formulation of the ill-posed inverse problem, but it creates a bizarre practical constraint: even when depth sensors are available at deployment (LiDAR on a phone, stereo on an AR headset, structured light on a robot), the model has no pathway to use them. The sensor sits idle while the model struggles with scale ambiguity that the sensor could resolve.

WildDet3D's architecture—the dual-vision encoder and depth fusion module described in Section 3—makes depth optional at inference time through a deliberate design choice: the RGBD encoder accepts a 4-channel input where the depth channel is zero-filled when absent, and training uses a stochastic strategy (70% monocular, 20% sparse depth, 10% full depth) so the model learns to operate across the full spectrum from pure monocular to fully depth-enhanced. This is not a new depth estimation technique—the RGBD encoder uses the existing LingBot-Depth [49] backbone—but rather a deployment-oriented reframing of how depth should integrate into 3D perception.

What makes this intellectually distinctive is the underlying philosophy: graceful degradation as a first-class design constraint rather than an afterthought. Prior work implicitly assumed uniform deployment conditions—either you have a depth sensor (use a LiDAR-based detector) or you don't (use a monocular detector). WildDet3D recognizes that real deployment is heterogeneous: the same model running on an iPhone Pro (with LiDAR), a budget Android phone (no depth), and a Meta Quest 3 (stereo depth) should adapt seamlessly without architecture changes or model swapping. The depth fusion module's zero-initialized residual connection—starting as identity and gradually learning geometric corrections—is a practical instantiation of this philosophy at the implementation level, but the conceptual move is bigger: treating sensor availability as a continuous variable rather than a binary condition.

The evidence that this matters is stark: ground-truth depth at test time yields a ~2× performance improvement (22.6 → 41.6 AP on WildDet3D-Bench, Table 3), and real stereo depth on the zero-shot Stereo4D evaluation produces a 2.8× gain (7.5 → 27.7 AP, Table 6). But the deeper significance is that this reframing changes the research question from "how do we solve monocular 3D detection?" to "how do we build 3D perception that uses whatever geometric signals are available?"—a question that matters for every deployed system but was largely ignored by the monocular-focused literature.

Innovation 2: Multi-Prompt Unification as a Target Distribution Problem, Not a Multi-Head Architecture Problem

The standard approach to handling multiple interaction modalities—text queries, point clicks, bounding boxes—is to build a separate model or prediction head for each. 3D-MOOD [59] handles text queries; DetAny3D [64] handles box prompts; SAM-3D [50] handles mask prompts. Each is a distinct system with its own architecture, training data, and failure modes. A deployment requiring multiple interaction modes would need to run multiple models or maintain multiple code paths, with no mechanism for cross-modal transfer learning.

WildDet3D takes a fundamentally different approach: all prompt modalities are encoded into a unified token sequence that conditions a single shared detector. Text prompts go through a causal Transformer and get projected to 256-dimensional embeddings. Box and point prompts go through a geometry encoder that sums coordinate projections, ROI-pooled visual features, and sinusoidal positional encodings, then get refined by cross-attention to image features. Exemplar prompts reuse the box encoder with a distinguishing "visual" text token. All of these—text tokens, geometry tokens, exemplar markers—are concatenated into a single prompt sequence that serves as cross-attention memory for the transformer decoder.

The intellectual move here is conceptual, not architectural (the prompt encoding itself follows SAM 3 [8]): treating diverse interaction modalities as different projections into a shared semantic-geometric embedding space, rather than as separate tasks requiring separate architectures. The model learns that a text query "chair" and a box prompt around a chair region should both activate the same underlying object representation, even though they arrive through entirely different input pathways. The per-prompt batching strategy—constructing batches at the prompt level rather than the image level—ensures balanced training across modalities without modality-specific loss weighting or staged training.

Why does this matter beyond convenience? Because the unification enables emergent cross-modal generalization that separate models cannot achieve. When the model is trained on text queries for "car" and box prompts for "bicycle," the shared transformer decoder learns representations that transfer across modalities—the geometric reasoning learned from box prompts improves text-prompt detection, and the semantic understanding learned from text prompts improves box-prompt lifting. The evidence is in the results: the same model achieves 22.6 AP with text prompts and 24.8 AP with box prompts on WildDet3D-Bench (Table 3), and 34.2 AP text / 36.4 AP oracle on Omni3D (Table 4)—competitive with or exceeding modality-specialized methods in both settings, despite being a single model.

This is a fundamental shift from multi-task learning (where shared representations improve separate heads) to unified conditioning (where shared representations eliminate the distinction between modalities entirely). It makes WildDet3D deployable as a drop-in module for diverse downstream systems—a VLM's 2D grounding, a user's tap on a screen, a robot's language command—without modality-specific interfaces, which is precisely what the deployment demonstrations in Section 5 (iPhone, Quest 3, Franka arm, VLM integration) show.

Innovation 3: Identifying Annotation Pipeline Diversity as the Bottleneck for Open-World 3D Detection, Not Model Architecture

The paper makes an explicit diagnostic claim that is both obvious in retrospect and largely unaddressed by prior work: the primary obstacle to open-world 3D detection is not model design but data scale and category diversity. Omni3D [6], the largest prior unified benchmark, covers 98 categories—approximately the same vocabulary size as COCO [29] in 2014. In the intervening decade, 2D detection has expanded to 13K+ categories (V3Det [51]) through large-scale annotation efforts. 3D detection, by contrast, has remained stuck at two orders of magnitude fewer categories because annotating 3D boxes at scale has been considered prohibitively expensive.

The WildDet3D-Data pipeline is an engineering contribution—five candidate generators, rule-based filtering, VLM scoring, human verification—but the underlying insight is a reframing of the 3D annotation problem from an acquisition problem to a selection problem. Prior approaches to 3D annotation assumed that humans must create 3D boxes from scratch (expensive, requires skilled annotators, limited throughput) or that a single automatic method can produce adequate annotations (noisy, biased toward that method's failure modes). WildDet3D-Data reformulates annotation as: generate multiple noisy candidates from diverse methods, then select the best one. Humans no longer create 3D boxes; they choose among pre-generated options—a task that is faster, requires less spatial reasoning skill, and can be crowdsourced at scale on platforms like Prolific.

The pipeline validation in Table 2 provides the evidence that this reframing works. The five candidate generation methods have dramatically different quality profiles: SAM-3D contributes 40.4% of accepted annotations, RANSAC-PCA 28.2%, DetAny3D 14.5%, LabelAny3D 13.0%, and 3D-MOOD 3.8%. Rejection rates vary from 12.5% (RANSAC-PCA) to 42.9% (DetAny3D). No single method dominates, and the best method depends on the specific object and image—diversity of candidates is essential because different methods fail on different examples. A pipeline using only SAM-3D (the best single method by selection share) would miss the 60% of annotations where other methods produce the best candidate.

This is a diagnostic insight with implications beyond this paper: it suggests that the path to scaling 3D perception—and likely other perception tasks requiring expensive annotation (6-DoF pose, dense depth, material properties)—is not better individual models but better pipelines for combining diverse, imperfect sources. The 138× increase in category coverage (98 to 13.5K) and the 3.3× performance improvement on WildDet3D-Bench from adding WildDet3D-Data to Omni3D training (6.8 → 22.6 AP, Table 3) demonstrate that data scale is the dominant factor, not a minor contributor.

The finding that VLM scoring correlates perfectly with human rejection rates (Spearman ρ = −1.0, Table 2 bottom) but still misses 9.2% of human rejections at the maximum score (score 11) adds nuance: automatic scoring can dramatically reduce the annotation burden (896K images auto-verified vs. 103K human-verified) but cannot fully replace human judgment, particularly for the hardest cases where geometric ambiguity is highest.

Innovation 4: Unambiguous Rotation Normalization as a Training Simplification, Not a Symmetry-Aware Loss

Most work on oriented 3D bounding box prediction acknowledges the rotation-dimension ambiguity—a box with dimensions (w, h, l) rotated by θ is identical to one with dimensions (l, h, w) rotated by θ + 90°, and a 180° yaw flip preserves geometry for symmetric objects—and addresses it through symmetry-aware loss functions. The standard approach computes the loss against all equivalent ground-truth parameterizations and takes the minimum: L = min_{k ∈ {1,2,3,4}} L(pred, GT_k), where GT_k are the four equivalent representations of the same physical box. This is conceptually clean—the model can predict any valid representation—but creates a non-convex loss landscape with multiple local minima corresponding to different representational conventions, which empirically makes training harder and convergence slower.

WildDet3D does something simpler and, in retrospect, more pragmatic: normalize the ground truth into a canonical form before computing the loss. The two-step procedure (dimension ordering: if w > l, swap and rotate by 90°; yaw folding: fold yaw into [0, π) by applying 180° flips) maps all four equivalent representations to a single unambiguous target. The model only needs to learn one representation per box, with a convex L1 loss on the canonical parameters.

This is not a theoretical advance—the normalization is a simple geometric transformation that has been used in other 3D vision contexts (e.g., canonical pose representations for object pose estimation). What makes it an innovation in this context is the pragmatic reframing of the problem: rather than designing a loss function sophisticated enough to handle ambiguity, eliminate the ambiguity at the data level so any standard loss works. This is a design philosophy—preprocess to simplify the learning problem rather than complicate the loss—that runs counter to the dominant trend in deep learning toward ever-more-elaborate loss formulations.

The ablation evidence is indirect but supportive: the ablation on training objectives (Table 8) shows that removing one-to-many matching drops AP by 2.5, removing geometry loss drops AP by 1.7, and the 3D detection head architecture ablation (Table 7) shows that removing the 2D head collapses AP by 19.1. The rotation normalization contributes to the baseline performance but isn't ablated directly (probably because it's baked into the data preprocessing and would require retraining from scratch). The paper's empirical success—state-of-the-art on Omni3D with 6–10× fewer training epochs than prior methods (Section 4.3)—is circumstantial evidence that the training simplifications (including rotation normalization) contribute to efficiency, but the paper does not isolate its specific contribution. This is a methodological contribution rather than an empirical one: a principle for designing 3D detection systems that future work can adopt without needing the symmetry-aware loss machinery.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on four benchmarks spanning diverse detection settings. WildDet3D-Bench (Section 4.2) is the authors' newly introduced in-the-wild benchmark covering 700+ open-vocabulary categories across COCO, LVIS, and Objects365 images, with human-verified 3D annotations split by annotation frequency into rare (<5 samples, 464 categories), common (5–20 samples, 283 categories), and frequent (>20 samples, 63 categories). Omni3D [6] unifies six datasets—KITTI [17] (7.5K outdoor driving images), nuScenes [7] (28K multi-view driving images), SUNRGBD [46] (10K indoor RGB-D images), Hypersim [43] (100K+ synthetic indoor images), ARKitScenes [4] (55K mobile AR indoor images), and Objectron [1] (15K+ object-centric images)—with a unified label space of 98 categories and standardized 3D box annotations. Argoverse 2 [55] (outdoor driving, 26 classes) and ScanNet [12] (indoor, 18 classes) are used for zero-shot cross-dataset evaluation following the 3D-MOOD protocol [59], both unseen during training. Stereo4D [22] provides 383 in-the-wild images with real stereo depth maps across 78 categories, used to evaluate depth generalization zero-shot.
Base model(s). WildDet3D uses two frozen pretrained backbones. The Image Encoder is a ViT-H from SAM 3 [8], initialized from segmentation pretraining (first 28 of 32 blocks frozen). The RGBD Encoder is a DINOv2 ViT-L/14 from LingBot-Depth [49], pretrained for metric depth estimation on large-scale RGBD data (first 21 of 24 blocks frozen, last 3 trainable). The 3D detection head is trained from scratch. The complete model is trained in three stages using AdamW with base learning rate 10⁻⁴ and weight decay 10⁻⁴, on 4 nodes (32 GPUs) with per-GPU batch size 4 (effective batch size 128). Stage 1 trains on Omni3D for 12 epochs; Stage 2 trains on a mixture of Omni3D, WildDet3D-Data (human + synthetic), and supplementary datasets (CA-1M, Waymo, 3EED, FoundationPose) for 12 epochs; Stage 3 fine-tunes on Omni3D and human-verified WildDet3D-Data with mask-guided point/box training for 3 epochs. Input images are resized to 1008×1008 pixels.
Metrics. For Omni3D, the primary metric is AP3D: Average Precision at 3D bounding box IoU thresholds [0.05 : 0.50 : 0.05] (10 thresholds), computed via oriented bounding box overlap from 8 corner points in camera coordinates. Per-dataset breakdowns report AP at individual thresholds (AP15, AP25, AP50). For WildDet3D-Bench and Stereo4D, where open-vocabulary categories span wide size ranges, the metric is AP using center-distance matching: a prediction matches a ground truth if the 3D center distance is below a threshold proportional to the object radius, with thresholds [0.50 : 1.00 : 0.05] (11 thresholds). For zero-shot transfer on Argoverse 2 and ScanNet, the metric is ODS (Open Detection Score): ODS = (3·AP + (1−mATE) + (1−mAOE) + (1−mASE))/6, where mATE, mAOE, and mASE denote mean translation, orientation, and scale errors respectively, with AP contributing 50% of the score. WildDet3D-Bench further reports per-frequency AP: AP_rare, AP_common, AP_frequent.
Baselines. The paper compares against multiple prior methods spanning closed-set, open-vocabulary, and prompt-conditioned 3D detection:
- Cube R-CNN [6]: A strong monocular 3D detector operating in closed-set settings across multiple datasets.
- Uni-MODE [28]: A unified monocular 3D detection model trained across diverse indoor and outdoor scenarios.
- 3D-MOOD [59]: An open-vocabulary text-prompt 3D detector with Swin-T and Swin-B backbones, which runs open-vocabulary 2D detection and lifts detections to 3D.
- OVMono3D-LIFT [60]: An open-vocabulary text-prompt method that lifts open-vocabulary 2D detections into 3D.
- DetAny3D [64]: A box-prompt 3D detection foundation model that predicts 3D boxes from 2D bounding box prompts.
- WildDet3D w/ depth: The authors' own model evaluated with ground-truth or real depth provided at test time, serving as an upper bound for depth-enhanced performance.
Generation budget / compute accounting. There is no explicit "generation budget" in the sense of test-time compute scaling (this is a single-pass detection architecture, not a sampling-based method). Compute is measured in training epochs—WildDet3D trains for 12 epochs on Omni3D versus 80 epochs for DetAny3D and 120 epochs for 3D-MOOD, providing a FLOPs-matched comparison at training time. At inference, the comparison is per-image throughput, though specific latency numbers are not reported. The paper's key efficiency claim is that WildDet3D achieves superior results with 6–10× fewer training epochs than prior methods.
Cross-validation / statistical protocol. For WildDet3D-Bench evaluation, the paper follows the federated evaluation protocol of LVIS [18]: since annotations are not exhaustive (not every visible object has a valid 3D bounding box), a prediction that overlaps with a 2D-annotated object lacking a valid 3D box is treated as neutral rather than a false positive. For zero-shot transfer (Argoverse 2, ScanNet, Stereo4D), models trained only on Omni3D are evaluated directly on the target datasets without fine-tuning. Per-category NMS with a 2D IoU threshold of 0.6 is applied at test time, and predictions with a 2D objectness score below 0.05 are discarded. Maximum detections per image is capped at 100. No cross-validation over hyperparameters is reported; the training recipe (3 stages, learning rate schedule, data mixing ratios) appears to be fixed.

Main Quantitative Results

In-the-Wild Evaluation on WildDet3D-Bench (Section 4.2, Table 3)

Headline result: WildDet3D achieves 22.6 AP3D with text prompts and 24.8 AP3D with box prompts on WildDet3D-Bench, representing a 9.8× improvement over the prior state-of-the-art (3D-MOOD at 2.3 AP) for text-prompt detection, while ground-truth depth at test time nearly doubles performance to 41.6 AP (text) and 47.2 AP (box).

Training data scaling. When trained on Omni3D only with text prompts, WildDet3D achieves 6.8 AP, already 3.0× better than 3D-MOOD (2.3 AP). Adding supplementary datasets (+Others) and WildDet3D-Data pushes text-prompt AP to 22.6—a further 3.3× gain from data scale alone. The same pattern holds for box prompts: Omni3D-only training yields 8.4 AP (already competitive with OVMono3D-LIFT at 7.7 AP and DetAny3D at 7.8 AP), while full training data reaches 24.8 AP. This demonstrates that data diversity is the dominant factor for in-the-wild generalization, more impactful than architectural improvements over baselines.

Effect of ground-truth depth. Providing ground-truth depth at test time yields dramatic gains across all settings. For the Omni3D-only model, text-prompt AP jumps from 6.8 → 20.7 (+13.9) and box-prompt AP from 8.4 → 23.9 (+15.5). For the full model trained on all data, text-prompt AP increases from 22.6 → 41.6 (+19.0) and box-prompt AP from 24.8 → 47.2 (+22.4). These gains are consistent across frequency splits: AP_rare improves from 28.3 → 47.4 (+19.1), AP_common from 21.6 → 40.7 (+19.1), AP_frequent from 18.7 → 37.2 (+18.5). The near-constant additive gain across frequency groups (~19–22 AP) suggests depth provides a consistent geometric improvement regardless of category frequency.

Text vs. box prompt comparison. Box prompts consistently outperform text prompts when no depth is provided: 8.4 vs. 6.8 AP for Omni3D-only, 24.8 vs. 22.6 AP for the full model. This gap (~2 AP) reflects the inherent difficulty of end-to-end open-vocabulary text detection compared to oracle-provided 2D localization. However, with ground-truth depth, the gap narrows proportionally: 47.2 vs. 41.6 AP, suggesting that depth helps text-prompt detection close some of the 2D localization gap. The text-prompt results with depth (41.6 AP) are competitive with box-prompt results without depth for the full model (24.8 AP), but not with box-prompt results with depth (47.2 AP), indicating that 2D localization remains a bottleneck even with perfect depth.

Frequency-dependent performance. Improvements are largest on rare categories: AP_rare reaches 47.4 with box prompts and depth, compared to 2.4 for 3D-MOOD with text prompts only—a 19.75× improvement. This suggests WildDet3D's open-vocabulary capability (inherited from SAM 3's pretrained vision-language alignment) and the broad category coverage of WildDet3D-Data specifically benefit long-tail categories that prior closed-set methods cannot detect at all.

Results on Omni3D (Section 4.3, Table 4)

Headline result: WildDet3D achieves 34.2 AP3D with text prompts and 36.4 AP3D with oracle box prompts on Omni3D, surpassing prior state-of-the-art methods (3D-MOOD Swin-B at 30.0 AP text, DetAny3D at 34.4 AP oracle) while training for only 12 epochs versus 80–120 epochs for competing approaches. With depth at test time, performance reaches 41.6 AP (text) and 45.8 AP (oracle).

Text-prompt comparison. WildDet3D (34.2 AP) outperforms 3D-MOOD Swin-B (30.0 AP) by +4.2 AP and Cube R-CNN (23.3 AP) by +10.9 AP. The per-dataset breakdown reveals where the gains concentrate: indoor datasets show the largest improvements. On SUNRGBD, WildDet3D achieves 38.9 vs. 3D-MOOD Swin-B's 23.8 (+15.1). On ARKitScenes, WildDet3D reaches 64.6 vs. 53.9 (+10.7). On Hypersim, WildDet3D achieves 16.5 vs. 9.1 (+7.4). Outdoor driving datasets show more modest gains: KITTI at 37.0 vs. 31.4 (+5.6), nuScenes at 31.7 vs. 35.8 (−4.1). The regression on nuScenes relative to 3D-MOOD Swin-B is notable and unexplained—it may reflect WildDet3D's indoor-heavy training data distribution or differences in how multi-view driving data is handled.

Box-prompt (oracle) comparison. WildDet3D (36.4 AP) outperforms DetAny3D (34.4 AP) by +2.0 AP. The pattern is similar: large gains on indoor datasets (SUNRGBD: 43.1 vs. 46.1, ARKitScenes: 66.6 vs. 50.6, Objectron: 60.8 vs. 56.8), competitive performance on driving datasets (KITTI: 44.3 vs. 38.7, nuScenes: 35.3 vs. 37.6). The nuScenes deficit relative to DetAny3D (−2.3) mirrors the text-prompt pattern and suggests that WildDet3D's geometric estimation may be less well-calibrated for the multi-view driving domain.

Effect of depth at test time. With ground-truth depth provided at inference, WildDet3D reaches 41.6 AP (text, +7.4 over monocular) and 45.8 AP (oracle, +9.4 over monocular). The gains are dramatic on indoor datasets with native depth sensors: SUNRGBD jumps from 38.9 → 51.1 (text) and 43.1 → 58.7 (oracle); ARKitScenes from 64.6 → 73.3 (text) and 66.6 → 76.6 (oracle); Hypersim from 16.5 → 26.6 (text) and 17.3 → 30.4 (oracle). Outdoor datasets show minimal improvement: KITTI remains nearly flat at 37.0 → 36.1 (text) and 44.3 → 42.8 (oracle), with slight degradation possibly due to train-test domain mismatch in depth statistics. This pattern confirms that depth is most beneficial in indoor environments where metric scale is the primary source of ambiguity, while outdoor driving scenes—where objects are at consistent scales and depths relative to the camera—benefit less from explicit depth signals.

Training efficiency claim. WildDet3D achieves these results with 12 epochs on Omni3D, compared to 120 epochs for 3D-MOOD and 80 epochs for DetAny3D. This 6–10× reduction in training compute is attributed to the strong pretrained representations from SAM 3 and LingBot-Depth, which provide high-quality initialization for both detection and depth estimation. However, the paper does not report a controlled ablation where prior methods are also initialized from SAM 3-style pretraining, so it is unclear whether the efficiency gain comes from the architecture or simply from better pretraining. If 3D-MOOD or DetAny3D were reimplemented with comparable backbone initialization, their training efficiency might also improve substantially.

Zero-Shot Evaluation (Section 4.4, Table 5)

Headline result: Trained on Omni3D and evaluated zero-shot, WildDet3D achieves 40.3 ODS on Argoverse 2 and 48.9 ODS on ScanNet, outperforming 3D-MOOD Swin-B by +16.5 and +17.4 ODS respectively. The detection AP component is particularly strong: 43.4 vs. 14.8 on Argoverse 2 (+28.6) and 56.5 vs. 28.8 on ScanNet (+27.7).

Decomposing the ODS gains. The ODS formula weights AP at 50% and three error metrics (mATE, mASE, mAOE) collectively at 50%. On Argoverse 2, WildDet3D's AP advantage (+28.6 over 3D-MOOD Swin-B) is partially offset by only modest improvements in error metrics: mATE improves from 0.755 → 0.714 (−0.041), mASE from 0.680 → 0.645 (−0.035), and mAOE from 0.580 → 0.526 (−0.054). The large AP gain combined with small error metric improvements yields a substantial but not proportionally enormous ODS gain (+16.5). On ScanNet, the pattern is similar but with a striking mAOE improvement: 0.655 → 0.437 (−0.218), indicating that WildDet3D's unambiguous rotation normalization and geometric pretraining provide substantially better orientation estimation on indoor objects.

Generalization to unseen categories. ScanNet includes categories unseen during Omni3D training, making it a genuine test of open-vocabulary generalization. WildDet3D's 48.9 ODS on ScanNet—with AP of 56.5—indicates strong cross-category transfer, consistent with the rare-category performance on WildDet3D-Bench (Table 3). The paper notes that the model achieves "particularly large improvements on novel categories unseen during training" (Section 4.4), though per-category breakdowns on ScanNet are not provided.

Effect of ground-truth depth on zero-shot transfer. Adding ground-truth depth at test time produces asymmetric gains: on ScanNet, ODS improves from 48.9 → 50.2 (+1.3); on Argoverse 2, ODS is essentially flat at 40.3 → 40.4 (+0.1). The paper interprets this as evidence that "the model's monocular depth estimation is already well-calibrated for outdoor driving scenes at the scale and depth ranges present in Argoverse 2" (Section 4.4). This is consistent with the Omni3D results (Table 4), where KITTI and nuScenes showed minimal benefit from depth, and reinforces the finding that depth integration is most valuable for indoor environments with greater scale variation.

Comparison to Cube R-CNN. Cube R-CNN achieves only 8.9 ODS on Argoverse 2 and 19.5 ODS on ScanNet, dramatically below both 3D-MOOD and WildDet3D. This is expected since Cube R-CNN is a closed-set detector—it can only detect the 98 Omni3D categories—and Argoverse 2 and ScanNet have category spaces that only partially overlap with Omni3D. WildDet3D's open-vocabulary design is thus essential for zero-shot transfer.

In-the-Wild Evaluation with Real Depth (Section 4.5, Table 6)

Headline result: On Stereo4D, a zero-shot evaluation with real stereo depth maps (383 images, 78 categories), WildDet3D achieves 27.7 AP3D with real depth input, a 2.8× improvement over the best monocular method (OVMono3D-LIFT at 9.9 AP). Without depth, WildDet3D's monocular performance (7.5 AP) is competitive with DetAny3D (7.1 AP) but below OVMono3D-LIFT (9.9 AP).

Monocular comparison. The monocular ranking—OVMono3D-LIFT (9.9) > WildDet3D (7.5) > DetAny3D (7.1)—is notable because OVMono3D-LIFT is a text-prompt method while the other two are evaluated in box-prompt (oracle) mode, which typically has an advantage. The paper attributes OVMono3D-LIFT's higher monocular AP to "stronger monocular depth estimation on this low-resolution stereo domain" (Section 4.5), suggesting that WildDet3D's depth estimation, while generally strong, may be less calibrated for the specific resolution and depth distribution of Stereo4D images.

Real depth integration. The dramatic jump from 7.5 (monocular) to 27.7 (with depth)—a 20.2 AP gain—validates that WildDet3D's depth fusion module works with real sensor depth, not just ground-truth depth. The per-frequency breakdown shows AP_common benefiting most (6.3 → 31.1, +24.8), followed by AP_rare (8.1 → 26.2, +18.1) and AP_frequent (8.5 → 24.6, +16.1). The fact that common categories benefit more than frequent ones suggests that depth resolves scale ambiguity for objects with typical sizes and shapes, while rare objects may have more variable geometry that depth alone cannot fully resolve.

Significance for deployment. This is the only evaluation in the paper using real (non-ground-truth) depth from an actual stereo system, making it the most ecologically valid test of the depth fusion module. The 2.8× improvement over the monocular state-of-the-art demonstrates that the architecture's core premise—optional depth integration at inference time—translates from ground-truth depth (where gains of ~2× are typical in Tables 3–4) to real sensor depth.

Ablation Studies and Robustness Checks

All ablations use the oracle (box prompt) evaluation setting on Omni3D, training on Omni3D only, unless otherwise noted.

Joint 2D+3D detection head (Table 7): Removing the 2D head and predicting 3D boxes directly causes AP to collapse from 30.2 to 11.1 (−19.1 AP). Indoor datasets are hit hardest: SUNRGBD drops from 33.9 → 5.1 (−28.8), Objectron from 56.8 → 10.9 (−45.9), ARKitScenes from 59.4 → 28.5 (−30.9). Driving datasets degrade less catastrophically but still substantially: KITTI from 27.9 → 18.3 (−9.6), nuScenes from 28.2 → 15.6 (−12.6). This confirms a fundamental design principle: 2D detection provides essential spatial priors for anchoring 3D regression, and attempting to jointly localize objects in 2D and estimate 3D geometry from scratch is extremely difficult, particularly in cluttered indoor environments where objects are small and numerous.

3D confidence head (Table 7): Removing the 3D confidence head drops AP from 30.2 to 29.4 (−0.8). This is a small but consistent effect, confirming that geometry-aware re-ranking complements 2D objectness for distinguishing well-localized 3D predictions from inaccurate ones. The small magnitude suggests that 2D objectness alone captures most of the detection quality signal, with 3D confidence providing a modest tiebreaking function.

One-to-many (O2M) matching (Table 8): Removing O2M auxiliary matching is the most impactful training objective ablation, dropping AP from 30.2 to 27.7 (−2.5). Driving datasets are most affected: KITTI from 27.9 → 23.2 (−4.7), nuScenes from 28.2 → 23.9 (−4.3). Indoor datasets show smaller drops: SUNRGBD from 33.9 → 30.8 (−3.1), ARKitScenes from 59.4 → 56.8 (−2.6). This pattern is consistent with the paper's explanation that "dense, similarly-sized objects benefit most from the richer supervision signal that O2M provides" (Section 4.6)—driving scenes contain many objects at similar scales (cars, pedestrians) where O2M's denser gradients help the model distinguish fine-grained differences.

Geometry loss (Table 8): Removing explicit geometric supervision (depth and camera-ray losses) drops AP from 30.2 to 28.5 (−1.7). The effect concentrates on indoor scenes where depth accuracy is most critical: SUNRGBD drops from 33.9 → 28.6 (−5.3), Hypersim from 13.2 → 11.1 (−2.1). Outdoor datasets are essentially unaffected: KITTI actually improves slightly (27.9 → 28.3, +0.4) and nuScenes is flat (28.2 → 27.7). This reinforces the pattern from Table 4: depth estimation is the dominant bottleneck for indoor 3D detection but less critical for outdoor driving where consistent ground-plane geometry provides strong monocular cues.

Deep supervision (Table 8): Removing deep supervision (applying loss only at the final decoder layer) drops AP from 30.2 to 29.9 (−0.3). The effect is small and uniform across datasets, suggesting that deep supervision provides a minor regularization benefit—encouraging intermediate layers to develop 3D awareness—but does not fundamentally change what the model learns.

Ignore-aware suppression (Table 8): Removing ignore-aware suppression drops AP from 30.2 to 30.0 (−0.2). This is the smallest ablation effect, consistent with the paper's note that "ignore annotations are sparse in this benchmark." On WildDet3D-Bench, where partial 3D annotations are more common, the effect would likely be larger, but this is not evaluated.

Sparse depth at test time (Table 4, depth rows): When sparse depth measurements are available at inference, performance on Omni3D improves substantially: oracle AP increases from 36.4 → 45.8 (+9.4), text AP from 34.2 → 41.6 (+7.4). The gains are concentrated on indoor datasets with depth sensors (SUNRGBD, Hypersim, ARKitScenes), with outdoor datasets showing little to no improvement. This ablation confirms that the depth fusion module effectively leverages real depth signals when available, not just ground-truth depth, and that the benefit is domain-dependent.

Training data ablation (Table 3): Comparing Omni3D-only training (6.8 AP text, 8.4 AP box) to full training with +Others + WildDet3D-Data (22.6 AP text, 24.8 AP box) reveals a 3.3× gain from data scale on WildDet3D-Bench. This is the largest ablation—larger than any architectural or training objective change—confirming the paper's central thesis that data diversity is the primary bottleneck for open-world 3D detection.

Critical Assessment

Claim 1: WildDet3D establishes state-of-the-art on multiple benchmarks

What the experiments demonstrate: On Omni3D (Table 4), WildDet3D achieves 34.2 AP (text) and 36.4 AP (oracle), outperforming 3D-MOOD Swin-B at 30.0 AP and DetAny3D at 34.4 AP. On WildDet3D-Bench (Table 3), WildDet3D achieves 22.6/24.8 AP versus 3D-MOOD at 2.3 AP and DetAny3D at 7.8 AP. On zero-shot transfer (Table 5), WildDet3D achieves 40.3/48.9 ODS versus 3D-MOOD Swin-B at 23.8/31.5 ODS. These are genuine state-of-the-art numbers.

Caveats: The Omni3D comparison has an asymmetry in training data. WildDet3D is trained on Omni3D + Others + WildDet3D-Data for Stage 2/3, while baselines like 3D-MOOD and DetAny3D are trained only on Omni3D (or Omni3D + smaller supplementary sets). The paper reports Omni3D-only WildDet3D results for the in-the-wild benchmark (6.8 AP, Table 3) but does not report Omni3D-only WildDet3D results for the Omni3D benchmark itself. This makes it impossible to determine how much of the Omni3D improvement (34.2 vs. 30.0 for 3D-MOOD Swin-B) comes from the architecture versus from training on additional data that the baselines did not see. A fair comparison would train all methods on the same data mixture or ablate WildDet3D trained only on Omni3D for the Omni3D benchmark.

The zero-shot transfer is genuinely impressive and has a cleaner comparison—all methods are trained on Omni3D only—so the +16.5/+17.4 ODS gains over 3D-MOOD Swin-B are a fair measure of architectural and pretraining advantages.

Claim 2: Incorporating depth at inference yields ~2× performance gains

What the experiments demonstrate: This claim is supported across multiple benchmarks. On WildDet3D-Bench (Table 3), depth provides +19.0 AP (text) and +22.4 AP (box) for the full model—near 2× improvements. On Omni3D (Table 4), depth provides +7.4 AP (text) and +9.4 AP (oracle)—substantial but less than 2×. On Stereo4D with real depth (Table 6), depth provides +20.2 AP—a 2.8× improvement. On zero-shot transfer (Table 5), the gains are more modest but still present (+1.3 ODS on ScanNet, +0.1 ODS on Argoverse 2).

Caveats: With one exception (Stereo4D), all depth experiments use ground-truth depth, not real sensor depth. Ground-truth depth is perfect—no noise, no missing values, no calibration errors—and represents an upper bound on what real sensors could provide. The Stereo4D experiment (Table 6) is the only evaluation with real sensor depth and shows that the gains persist (2.8×), but this is on only 383 images across 78 categories. A more convincing demonstration would evaluate with real sensor depth on larger benchmarks and across more diverse sensor types (sparse LiDAR, ToF, structured light).

Additionally, the depth experiments on Omni3D (Table 4) show the gains are highly domain-dependent: indoor datasets (SUNRGBD, Hypersim, ARKitScenes) benefit enormously (often 10–20 AP), while outdoor driving datasets (KITTI, nuScenes) show negligible or slightly negative effects. The "~2×" claim masks this heterogeneity—depth is transformative for indoor 3D detection but largely irrelevant for outdoor driving under the evaluation conditions tested.

Claim 3: WildDet3D generalizes zero-shot to novel categories and datasets

What the experiments demonstrate: The zero-shot transfer results (Table 5) are the strongest evidence for this claim. Trained on Omni3D (98 categories), WildDet3D achieves 40.3 ODS on Argoverse 2 (26 classes, outdoor driving) and 48.9 ODS on ScanNet (18 classes, indoor), both unseen during training. The detection AP alone—43.4 on Argoverse 2 and 56.5 on ScanNet—indicates genuine generalization, not just metric manipulation. The WildDet3D-Bench results broken down by frequency (Table 3) show strong performance on rare categories (AP_rare = 28.3 with text prompts, 47.4 with depth), which are explicitly long-tail objects with few training examples.

Caveats: The claim of "novel categories" in zero-shot transfer requires careful examination. Argoverse 2 and ScanNet categories partially overlap with Omni3D categories (e.g., "car," "chair," "table" appear in both). The paper acknowledges this implicitly by noting "particularly large improvements on novel categories unseen during training" (Section 4.4) but does not provide a per-category breakdown showing performance on truly novel versus overlapping categories. Without this breakdown, it is unclear whether the zero-shot gains come from better detection of shared categories (which would be less impressive—it's just better within-distribution performance) or from genuine generalization to completely unseen object types.

Additionally, the WildDet3D-Bench results are on categories that all appear in WildDet3D-Data training, so they test generalization to rare-but-seen categories, not zero-shot transfer to unseen ones. A true zero-shot evaluation on WildDet3D-Bench—holding out a subset of categories entirely during training—would be a stronger test of open-world generalization.

Claim 4: WildDet3D-Data enables 138× category coverage and 3–4× performance gains

What the experiments demonstrate: Table 1 confirms the 13,499 categories in WildDet3D-Data versus Omni3D's 98—a 138× increase. Table 3 shows that adding WildDet3D-Data to training improves WildDet3D-Bench performance from 6.8 → 22.6 AP (text, 3.3×) and 8.4 → 24.8 AP (box, 3.0×).

Caveats: The 138× number refers to category vocabulary size, not to number of annotations per category or annotation quality. Most of the 13,499 categories likely have very few annotations (the val/test sampling ensures ≥3 per category, but training may have similarly sparse coverage for rare categories). The "3–4× performance gain" is measured on WildDet3D-Bench, which is constructed from the same datasets that WildDet3D-Data draws from (COCO, LVIS, Objects365). This creates potential for train-test leakage if the sampling is not perfectly stratified—images from the same scene or of the same object instance could appear in both train and test. The paper describes a careful sampling strategy (Section 3.4 and Appendix D) but does not provide analysis of instance-level overlap between train and test.

Furthermore, the paper does not report results on Omni3D after training on WildDet3D-Data. Table 4 reports Omni3D results for the full model (trained on Omni3D + Others + WildDet3D-Data), but the Omni3D-only ablation is not provided. This makes it impossible to determine whether WildDet3D-Data improves Omni3D performance (suggesting broad transfer) or mainly improves WildDet3D-Bench performance (suggesting narrower transfer within the same image sources).

Claim 5: WildDet3D achieves these results with 6–10× fewer training epochs

What the experiments demonstrate: Table 4 reports 12 epochs for WildDet3D versus 80 epochs for DetAny3D and 120 epochs for 3D-MOOD, with WildDet3D achieving superior performance.

Caveats: Training epochs are not a standardized compute metric. Different methods use different batch sizes, image resolutions, and model sizes, so epoch count is not comparable across methods without additional context. The paper reports using 4 nodes (32 GPUs) with batch size 128 for WildDet3D but does not report the hardware configuration for baselines (which are evaluated using their publicly released models or reimplemented per their papers). More importantly, the pretraining FLOPs for SAM 3 (ViT-H) and LingBot-Depth (ViT-L) are not accounted for in the training efficiency comparison. If WildDet3D benefits from massive pretraining that the baselines do not, then the 6–10× reduction in fine-tuning epochs is not a fair measure of total compute—it measures fine-tuning efficiency given powerful pretrained initialization, not end-to-end training efficiency. A fair comparison would account for the total FLOPs of pretraining plus fine-tuning or would reimplement baselines with comparable pretrained backbones.

Missing Experiments

Several experiments would strengthen the paper's claims but are not reported:

WildDet3D trained on Omni3D only, evaluated on Omni3D. This would isolate the architectural contribution from the data contribution for the Omni3D benchmark.
Baselines trained on WildDet3D-Data. Training 3D-MOOD or DetAny3D on the same data as WildDet3D would test whether the performance gains are architectural or data-driven. If baselines also improve substantially with WildDet3D-Data, the data contribution would be validated; if they don't, the architecture's ability to leverage diverse data would be validated.
Per-category breakdown on zero-shot transfer. Which categories drive the gains on Argoverse 2 and ScanNet—shared Omni3D categories or genuinely novel ones? Without this, the "zero-shot generalization to novel categories" claim is only partially supported.
Real sensor depth evaluation on larger benchmarks. Stereo4D (383 images) is the only real-depth evaluation. Evaluating with sparse LiDAR on KITTI or with RGB-D sensors on SUNRGBD would test the depth fusion module's robustness to realistic depth quality.
Ablation of rotation normalization. The unambiguous rotation normalization is presented as a key design choice (Section 2.3) but is never ablated. Training with and without it would quantify its contribution to rotation accuracy (mAOE) and overall AP.
Computational cost analysis. The paper mentions that "the dual-backbone design increases memory and compute requirements compared to 2D-only detectors" (Section 7) but provides no latency, memory, or FLOPs measurements. For a system intended for deployment on iPhone and AR headsets (Section 5), these numbers are essential for assessing practicality.
VLM-filtered vs. human-verified training data ablation. WildDet3D-Data contains both human-verified (103K images) and VLM-filtered (896K images) annotations. Training with only one or the other would quantify the value of human verification versus the scale of automatic annotation. Table 2 suggests VLM filtering leaves 16.7% error at score 10—does training on this noisy data help or hurt relative to training on the smaller but cleaner human-verified subset?

Summary Assessment

The experiments broadly support the paper's core claims: WildDet3D achieves state-of-the-art on multiple benchmarks, depth integration provides substantial gains, zero-shot transfer is strong, and WildDet3D-Data enables broader category coverage. However, the evidence for architectural innovation (as opposed to data scale and pretraining quality) is confounded by the lack of controlled comparisons: the Omni3D-only ablation on Omni3D is missing, baselines are not trained on the same data, and pretraining compute is not accounted for in training efficiency claims. The paper's most robust finding—that data diversity is the primary bottleneck for open-world 3D detection—is well-supported by the WildDet3D-Bench results with and without WildDet3D-Data, but this finding is about the problem formulation and dataset, not about the model architecture. The architecture's specific contributions (dual encoders, depth fusion, promptable detector) are harder to isolate from the background factors of strong pretraining and large-scale training data.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted for in Reported Efficiency Claims

The assumption or constraint: The compute-optimal framework as deployed—specifically the PRM-based difficulty estimation—requires generating 2048 samples per question and scoring them with the PRM to assign a difficulty bin. The paper acknowledges this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The 2048-sample cost of difficulty estimation exceeds the largest test-time budgets studied (256–512 generations) by a factor of 4–8×. In the search experiments (Section 5), the reported 4× efficiency gain—e.g., compute-optimal at 16 generations matching best-of-N at 64 generations (Figure 4)—is computed after difficulty is known, without amortizing the cost of learning it.

The consequence: In a realistic deployment, the total compute would be difficulty_estimation_cost + strategy_execution_cost. Since the estimation cost dominates the execution cost, the true efficiency of the compute-optimal approach relative to a uniform best-of-N baseline is unknown and likely dramatically lower than the 4× headline figure. An organization deploying this system would need to decide whether the per-query overhead of estimating difficulty (which could be done once and cached for frequently-seen prompts) is justified by downstream savings—a trade-off the paper does not evaluate.

What evidence exists in the paper: The difficulty estimation cost is mentioned in Section 3.2 but is excluded from all budget calculations in Figures 4 and 8. The predicted difficulty bins (using PRM scores instead of ground-truth labels) still require 2048 samples—they only eliminate the need for ground-truth correctness, not the generation cost. The paper does not report any experiment measuring how performance degrades when difficulty is estimated from fewer samples (e.g., 16, 64, 256), which would directly quantify the cost-accuracy trade-off.

Mitigation status: The authors explicitly flag this as "a key avenue for future work" (Section 3.2) and suggest training models to predict difficulty directly from the question text, but no such model is developed or evaluated. The current results therefore represent an upper bound on achievable efficiency rather than a realized deployment gain. A reader should interpret the 4× figure as the theoretical maximum improvement if difficulty could be estimated at zero cost.

6.2 The Data Scale vs. Architecture Claim Is Confounded by Missing Controlled Comparisons

The assumption or constraint: The paper makes a strong claim that WildDet3D's performance gains come from its architecture—the dual-vision encoders, depth fusion module, promptable detector, and deeply-supervised 3D head. However, the evidence confounds architecture, pretraining quality, and training data scale. Specifically:

The Omni3D benchmark results (Table 4) report WildDet3D trained on Omni3D + Others + WildDet3D-Data, while baselines (3D-MOOD, DetAny3D, Cube R-CNN) are trained on Omni3D only or smaller supplementary sets. The paper does not report Omni3D results for WildDet3D trained on Omni3D only.
The training efficiency claim (12 epochs vs. 80–120 for baselines) does not account for the massive pretraining of the SAM 3 ViT-H and LingBot-Depth ViT-L backbones. If baselines were reimplemented with comparable pretrained initialization, their training efficiency might also improve substantially.
No baseline method is trained on WildDet3D-Data. It is therefore impossible to determine whether 3D-MOOD or DetAny3D would also achieve substantial gains from the additional 1M training images—which would validate the data-centric thesis but weaken the architectural contribution claim.

The consequence: The paper's contribution could be reinterpreted as primarily a data engineering result (building WildDet3D-Data) rather than an architectural innovation. If a practitioner reimplemented 3D-MOOD or DetAny3D with SAM 3-style pretraining and trained it on WildDet3D-Data, they might achieve comparable or even superior performance with a simpler architecture. The paper does not provide evidence to distinguish between these two interpretations. The 3.3× gain from adding WildDet3D-Data to training (6.8 → 22.6 AP on WildDet3D-Bench, Table 3) is larger than any architectural ablation in Tables 7–8, suggesting data scale may be the dominant factor.

What evidence exists in the paper: The ablation in Table 3 shows that training WildDet3D on Omni3D only yields 6.8 AP (text) on WildDet3D-Bench, already 3.0× better than 3D-MOOD (2.3 AP) trained on the same data—this is a fair architectural comparison. However, the equivalent Omni3D-only comparison on the Omni3D benchmark itself is missing. Tables 7–8 ablate architectural components (joint 2D+3D head, 3D confidence head, O2M matching, geometry loss) but only for the Oracle-prompt setting on Omni3D, and only when trained on Omni3D—not on the full WildDet3D-Data. This means we cannot assess whether these architectural components matter more or less when training data is abundant.

Mitigation status: The paper is transparent about its training data mixtures (Table 9) but does not acknowledge this confound as a limitation. The missing Omni3D-only WildDet3D evaluation on Omni3D is a significant gap. Until baselines are trained on the same data with comparable pretrained backbones, the claim that WildDet3D's architecture is state-of-the-art—rather than its training data—remains incompletely supported. This is a methodological weakness that affects the strength of the central architectural contribution claim.

6.3 Hard Problems (Bin 5) Remain Essentially Unsolved—Test-Time Compute Cannot Create Capability

The constraint: Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%. The paper acknowledges this directly:

"On the hardest questions (bin 5), no method makes meaningful progress—the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." (Section 5.3)

The consequence: Test-time compute amplifies existing capability but does not create it. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help—there are no correct solutions in the proposal distribution to find or refine. This places a hard boundary on the applicability of compute-optimal test-time scaling: it is effective only for problems within the base model's rough capability range. For genuinely novel, out-of-distribution, or fundamentally harder reasoning tasks that the base model cannot solve at all, pretraining remains the only viable path. In the FLOPs-matched analysis (Section 7, Figure 9), this translates to pretraining being universally preferable on bin 5 problems regardless of the inference-to-pretraining ratio R.

What evidence exists in the paper: Figure 3 (right) shows bin 5 accuracy at 1–3% for both best-of-N and beam search at all budgets from 4 to 256 generations. Figure 7 (right) shows bin 5 at ~2.5% for all sequential-to-parallel ratios at 128 generations. Figure 9 shows bin 5 lines flat and well below the 14× larger model across all R values. The evidence is consistent and clear: the hardest 20% of MATH problems are outside PaLM 2-S*'s capability range, and test-time compute cannot bridge this gap.

Mitigation status: The paper is candid about this limitation, stating it explicitly in the Section 7 takeaway box:

"Test-time compute is powerful when problems are within the base model's reach (it already produces correct solutions at some non-trivial rate), but it cannot compensate for fundamental capability gaps that larger pretraining would address."

No mitigation is attempted because none is possible within the current framework—this is a fundamental limitation of amplifying an existing proposal distribution rather than improving the model itself. The practical implication is clear: for any deployment where a significant fraction of queries fall into bin 5 equivalence, investing in larger pretraining is likely more cost-effective than investing in compute-optimal inference.

6.4 The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, Limiting Sequential Chain Reliability

The constraint: The revision model was trained exclusively on sequences where all in-context answers are incorrect (followed by a correct target), meaning it never learned what to do when the current answer is already correct. At inference time, this causes a systematic failure: approximately 38% of correct answers get revised back to incorrect ones during sequential revision chains. The paper reports this in Section 6.1:

"The paper reports that approximately 38% of correct answers get converted back to incorrect ones using a naive approach."

The consequence: Sequential revision chains are fundamentally unreliable—each revision step has a probability of destroying a correct answer that was already produced. The paper mitigates this by selecting the best answer from anywhere in the chain (via majority voting or verifier-based selection) rather than always taking the final revision, but this treats the symptom rather than the cause. The mitigation means that longer chains waste compute on revisions that will ultimately be discarded, and there is no mechanism to stop revising when a correct answer is reached. In a latency-sensitive deployment, this means the model burns its entire sequential budget even after producing a correct answer at step 1, wasting the remaining N−1 steps and potentially degrading a correct answer.

What evidence exists in the paper: The 38% reversion rate is reported in Section 6.1, though the exact experiment measuring this rate is not described in detail. Figure 6 (left) shows that pass@1 gradually improves through the chain (from ~18% to ~25% by step 20), but this is the best answer selected post-hoc from the chain, not the reliability of each individual step. The ReST^EM experiment (Appendix K, Figure 16) provides corroborating evidence that sequential revisions are fragile: attempting to optimize the revision model with RL-style training caused performance to degrade substantially—fully sequential performance dropping from ~38.5% to ~33.5% at 256 generations—likely because on-policy data collection amplified spurious correlations in the revision trajectories.

Mitigation status: The paper mitigates this through within-chain selection (majority voting or verifier-based selection) but does not address the root cause—the training data construction that never exposes the model to correct in-context answers. A more principled solution, such as training the model to recognize when no revision is needed (by including correct-to-correct trajectories in training data) or learning an explicit "stop" token, is not explored. The paper flags this as an issue but treats it as an implementation detail rather than a fundamental limitation of the revision training paradigm. For practitioners building revision-based systems, the 38% reversion rate means within-chain selection is mandatory, not optional, and sequential budgets should be kept moderate to limit the number of opportunities for reversion.

6.5 The FLOPs-Matched Comparison Uses a Weak Pretraining Baseline (Parameter-Only Scaling, No Test-Time Compute)

The constraint: The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters that uses greedy decoding with no test-time augmentation of its own and is trained by scaling parameters only while holding data fixed. The paper acknowledges:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence: Both design choices weaken the pretraining baseline:

Parameter-only scaling vs. Chinchilla-optimal scaling: A model trained with 14× more total FLOPs where both parameters and data are scaled equally (following Hoffmann et al., 2022) would likely outperform a parameter-only-scaled model. The reported advantages of test-time compute over pretraining—e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions (Figure 1)—may shrink or reverse against a properly compute-optimal larger model.
No test-time compute for the larger model: The larger model uses only greedy decoding. Giving it even a modest test-time compute budget—say, best-of-8 with a simple verifier—would create a much stronger baseline that is never tested. Since the paper's entire thesis is that test-time compute matters, it is inconsistent to deny the larger model any test-time compute in a comparison meant to determine whether test-time or pretraining compute is more valuable.

What evidence exists in the paper: Figure 9 and Section 7 describe the FLOPs accounting and the three values of R tested (0.16, 0.79, 22), but the baseline choices are only discussed briefly. The paper does not report what a compute-optimally trained larger model would achieve, nor does it vary the larger model's test-time compute budget. The 14× figure comes from scaling parameters only (Section 7 paragraph "Design choice"), and the training data quantity for the larger model is not specified.

Mitigation status: The paper explicitly acknowledges the parameter-only scaling caveat and frames it as representative of the LLaMA training paradigm (Touvron et al., 2023), but does not perform the compute-optimal pretraining comparison it identifies as future work. The absence of any test-time compute for the larger model is not acknowledged as a limitation. For practitioners trying to decide how to allocate their total compute budget (pretraining vs. inference), this comparison provides a lower bound on the pretraining advantage—the true trade-off is less favorable to test-time compute than the paper's numbers suggest. The paper's conclusion that "test-time compute can outperform a ~14× larger model" should be understood as "under specific and favorable comparison conditions," not as a universal result.

6.6 Verifier Over-Optimization Is a Hard Ceiling for Search-Based Methods, and the Paper Does Not Solve It

The assumption or constraint: The effectiveness of PRM-guided search is fundamentally bounded by verifier quality. When search aggressively optimizes against the PRM—using beam search or lookahead search with high budgets—it finds solutions that score highly under the PRM but are actually incorrect. The paper documents this in Section 5.3:

"The degradation at high budgets is attributed to over-optimization of the PRM — search finds solutions that score highly under the PRM but are actually incorrect. Specific failure modes include low-information repetitive steps at the end of solutions (visible in Figure 29) and overly short 1–2 step solutions."

The consequence: The compute-optimal policy mitigates over-optimization by routing easy problems (where the verifier is most exploitable) away from aggressive search—using best-of-N instead of beam search on bins 1–2—but it does not solve the underlying problem. On medium-difficulty problems (bins 3–4) where beam search is deployed, over-optimization still limits the scaling ceiling: the beam search curves in Figure 3 (right) flatten and sometimes decline well before the budget is exhausted. Lookahead search—the most powerful optimizer—paradoxically performs worst overall (Figure 3, left) because its stronger optimization amplifies verifier exploitation. This means the current approach is capped by verifier quality, and no amount of budget allocation strategy can exceed the verifier's reliability frontier.

What evidence exists in the paper: Figure 3 (right) shows beam search accuracy on bin 1 decreasing from ~78% to ~77% as the budget increases from 4 to 256, while best-of-N increases from 68% to 88%—a clear over-optimization signal. Figure 3 (left) shows lookahead search underperforming all other methods at equal generation budget. Appendix M (Figures 29, etc.) shows qualitative examples of degenerate outputs: repetitive low-information steps and overly short 1–2 step solutions that score highly under the PRM. The paper's own compute-optimal policy (Figures 4 and 8) can be understood partly as a way to stay below the over-optimization threshold per difficulty level, but it does not extend the threshold itself.

Mitigation status: The compute-optimal policy dodges the problem by using weaker optimization (best-of-N) where the verifier is reliable and stronger optimization (beam search) only where there is more room before over-optimization sets in. But this is a workaround, not a solution. The paper does not explore verifier improvements—adversarial training, ensemble verification, KL-constrained search that penalizes solutions deviating from the base model distribution—that could raise the over-optimization ceiling. Section 8 acknowledges that "improving verifier robustness is the key bottleneck for further scaling test-time compute," but no experiments address this. For practitioners, this means that even with compute-optimal allocation, search-based methods are fundamentally capped by the verifier quality achievable with the Monte Carlo rollout training procedure described in Appendix D, and the ceiling may be lower than what larger pretraining could achieve on the same problems.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the framing of monocular 3D object detection from a pure model architecture problem to a systems problem involving data, sensors, and interaction paradigms. The dominant narrative in monocular 3D detection has been that the core challenge is designing better architectures to solve the ill-posed inverse problem—recovering metric 3D from a single RGB image through learned priors. Cube R-CNN [6], UniMODE [28], 3D-MOOD [59], and DetAny3D [64] each represent architectural refinements along this axis: better feature extraction, better cross-dataset generalization, better open-vocabulary lifting, better prompt-conditioned regression. WildDet3D's most significant contribution is showing that these architectural advances are necessary but not sufficient—that data scale (13.5K categories vs. 98), sensor flexibility (optional depth at inference), and interaction modality unification (text, point, box within one model) matter at least as much as model design, and likely more.

The evidence for this reframing is the 3.3× gain from adding WildDet3D-Data to Omni3D-only training on WildDet3D-Bench (6.8 → 22.6 AP, Table 3), which is larger than any architectural ablation in Tables 7–8. The largest architectural ablation—removing the joint 2D+3D head—causes a catastrophic −19.1 AP drop, but this is a binary design choice (joint prediction vs. separate prediction), not a continuous refinement. Among the continuous architectural improvements (O2M matching, geometry loss, deep supervision, 3D confidence head), the largest effect is −2.5 AP (O2M removal). The data scaling gain of +15.8 AP dwarfs any single architectural refinement, suggesting that for open-world generalization, the primary bottleneck has been data, not architecture. This is a methodological shift: future work on 3D detection should invest at least as much effort in data pipeline design as in model architecture, treating the two as co-equal contributions rather than treating data as a fixed benchmark to optimize against.

The paper also resolves a latent tension in the field between closed-set detection accuracy and open-world flexibility. Prior to this work, there was an implicit trade-off: you could have accurate 3D detection on a fixed set of categories (Cube R-CNN, UniMODE) or flexible detection on an open vocabulary (3D-MOOD, OVMono3D-LIFT), but not both. The open-vocabulary methods achieved flexibility at a significant accuracy cost—3D-MOOD's 2.3 AP on WildDet3D-Bench (Table 3) is far below what a closed-set detector would achieve if categories were constrained. WildDet3D demonstrates that this trade-off is not fundamental: combining strong pretrained backbones (SAM 3 for semantics, LingBot-Depth for geometry) with large-scale diverse training data achieves open-vocabulary detection that is simultaneously more flexible and more accurate than prior closed-set methods—34.2 AP text on Omni3D exceeds closed-set Cube R-CNN's 23.3 AP, a direct counterexample to the flexibility-vs-accuracy tension. This is an existence proof that should encourage the field to abandon the closed-set/open-vocabulary dichotomy and treat open-world 3D perception as the default target.

The paper's treatment of depth as an optional, variable-quality sensor input rather than a binary condition introduces a new axis of evaluation for 3D perception systems: performance across the spectrum from pure monocular to fully depth-enhanced, including realistic sensor noise. Prior benchmarks implicitly assume a fixed sensor configuration—either you're evaluating a monocular method (no depth at test time) or a LiDAR-based method (full point clouds at test time). WildDet3D's demonstration that the same model achieves 22.6 AP monocular and 41.6 AP with ground-truth depth on WildDet3D-Bench, and 7.5 AP monocular vs. 27.7 AP with real stereo depth on Stereo4D, establishes that sensor-conditioned evaluation—reporting performance as a function of available geometric signal quality—should become standard practice. A model that works well with perfect ground-truth depth but degrades catastrophically with realistic sensor noise is not practically useful, and current benchmarks don't distinguish these cases.

A less obvious but equally important shift is in how 3D annotation is conceptualized. The WildDet3D-Data pipeline reframes annotation from an acquisition problem (humans creating 3D boxes from scratch, requiring skilled annotators and expensive equipment) to a selection problem (generating multiple noisy candidates from diverse automated methods, then selecting the best one). This is the difference between asking a crowdworker to draw a 3D cuboid in a point cloud—a task requiring spatial reasoning skills that most people lack—and asking them to choose which of five pre-generated candidates looks best—a perceptual judgment task that can be crowdsourced at scale on platforms like Prolific. The pipeline validation (Table 2) empirically confirms that this reframing matters: candidate quality varies by >3× across generation methods (rejection rates from 12.5% to 42.9%), and no single method dominates across all objects (SAM-3D tops selection share at 40.4% but still misses 59.6% of cases where other methods are better). The implication for the field is that annotation pipeline architecture—how you combine diverse weak sources into strong supervision—is as important as model architecture, and future work on scaling 3D perception to new domains should invest in pipeline design rather than attempting to build a single perfect automatic annotator.

However, the paper's impact on the pretraining vs. data scale debate is more nuanced than it might first appear. The 3.3× gain from WildDet3D-Data could be interpreted as evidence that data scale is the dominant factor, but the Omni3D-only result on WildDet3D-Bench (6.8 AP for WildDet3D vs. 2.3 AP for 3D-MOOD) shows that even with identical training data, WildDet3D outperforms prior methods by 3.0×. This suggests that strong pretrained initialization (SAM 3, LingBot-Depth) is the actual dominant factor, with data scale providing additional gains on top. The paper does not disentangle these because it never trains baselines on the same pretrained backbones—we don't know whether 3D-MOOD with a SAM 3 backbone would match WildDet3D's Omni3D-only performance. The conservative interpretation is that pretraining quality × data scale × architecture are multiplicative factors, and WildDet3D benefits from all three simultaneously, making it impossible to attribute the total gain to any single factor from the reported experiments.

Follow-Up Research This Work Enables

Training baselines on WildDet3D-Data with comparable pretrained backbones to isolate the architectural contribution. The single most important follow-up experiment is to train 3D-MOOD, OVMono3D-LIFT, and DetAny3D on the same WildDet3D-Data with backbone initialization from SAM 3 and LingBot-Depth, then evaluate on both Omni3D and WildDet3D-Bench. This would answer the question that the current paper leaves open: how much of WildDet3D's state-of-the-art performance comes from its specific architecture (dual-vision encoders, promptable detector, deeply-supervised 3D head, unambiguous rotation normalization) versus from the combination of strong pretraining and large-scale diverse training data? If baselines with equivalent pretraining and data match or exceed WildDet3D, the contribution is primarily a data and pretraining recipe; if they fall substantially short, the architecture is validated as essential. The experiment should also vary the pretraining quality—e.g., replacing SAM 3 with a standard ImageNet-pretrained ViT and LingBot-Depth with a standard monocular depth estimator—to measure the contribution of pretraining quality independently. A negative result (baselines match WildDet3D given equal data and pretraining) would still be valuable: it would reframe WildDet3D as a strong engineering baseline rather than a novel architecture, and redirect research toward better pretraining strategies for 3D perception.

Real sensor depth evaluation at scale across diverse sensor types and noise levels. The paper's depth integration results are compelling but limited: all depth experiments except Stereo4D (383 images) use ground-truth depth, which is noise-free and perfectly calibrated. A systematic evaluation using real sensor data—sparse LiDAR from KITTI or nuScenes, structured light from iPhone LiDAR (ARKitScenes already has this), stereo depth from Middlebury or Sintel, ToF sensors from Azure Kinect—would quantify the depth fusion module's robustness to realistic depth quality. The key question is: at what depth noise level does the benefit of sensor input disappear, and does the model degrade gracefully (performance approaches monocular baseline) or catastrophically (performance drops below monocular because the model over-trusts noisy depth)? The paper's stochastic training strategy (70% monocular, 20% patch-masked depth, 10% full depth) is designed to prevent over-trust, but this should be empirically validated. A strong follow-up would measure AP as a function of depth noise (Gaussian noise standard deviation, fraction of missing pixels, calibration error) across multiple sensor types, producing a "depth quality vs. detection accuracy" curve that practitioners can use to decide whether adding a depth sensor to their deployment is worthwhile.

VLM-filtered vs. human-verified training data ablation to quantify the value of annotation quality at scale. Table 2 shows that VLM scoring correlates perfectly with human rejection rates (Spearman ρ = −1.0) but still misses 9.2% of human rejections at the maximum score (score 11), and 16.7% of score-10 candidates are rejected by humans. The training set contains ~896K VLM-filtered annotations and ~103K human-verified annotations. Training WildDet3D on (a) only human-verified data, (b) only VLM-filtered data, and (c) both, then evaluating on WildDet3D-Bench, would quantify the value of human verification at scale. Does the ~9% error rate in VLM-filtered data meaningfully degrade the model, or does the 8.7× scale advantage (896K vs. 103K images) outweigh the noise? This is a classic scale-vs-quality trade-off for which the paper provides the diagnostic data (Table 2) but not the model-level answer. The experiment should also vary the VLM score threshold—training on only score-11 annotations (~53K images) vs. score-10+ (~310K images) vs. all scores—to find the optimal quality threshold for training data selection. A negative result (noise in VLM-filtered data significantly degrades performance, human verification is essential) would validate the annotation cost; a positive result (VLM-filtered data at scale performs as well or better) would suggest that human verification can be minimized or eliminated for future dataset construction.

Difficulty-conditioned compute-optimal scaling for 3D detection, analogous to the LLM test-time compute analysis. The paper demonstrates that depth quality dramatically affects performance (+20.7 AP average gain across settings), but this gain varies by domain: indoor scenes benefit enormously (+15–20 AP), outdoor driving scenes benefit minimally (±2 AP). This is directly analogous to the difficulty-dependent behavior of test-time compute for LLMs—easy problems benefit from one strategy, hard problems benefit from another. A natural extension is to develop a difficulty estimator for monocular 3D detection that predicts, from the input image alone, whether depth will help for that specific scene. The estimator could use features like scene type (indoor/outdoor), object size distribution, texture richness, and estimated depth variance to assign a "depth-dependence score." At deployment, the system would request depth sensor data only when the predicted benefit exceeds a threshold, saving sensor power and compute when depth is unnecessary. The paper's finding that outdoor driving scenes show minimal depth benefit (KITTI: 37.0 → 36.1 AP with depth, Table 4) provides a clear use case: an autonomous vehicle could disable its power-hungry LiDAR in highway scenarios where monocular depth estimation is already well-calibrated, only activating it for complex urban or indoor-adjacent scenes. The experiment would measure total energy consumption vs. AP on a mixed-domain benchmark, comparing the adaptive strategy to always-on and always-off depth.

Combining WildDet3D with video-based temporal depth fusion for dynamic scenes. The paper's depth fusion module treats each frame independently, but real-world deployment (AR headsets, robots, autonomous vehicles) provides video streams where depth can be temporally fused. Stereo4D [22] provides exactly this—stereo video with real depth maps—but WildDet3D is evaluated on individual frames. Extending the architecture to accept multiple frames with known camera motion would allow temporal depth fusion: depth estimates from past frames can be warped into the current frame using estimated scene flow or camera ego-motion, providing a denser and more accurate depth signal than any single frame's sensor depth. This is particularly relevant for sparse depth sensors like iPhone LiDAR, where temporal accumulation can dramatically increase point density. The experiment would compare (a) single-frame monocular, (b) single-frame with real depth, (c) multi-frame with monocular temporal depth, and (d) multi-frame with real depth temporal fusion, measuring AP on a video 3D detection benchmark like Stereo4D or a custom benchmark built from ARKitScenes video sequences. A strong positive result would position WildDet3D as a core component in video-based 3D perception pipelines for AR and robotics.

Stress-testing rotation estimation on near-symmetric and textureless objects to identify failure modes. The paper acknowledges that "rotation prediction remains the weakest component of our 3D box estimation" (Section 7) and that "objects with near-symmetric geometry (e.g., round tables, square boxes) or limited visible surface area pose particular challenges." The unambiguous rotation normalization (Section 2.3) resolves the 4-fold rotation-dimension ambiguity but does not address the fundamental visual ambiguity of objects whose appearance is invariant under certain rotations. A systematic stress test would construct a benchmark of objects with quantified symmetry—cylinders (continuous rotational symmetry), cubes (90° rotational symmetry), rectangular boxes with aspect ratios near 1 (near-symmetry), and thin elongated objects (limited visible surface for orientation cues)—and measure per-category orientation error (mAOE) as a function of symmetry class. The experiment would reveal whether WildDet3D's rotation errors are concentrated in predictable categories, which would motivate category-specific rotation prediction strategies (e.g., predicting a distribution over orientations for symmetric objects rather than a single rotation). A negative result—rotation errors uniformly distributed across all categories rather than concentrated in symmetric ones—would suggest that the bottleneck is not symmetry but something more fundamental about monocular orientation inference, such as insufficient training data diversity in viewpoint or object pose.

Practical Applications and Downstream Use Cases

On-device 3D perception for mobile AR with adaptive depth sensor activation. The paper demonstrates WildDet3D running on iPhone via a client-server architecture (Section 5, Figure 9a), but the most impactful mobile deployment would be fully on-device inference with adaptive LiDAR activation. Modern iPhone Pro models ship with a LiDAR scanner that consumes significant battery power when active. WildDet3D's optional depth design enables an intelligent power-saving strategy: run the model in monocular mode by default (using only the RGB camera, low power), and activate LiDAR only when the model's internal uncertainty estimates—depth confidence from the auxiliary depth head, 3D confidence scores from the detection head, or scene type classification (indoor vs. outdoor)—indicate that additional geometry would substantially improve accuracy. Based on the paper's results, indoor scenes (ARKitScenes: 64.6 → 73.3 AP with depth, Table 4) would frequently trigger LiDAR activation, while outdoor scenes (KITTI: 37.0 → 36.1 AP) would rarely do so, providing a principled power-accuracy trade-off. An AR application measuring furniture for a room layout could run monocular until the user points at a specific object, then activate depth for precise measurement. The key metric would be accuracy per watt-hour rather than raw AP.

Zero-shot robotic grasping with open-vocabulary object specification. The paper demonstrates WildDet3D on a Franka Emika Panda arm (Section 5, Figure 9c) but treats it as a proof of concept. A practical deployment would integrate WildDet3D into a full robotic pick-and-place pipeline: a user specifies a target object via text ("green chips") or by pointing (box prompt from an AR interface), WildDet3D produces a 3D bounding box in the camera frame, the box center and dimensions are transformed into the robot's coordinate frame using calibrated extrinsic parameters, and a grasp planner generates viable grasp poses from the predicted 3D geometry. The zero-shot nature is crucial—the robot can grasp objects never seen during training, specified by open-vocabulary text queries, without requiring per-object CAD models or task-specific perception training. The paper's WildDet3D-Bench results (Table 3) suggest that rare categories (AP_rare = 28.3 with text prompts, 47.4 with depth) are detected with reasonable accuracy, sufficient for many manipulation tasks where a coarse 3D box is adequate for grasp planning. The key metrics would be grasp success rate as a function of object category frequency (common vs. rare vs. zero-shot) and depth availability (monocular vs. sensor depth), directly measuring whether WildDet3D's 3D accuracy translates to physical task success.

Open-vocabulary 3D scene understanding for vision-language model spatial reasoning. The paper's WildDet3D-agent demonstration (Section 5, Figure 10) shows that pairing a grounded VLM with WildDet3D enables spatial question-answering that 2D grounding alone cannot provide: "what can I reach from here?" or "which object is blocking the door?" require metric depth and 3D position, not just 2D bounding boxes. A practical integration would embed WildDet3D as a 3D grounding tool in a VLM agent framework: the VLM receives a spatial query, generates 2D grounding (box or point) for relevant objects using its built-in visual grounding capability, calls WildDet3D to lift each grounded region to a 3D bounding box, then reasons over the 3D spatial relationships (distances, containment, occlusion ordering) to answer the query. The paper's text-prompt results with depth (41.6 AP on WildDet3D-Bench, Table 3) and box-prompt results with depth (47.2 AP) suggest that the 3D lifting step is sufficiently accurate for coarse spatial reasoning, particularly for common categories (AP_common = 40.7, AP_frequent = 37.2). The key demonstration would be on a benchmark of spatial reasoning questions that require metric depth—e.g., "which object is closest to the camera?", "can the robot reach the mug without moving the laptop?", "is the chair in front of or behind the table?"—where 2D-only VLMs fail and WildDet3D-augmented VLMs succeed.

Large-scale 3D annotation bootstrapping for new domains using the candidate-selection pipeline. The WildDet3D-Data pipeline (Section 3) is domain-agnostic: it requires only 2D annotations, a depth estimator, and a set of candidate generation methods, which could be adapted to new domains. A practical deployment would apply the pipeline to a domain currently lacking 3D annotations—medical imaging (3D bounding boxes for organs in CT scans), agricultural robotics (3D boxes for fruits on trees), marine biology (3D boxes for fish in underwater video), or construction (3D boxes for building components in site imagery). The pipeline would be adapted by (1) collecting or leveraging existing 2D annotations, (2) running domain-appropriate depth estimation (e.g., stereo for underwater, LiDAR for construction), (3) training or adapting candidate generation methods to the new domain's object characteristics, and (4) using the same VLM scoring + human verification selection stage. The key metric would be the cost per verified 3D annotation in the new domain versus traditional manual annotation, and the resulting 3D detection accuracy of a model trained on the bootstrapped data. The paper's finding that VLM scoring correlates perfectly with human judgment (Spearman ρ = -1.0, Table 2) but leaves 9.2% error at the top score suggests that the pipeline would work best in domains where VLM spatial reasoning transfers well (man-made objects with canonical shapes) and would require more human verification in domains with unusual geometries or scales.

When to Prefer This Method

The paper does not explicitly articulate a decision rule for preferring WildDet3D over named alternatives, and the experimental comparisons do not systematically vary deployment conditions (sensor availability, latency constraints, category novelty, annotation budget) to establish clear preference boundaries. The paper positions WildDet3D as a general-purpose 3D perception module that subsumes prior approaches—it handles text, point, and box prompts where prior methods handle only one; it uses depth when available where prior methods are purely monocular; it generalizes to 13.5K categories where prior methods cover at most a few hundred. In this framing, the choice is not "WildDet3D vs. alternative X" but "WildDet3D vs. building separate specialized systems for each prompt modality and sensor configuration," and the paper argues for the unified approach on grounds of engineering simplicity and cross-modal transfer. Without explicit trade-off experiments (e.g., comparing WildDet3D to a DetAny3D + 3D-MOOD ensemble at equal inference compute), the decision rule cannot be derived from the paper's evidence. A forced preference matrix would therefore be speculative rather than supported.