Manipulation as in Simulation: Enabling Accurate Geometry Perception in Robots

ArXiv: 2509.02530

🎯 Pitch

A camera-specific depth denoising model, trained purely on simulated noise patterns and a guided filter correction, lifts consumer depth cameras to near-simulation accuracy. For the first time, a visuomotor policy trained on raw simulated depth transfers directly to a real robot without any real-world fine-tuning, matching or exceeding simulation success rates on long-horizon tasks involving glass, reflective metal, and slender objects.

---

1. Executive Summary

This paper proposes Camera Depth Models (CDMs), a plug-in solution that enhances the geometric accuracy of consumer depth cameras by taking raw RGB images and noisy depth signals as input and outputting denoised, accurate metric depth. A neural data engine is developed to train CDMs by modeling each depth camera’s specific noise pattern—both hole noise (missing data, e.g., from glass or reflective surfaces) and value noise (biases, blur, and scale distortions)—and synthesizing high-quality paired data from simulation, with a novel guided filter approach to correct the scale mismatch introduced by the learned noise. CDMs achieve nearly simulation-level 3D accuracy, and the experiments demonstrate, for the first time, that a depth-only visuomotor policy trained on raw simulated depth—without any added noise or real-world fine-tuning—generalizes seamlessly to real robots on two challenging long-horizon manipulation tasks (a kitchen task involving articulated and glass objects, and a canteen task requiring grasping reflective and slender objects such as a metal fork), achieving success rates that match or exceed simulation performance (e.g., 26/30 overall for the kitchen task with CDM-D435 versus 30/50 in simulation), establishing that the sim-to-real geometry gap can be bridged effectively only when the depth perception itself is brought to near-simulation fidelity.

2. Context and Motivation

The Core Problem: Depth Cameras Produce Unreliable Geometry, and Robots Cannot Afford That

The fundamental problem this paper tackles is deceptively simple: consumer depth cameras are not accurate enough for robotic manipulation. While this might sound like an engineering footnote rather than a research problem, the implications are profound. Modern manipulation policies—whether learned via imitation or reinforcement learning—rely on visual observations to perceive object locations, shapes, and spatial relationships. When those observations are geometrically corrupted, the policy receives systematically wrong information about where objects are and what they look like in 3D space. The consequences range from degraded performance to complete task failure, particularly for tasks requiring precise interactions with small, reflective, transparent, or articulated objects.

The authors articulate this in their introduction with a direct observation about the state of the field:

"Although recent studies have integrated 3D representations into robotic manipulation, performance remains limited by the poor quality of depth data produced by such devices. Consequently, evaluations are typically restricted to simulation environments, where clean and accurate depth is available; or rely on downsampled point clouds to mitigate noise in real-world scenarios."

This is an important research bottleneck. The field wants to use 3D information—point clouds, depth maps, neural radiance fields—because geometry is texture-agnostic (a bowl is still a bowl regardless of its color or pattern, and depth-only policies naturally generalize across visual appearances). But the 3D data available from real sensors is so noisy that researchers are forced into two unsatisfying compromises: either stay in simulation (where depth is perfect but the real world is absent) or aggressively preprocess the data (downsampling point clouds, clipping depth ranges, hole-filling, temporal filtering) which throws away the geometric fidelity that motivated using 3D in the first place.

The paper's Figure 1 makes this concrete. Looking at the raw depth from a RealSense D435 camera—one of the most widely used depth sensors in robotics research—we see that the microwave's glass door is completely missing (rendered as holes), the metal fork is essentially invisible, and surface depths are speckled with noise. This is not a minor imperfection; it means that a robot's geometric understanding of the scene is qualitatively wrong about critical task-relevant objects.

Why This Problem Matters: Simulation Is the Bottleneck, Not the Solution

The significance of this problem extends well beyond depth estimation accuracy. It sits at the intersection of three critical challenges in modern robotics.

First, the sim-to-real gap in manipulation remains largely unsolved for geometry. Sim-to-real transfer is arguably the central challenge in scaling robot learning. Simulation offers unlimited data, perfect ground truth, and safe exploration—but policies trained in simulation often fail catastrophically when deployed on real hardware. The research community has made substantial progress on two fronts: domain randomization for physics (randomizing friction, mass, damping to cover real-world variability) and photorealism for RGB observations (neural rendering, Gaussian splatting reconstruction of real scenes). However, as the authors note in Section 2.3:

"Simulating a real-world camera or adding noise in the simulation is a last resort, as it may deteriorate the rich geometry information and precise manipulation."

The existing approach for depth-based sim-to-real is to add synthetic noise to simulated depth images—Gaussian noise, dropout, random perturbations—to make them "look like" real sensor data. This creates a fundamental tension: you degrade the geometric information during training to make the policy robust to degraded information at test time, but in doing so, you prevent the policy from ever learning the precise manipulation skills that accurate geometry would enable. The authors identify this as a "last resort" for good reason: it is training the policy to be forgiving of bad data rather than providing it good data to begin with. The paper's proposed alternative—making real-world depth accurate rather than making simulated depth noisy—changes the entire premise of the sim-to-real transfer problem.

Second, the inability to use simulation data effectively constrains scalability. The most powerful recent results in robot learning—generalist policies that work across many tasks, embodiments, and environments—require enormous amounts of diverse training data. Real-world data collection at this scale is prohibitively expensive (the paper's imitation learning experiments use 50 real-world demonstrations; the sim-to-real experiments use ~680 and ~800 simulated demonstrations, two orders of magnitude more easily obtained). If simulation data cannot be made usable for real-world deployment due to geometry mismatches, then the scaling benefits of simulation are fundamentally limited for precision manipulation tasks. By bridging the sim-to-real geometry gap—making real depth images look like simulated ones—CDMs unlock the possibility of training policies entirely in simulation using unlimited high-quality data and deploying them without modification, creating a pathway to data scaling that does not require proportional real-world data collection.

Third, 3D understanding is necessary for generalization but unreliable in practice. The paper builds on a well-established intuition in the robotics community: geometry-based policies generalize better than appearance-based ones. A policy trained on depth images of bowls will work with bowls of any color, pattern, or material, because depth is insensitive to texture (Table 2 and Figure 5 explicitly demonstrate this—the depth-only policy generalizes to unseen bowl sizes when trained with CDM-processed depth, but fails without CDM). However, this promise has been underdelivered because the actual depth data available from real sensors is so degraded that any generalization benefit is overwhelmed by geometric inaccuracies. The authors frame this explicitly: their goal is "not to identify if depth is a better visual modality than color, but to validate whether accurate geometry information contained in a more precise depth image can benefit manipulation" (Section 5). By fixing the quality of the depth signal, they aim to unlock the generalization benefits that geometry promises.

Prior Approaches and Where They Fall Short

The paper identifies three categories of prior work relevant to its contribution, each with specific limitations that CDMs address.

Metric Depth Estimation: Scale Ambiguity Remains Unresolved

The computer vision community has made remarkable progress on monocular depth estimation. The Depth Anything series (Yang et al., 2024a,b), cited extensively in the paper, can predict high-quality relative depth from single RGB images, learning from massive datasets of unlabeled images. However, robots need metric depth—absolute distances in physical units—not just which object is closer than another. The paper identifies the core challenge with converting relative to metric depth:

"Simply fine-tuning DA models to predict metric depth remains constrained by a fixed depth scale and is susceptible to scale ambiguities."

Scale ambiguity refers to the inherent impossibility of determining absolute size and distance from a single monocular image without additional information. A bowl close to the camera and a larger bowl far from the camera can produce identical images. Prior approaches have attempted to resolve this by incorporating scale cues. Guizilini et al. (2023) and Piccinelli et al. (2024) introduce camera intrinsics as explicit scale information (focal length tells the model how image geometry relates to real-world geometry). MoGe (Wang et al., 2024b, 2025a,b) uses affine-invariant techniques—training models to predict depth up to an unknown scale and shift, then resolving these using a sparse set of known-depth points.

The more directly relevant prior work is PromptDA (Lin et al., 2025) and PriorDA (Wang et al., 2025c), which take a different approach: rather than trying to resolve scale from intrinsics or sparse points, they accept a low-quality depth image as a "prompt" that provides coarse metric information, and refine it using semantic information from the RGB image. This is in the same spirit as CDMs—camera depth as a scale cue, RGB as a semantic refiner.

However, these methods have critical limitations that CDMs directly address:

• Hole-filling dependency: Both PromptDA and PriorDA require that the low-quality depth image be pre-processed to fill holes (missing values) before being fed into the model. This preprocessing is computationally expensive (adding latency to real-time robot inference), lossy (hole-filling inpainting can introduce errors that propagate through the model), and architecturally limiting (the model never learns to handle missing data natively). The paper's quantitative results in Table 1 demonstrate this starkly: PromptDA's L1 error quadruples (0.1703 vs. 0.0396) when holes are not pre-filled on the D435 test split.

• Handcrafted training data: Both methods train their models by synthesizing prompt depth images—adding handcrafted noise patterns to clean ground-truth depth. The paper argues this cannot capture the complex, camera-specific noise patterns of real sensors:

"other factors are hard to be accurately modeled in the simulation, for instance, the camera parameters, the implementation and optimization details in each depth camera hardware and software are case by case, resulting distinct noise behaviors"

A RealSense D435 (active IR stereo) has fundamentally different failure modes than a RealSense L515 (LiDAR-based time-of-flight) than a ZED 2i (passive stereo). Handcrafted noise rules (e.g., adding Gaussian noise + random dropout) cannot capture that the D435 fails on glass because stereo matching has no texture to correlate, while the L515 fails on reflective metal because LiDAR pulses bounce away from the sensor. The paper's approach of learning noise models from real data per camera directly addresses this limitation.

• Shallow fusion: Both PromptDA and PriorDA fuse the depth prompt relatively late in the decoding phase. The paper argues (Section 3.2) that this limits the model's ability to deeply integrate scale information with semantic understanding, particularly when the depth prompt has large missing regions where global context is needed.

Manipulation with 3D Representations: Aggressive Preprocessing Masks the Problem

A parallel line of work attempts to use 3D data directly in robot policies. Ze et al. (2024a,b) developed 3D Diffusion Policy, using point clouds as the visual input to diffusion-based visuomotor policies. Hua et al. (2024) used point cloud representations in a multi-task setting. Liu et al. (2024) used depth-only policies for quadrupedal mobile manipulation.

The critical observation the paper makes about all this work is:

"these methods still require point cloud downsampling, calibration, and table-top cropping to mitigate noise from depth cameras"

In other words, the pipeline includes aggressive noise mitigation steps, but the problem of depth inaccuracy is treated as a preprocessing issue rather than a perception problem to be solved. The policy receives a degraded and filtered geometric representation—better than raw sensor data, but still a lossy version of the true scene geometry. The paper implicitly argues that this approach is fundamentally limited: you cannot recover geometric precision that was destroyed at the sensor level.

The work closest in spirit to CDMs in the manipulation literature is Liu et al. (2024), which also uses depth-only policies for sim-to-real transfer on loco-manipulation tasks. However, their approach requires object segmentation (a separate perception module that isolates task-relevant objects) to further narrow the sim-to-real gap, adding complexity and potential failure modes. CDMs sidestep the need for segmentation by making the whole depth image accurate, not just selected object regions.

Visual Sim-to-Real: "Add Noise" as the Default (and Flawed) Strategy

The dominant approach for sim-to-real transfer with depth, as documented in the paper's Section 2.3, is to train policies on simulated depth with added noise, and then apply post-processing (clipping, hole-filling, temporal filtering) to real-world depth at inference time. This is what works like Cheng et al. (2023), Zhuang et al. (2023, 2024), and He et al. (2024) do for legged locomotion, and what Liu et al. (2024) does for manipulation.

The authors identify this as a fundamentally unsatisfying compromise:

"Simulating a real-world camera or adding noise in the simulation is a last resort, as it may deteriorate the rich geometry information and precise manipulation."

The logic is clear: if you add noise during training, the policy learns to be robust to noisy depth—but it never learns what accurate depth looks like. It cannot exploit the fine geometric details that distinguish, for example, a fork resting on a plate from a fork embedded in the plate. By contrast, CDMs act as a preprocessing step that makes real-world depth look like simulation depth, enabling the policy to be trained on clean depth and receive clean depth at inference time. This flips the sim-to-real strategy from "make simulation worse" to "make reality better."

There is one prior exception the paper acknowledges: Tao et al. (2025) introduced a computation-intensive method to simulate depth with typical noise patterns rendered by a real-world stereo camera. However, this approach is specific to stereo cameras, requires modeling the camera's stereo matching algorithm, and is computationally expensive—not a general solution for the diverse camera types commonly used in robotics.

How This Paper Positions Itself

The paper positions CDMs not as a new depth estimation algorithm in the computer vision sense, but as a perception plugin that bridges the geometry gap between simulation and reality for robotic manipulation. This framing is critical for understanding the paper's contribution. The authors are not claiming to advance the state of the art in depth estimation on standard benchmarks (though CDMs do outperform baselines on the Hammer dataset, per Table 1). They are claiming to solve a specific, high-impact integration problem: how to make the depth camera output good enough that a policy trained on perfect simulated depth works on real hardware without modification.

The key conceptual move is to treat depth camera noise not as random corruption to be averaged away (through temporal filtering) or ignored (through downsampling), but as a camera-specific, learnable pattern that can be modeled and inverted. This is what Section 3.1 frames as the tension between relying on sensor data and being skeptical of its inaccuracies:

"Balancing reliance on sensor data with skepticism of its inaccuracies poses a significant challenge, making a generalizable solution nontrivial. This motivates the development of camera-specific depth models (CDMs) tailored to individual depth cameras."

By learning separate noise models and separate CDMs for each camera type, the approach acknowledges that a RealSense D435 and a RealSense L515 have fundamentally different failure modes requiring different correction strategies. A single general-purpose depth refinement model would need to learn all these patterns implicitly; CDMs make the camera specificity explicit in the architecture and training pipeline.

The paper also positions itself relative to the broader trajectory of robot learning. The end of Section 6 explicitly frames CDMs as infrastructure for foundation models:

"Moreover, by achieving simulation-level 3D perception in the real world and aligning sim-real geometry gaps, CDMs enable seamless integration of simulation and real-world 3D data. This approach could lead to more efficient data utilization strategies, fostering the development of large-scale robotic foundation models with human-level generalization ability."

This is an ambitious claim. The scaling hypothesis in robot learning is that more data + larger models = generalization. But this hypothesis breaks down if simulation data (the primary scalable source) cannot be made compatible with real-world deployment. CDMs are proposed as a compatibility layer that makes simulation-trained models directly deployable, potentially enabling the same scaling dynamics that have transformed computer vision and natural language processing to be applied to robotic manipulation with geometric perception.

Finally, it is worth noting what the paper explicitly does not claim. It does not claim that depth is better than RGB for manipulation in general (Section 5 disclaims this). It does not claim that CDMs are a general depth estimation solution for all cameras (the models are camera-specific and require per-camera noise model training). And it does not claim that the current difficulty estimation and noise modeling pipeline is optimal or complete (the failure case in Figure 18 shows that when the raw depth is completely wrong over large regions and RGB provides insufficient semantic information, CDMs can be misled). The contribution is focused and specific: for the cameras commonly used in robotics, CDMs can make real-world depth accurate enough to enable sim-to-real transfer of depth-only policies, which has not been demonstrated before.

3. Technical Approach

3.1 Reader Orientation

The system being built is a camera-specific neural network plugin that sits between a consumer depth camera and a robot's visuomotor policy, transforming the camera's noisy, incomplete depth output into clean, metric-accurate depth that looks like it came from a simulation renderer. The core problem it solves is this: robot policies trained on perfect simulated depth fail in the real world because real depth cameras produce geometrically corrupted data, but training policies on noisy simulated depth prevents them from ever learning precise manipulation skills. The "shape" of the solution is to learn the specific noise signature of each depth camera from real data, synthesize that noise onto clean simulation depth to create paired training data, and then train a model that inverts this noise—producing simulation-quality depth from real camera input at inference time.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, connected in a pipeline that flows from real-world data collection through model training to real-time inference:

1. ByteCameraDepth Dataset (data collection): A multi-camera mount captures simultaneous RGB and raw depth from seven cameras across ten depth modes in seven indoor scenes, producing over 170,000 paired examples. This dataset contains only noisy real data—no ground-truth depth—and serves as the raw material for learning noise patterns.

2. Noise Models (data synthesis): Two neural networks are trained per camera on the ByteCameraDepth data: a hole noise model that predicts where depth values will be missing (zero-valued pixels), and a value noise model that predicts the distorted depth values at non-hole locations, including biases, blur, and scale errors specific to that camera's sensing principle. These models learn to produce realistic noise when given only an RGB image as input.

3. Synthesized Training Data: The noise models are applied to clean simulated RGB-D datasets (HyperSim, DREDS, HISS, IRS, totaling 280,000+ images). For each simulated scene, the RGB image is fed through both noise models, producing a synthetic "raw camera depth" image. A guided filter corrects the scale mismatch between the value noise output and ground-truth depth, and handcrafted high-frequency noise is added on top. The result is a triplet of (RGB, noisy depth, ground-truth depth) for training.

4. Camera Depth Model (CDM): A dual-branch ViT architecture that takes an RGB image and a raw (unfilled, hole-containing) depth image as input and predicts clean metric depth. One ViT encoder processes RGB semantics; a separate ViT encoder processes the depth prompt; their token-level features are fused via per-location multi-head attention; the fused features are concatenated with RGB features and decoded through a DPT head to produce the output depth map. The model is initialized from DINOv2 weights and DPT architecture, trained with L1 + gradient loss on the synthesized data.

5. Inference-Time Deployment: At runtime on the real robot, the CDM is inserted as a plugin between the depth camera and the policy. For each frame, the camera's RGB image and raw depth image (with holes, noise, and biases intact—no hole-filling) are fed to the CDM, which outputs a clean depth image in under 151ms on a single 4090 GPU. This clean depth is then passed directly to the visuomotor policy for action prediction.

Information flows as follows: real camera data $\rightarrow$ ByteCameraDepth dataset $\rightarrow$ trained hole/value noise models $\rightarrow$ synthesized noisy training data from simulation $\rightarrow$ CDM training $\rightarrow$ real-time CDM inference on the robot $\rightarrow$ clean depth $\rightarrow$ policy action.

3.3 Roadmap for the Deep Dive

• First, the noise taxonomy and the fundamental tension (Section 3.1 in the paper): what types of noise depth cameras produce, why they are camera-specific, and why naive denoising is insufficient. This establishes what CDMs must overcome and motivates the camera-specific design.

• Second, the CDM architecture (Section 3.2): the dual-branch ViT design, the token-level fusion mechanism, and why this structure avoids the hole-filling dependency and shallow fusion limitations of prior work. Understanding this architecture is prerequisite to understanding how the model integrates semantics with scale.

• Third, the ByteCameraDepth dataset (Section 3.3): the multi-camera mount, the seven cameras and ten modes, the seven indoor scenes, and what the dataset contains and does not contain (noisy real data only, no ground truth). This is the raw material from which noise models learn.

• Fourth, the noise models and data synthesis pipeline (Section 3.4): the hole noise model as binary classification, the value noise model as stylized depth prediction with a Depth Anything V2 backbone, the synthesis equation that combines them, the guided filter for scale correction, and the high-frequency augmentation. This is the engine that produces the training data for CDMs.

• Fifth, CDM training (Section 3.5): the loss function, initialization strategy, training datasets, and the rationale for using disparity as the training target.

This ordering mirrors the data flow: we start with the problem (noise), then the solution architecture (CDM design), then how we get training data (dataset collection $\rightarrow$ noise model learning $\rightarrow$ synthesis), and finally how the model is trained. The architecture is introduced before the training pipeline because understanding the model's input requirements clarifies why the noise synthesis must produce unfilled, hole-containing depth.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and perception paper whose core idea is that the geometry gap between simulation and reality can be closed by treating each depth camera's noise as a learnable, invertible pattern rather than as random corruption, and that doing so at the sensor output level fundamentally changes the sim-to-real transfer problem from "make simulation worse" to "make reality better."

---

The Noise Taxonomy: Why Camera-Specific Modeling Is Necessary

Before building a solution, the paper precisely characterizes the problem. Section 3.1 defines two categories of noise that every consumer depth camera exhibits:

Hole noise manifests as missing data—pixels where the depth camera returns zero or no value. The paper identifies distinct physical causes for holes depending on the camera's sensing principle. Stereo matching cameras (like the RealSense D435, which uses active IR stereo) produce holes at object boundaries where occlusion prevents one camera from seeing what the other sees, and on textureless surfaces where the matching algorithm finds no features to correlate. LiDAR-based time-of-flight cameras (like the RealSense L515, which uses direct time-of-flight) produce holes on black or highly reflective surfaces: black surfaces absorb the emitted laser pulse so no return signal is detected; specular surfaces reflect the pulse away from the sensor. Both camera types fail on transparent or mirror-like objects such as glass, because the physical assumptions underlying depth reconstruction (light reflects diffusely from opaque surfaces) are violated. The paper states explicitly:

"Both types perform poorly on transparent or mirror-like objects, such as glass."

Value noise encompasses all inaccuracies in the depth readings that are actually returned—pixels where the camera provides a number, but that number is wrong. This includes systematic biases (a camera might consistently overestimate or underestimate depth at certain ranges), blur (spatial smoothing of depth discontinuities), jitter (temporal and spatial random fluctuations), and scale distortions (errors in the absolute metric scale). The paper emphasizes that these noise patterns are not universal:

"These noise patterns depend on the camera's intrinsic parameters and physical installation."

This dependence on intrinsics and installation is the key argument for camera-specific modeling. A RealSense D435 (active IR stereo, ~87mm baseline) and a ZED 2i (passive RGB stereo, 120mm baseline) are both stereo cameras, but their baseline distances, IR projector patterns, matching algorithms, and post-processing pipelines are entirely different, leading to different hole distributions, different depth ranges, different edge behavior, and different bias profiles. A single general-purpose depth enhancement model would need to implicitly learn to recognize which camera produced the input and apply the appropriate correction—a harder learning problem than training separate models with camera identity as explicit prior knowledge.

The paper also identifies a fundamental tension that any effective depth enhancement model must navigate: the model must simultaneously trust the camera depth enough to use it as a metric scale reference, and distrust it enough to override specific erroneous readings using semantic information from the RGB image. The raw depth provides scale—it tells the model that an object is, say, 0.8 meters away rather than 2 meters—but the value at any given pixel may be wrong. The RGB image provides semantic grouping—it tells the model that these pixels belong to a continuous surface that should have smooth depth, or that this region is glass and the reported depth is likely spurious. The architecture must fuse these two information sources without letting errors in either corrupt the final output.

---

The Camera Depth Model (CDM) Architecture

The CDM is designed to take two inputs and produce one output:

• Input 1: An RGB image $\mathbf{I} \in \mathbb{R}^{3 \times H \times W}$, with three color channels.
• Input 2: A raw depth image $\mathbf{D} \in \mathbb{R}^{H \times W}$ from the depth camera, containing holes (zero-valued pixels), value noise, biases, and all other imperfections—with no preprocessing such as hole-filling, clipping, or temporal filtering.
• Output: A predicted high-quality metric depth image $\hat{\mathbf{D}} \in \mathbb{R}^{H \times W}$.

The architecture, illustrated in Figure 2 (right), has four main stages: dual-branch encoding, token-level fusion, feature concatenation, and dense prediction decoding.

Dual-Branch ViT Encoding

The model uses two separate Vision Transformer (ViT) encoders, both initialized from DINOv2 weights (Oquab et al., 2023). The separation is deliberate—each branch processes fundamentally different information:

$\mathbf{X}^{\mathbf{I}} = \text{ViT}^{\mathbf{I}}(\mathbf{I}),\quad \mathbf{X}^{\mathbf{D}} = \text{ViT}^{\mathbf{D}}(\mathbf{D})$

where $\mathbf{X}^{\mathbf{I}} = \{\mathbf{X}^{\mathbf{I}}_1, \ldots, \mathbf{X}^{\mathbf{I}}_N\}$ is the set of $N$ feature tokens produced by the RGB branch, and $\mathbf{X}^{\mathbf{D}} = \{\mathbf{X}^{\mathbf{D}}_1, \ldots, \mathbf{X}^{\mathbf{D}}_N\}$ is the corresponding set from the depth branch. Each token $\mathbf{X}^{\mathbf{I}}_i$ or $\mathbf{X}^{\mathbf{D}}_i$ corresponds to a specific spatial location (patch) in the input image.

What this computes: the RGB branch maps color and texture patterns to semantic feature representations—it learns that these particular visual patterns correspond to "glass microwave door," "metal fork," "wooden table," etc. The depth branch maps the raw depth values (including zeros at holes) to geometric feature representations—it learns that these particular depth patterns (with their characteristic noise) correspond to certain 3D structures. The two branches operate independently at this stage, each extracting information from its own modality.

Why this form: the separation prevents the presence of depth holes from corrupting the semantic feature extraction. If RGB and depth were concatenated and fed through a single encoder (as in earlier approaches), holes in the depth channel would create artificial discontinuities in the input that the model would need to learn to ignore—a wasted modeling capacity. Additionally, since DINOv2 was pretrained on RGB images, initializing the RGB branch with DINOv2 weights gives it strong semantic representations from the start; the depth branch, processing single-channel images, starts from the same DINOv2 initialization but adapts to the depth modality during fine-tuning. For the single-channel depth images, the paper notes that "by default, they are copied three times before being fed into the network" to match the expected 3-channel input of the ViT encoder.

Token-Level Feature Fusion

After encoding, the RGB and depth tokens are fused. The fusion is not a simple concatenation or element-wise addition; it is a structured attention mechanism that operates on corresponding spatial locations:

$\tilde{X} = \sum_i \text{MHA}(\{[\mathbf{X}^{\mathbf{I}}_i; \mathbf{X}^{\mathbf{D}}_i]\}_{i=1}^N)$

where MHA stands for multi-head attention, and $[\cdot; \cdot]$ is concatenation along the feature dimension. For each spatial position $i$, the RGB token $\mathbf{X}^{\mathbf{I}}_i$ and depth token $\mathbf{X}^{\mathbf{D}}_i$ are concatenated to form a combined token. Multi-head attention is then applied across all $N$ combined tokens, and the results are summed to produce the fused feature $\tilde{X}$.

What this computes: the attention mechanism allows each spatial location to attend to information from all other spatial locations, using both semantic and geometric features simultaneously. A token representing a pixel on the glass microwave door (where the depth branch sees a hole) can attend to tokens representing the microwave frame (where the depth branch sees valid depth), using the RGB branch's semantic grouping to infer that the glass and frame belong to the same physical object and should have related depths. The summation across attention heads aggregates multiple relationship patterns (e.g., one head might learn surface continuity, another might learn occlusion boundaries).

Why this form: concatenation followed by attention is more expressive than late fusion alternatives. The paper explicitly contrasts this with PromptDA (Lin et al., 2025) and PriorDA (Wang et al., 2025c), which "fuse the prompted depth information simply in the shallow decoding phase." Fusing only in the decoder means that the encoder's feature extraction operates entirely independently on RGB and depth, and the integration of scale information with semantic understanding happens only at the output level—too late for the model to, for example, use depth scale to resolve semantic ambiguities (is this a small nearby object or a large distant one?). The multi-level, token-level fusion in CDMs allows the integration to happen at multiple scales, which the paper argues is crucial "when the camera depth prompt has large missing regions, where global-scale information is particularly needed."

Feature Concatenation and Dense Prediction

The fused features do not replace the original RGB features; they augment them:

$\hat{\mathbf{D}} = \text{DPT}([\mathbf{X}^{\mathbf{I}}; \tilde{X}])$

The DPT (Dense Prediction Transformer) head, following Ranftl et al. (2021), takes the concatenation of the original RGB tokens and the fused tokens and produces the final metric depth prediction $\hat{\mathbf{D}}$.

What this computes: the DPT head is a convolutional decoder that upsamples the token-level features back to the original image resolution through a series of fusion blocks that combine features from different transformer layers with progressively higher-resolution feature maps. The skip connection from the original RGB tokens $\mathbf{X}^{\mathbf{I}}$ ensures that semantic information is not lost in the fusion process—even if the depth prompt contains no useful information for a particular region, the RGB branch's understanding of what object occupies that region is preserved and can guide depth prediction.

Why this form: the concatenation $[\mathbf{X}^{\mathbf{I}}; \tilde{X}]$ rather than using $\tilde{X}$ alone prevents information loss. The paper states this explicitly: "Additionally, we concatenate the original RGB feature tokens into the fused feature tokens. These fused feature tokens, along with the RGB feature tokens, are concatenated to prevent loss of semantic information." This is a form of residual connection at the feature level—the RGB information flows directly to the decoder regardless of what happens in fusion, so the model can always fall back on monocular depth cues when the camera depth is unreliable.

An important practical design choice: the CDM takes the raw depth image directly, without hole-filling. This simplifies the inference pipeline (no need to run a separate inpainting step) and allows the model to learn its own strategy for handling missing data. The paper's quantitative results in Table 1 show that competing methods (PromptDA, PriorDA) fail catastrophically without hole-filling (PromptDA L1 error increases from 0.0396 to 0.1703 on D435 data when holes are present), while CDMs handle holes natively.

---

The ByteCameraDepth Dataset

To train the noise models, the paper needs real-world data showing what noise looks like for each specific camera. The ByteCameraDepth dataset is the solution—a large-scale multi-camera capture that records RGB and raw depth simultaneously from seven cameras.

Hardware setup. The authors designed a multi-camera mount (Figure 3) that rigidly holds all seven cameras, allowing them to capture the same scene from approximately the same viewpoint simultaneously. The seven cameras span the most common depth sensing technologies used in robotics:

• RealSense D405 (stereo depth, short range)
• RealSense D415 (active IR stereo)
• RealSense D435 (active IR stereo with wider FOV)
• RealSense D455 (active IR stereo with extended range)
• RealSense L515 (LiDAR direct time-of-flight, D-Tof)
• ZED 2i (passive RGB stereo)
• Azure Kinect (time-of-flight)

For the ZED 2i specifically, the paper records raw data and replays it offline to generate depth in four different modes: performance, ultra, quality, and neural. These modes represent different tradeoffs between depth accuracy, resolution, and computational cost, each with its own noise characteristics. This brings the total to 10 distinct depth modes across the seven physical cameras.

The authors note a practical constraint: "we use two computers to capture all the data due to the USB bus bandwidth limits." Seven depth cameras, several of which stream high-resolution RGB and depth at high frame rates, saturate a single USB controller's bandwidth. This is a real-world engineering detail that matters for reproducibility.

Data scale and diversity. The dataset contains over 17,000 images for each camera mode, sampled from video recordings at 5 Hz. The scenes cover seven indoor environment types: kitchens, living rooms, markets, bedrooms, bathrooms, offices, and breakrooms. Figure 4 shows examples of the raw depth data, revealing the characteristic noise patterns of each camera: D405 and D415 show granular noise and structured artifacts; D435 shows smoother but biased depth with edge holes; L515 shows systematic failures on dark and reflective surfaces; Kinect shows blocky artifacts from its lower resolution; ZED 2i modes show varying degrees of speckle noise depending on the quality setting.

What the dataset is and is not. Critically, ByteCameraDepth contains only noisy real data—RGB images and raw depth from real sensors. There is no ground-truth depth. The dataset's purpose is not to directly train CDMs (which require clean ground truth), but to train noise models that can then be applied to simulation data that does have ground truth. This is a clever separation of concerns: collect real noise patterns without needing expensive ground-truth depth capture (which would require laser scanners or structured light systems far more accurate than the cameras being studied), then synthesize the learned noise onto clean simulated data to create training pairs.

---

Noise Models and Data Synthesis

This is the most technically involved part of the pipeline. The goal is to take a clean simulated RGB-D image and generate a synthetic "raw camera depth" image that is statistically indistinguishable from what the real camera would produce for that scene. The approach trains two separate neural networks per camera—one for holes, one for values—and then combines their outputs with a guided filter and handcrafted high-frequency augmentation.

Hole Noise Model

The hole noise model $N_{\text{hole}}$ is trained as a pixel-wise binary classifier. Given only an RGB image $\mathbf{I}$ as input, it predicts for each pixel whether the corresponding depth value would be a hole (zero) if captured by the target camera.

The training objective is binary cross-entropy:

$\ell(N_{\text{hole}}(\mathbf{I})) = \sum_{i=0,j=0}^{H,W} \left[y_{i,j} \log \sigma(x_{i,j}) + (1 - y_{i,j}) \log(1 - \sigma(x_{i,j}))\right]$

where $x_{i,j} = N_{\text{hole}}(\mathbf{I})_{i,j}$ is the model's raw output (logit) at pixel $(i,j)$, $y_{i,j} = \mathbb{I}(\mathbf{D}_{i,j} = 0)$ is the binary label indicating whether the real camera's depth is zero at that pixel, and $\sigma$ is the sigmoid function that maps logits to probabilities in $[0, 1]$.

What this computes: for each pixel in the input RGB image, the model outputs a probability that a hole would appear at that location in the target camera's depth output. The loss compares this predicted probability to the actual presence or absence of a hole in the real camera data. The model is encouraged to output high probability for pixels that are holes in the real data (glass, reflective surfaces, depth boundaries) and low probability otherwise.

Why this form: the key insight is that hole locations are not random—they are predictable from RGB alone because they are caused by material properties (transparency, specularity, darkness) and geometric configurations (occlusion boundaries, surface normals facing away from the emitter) that are visible in the color image. A pretrained DINOv2 backbone with a DPT head provides strong semantic features that can recognize glass, metal, dark surfaces, and depth edges. Binary cross-entropy is the standard objective for probabilistic binary classification and produces well-calibrated probability estimates (when combined with appropriate regularization), which matters because these probabilities are used for thresholding during synthesis (Equation 6).

Value Noise Model

The value noise model $N_{\text{value}}$ predicts the actual (noisy) depth values at non-hole locations. Rather than predicting absolute metric depth directly—which would require learning to map from RGB to physical distance, a hard monocular depth estimation problem—the model is framed as a stylized relative depth prediction problem, leveraging the Depth Anything V2 (DAV2) architecture.

The training procedure is:

1. Take the real camera's depth image $\mathbf{D}$ (with holes) as the target.
2. First, fine-tune a DAV2 model to predict this noisy depth from RGB.
3. Use the fine-tuned DAV2 model as the value noise model $N_{\text{value}}$.

The training objective uses an affine-invariant transformation to normalize the ground-truth depth before computing the loss, which handles the scale ambiguity between the clean ground-truth depth and the camera's biased output:

$\ell(N_{\text{value}}) = L_1(f(\overline{\mathbf{D}}), \hat{\mathbf{D}}_{\text{value}})$

where $\overline{\mathbf{D}}$ is the ground-truth depth, $f$ is the affine-invariant normalization function (from Wang et al., 2025a), $\hat{\mathbf{D}}_{\text{value}} = N_{\text{value}}(\mathbf{I})$ is the model's prediction, and $L_1$ is mean absolute error.

What this computes: the affine-invariant normalization $f$ transforms the ground-truth depth by subtracting a global shift and dividing by a global scale, so that the absolute metric values are discarded and only the relative depth ordering is preserved. The model then learns to predict the noise-corrupted, camera-specific depth appearance—the characteristic biases, blur patterns, edge artifacts, and systematic errors—without needing to reproduce the exact metric scale. The $L_1$ loss (mean absolute error) is more robust to outliers in depth prediction than $L_2$ (mean squared error) and is standard in depth estimation.

Why this form: training DAV2 as the backbone of the value noise model is a strategic choice. DAV2 was trained on enormous amounts of unlabeled data to produce high-quality relative depth from single RGB images. Fine-tuning it on the ByteCameraDepth data with the noisy depth as target teaches it to degrade its clean predictions into camera-specific noisy patterns. The affine-invariant normalization decouples the learning of relative depth structure (which DAV2 already does well) from the learning of absolute metric scale (which each camera distorts differently). Without this normalization, the model would receive conflicting signals—the same physical distance would map to different depth values across cameras—making the learning problem ill-posed.

Guided Filter for Scale Correction

A critical problem emerges when the value noise model is applied to simulation data: the noise model, trained with affine-invariant normalization, predicts depth values that have the correct relative structure (noise patterns, edge blur, surface artifacts) but are in the wrong absolute metric scale. The model might predict that an object's depth decreases smoothly from front to back, matching the real camera's bias pattern, but the numerical values might be systematically offset or scaled relative to the ground-truth depth. If this synthetic "noisy" depth is used as the prompt for training the CDM, the CDM learns that the camera prompt's scale is unreliable and learns to ignore it—exactly the wrong behavior, since the whole purpose of using camera depth as prompt is to provide metric scale information.

The solution is the guided filter (He et al., 2012), an edge-aware smoothing operator. The guided filter models the output image $\mathbf{B}$ as a local linear transformation of a guidance image $\mathbf{G}$:

$b_i = x_k g_i + y_k$

where $b_i$ is the $i$-th pixel of the output $\mathbf{B}$, $g_i$ is the corresponding pixel of the guidance image $\mathbf{G}$, and $x_k, y_k$ are scalar parameters that are constant within a local window $\omega_k$ centered at pixel $k$. The parameters are optimized per window to minimize:

$\sum_{i \in \omega_k} \left((x_k g_i + y_k - a_i)^2 + \epsilon x_k^2\right)$

where $a_i$ is the $i$-th pixel of the input image $\mathbf{A}$ to be filtered, and $\epsilon$ is a regularization parameter that prevents $x_k$ from becoming too large.

What this computes: within each local window $\omega_k$, the filter finds the best linear mapping ($x_k$ scale, $y_k$ shift) from the guidance image values $g_i$ to the input image values $a_i$, with a regularization term $\epsilon x_k^2$ that penalizes large scale factors (encouraging smoothness). The output $b_i$ is then $x_k g_i + y_k$ using the parameters from the window centered at $i$. Because each pixel is covered by multiple overlapping windows, the final output averages the predictions from all windows containing that pixel.

Why this form: in the paper's usage, the guidance image $\mathbf{G}$ is the value noise model's output (wrong scale, right structure), and the input image $\mathbf{A}$ is the ground-truth depth (right scale, clean structure). The guided filter adjusts the value noise to match the ground-truth's local metric scale while preserving the noise structure (edges, smooth regions, biases) from the value noise. A key control knob is the kernel size $k$, which determines the spatial extent of the local windows:

• Small $k$: the filter adjusts scale very locally, making the output match ground truth closely—but this erases the camera-specific noise structure.
• Large $k$: the filter adjusts scale more globally, preserving the noise structure but only approximately matching ground-truth scale.

The paper uses a randomized kernel size $k$ (ranging from small to large) as an augmentation strategy during training data synthesis. This creates a distribution of training examples: some where the prompt depth closely matches ground truth (encouraging the CDM to trust the prompt) and some where it is less reliable (encouraging the CDM to rely on RGB semantics). The maximum value of $k$ controls the overall trust balance. Additionally, because both noise models struggle to capture high-frequency noise patterns (neural networks tend to produce smooth outputs, and DAV2's training objective does not emphasize fine texture), the paper adds handcrafted high-frequency noise as an additional augmentation—likely Gaussian noise or Perlin noise added at pixel level to simulate the sensor's quantization and thermal noise characteristics.

Synthesizing Camera Depth

With both noise models trained and the guided filter applied to correct scale, the synthetic "raw camera depth" $\tilde{\mathbf{D}}$ is produced from an RGB image $\mathbf{I}$ via:

$\tilde{\mathbf{D}} = \mu(V(\mathbf{I})) \cdot (H(\mathbf{I}) < 0.5)$

where $V(\mathbf{I})$ is the value noise model's output, $H(\mathbf{I})$ is the hole noise model's output, $\mu$ is the affine-invariant unscaling function (the inverse of the normalization $f$, which recovers metric scale by referencing the ground-truth depth), and $(H(\mathbf{I}) < 0.5)$ is a binary mask that is 1 where the hole model predicts probability less than 0.5 (non-hole) and 0 where it predicts probability $\geq$ 0.5 (hole).

What this computes: first, the value noise model $V$ produces a relative-depth prediction from the RGB image, with the characteristic noise patterns of the target camera but in an affine-invariant normalized scale. The unscaling function $\mu$ converts this relative depth back to absolute metric depth by applying the inverse of the normalization transformation (which requires knowing the ground-truth depth's metric scale for the reference). Next, the hole model $H$ produces a probability map indicating where holes should appear. The binary mask $(H(\mathbf{I}) < 0.5)$ sets those pixels to zero in the final output—these become the "holes" in the synthetic camera depth. The multiplication $\cdot$ applies this masking element-wise: where the mask is 1 (predicted non-hole), the value noise depth is preserved; where it is 0 (predicted hole), the output becomes zero.

Why this form: the separation of hole and value noise into independently trained models and their combination through this equation is the core of the data synthesis approach. It mirrors the physical reality of depth cameras: the physical processes that cause holes (light absorption, specular reflection, occlusion) are different from those that cause value noise (sensor noise, quantization error, algorithmic bias), so modeling them separately is more natural and more controllable than training a single model to predict everything end-to-end. The threshold of 0.5 for the hole mask is a natural choice (50% probability as the decision boundary for binary classification), though the paper does not explore sensitivity to this threshold.

---

CDM Training

With the synthesized training data prepared—RGB images paired with synthetic noisy depth and ground-truth clean depth—the CDM itself is trained.

Training data sources. The noise models are applied to four simulated datasets: HyperSim (Roberts et al., 2021), DREDS (Dai et al., 2022), HISS (Wei et al., 2024), and IRS (Wang et al., 2019). Together these provide over 280,000 images spanning diverse indoor scenes with accurate ground-truth depth. HyperSim contributes photorealistic indoor environments; DREDS focuses on specular and transparent objects (exactly the challenging cases CDMs need to handle); HISS provides stereo data; IRS is a large indoor robotics stereo dataset.

Loss function. The CDM is trained with a combination of $L_1$ loss and gradient loss, following Lin et al. (2025):

$\ell(M) = L_1(\overline{\mathbf{D}}, \hat{\mathbf{D}}) + \ell_{\text{grad}}(\overline{\mathbf{D}}, \hat{\mathbf{D}})$

where $L_1$ is the mean absolute error between the predicted depth $\hat{\mathbf{D}}$ and ground-truth depth $\overline{\mathbf{D}}$, and the gradient loss is defined as:

$\ell_{\text{grad}}(\overline{\mathbf{D}}, \hat{\mathbf{D}}) = \left|\frac{\partial(\hat{\mathbf{D}} - \overline{\mathbf{D}})}{\partial x}\right| + \left|\frac{\partial(\hat{\mathbf{D}} - \overline{\mathbf{D}})}{\partial y}\right|$

where $\frac{\partial}{\partial x}$ and $\frac{\partial}{\partial y}$ are spatial derivatives (implemented as finite differences between neighboring pixels in the horizontal and vertical directions).

What this computes: the $L_1$ term penalizes any deviation between predicted and ground-truth depth at any pixel—it encourages global accuracy. The gradient loss penalizes deviations in the edges of the depth map: if the ground-truth depth has a sharp edge (a depth discontinuity at an object boundary) but the prediction smooths it out, the spatial derivative of the error will be large at that location, incurring a penalty. Conversely, if the prediction hallucinates an edge that does not exist, it is also penalized.

Why this form: $L_1$ alone produces blurry depth predictions because the model hedges its bets at depth discontinuities—predicting an intermediate value is safer (lower $L_1$) than committing to a sharp edge that might be in the wrong place by one pixel. The gradient loss directly incentivizes sharp edges by penalizing any spatial variation in the error map. This is particularly important for robotics, where precise object boundaries matter for grasping (a fork on a plate needs to be distinguishable from the plate) and for collision avoidance. The paper adopts this loss combination from Lin et al. (2025), who found it effective for high-resolution depth prediction.

Training target and initialization. The CDM is trained to predict disparity (inverse depth, $1/\text{depth}$) rather than depth directly. Disparity has the property that equal changes in disparity correspond to equal changes in perceived object size in the image, making it a more natural target for convolutional networks that process images in pixel space. After prediction, disparity is converted back to depth.

The ViT encoder weights for both the RGB and depth branches are initialized from DINOv2 pretraining (Oquab et al., 2023). DINOv2 provides strong visual features learned from massive self-supervised pretraining on diverse images. The DPT decoder is trained from scratch. For the single-channel depth images, they are "copied three times before being fed into the network" to match the 3-channel input expected by the ViT which was originally designed for RGB images.

Why DINOv2 initialization matters: starting from pretrained weights rather than random initialization dramatically reduces the amount of training data needed. DINOv2 already knows about object boundaries, material properties, and scene layout from its pretraining—fine-tuning it for depth prediction means the model only needs to learn how to project these semantic features into metric depth and how to integrate the scale information from the camera prompt, not how to recognize objects from scratch.

The training is conducted entirely on the synthesized data—the CDM never sees real-world ground-truth depth during training. This means that the quality of the synthesized data (how accurately the noise models replicate real camera noise) directly determines the quality of the CDM's real-world performance. The paper's strong real-world results (Tables 2 and 3) validate that the noise modeling approach successfully captures the distribution of real camera noise well enough for the CDM to generalize.

4. Key Insights and Innovations

Innovation 1: The "Make Reality Better" Reversal of the Standard Sim-to-Real Strategy

The dominant sim-to-real paradigm for depth-based manipulation is to add synthetic noise to simulated depth during training so the policy becomes robust to sensor noise at deployment time—what the paper calls a "last resort." This approach is conceptually flawed in a way the paper makes explicit: degrading the training signal prevents the policy from ever learning the precise manipulation skills that accurate geometry enables. The policy is trained to be forgiving of bad data rather than exploitative of good data.

CDMs implement the reverse strategy: instead of making simulation worse to match reality, make reality better to match simulation. At a technical level, this is a depth enhancement model. At a conceptual level, it is a complete inversion of how the community has approached the sim-to-real geometry problem. The paper demonstrates that when real-world depth is brought to simulation-level fidelity, a policy trained on clean simulated depth transfers directly without any noise augmentation, any fine-tuning, or any real-world adaptation. The results in Table 3 make this empirically vivid: the kitchen task achieves 26/30 overall success with CDM-D435, closely matching the 30/50 simulation success rate, while the same policy using raw camera depth scores 0/30 across all subtasks. The policy does not need to learn to cope with noise because the noise is removed before the policy sees the data.

This reversal has implications beyond the specific results. If the geometry gap can be closed at the perception layer, then the entire existing infrastructure for simulation-based training—generating demonstrations, learning policies, evaluating in simulation—becomes directly transferable to reality without the per-task sim-to-real adaptation engineering that currently dominates robot learning research. The paper frames this explicitly as a path to scalable data utilization: simulation data, which is effectively unlimited, can now be used to train policies that work on real hardware without degradation. This is not an incremental improvement in depth estimation accuracy; it is a re-architecting of the relationship between simulation and reality in robot learning.

Innovation 2: The Noise Model as a Camera-Specific, Learnable, Invertible Pattern

Prior work treats depth camera noise as generic corruption—add Gaussian noise, random dropout, maybe some Perlin noise for texture—and applies denoising or inpainting as a post-hoc cleanup. This framing has two consequences: it produces suboptimal denoising (since the noise model does not match the actual camera), and it treats the noise as something to be removed rather than something that carries information about the sensor that produced it.

The paper's key conceptual move is to reframe depth camera noise not as random corruption but as a camera-specific signature that can be learned from data and artificially reproduced on clean simulation images. The hole noise model learns that the RealSense D435's matched stereo algorithm produces holes at textureless regions and depth discontinuities, while the L515's LiDAR sensor produces holes on dark and specular surfaces. These are physically different failure modes arising from different sensing principles, and a model trained only on D435 noise patterns will not generalize to L515 data (which Table 1 confirms: CDM-L515 outperforms CDM-D435 on L515 data, and vice versa). The value noise model, trained by fine-tuning Depth Anything V2 to predict the camera's actual noisy depth output from RGB, captures the systematic biases, edge blur characteristics, and scale distortions that are unique to each camera's hardware and firmware implementation.

This framing turns the noise model from an adversary (something to fight against) into a tool (something that enables unlimited training data generation). By learning to produce realistic noise from RGB alone, the hole and value noise models allow the creation of paired training data—noisy depth + ground-truth clean depth—on any simulated dataset with ground-truth depth. This is what the paper calls a "neural data engine," and it solves the fundamental data bottleneck for training camera-specific depth enhancement models: you do not need to collect real-world data with ground-truth depth (which would require expensive laser scanners and careful calibration), because the noise models learn from real RGB-D pairs alone (no ground truth) and are then applied to simulation data (which has ground truth by construction).

The distinction between hole noise and value noise as independently modeled phenomena is also conceptually significant. Holes are binary, structural, and predictable from surface material and geometry; value noise is continuous, statistical, and predictable from the interaction of scene structure with sensor physics. Modeling them separately rather than with a single end-to-end noise predictor gives the synthesis pipeline modularity and interpretability—the hole model decides where data is missing, the value model decides what wrong value to report, and the guided filter ensures the wrong values are at the right metric scale. This separation mirrors the actual physics of depth sensing and makes the noise models more robust than a monolithic predictor would be.

Innovation 3: The Guided Filter as a Training-Time Regularizer for Scale Dependence

The paper identifies a subtle but important problem in using learned value noise models for training data synthesis: the noise model, trained with affine-invariant normalization, produces outputs with the right relative structure but the wrong absolute metric scale. If these scale-erroneous synthetic depth images are used as prompts for training the CDM, the CDM learns that the camera depth's scale is unreliable and ignores it—defeating the purpose of using camera depth as a metric scale reference.

The standard approach would be to add a scale-correction module after the value noise model, but this would either be handcrafted (and therefore approximate) or learned (and therefore require additional data). The paper instead repurposes the guided filter, an edge-aware smoothing operator originally designed for image enhancement, as a scale regularizer in the data synthesis pipeline. The guided filter performs a local linear transformation from the value noise (incorrect scale, correct structure) to the ground-truth depth (correct scale, clean structure). Critically, the kernel size $k$ controls the fidelity of this transformation: small $k$ makes the output match ground truth locally (erasing noise structure but providing correct scale), large $k$ preserves the noise structure but only approximately matches ground-truth scale.

The innovation is not the guided filter itself—it is the use of randomized kernel size as a training augmentation to control the CDM's learned reliance on prompt scale. By mixing examples where the prompt depth is nearly ground-truth (small $k$) and where it is weakly correlated (large $k$), the training distribution teaches the CDM a nuanced strategy: trust the prompt's scale when it is reliable, fall back on RGB semantics when it is not, and smoothly interpolate between these regimes. The maximum kernel size becomes a hyperparameter that controls the overall trust balance, enabling the CDM to be tuned for different deployment scenarios (e.g., a camera known to have good scale calibration vs. one that drifts).

This is a conceptually elegant solution to a problem that is easy to overlook but critical in practice. Without scale correction, the entire premise of using camera depth as a metric prompt collapses—the CDM would learn that the prompt is unreliable and default to monocular depth estimation, losing the benefit of the sensor entirely. The guided filter augmentation ensures that the CDM learns to use the prompt's scale information when it is available without being misled when it is wrong, and does so through a simple data augmentation rather than through architectural complexity or additional loss terms.

Innovation 4: The Demonstration That Accurate Geometry Alone Enables Zero-Shot Sim-to-Real Transfer

The paper makes a strong empirical claim that has not been demonstrated before: a depth-only visuomotor policy, trained on raw simulated depth with no noise augmentation and no real-world fine-tuning, transfers to real hardware with minimal performance degradation—provided the real-world depth is made accurate first. The results are striking: across two long-horizon tasks involving articulated (microwave door), transparent (glass), reflective (metal fork), and slender (fork, plate) objects, the CDM-processed depth enables success rates that substantially match simulation performance, while the same policies using raw depth succeed zero times (0/30 on all subtasks for both tasks).

The significance of this result lies not in the absolute success rates but in what it reveals about the sim-to-real transfer problem. The fact that a policy trained on clean depth works immediately when real depth is made clean suggests that the geometry gap is the dominant obstacle for depth-based manipulation policies—not physics discrepancies, not actuator dynamics, not camera calibration errors (which were addressed through randomized camera poses during data generation), but the simple fact that real depth data is geometrically corrupted. When that corruption is removed, the remaining sim-to-real gap is small enough to be bridged by modest domain randomization in camera pose and object placement.

This finding has implications for how the field allocates research effort. If accurate geometry solves most of the sim-to-real problem for depth-based policies, then investment in better depth perception (through models like CDMs, through better sensors, through sensor fusion) may yield higher returns than investment in more sophisticated sim-to-real adaptation techniques (domain randomization of physics parameters, adversarial training, real-world fine-tuning). The paper does not claim that geometry is the only gap—physics randomization and demonstration quality matter too, as shown by the improvement from MimicGen to WBCMimicGen in Figure 17—but it demonstrates that geometry is the bottleneck whose removal unlocks the transfer of simulation-trained policies.

The generalization results in Table 2 and Figure 5 further underscore the value of accurate geometry. The depth-only policy trained with CDM-processed depth on a normal-sized bowl generalizes to bowls of four unseen sizes, while the policy trained without CDM fails to generalize. This is not because the CDM policy is more sophisticated—it is the same policy architecture—but because accurate depth captures the 3D shape of objects, which is the invariant property that enables cross-instance generalization. A bowl is recognized as a bowl by its concave geometry, regardless of its diameter or depth; if the depth data is too noisy to perceive that concavity, the policy cannot generalize.

Innovation 5: The Multi-Level Token Fusion Architecture as a Native Solution to Incomplete Depth Prompts

Prior depth refinement models (PromptDA, PriorDA) require hole-filling preprocessing before the depth prompt is fed into the model. This is not an incidental implementation detail—it reflects an architectural limitation. When holes are filled before processing, the model never learns to distinguish between actual depth measurements (which carry metric scale information) and inpainting artifacts (which carry no physical information and may be systematically wrong). The model must trust the filled regions as if they were real measurements, creating a pathway for errors to propagate from the hole-filling algorithm into the depth prediction.

CDMs avoid this by accepting raw, holed depth images directly and using a dual-branch ViT architecture with per-location token fusion that processes RGB semantics and depth geometry through separate encoders before fusing them. The separation means that holes in the depth input do not corrupt the semantic feature extraction pathway (which operates on RGB alone), and the fusion mechanism—multi-head attention over concatenated RGB-depth tokens at corresponding spatial locations—allows the model to reason explicitly about where depth information is reliable and where it must be inferred from context.

The conceptual advance is architectural rather than algorithmic: the model structure itself encodes the prior that RGB and depth carry qualitatively different information (semantics vs. metric scale) that should be processed separately and integrated deliberately. This is distinct from simply concatenating RGB and depth channels at the input or fusing them in a late decoder—both of which conflate the two modalities before their distinct properties can be exploited. The experimental validation comes from Table 1, where PromptDA and PriorDA show dramatic performance degradation when holes are not pre-filled (PromptDA L1 increases from 0.0396 to 0.1703 on D435 data), while CDMs handle raw holed depth naturally because the architecture was designed for it. This is not a small engineering improvement; it is a structural difference that determines whether the model can serve as a practical real-time plugin in a robotics pipeline, where the latency cost of hole-filling preprocessing is non-trivial (the paper reports 0.151s total latency for CDMs versus 0.154s for PriorDA and 0.188s for PromptDA, all on a 4090 GPU—but PriorDA and PromptDA require additional hole-filling time not included in those numbers).

5. Experimental Analysis

Evaluation Methodology

• Dataset (depth accuracy). The Hammer dataset (Jung et al., 2023) is used for zero-shot evaluation of metric depth prediction accuracy. It contains warped depth data paired with RGB images collected by three depth sensors: RealSense D435 (active IR stereo), RealSense L515 (LiDAR D-Tof), and Lucid Helios (I-Tof). The dataset is not used for training any model, providing a zero-shot evaluation of generalization. Notes on the dataset itself are sparse—the paper does not report the number of test images per camera split or the scene diversity, only that it is a "real-world dataset" with warped depth.

• Depth accuracy metrics. Four standard depth estimation metrics are used: L1 (mean absolute error), RMSE (root mean squared error), AbsRel (absolute relative error), and thresholded accuracy $\delta_t$ (fraction of pixels where $\max(\frac{\hat{D}_i}{D_i}, \frac{D_i}{\hat{D}_i}) < t$, for $t \in \{1.25, 1.25^2 = 1.5625^{[1]}\}$, reported as $\delta_{0.5}$ and $\delta_1$ in Table 1). All metrics are computed directly from model outputs without any alignment postprocessing—unlike the affine-invariant evaluation protocols common in monocular depth estimation, this tests absolute metric accuracy.

  Correction note: The paper labels threshold columns as $\delta_{0.5}$ and $\delta_1$, but in standard depth estimation literature these are $\delta_{1.25}$ and $\delta_{1.25^2}$ respectively (i.e., $\delta_{1.25}$ is the fraction of pixels where the ratio is within 1.25). The $0.5$ and $1$ labels are non-standard and may be typographic errors. I will report them as they appear in the paper.

• Robot manipulation tasks. Three task categories are used, each with its own evaluation protocol:

  1. Real-world imitation learning (Section 5.2): Two tasks (Toothpaste-and-Cup, Stack-Bowls) with 50 teleoperated demonstrations each, tested at five positions with three trials each (15 total tests per subtask). For Stack-Bowls, policy is trained on a normal-sized bowl and tested on bowls of five different sizes (four unseen). Success is reported as fraction of trials where the subtask is completed.
  2. Zero-shot sim-to-real (Section 5.3): Two long-horizon multi-stage tasks (Kitchen task with three stages, Canteen task with five stages) trained entirely in simulation (~680 demos for Kitchen, ~800 for Canteen). Evaluation is at 10 real-world positions with three trials each (30 tests total), reporting success per stage and cumulative task success. Simulation baselines are reported as 50 trials.
  3. Simulation policy comparison (Appendix, Figure 17): MimicGen vs. WBCMimicGen evaluated on 50 simulation trials for both Kitchen and Canteen tasks, reporting success rate and trajectory smoothness metrics (mean absolute acceleration in rad/s², RMS jerk in rad/s³).

• Camera depth models (CDMs) evaluated. The primary models are CDM-D435 (trained on synthesized D435 noise) and CDM-L515 (trained on synthesized L515 noise). For the Hammer dataset evaluation, both are tested on all three camera data splits (D435, L515, Helios) to assess cross-camera generalization. Three additional CDMs are trained (D405, Azure Kinect, ZED 2i Neural mode) but evaluated only qualitatively (Figure 8) due to lack of ground-truth depth data for those cameras.

• Baselines (depth estimation).

  • PromptDA (Lin et al., 2025): A metric depth estimation method that uses a low-quality depth image as a scale prompt. Tested in two configurations: with hole-filling preprocessing (standard usage) and without (to assess robustness to missing data). Also tested with fine-tuning on the paper's synthesized datasets using the same augmentation strategy as CDMs.
  • PriorDA (Wang et al., 2025c): A similar prompt-based metric depth method with a different architecture.
  • Raw Depth: Using the depth camera's output directly without any model-based enhancement. Serves as the lower bound baseline.
  • Cross-camera CDMs: CDM-D435 tested on L515 and Helios data, CDM-L515 tested on D435 and Helios data. This is not a traditional baseline but reveals whether CDM training is strictly camera-specific or provides broader robustness.

• Baselines (robot manipulation).

  • None (raw depth): The policy receives the raw depth camera output directly. For D435 and L515 cameras, this is the unprocessed, hole-containing, noisy depth.
  • PromptDA: The same PromptDA depth model used as a preprocessing plugin instead of CDMs.
  • PriorDA: The same PriorDA depth model used as a preprocessing plugin. Both PromptDA and PriorDA serve as "state-of-the-art prompt-based depth model" baselines for the manipulation pipeline.

• Compute budget / latency measurement. The key efficiency metric is total latency from camera input to depth output, measured on a single NVIDIA 4090 GPU in Float32 precision. Measurements include preprocessing, model inference, and postprocessing time, reported as mean ± standard deviation (Table 4). The paper notes that quantization and further engineering optimization could reduce this further. For policy training, compute is not the primary budget—the relevant resource is the number of demonstrations (50 for imitation, ~680-800 for sim-to-real).

• Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. The Hammer dataset evaluation is zero-shot (no training on that dataset), so no data leakage concern exists. For the robot experiments, results are reported as raw success counts (X/15 or X/30) without confidence intervals, standard errors, or multiple random seeds. The simulation baselines use 50 trials, the real-world tests use 30 trials, but no statistical comparison between these numbers is provided. This is a limitation—particularly for claims about performance "matching" or exceeding simulation, since 30 trials is a small sample for drawing strong conclusions about equivalence.

---

Main Quantitative Results

Depth Prediction Accuracy on the Hammer Dataset (Table 1)

Headline result: CDMs achieve the best metric depth accuracy on their corresponding camera data splits, substantially outperforming prior methods, and exhibit unexpected cross-camera generalization.

On D435 data (active IR stereo camera):

• CDM-D435 achieves the best overall performance: L1 = 0.0258 (Filled) / 0.0404 (Holed), RMSE = 0.0312 (Filled) / not separately reported for Holed, AbsRel = not reported in the table, $\delta_{0.5} = 0.9842$, $\delta_1 = 0.9951$ (Holed configuration).
• CDM-L515 (trained on L515 noise, tested on D435 data) performs slightly better than CDM-D435 in the Filled configuration: L1 = 0.0182 vs. 0.0258, RMSE = 0.0338 vs. 0.0312. This is flagged as a surprising result—a model trained on a LiDAR camera generating data that matches or exceeds the accuracy of a model trained specifically on stereo camera data.
• PromptDA with hole-filling: L1 = 0.0396, RMSE = 0.0691, $\delta_{0.5} = 0.9503$, $\delta_1 = 0.9772$. PromptDA without hole-filling shows catastrophic degradation: L1 = 0.1703, RMSE = 0.2971, $\delta_{0.5}$ drops to 0.6704—quantifying why hole-filling dependency is a practical barrier for real-time robotics.
• PromptDA fine-tuned on CDM's synthesized data (PromptDA*): L1 = 0.0434 (without hole-filling, not available). This is worse than CDM-D435, confirming that the architectural difference (dual-branch fusion vs. shallow decoder fusion) accounts for the performance gap beyond just training data quality.
• PriorDA: L1 = 0.0388 (Filled), showing similar hole-filling dependency to PromptDA, though slightly better on some metrics.
• Raw Depth: L1 = 0.0550, RMSE = 0.1458, establishing the baseline that CDMs improve upon.

On L515 data (LiDAR time-of-flight camera):

• CDM-L515 achieves best performance: L1 = 0.0156 (Filled) / 0.0297 (Holed), $\delta_{0.5} = 0.9754$, $\delta_1 = 0.9919$ (Filled).
• CDM-D435 tested on L515 data: L1 = 0.0165 (Filled) / 0.0349 (Holed), slightly worse than CDM-L515 but still better than all baselines. The cross-camera generalization is bidirectional.
• PromptDA with hole-filling: L1 = 0.0207, significantly worse than CDM-L515.
• PromptDA without hole-filling: L1 = 0.0483, showing the same failure pattern as on D435.
• PromptDA* (fine-tuned on synthesized L515 data): L1 = 0.0235, better than standard PromptDA but worse than CDM-L515.
• PriorDA: L1 = 0.0177 (Filled), competitive with CDM-L515 on this metric but worse on RMSE (0.0385 vs. 0.0297).
• Raw Depth: L1 = 0.0312, again showing CDMs provide substantial improvement.

On Helios data (I-Tof, unseen camera during training):

• CDM-L515 generalizes best: L1 = 0.0248, CDM-D435 slightly worse: L1 = 0.0272. Both outperform the baselines (PriorDA: 0.0324, Raw Depth: 0.0312). This is the paper's key claim of zero-shot generalization to unseen sensor types—neither CDM was trained on Helios noise patterns, yet both outperform the baselines.
• The paper interprets this as evidence that "CDMs solve some common noise problems among depth cameras," and notes it as surprising since PromptDA does not show comparable cross-camera generalization.

Key patterns from Table 1:

1. Hole-filling dependency is a major practical barrier for PromptDA and PriorDA. The performance gap between Filled and Holed configurations is dramatic (e.g., PromptDA L1: 0.1703 Holed vs. 0.0396 Filled on D435). CDMs match or exceed the Filled performance of baselines without requiring hole-filling.
2. Cross-camera generalization is strong and unexpected. CDM-L515 matches or exceeds CDM-D435 on D435 data, and both generalize to Helios data better than baselines. The paper does not fully explain this—it is presented as an empirical finding rather than a designed property.
3. Architecture matters beyond data quality. PromptDA fine-tuned on CDM's synthesized data (PromptDA*) still underperforms CDMs, confirming the dual-branch ViT architecture with token-level fusion provides benefits beyond simply having better training data.
4. CDMs improve over raw depth substantially: L1 improves from 0.0550 to 0.0258 (D435, Filled) and from 0.0312 to 0.0156 (L515, Filled)—roughly a 2× reduction in mean absolute error.

---

Real-World Imitation Learning with Depth-Only Policies (Table 2, Figure 5)

Headline result: CDM-processed depth enables depth-only visuomotor policies to succeed at manipulation tasks where the same policy using raw depth fails completely or nearly completely.

Toothpaste-and-Cup task (two stages, 15 trials each):

• Pick Toothpaste: CDM-D435 achieves 10/15, baseline (None) achieves 0/15. The policy cannot locate or grasp the toothpaste using raw depth alone.
• Put Toothpaste into Cup: CDM-D435 achieves 6/15, baseline 0/15. The precision required to place the toothpaste into the cup exceeds what raw depth can support—even if the toothpaste were grasped (which it is not with raw depth), the placement would likely fail due to inaccurate geometry.

Stack-Bowls task (two stages, 15 trials each):

• Pick Bowl: CDM-D435 achieves 11/15, baseline 6/15. The bowl is a larger, more geometrically distinctive object that the raw depth policy can sometimes locate (6/15 is better than 0/15), but CDM substantially improves reliability.
• Stack Bowl: CDM-D435 achieves 9/15, baseline 3/15. The stacking action requires precise alignment of one bowl with another—a task that depends critically on accurate depth perception of both objects' 3D positions and orientations.

Overall observation: The success rate improvement from CDMs is largest for subtasks requiring fine spatial precision (putting toothpaste into cup: baseline 0/15, CDM 6/15; stacking bowls: baseline 3/15, CDM 9/15) and more modest for tasks involving larger, more easily perceived objects (picking bowl: baseline 6/15, CDM 11/15). This is consistent with the paper's central thesis: accurate geometry matters most when tasks require precise spatial reasoning.

Size generalization (Figure 5, Stack-Bowls task):

• The policy trained without CDM shows near-zero success on bowl sizes beyond the training size (the histogram bars for unseen sizes are effectively zero or negligible).
• The policy trained with CDM generalizes to all five sizes, with some degradation on the extreme sizes but maintaining non-trivial success throughout, demonstrating that accurate geometry captures the shape invariant (concavity of a bowl) that enables cross-instance generalization, and that raw depth loses this invariant to noise.

The results in Table 2 use only 50 demonstrations per task—a small number by imitation learning standards—yet CDMs enable non-trivial performance. This suggests the bottleneck is not policy expressivity or data quantity but perceptual quality.

---

Zero-Shot Sim-to-Real Manipulation (Table 3)

Headline result: Policies trained entirely in simulation on raw simulated depth (no noise augmentation) transfer to real robots with minimal performance degradation when CDMs are used as a depth preprocessing plugin, achieving success rates that match or slightly exceed simulation baselines, while the same policies fail completely (0/30 across all stages) with raw camera depth.

Kitchen Task (D435 camera view, 30 real-world trials):

• Pick Bowl stage: CDM-D435 scores 29/30 vs. simulation 43/50 (86.0% vs. 86.7%—essentially equivalent). CDM-L515 scores 29/30. Raw depth scores 0/30. PromptDA scores 11/30, PriorDA 16/30.
• Put Bowl into Microwave: CDM-D435 26/30, CDM-L515 22/30, simulation 33/50, raw depth 0/30.
• Close Microwave: CDM-D435 26/30, CDM-L515 16/30, simulation 32/50, raw depth 0/30.
• Overall (all three stages completed): CDM-D435 26/30 (86.7%), simulation 30/50 (60.0%), raw depth/PromptDA 0/30, PriorDA 7/30.

The CDM-D435 real-world success rate (86.7%) exceeds the simulation success rate (60.0%). The paper attributes this to broader randomization ranges in simulation making some configurations particularly difficult, suggesting the real-world test conditions were effectively easier than the hardest simulation configurations. This is not a problem per se, but it means the "matches or exceeds simulation" claim needs the qualification that simulation and real-world test distributions were not strictly matched.

Kitchen Task (L515 camera view, 30 real-world trials):

• Overall: CDM-L515 18/30 (60.0%), CDM-D435 9/30, PriorDA 2/30, PromptDA and raw depth 0/30. Simulation baseline: 32/50 (64.0%).
• The cross-camera pattern mirrors the depth metrics: CDM-L515 works best on L515 data, CDM-D435 works reasonably well but worse. This confirms the earlier finding (Table 1) that camera-specific training matters for downstream task performance.
• CDM-D435 on the L515 camera view (9/30) is substantially worse than CDM-L515 (18/30), even though Table 1 showed CDM-D435 achieving competitive depth metrics on L515 data. This suggests the downstream manipulation task is a stricter test of depth quality than the standard metrics—small depth errors that look similar in aggregate metrics may have outsized impacts on task success.

Canteen Task (D435 camera view, 30 real-world trials):

This task is the harder test: it involves a slim metal fork (reflective, geometrically small) and a thin plate, both challenging for depth cameras.

• Pick Fork: CDM-D435 30/30, CDM-L515 30/30, PriorDA 30/30, PromptDA 17/30, raw depth 0/30. The fork is a small object, yet all depth-enhanced methods locate it perfectly—likely because the fork is in a consistent position relative to the plate and box, providing strong contextual cues.
• Place Fork: CDM-D435 30/30, CDM-L515 29/30, PriorDA 30/30, raw depth 0/30.
• Pick Plate: CDM-D435 15/30, CDM-L515 0/30, PriorDA 1/30, raw depth 0/30. This is the failure point for CDM-L515—the plate, a thin reflective object, challenges the L515's LiDAR sensor in ways the CDM cannot fully correct, and cross-camera transfer (CDM-L515 on D435 camera data) is insufficient.
• Dump Plate: CDM-D435 14/30, CDM-L515 0/30.
• Place Plate: CDM-D435 14/30, CDM-L515 0/30.
• Overall: CDM-D435 14/30 (46.7%), simulation 21/50 (42.0%), all other methods 0/30 or 1/30 (PriorDA).

Canteen Task (L515 camera view, 30 real-world trials):

• Overall: CDM-L515 22/30 (73.3%), CDM-D435 9/30, PriorDA 3/30, PromptDA and raw depth 0/30. Simulation baseline: 20/50 (40.0%).
• Again, CDM-L515's real-world success (73.3%) exceeds simulation (40.0%), likely due to distribution differences.
• CDM-D435 on L515 camera data (9/30) performs substantially worse than CDM-L515 (22/30), confirming the camera-specific effect is particularly important for challenging tasks with reflective/slender objects.

Key patterns from Table 3:

1. Raw depth yields zero success everywhere. Across both tasks, both cameras, and all stages, policies using raw depth achieve 0/30. This is the paper's strongest empirical justification: without depth enhancement, sim-to-real transfer of depth-only policies is simply not feasible for these tasks.
2. CDMs achieve task-level success rates comparable to or exceeding simulation. The best camera-matched pairings (CDM-D435 on D435 view, CDM-L515 on L515 view) produce overall success rates within a few percentage points of simulation baselines, and in several cases exceed them. This is the paper's central empirical claim.
3. Camera matching matters substantially. CDM-D435 on L515 data drops from 26/30 to 9/30 (Kitchen) and 14/30 to 9/30 (Canteen). CDM-L515 on D435 data drops from 18/30 to 14/30 (Kitchen) and 22/30 to 0/30 (Canteen—catastrophic failure on Pick Plate). While Table 1 showed cross-camera generalization in depth metrics, the manipulation results reveal limits to this generalization that aggregate metrics do not capture.
4. PromptDA and PriorDA, while better than raw depth, fail to achieve the reliable performance needed for long-horizon tasks. PriorDA achieves some non-zero results (7/30 Kitchen-D435, 1/30 Canteen-D435) but success degrades severely for multi-stage tasks where each stage depends on the previous one.
5. The Canteen task reveals specific failure modes. The Pick Plate stage is the primary bottleneck: CDM-L515 fails here on D435 data (0/30), and CDM-D435 achieves only 15/30. The paper's point cloud visualizations (Figures 14, 15) confirm that the fork and plate geometry is partially recovered but not perfectly—the fork in particular "is integrated within the plate" in raw depth and is partially but imperfectly separated by CDM.

---

Latency Comparison (Table 4)

Headline result: CDMs achieve total latency of 0.151s ± 0.002s on a single RTX 4090 GPU, comparable to PriorDA (0.154s ± 0.005s) and faster than PromptDA (0.188s ± 0.005s).

Crucially, CDMs achieve this latency while accepting raw, unfilled depth images, whereas PriorDA and PromptDA require additional hole-filling preprocessing time not included in their reported latency numbers. This means the practical runtime advantage of CDMs is larger than the raw numbers suggest—the hole-filling step would add additional latency to the baseline methods in a deployed system. At 0.151s per frame, the CDM supports policy inference at >6 Hz (1/0.151 ≈ 6.62 Hz), which the paper notes is sufficient for real-time robot control without further optimization, and can be improved through quantization and engineering.

---

Data Generation Quality (Appendix Table 5, Figure 17)

Headline result: WBCMimicGen produces substantially smoother demonstration trajectories than MimicGen, and policies trained on these smoother demonstrations achieve higher simulation success rates.

Smoothness metrics (Table 5): Across all six UR5 joints, WBCMimicGen reduces mean absolute acceleration by 4-87% and RMS jerk by 12-98%, with the largest improvements on joints 4-5 (wrist joints) where MimicGen's linear interpolation produces sharp velocity changes.

Simulation success rates (Figure 17):

• Kitchen Task: WBCMimicGen-trained policy achieves ~72% vs. MimicGen's ~56%. The WBCMimicGen policy also shows lower RMS jerk (visible in the bar chart comparison).
• Canteen Task: WBCMimicGen-trained policy achieves ~42% vs. MimicGen's ~24%, again with substantially lower trajectory jerk.

This ablation supports a secondary claim: trajectory quality in demonstration data matters for downstream policy performance, and WBC-based trajectory generation produces data that is both safer (lower acceleration reduces risk of dropping objects) and more learnable (smoother trajectories are easier to model with behavior cloning). The connection to CDMs is indirect but important—CDMs handle the perception side of sim-to-real, while WBCMimicGen handles the action quality side. Both are necessary for the overall pipeline to work.

---

Ablation Studies and Robustness Checks

• Hole-filling vs. raw depth input (Table 1): The Filled vs. Holed comparison for PromptDA and PriorDA demonstrates that these methods critically depend on hole-filling preprocessing. PromptDA's L1 error on D435 data increases from 0.0396 (Filled) to 0.1703 (Holed)—a >4× degradation. CDMs are designed to accept raw depth and show no such degradation. This ablation justifies the architectural design choice of dual-branch encoding rather than single-branch concatenation.

• Fine-tuning PromptDA on CDM's synthesized data (Table 1, PromptDA entries):* PromptDA fine-tuned on the paper's synthesized datasets with the same augmentation strategy (labeled PromptDA*) improves over standard PromptDA but still underperforms CDMs. On D435 data (Filled): PromptDA* L1 = 0.0434 vs. CDM-D435 L1 = 0.0258; on L515 data: PromptDA* L1 = 0.0235 vs. CDM-L515 L1 = 0.0156. This demonstrates that the CDM's architectural advantages (dual-branch encoding, multi-level token fusion) contribute to performance beyond just having better training data.

• Cross-camera CDM generalization (Table 1, cross-camera rows): CDM-L515 tested on D435 data achieves L1 = 0.0182 (Filled), which is better than CDM-D435's own performance (L1 = 0.0258). CDM-D435 tested on L515 data achieves L1 = 0.0165, slightly worse than CDM-L515 (0.0156). Both CDMs generalize to the unseen Helios camera better than PriorDA. This is an unexpected finding—the paper does not claim cross-camera generalization as a designed property, but observes it empirically and hypothesizes that "CDMs solve some common noise problems among depth cameras."

• Camera-matched vs. cross-camera CDM in manipulation (Table 3): The manipulation results test whether the cross-camera generalization seen in depth metrics translates to downstream task performance. It does partially, but with important failures. On Kitchen-D435: CDM-L515 achieves overall 14/30 vs. CDM-D435's 26/30—a substantial drop despite good depth metrics in Table 1. On Canteen-D435: CDM-L515 scores 0/30 overall (failing entirely on Pick Plate onward), while CDM-D435 scores 14/30. Cross-camera CDM deployment is possible but unreliable, especially for tasks involving challenging objects (thin reflective plates). Camera-matched CDMs are strongly preferred for reliable manipulation.

• CDM for unseen cameras (Appendix Figure 8): CDM-D435 (trained on D435 noise models) is applied to raw depth from D415 and D455 cameras, which share similar stereo sensing principles with D435. Qualitative visualization shows "good predictions." This is a weak test (no quantitative metrics, no downstream task evaluation), but suggests that noise models trained on one stereo camera may transfer to other stereo cameras with similar hardware, reducing the data collection burden for new cameras within the same product family.

• Depth accuracy vs. distance (Appendix Figure 11): The paper evaluates CDM-D435 and CDM-L515 accuracy as a function of distance on the Hammer dataset. Raw depth accuracy degrades with distance beyond manufacturer specifications (the D435 "should have a less than 2% error rate when working under 1~2 meters"). CDMs achieve high accuracy whose trend "follows the accuracy of the original prompted depth"—meaning CDMs improve accuracy at all distances but the relative shape of the accuracy-vs-distance curve mirrors the raw sensor, suggesting CDMs cannot overcome fundamental sensor limitations (e.g., reduced LiDAR return signal at long range) that manifest as systematic depth errors rather than local noise.

• Guided filter kernel size as CDM trust control (Section 3.5, qualitative claim): The paper states that "adjusting the maximum value of $k$ controls the model's reliance on the prompt depth: a smaller $k$ aligns the prompt closer to the ground truth, encouraging the model to depend more on the prompt." No quantitative ablation is provided for this claim—no sweep over kernel sizes, no measurement of how CDM performance changes with trust level. This is a design parameter whose effects are asserted but not experimentally validated in the paper.

• WBCMimicGen vs. MimicGen (Appendix Figure 17, Table 5): The data generation method comparison shows that WBC-based trajectory generation produces smoother demonstrations (lower acceleration and jerk) and that policies trained on these demonstrations achieve higher success rates in simulation (Kitchen: ~72% vs. ~56%, Canteen: ~42% vs. ~24%). This ablation validates the choice of WBCMimicGen as the demonstration generation method, though it does not directly test the CDM's contribution—it establishes that trajectory quality matters independently of perception quality.

---

Critical Assessment

This paper makes three central claims, each of which needs to be evaluated against the evidence presented.

Claim 1: CDMs achieve nearly simulation-level accuracy in depth prediction, substantially outperforming prior prompt-based depth refinement methods.

What the experiments show: Table 1 provides clear quantitative evidence that CDMs outperform PromptDA and PriorDA on the Hammer dataset across all three camera splits, with particularly dramatic margins when holes are not pre-filled (PromptDA L1: 0.1703 vs. CDM: 0.0404 on D435 Holed). The architectural ablation (PromptDA* trained on the same synthesized data) shows that the performance difference is not just better training data—the dual-branch ViT architecture with token-level fusion provides additional gains.

What the experiments do not show: The Hammer dataset is the only quantitative depth benchmark used (though the paper mentions DREDS, HISS, HyperSim, IRS as training data sources, none are used for evaluation). We do not know the size of the Hammer test splits (the paper reports no image counts), making it impossible to assess statistical significance of the improvements. The cross-camera generalization result (CDM-L515 outperforming CDM-D435 on D435 data) is presented as a positive finding but could equally indicate that the D435 data split in Hammer is not representative of real D435 noise, or that the CDM-D435 training was suboptimal. No ablation studies vary training data quantity, architecture depth, or fusion mechanism to understand which components drive the performance improvements reported.

Additionally, the paper claims "nearly simulation-level accuracy" but never directly compares CDM output to simulated depth on the same scenes—no sim-vs-CDM depth difference maps are computed, no tolerance thresholds for "simulation-like" are defined. The manipulation results (policies working in reality) are the implicit validation, but these are downstream task evaluations, not direct depth accuracy measurements against simulation ground truth.

Claim 2: CDMs enable zero-shot sim-to-real transfer of depth-only visuomotor policies, demonstrating sim-to-real manipulation with no noise augmentation, no real-world fine-tuning, and minimal performance degradation.

What the experiments show: Table 3 provides strong evidence that CDM-processed depth enables policies to achieve non-trivial real-world success on challenging long-horizon tasks, while the same policies using raw depth achieve 0/30 across all stages. This is a compelling demonstration that depth quality is necessary for these policies to work, and that CDMs provide sufficient quality to make them work.

What the experiments do not show—and what weakens the claim:

1. The "matches or exceeds simulation" framing is misleading without matched test distributions. The real-world success rates often exceed simulation rates (e.g., Kitchen-D435: 26/30 or 86.7% real vs. 30/50 or 60.0% sim). The paper attributes this to simulation having broader randomization ranges. This is plausible, but it means the paper has not demonstrated matched performance on matched distributions. A stronger experiment would have tested the physical configurations that correspond to the hardest simulation configurations, or at minimum would have reported the distribution of object positions actually tested in reality and confirmed they span the same range as simulation.

2. No noise-augmented baseline is included. The paper's central narrative contrasts "make reality better" (CDMs) with "make simulation worse" (noise augmentation). But the experiments never train a noise-augmented policy and compare it against the CDM approach. Would a policy trained on simulated depth with realistic noise augmentation (using the paper's own noise models to generate realistic training noise) achieve comparable real-world performance without needing CDMs at inference time? This is a critical missing baseline—it is possible that noise augmentation during training is equally effective as CDM preprocessing at test time, and the paper provides no evidence either way.

3. The experiments use only two tasks, one robot morphology (UR5), and one camera per test condition. With 30 trials per condition and no statistical testing, the results are suggestive but do not constitute proof of general sim-to-real transfer capability. The tasks, while long-horizon and challenging, are both tabletop pick-and-place variants with rigid objects. Tasks involving deformable objects, dynamic environments, or mobile manipulation are not tested.

4. Policy performance is the evaluation metric, not sim-to-real perception gap directly. The claim is that CDMs bridge the "sim-to-real geometry gap," but what is actually measured is downstream task success—which conflates perception quality with policy robustness, physics discrepancies, control accuracy, and demonstration quality. A policy that succeeds with CDM depth and fails with raw depth proves that CDM depth is better, but does not isolate how much of the residual gap (sim success minus real success) is due to remaining depth errors vs. other factors.

5. The "without any added noise" claim is technically true but potentially overstated. While the simulated depth images used for training are not artificially corrupted with noise, the paper's WBCMimicGen data generation and the domain randomization of camera poses and object positions are forms of augmentation that contribute to sim-to-real robustness. The policy is not trained on a single static simulation configuration and deployed—it already incorporates robustness mechanisms. This is standard good practice, but it means the sim-to-real success is not purely attributable to CDMs.

Claim 3: CDMs are a simple plug-in that works across multiple camera types and tasks with low latency.

What the experiments show: Table 4 demonstrates practical latency (0.151s on a 4090). Table 3 shows that camera-specific CDMs work for their target cameras (D435 and L515). The point cloud visualizations (Figures 13-15) qualitatively show improved geometry. The paper releases model weights for five CDMs.

What the experiments do not show:

1. "Simple plug-in" implies ease of deployment, but the training pipeline is complex. Training a new CDM for a new camera requires: collecting a ByteCameraDepth-style dataset for that camera (multi-scene, thousands of images), training hole and value noise models, setting up the guided filter augmentation with appropriate kernel size ranges, and training the CDM on synthesized data from multiple simulation datasets. This is a substantial engineering effort—not a "simple" addition. The paper does not provide guidance on minimum data requirements, training time, or sensitivity to data collection quality.

2. The cross-camera generalization is unreliable for manipulation. While Table 1 shows decent cross-camera depth metrics, Table 3 shows CDM-L515 completely failing on Canteen-D435 (0/30 Pick Plate onward). A "plug-in" that requires matching the camera it was trained for is less practical than one that works robustly across cameras—and the paper does not test whether a combined CDM trained on multiple cameras' noise patterns would achieve the best of both.

3. Only two of the ten camera modes have quantitative manipulation results. The paper trains CDMs for D405, Azure Kinect, and ZED 2i but evaluates them only qualitatively (Appendix Figure 8). The claim of broad camera support is not backed by manipulation experiments for these additional cameras.

What Would Strengthen the Paper

• A noise-augmented training baseline: Train the same policy architecture on simulated depth with realistic noise augmentation (using the paper's own learned noise models) and compare real-world performance against the CDM approach. This would directly test whether making reality better outperforms making simulation worse, which is the paper's central conceptual claim.

• Matched sim-real test distributions: For the zero-shot transfer experiments, replicate the exact simulation test configurations (object positions, camera poses) in the real world and report success rates side by side. The current comparison (30 real trials vs. 50 simulation trials with different randomization ranges) does not support precise claims about degradation.

• Depth accuracy evaluation on the robot tasks themselves: The Hammer dataset provides standardized depth metrics, but these are on static scenes not from the robot tasks. Measuring depth accuracy (against a high-accuracy reference, like a laser scanner) on the actual manipulation scenes would provide a direct measure of whether CDM depth on task-relevant objects achieves simulation-level accuracy.

• Sensitivity to camera calibration: The paper uses a differentiable-rendering-based calibration method and randomizes camera pose during training to mitigate misalignment. An ablation that tests CDM-empowered policies under deliberate calibration errors would reveal how robust the "geometry gap bridging" is to practical calibration imperfections.

• Statistical reporting: Success rates of X/30 without confidence intervals make it difficult to assess whether differences (e.g., CDM-D435 26/30 vs. PriorDA 7/30 on Kitchen) are statistically significant or could arise from sampling variation. With 30 trials, a 95% confidence interval on a proportion of 26/30 spans approximately 69% to 96%—wide enough that many comparisons in Table 3 overlap.

• Ablation of data requirements: How does CDM performance vary with the number of real images used to train noise models? With the number of synthesized training images? With the diversity of training scenes? These are practical questions for anyone wanting to deploy CDMs on a new camera.

Summary

The paper provides compelling evidence that CDMs substantially improve depth quality from consumer cameras and that this improved depth enables depth-only manipulation policies to work in the real world where they otherwise fail completely. The zero-shot sim-to-real results are genuinely impressive—policies trained on clean simulated depth working on real hardware with articulated, transparent, and reflective objects is a result that has not been demonstrated at this level of reliability before.

However, the paper's strongest claims—that CDMs enable "seamless" sim-to-real transfer "without any performance degradation," that they are a "simple plug-in," and that the approach is superior to noise augmentation—are not fully supported by the experiments as designed. The missing noise-augmented baseline is the most significant gap: without it, we cannot conclude that the CDM approach is better than the standard paradigm, only that CDM depth is good enough when the standard paradigm (raw depth) is not. The paper establishes that accurate depth perception is critical for sim-to-real transfer, and that CDMs are an effective way to achieve it—but the broader methodological claim (that perception-side solutions are universally preferable to training-side robustness) remains an interesting hypothesis rather than an empirically validated principle.

6. Limitations and Trade-offs

CDM Training Requires Per-Camera Data Collection and a Complex Synthesis Pipeline

The assumption or constraint. The paper presents CDMs as "a simple plug-in on daily-use depth cameras" (Abstract), but this simplicity describes inference-time use, not model creation. Training a CDM for a new camera requires building a multi-camera mount, collecting thousands of real-world RGB-depth pairs across diverse scenes to train camera-specific noise models, then applying those noise models to multiple simulation datasets to synthesize paired training data with ground-truth depth. The paper is transparent about this pipeline's complexity in Section 3.3–3.5, but the gap between the inference-time plug-in framing and the training-time infrastructure is substantial. The data collection alone involves "over 17,000 images for each camera, sampled from videos at 5Hz, covering 7 different scenes" (Section 3.3), and the paper notes practical constraints like USB bandwidth saturation requiring multiple computers (Section 3.3). The noise model training requires a pretrained Depth Anything V2 model fine-tuned per camera, a pretrained DINOv2 backbone with a DPT head for the hole model, and the guided filter pipeline with tuned kernel size ranges. For the CDM itself, training uses "280,000+ images" synthesized across four simulation datasets (Section 3.5).

The consequence. Anyone wanting to deploy CDMs for a camera not already covered by the paper—or on a different hardware configuration of a known camera (different firmware, different baseline, different resolution)—must replicate substantial portions of this pipeline. There is no guidance on minimum data requirements, tuning sensitivity, or whether the noise models generalize across instances of the same camera model (manufacturing variance in sensor alignment, IR projector patterns, or calibration). A practitioner cannot simply download a pretrained model for a RealSense D435, observe that it does not work well for their specific unit, and "fine-tune" it with a few dozen calibration images—the paper provides no evidence that the learned noise patterns are consistent across individual camera units of the same model. This limits CDMs' practical deployability to the exact camera models and modes the authors have already trained and released, and creates an open question about whether the approach scales to the long tail of depth cameras and sensing configurations used in robotics labs and products.

What evidence exists in the paper. The paper does not measure the sensitivity of CDM performance to any aspect of the data collection or training pipeline. No ablation varies the number of real-world images used to train noise models, the number of synthesized training images, the diversity of training scenes, or the guided filter kernel size (the paper states that adjusting kernel size "controls the model's reliance on the prompt depth" in Section 3.5 but provides no quantitative ablation). The cross-camera generalization results in Table 1 and Table 3 provide indirect evidence that noise patterns differ across camera models—a CDM trained on L515 noise does not transfer reliably to D435 for manipulation tasks (Canteen-D435 with CDM-L515: 0/30 on Pick Plate, Table 3)—but this measures camera-to-camera generalization, not instance-to-instance robustness within a camera model.

Mitigation status. The paper releases pre-trained CDM weights for five cameras (D405, D435, L515, Azure Kinect, ZED 2i) to allow the community to test and build on them (Appendix, Section 8). It demonstrates that CDM-D435 trained on D435 noise qualitatively works on D415 and D455 cameras (Figure 8), suggesting that noise models trained on one camera in a product family may transfer to similar models. However, this is only a qualitative visualization, and no quantitative metrics or manipulation results are provided for these cross-family transfers. The paper does not specify minimum data requirements, provide training recipes for new cameras, or test sensitivity to data quantity or diversity. The claim that CDMs are a "simple plug-in" (Abstract) accurately describes the inference interface but substantially understates the training burden, creating a deployment gap between the released models and adoption for new sensors.

---

The Missing Noise-Augmented Training Baseline Leaves the Central Claim Unvalidated

The assumption or constraint. The paper frames its contribution as an alternative—and implicitly, a superior one—to the dominant sim-to-real paradigm of adding noise to simulated depth during training so that policies become robust to sensor noise. The authors explicitly characterize noise augmentation as "a last resort, as it may deteriorate the rich geometry information and precise manipulation" (Section 2.3, also Section 4). The paper's proposed alternative is to instead make real-world depth accurate enough that policies trained on clean simulated depth transfer directly. This is the paper's central conceptual claim: that perception-side solutions (improving the sensor output) are preferable to training-side solutions (teaching the policy to tolerate sensor degradation).

The consequence. Without a noise-augmented baseline, we cannot determine whether the CDM approach is actually better than the standard paradigm, or merely sufficient while the standard paradigm fails with raw depth. The paper shows that CDM depth works (policies succeed in reality) while raw depth fails (policies achieve 0/30 everywhere, Table 3). But this comparison does not isolate whether the benefit comes from the specific CDM approach or from the more general principle of providing accurate depth. A policy trained on simulated depth with realistic noise augmentation—using the paper's own learned noise models to generate training noise that matches real sensor characteristics—would test directly whether teaching the policy to be robust to sensor degradation is equally effective. If that noise-augmented policy achieved comparable real-world performance, the CDM approach would be an implementation choice (where to handle the noise) rather than a methodological advance. If it performed worse, the paper's claim that noise augmentation "deteriorates the rich geometry information and precise manipulation" would be empirically validated.

This is not a hypothetical concern. The paper has already demonstrated the ability to generate realistic synthetic noise via the hole and value noise models (Section 3.4). The guided filter can control noise severity (kernel size as a trust parameter, Section 3.5). It would be straightforward to train a policy on simulated depth data augmented with the same noise models used to synthesize CDM training data, and evaluate it under the same real-world conditions. The absence of this experiment is the single most significant gap in the paper's empirical validation of its central thesis.

What evidence exists in the paper. None. The paper does not train or evaluate any policy with noise-augmented simulated depth. The only training conditions compared in Table 3 are: policies trained on clean simulated depth (used for all real-world evaluations) and policies that receive different depth preprocessing at test time (raw, PromptDA, PriorDA, CDM variants). The simulation baseline is always clean depth. The policy architecture, training data quantity, and training procedure are held constant across these conditions—the only variation is the test-time depth quality. The paper cannot distinguish between the hypothesis "CDM depth is uniquely effective at enabling sim-to-real transfer" and the weaker hypothesis "accurate depth of any origin enables sim-to-real transfer." Both hypotheses predict success for CDMs and failure for raw depth. Only a noise-augmented baseline can separate them.

Mitigation status. Not addressed. The paper does not acknowledge this as a missing experiment or suggest it as future work. The closest the paper comes to discussing noise augmentation as a baseline is the statement in Section 5.3 that "the policies designed in our experiments are depth-only, excluding the effect of color information," and the framing in Section 2.3 that noise augmentation is a "last resort." These are rhetorical positions, not experimental controls. The paper's contribution would be substantially stronger if it included this experiment, even if the noise-augmented policy performed competitively—either outcome would be informative and would sharpen the contribution's framing (as a specific solution for specific cameras vs. as a methodological principle).

---

Cross-Camera Generalization Is Inconsistent and Tasks Involving Thin, Reflective Objects Partially Fail

The assumption or constraint. The paper claims that CDMs "achieve nearly simulation-level accuracy in depth prediction, effectively bridging the sim-to-real gap for manipulation tasks" (Abstract) and demonstrates "for the first time, that a policy trained on raw simulated depth, without the need for adding noise or real-world fine-tuning, generalizes seamlessly to real-world robots" (Abstract). These claims imply that CDM depth is both accurate across a range of scenes and objects, and sufficient for reliable manipulation.

The consequence. The experimental results reveal important boundaries on this reliability. Cross-camera deployment (using a CDM trained for one camera on another camera's data) produces inconsistent manipulation performance despite promising depth metrics. Table 3 shows: CDM-L515 on the D435 camera view achieves 0/30 on the Canteen task (failing entirely at the Pick Plate stage and beyond), while CDM-D435 on the same task achieves 14/30. Even camera-matched CDMs struggle with certain objects: CDM-D435 on the D435 view achieves only 15/30 on Pick Plate and 14/30 on subsequent stages (Table 3, Canteen task). The paper acknowledges this: "the geometry of the fork from the raw camera depth is integrated within the plate; the one predicted by the CDM is better, but still inaccurate" (Appendix, Section 12). The failure case in Figure 18 further illustrates that when the raw depth contains large hole regions and the RGB image provides insufficient semantic information to disambiguate the geometry, CDM predictions can be misled—and the paper provides only one such failure case, so the frequency and diversity of failure modes are unknown.

These failures matter because the tasks and objects involved are not exotic. Thin metal forks, reflective plates, glass microwave doors, and metallic drawer handles are common in household and industrial manipulation scenarios. A practitioner deploying CDMs for tabletop manipulation will encounter these object types routinely. The paper demonstrates that CDMs substantially improve over raw depth for these objects (raw depth scores 0/30 everywhere, while CDMs achieve non-trivial success), but the residual failure rate on hard objects is high enough to prevent reliable long-horizon task completion. In the Canteen task with CDM-D435, a policy that successfully picks the fork 30/30 times still only completes the full five-stage task 14/30 times, because picking the plate and subsequent stages fail roughly half the time. For applications requiring high reliability—which most real-world deployments do—this represents a significant remaining gap.

What evidence exists in the paper. Table 3 provides explicit per-stage success counts, revealing where CDMs fail. Cross-camera failures: CDM-L515 on D435 Canteen: Pick Fork 30/30, Place Fork 29/30, Pick Plate 0/30, Dump Plate 0/30, Place Plate 0/30, Overall 0/30. Camera-matched failures: CDM-D435 on D435 Canteen: Pick Fork 30/30, Place Fork 30/30, Pick Plate 15/30, Dump Plate 14/30, Place Plate 14/30, Overall 14/30. The point cloud visualizations in Figures 14 and 15 show that "the geometry of the fork from the raw camera depth is integrated within the plate; the one predicted by the CDM is better, but still inaccurate." The failure case in Figure 18 shows a metal plane causing the L515 to report holes and CDM-L515 failing to recover correct geometry because "the RGB image does not bring informative semantic information for the CDM to fix that error." The depth accuracy vs. distance curves in Figure 11 show that CDM accuracy "follows the accuracy of the original prompted depth"—CDMs improve accuracy at all distances but cannot compensate for fundamental sensor limitations that cause systematic errors.

Mitigation status. Partially addressed through cross-camera experimentation and explicit failure case documentation. The authors are transparent about the fork/plate limitation (Appendix, Section 12), the cross-camera performance degradation (Section 5.3 notes that CDMs "work better under their specific camera type"), and the metal plane failure case (Appendix, Section 15). They encourage readers to visit the project page for interactive point cloud demos. However, no solution is proposed for these failure modes beyond using the correct camera-matched CDM—which helps but does not eliminate failures, as the CDM-D435 on D435 Canteen results show. No analysis is provided of whether additional training data, architectural modifications, or sensor fusion could reduce these failures. The paper frames CDMs as bridging the sim-to-real gap, but the residual gap for thin, reflective, and transparent objects suggests a more accurate characterization is that CDMs substantially narrow but do not close the geometry gap for the most challenging object types.

---

Difficulty Estimation and the "Plug-in" Cost Are Not Accounted for in the Manipulation Pipeline

The assumption or constraint. The paper treats CDMs as a drop-in preprocessing step that takes camera output and produces clean depth, which is then fed to a visuomotor policy. The inference latency is reported as 0.151s on a single RTX 4090 GPU (Table 4), which the paper notes enables "policy to be run at a rate greater than 6Hz" (Section 5.3). The implicit assumption is that this is fast enough for real-time manipulation and that the computational overhead is the only deployment cost.

The consequence. There are two unaccounted deployment costs that matter in practice. First, GPU dependency: CDMs require a dedicated GPU for inference. The latency numbers are reported on an RTX 4090 (a high-end consumer GPU, circa 2022, with significant power draw and physical footprint). The paper does not report CDM latency on edge devices (Jetson Orin, Qualcomm platforms, or other on-robot compute common in mobile manipulation), on CPU-only inference, or under quantization. For deployment scenarios where the robot's onboard compute is limited—mobile manipulators, low-cost platforms, battery-powered systems—the requirement for a GPU running a ViT-based model at 6+ Hz may be prohibitive. The paper acknowledges that "additional quantization and other optimizations could further decrease inference times" (Section 5.3) but provides no measurements or projections for what latency or accuracy would result from such optimizations.

Second, pipeline integration complexity: CDMs are evaluated as a standalone module whose output feeds a depth-only policy. Real manipulation systems often fuse information from multiple modalities and multiple cameras. CDMs produce metric depth images, but the paper does not test whether CDM-processed depth is geometrically consistent across multiple viewpoints (a requirement for multi-camera point cloud fusion) or whether CDM output integrates cleanly with RGB-based policies (which would need synchronized, aligned RGB and depth). The paper's experiments are designed to isolate the depth modality—the policies are depth-only by design—but most practical manipulation systems use RGB, depth, or both, and the interaction between CDM-processed depth and RGB observations (are they spatially aligned? temporally synchronized? does the CDM introduce frame-to-frame jitter that affects temporal filtering?) is unexplored.

What evidence exists in the paper. Table 4 reports latency on a 4090 GPU without comparison to edge hardware. No power, memory, or throughput measurements are provided. The pipeline integration question is not addressed: all experiments use a single camera and a depth-only policy. The paper notes that "we choose depth instead of pointclouds as the observation" (Section 4) partially because "pointclouds fused from multi-view cameras require careful camera calibrations," implying that CDMs are primarily tested and validated in the single-view depth setting. Multi-view consistency, frame-to-frame temporal stability, and integration with RGB pipelines are not evaluated.

Mitigation status. Minimal. The paper suggests quantization as future work but provides no experimental support. The release of model weights (Appendix, Section 8) allows the community to perform these evaluations, but the paper does not provide benchmarks or guidance for deployment on resource-constrained hardware. For a method positioned as a "simple plugin" for "daily-use depth cameras" (Abstract), the absence of edge-deployment characterization—or even a projection of what latency might look like after quantization—is a significant gap for practitioners evaluating whether to integrate CDMs into their systems.

---

Small Sample Sizes and Unmatched Test Distributions Weaken the Statistical Basis for the Central Sim-to-Real Claim

The assumption or constraint. The paper's central empirical claim is that policies trained on clean simulated depth "generalize seamlessly" to real robots when CDMs are used, "with little to no performance degradation" (Abstract). This claim is supported by comparing real-world success rates (30 trials per condition) against simulation success rates (50 trials per condition) in Table 3.

The consequence. The comparison between real and simulation performance is undermined by two statistical issues. First, small sample sizes without confidence intervals: with 30 real-world trials, a 95% Clopper-Pearson confidence interval for a proportion of 26/30 spans approximately [0.69, 0.96]—wide enough that many comparisons in Table 3 are not statistically distinguishable. For example, the Kitchen task with CDM-D435 achieves 26/30 real (86.7%) vs. 30/50 simulation (60.0%). The real-world success rate is higher, which could indicate that CDM depth enables performance exceeding simulation (a strong positive result) or that the real-world test conditions were easier than the average simulation condition (a confound that inflates apparent performance). Without matched test distributions, we cannot distinguish these explanations. The paper acknowledges that "the randomized position in the simulation is bigger, and some of them are difficult to complete all tasks" (Section 5.3), which supports the confound interpretation—the real-world test may have sampled an easier subset of the simulation's test distribution—but provides no quantification of how much easier.

Second, no repeated evaluations or multiple seeds: the paper reports a single set of 30 trials per condition with no replication across different days, lighting conditions, or robot calibration states. Real robot experiments are inherently noisy due to hardware variability, environmental changes, and stochastic policy behavior. Without repeated evaluations, we cannot estimate the variance in success rates or assess whether reported differences are stable. This is a common limitation in real-robot papers (30 trials is more than many works report), but it is particularly relevant here because the paper makes a strong comparative claim (CDMs enable near-simulation-level performance) that requires precise estimation of the real-vs-sim gap.

What evidence exists in the paper. Table 3 reports raw counts: X/30 for real-world, X/50 for simulation. No confidence intervals, standard errors, or statistical tests are reported anywhere in the paper—not for depth metrics (Table 1, which also lacks sample sizes), not for imitation learning (Table 2, 15 trials per subtask), not for sim-to-real (Table 3, 30 trials), and not for simulation comparisons (Figure 17, 50 trials but reported only as bar chart heights). For the data generation comparison (Table 5), the paper reports smoothness metrics but no sample sizes or variance estimates. The paper simply does not engage with statistical uncertainty in any quantitative result.

Mitigation status. Not addressed. The paper does not discuss sample size limitations, does not provide confidence intervals, and does not test for statistical significance of any comparison. This is a methodological weakness that affects the strength of essentially every quantitative claim in the paper. In fairness, the gross pattern—0/30 for raw depth vs. 14-26/30 for CDMs—is unambiguous (Fisher's exact test would yield P < 0.0001 for any comparison of 0/30 vs. ≥14/30), so the necessity of CDMs for any non-zero performance is well-supported. But the more nuanced claims about the magnitude of the sim-to-real gap and whether performance "matches or exceeds simulation" are not statistically justified given the data presented. The small sample sizes also limit the generalizability of the specific success rate numbers—a practitioner should interpret the reported rates as indicative of the performance regime (CDMs enable reliable manipulation, raw depth does not) rather than as precise estimates of expected success probability.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new policy architecture, a new sim-to-real algorithm, or a new manipulation primitive. It does something more fundamental and arguably more impactful: it identifies the depth perception quality as the binding constraint on sim-to-real transfer for geometry-based manipulation, and provides a concrete, camera-specific solution that removes that constraint well enough to make simulation-trained depth-only policies work on real hardware for the first time. The shift is methodological rather than algorithmic—a re-architecting of where in the stack the noise problem is addressed (at the sensor output, not in the policy training), with the downstream consequence that simulation data becomes directly usable for real-world deployment without the per-task adaptation engineering that currently dominates robot learning pipelines.

The paper resolves a tension that has quietly structured the field's approach to 3D perception for manipulation. On one side, substantial evidence shows that geometric representations—point clouds, depth maps, 3D voxels—improve policy generalization across object appearances and enable precise spatial reasoning (Ze et al., 2024a,b; Zhu et al., 2024). On the other side, every practical deployment of these methods requires aggressive noise mitigation: downsampling point clouds, clipping depth ranges, hole-filling, temporal filtering, and adding synthetic noise during training to build robustness. The net effect is that policies never actually see accurate geometry—they see a degraded, filtered version that retains some structural information but loses the fine details that motivated using 3D in the first place. The field settled into a local equilibrium: 3D representations help a bit, but the sensor noise prevents them from delivering on their full promise, so RGB-based policies remain the default.

CDMs break this equilibrium by demonstrating that when you fix the depth signal at the source, the downstream benefits compound. Table 3 shows that policies trained on clean simulated depth achieve 0/30 success with raw depth, 0-7/30 with prior depth enhancement methods (PromptDA, PriorDA), and 14-26/30 with camera-matched CDMs—a discontinuous jump from "does not work" to "works reliably." This is not an incremental improvement in depth estimation metrics (though Table 1 shows those too); it is evidence that a threshold of geometric fidelity exists below which depth-based policies simply cannot function, and that CDMs cross that threshold for the tested cameras and tasks.

The paper also reframes what counts as a "sim-to-real" contribution. A growing body of work addresses the sim-to-real gap by making simulation more realistic—better rendering (NVIDIA Isaac Sim, Xiang et al., 2020), neural reconstruction of real scenes (Re³Sim, Han et al., 2025; Robo-GS, Lou et al., 2024), and adversarial training. CDMs pursue the opposite strategy: make reality more like simulation. This flips the cost structure. Making simulation realistic enough to match every real-world sensor's idiosyncratic noise patterns is a moving target that must be recomputed for each new environment, each new camera, each new lighting condition. Making reality simulation-like requires training one CDM per camera, after which deployment is camera-consistent regardless of the scene. The paper does not prove that the CDM approach is superior to simulation-realism approaches—the experiment comparing these strategies head-to-head was not conducted—but it establishes a new axis along which the tradeoff can be evaluated, distinct from the dominant "improve simulation fidelity" paradigm.

Perhaps the most consequential shift is what this work implies about data scaling for robot learning. Foundation models in vision and language succeeded in large part because internet-scale data is naturally compatible with model training—a pretrained ViT processes any JPEG, a pretrained LLM processes any text. Robotics lacks this property because real-world sensor data and simulation data inhabit different distributions, and the effort required to bridge them scales with the number of tasks, environments, and sensor configurations. By making one major sensor modality (depth) produce consistent output regardless of whether the underlying scene is real or simulated, CDMs effectively create a canonical observation space in which simulation data and real data are interchangeable. This does not solve the whole sim-to-real problem—physics, actuation, and contact dynamics remain distinct—but it isolates the visual perception component and solves it in a way that is decoupled from any specific task or environment. If this approach proves robust across a wider range of cameras and tasks, it could meaningfully change how the field allocates effort between perception engineering and policy engineering, and make simulation a genuinely scalable source of training data for geometric manipulation skills.

---

Follow-Up Research This Work Enables

Training a noise-augmented policy baseline to test the central thesis. The paper argues that making simulation worse through noise augmentation is a "last resort" that "deteriorates the rich geometry information and precise manipulation" (Section 2.3), but never validates this claim against a noise-augmented policy. A clean follow-up experiment would use the paper's own learned noise models (hole and value models, Section 3.4) to augment simulated depth during policy training, matching the noise distribution that real cameras produce, and compare the resulting policy's real-world performance against the CDM-preprocessed approach on the same Kitchen and Canteen tasks with the same test configurations. If the noise-augmented policy achieves comparable success rates (within, say, 5-10 percentage points of CDM performance on the overall task), the CDM approach would be an implementation choice rather than a methodological necessity—you could handle noise in training or at test time, with different latency/throughput tradeoffs but equivalent task performance. If the noise-augmented policy performs substantially worse, the paper's central conceptual claim would be validated, and the result would quantify the cost of degrading geometric information during training. This experiment requires no new hardware, no new data collection, and uses models the paper has already built—it is purely a training-time variation on the existing pipeline.

Quantifying how much geometry accuracy is "good enough" for manipulation tasks. The paper demonstrates that CDM-processed depth crosses a threshold that raw depth and prior enhancement methods do not, but it does not measure where that threshold lies. A systematic study could degrade CDM depth output in controlled ways—adding back Gaussian noise at increasing magnitudes, downsampling depth resolution, introducing systematic scale offsets, applying spatial blur of increasing kernel size—and measure at what point downstream task success degrades to some target level (e.g., 50% of the clean-depth success rate). The guided filter kernel size (Section 3.5) provides a natural knob for controlling depth fidelity continuously rather than discretely, enabling a sweep that maps geometric accuracy to manipulation success. This would produce a tolerance curve for depth quality in manipulation, telling practitioners how accurate a depth sensor needs to be for reliable task performance, and informing both sensor selection and future depth model design—do we need to chase the last millimeter of accuracy, or is "good enough" geometrically quantifiable?

Training a multi-camera CDM to handle cross-camera deployment robustly. The paper shows that cross-camera CDM deployment is unreliable: CDM-L515 fails entirely on the Canteen task with D435-view data (0/30 overall, Table 3), and CDM-D435 degrades substantially on L515-view data (Kitchen: 9/30 vs. CDM-L515's 18/30). A natural extension is to train a single CDM on synthesized data from multiple cameras' noise models simultaneously—for example, mixing D435-style and L515-style synthesized noisy depth in the training set, potentially with a camera-type conditioning token. The experiment would test whether a unified model can achieve camera-matched performance without knowing which camera is connected at inference time. This is practically important because deployment environments often have multiple camera types, and the current per-camera model approach requires the system to know which camera is active and load the correct weights—a solvable but inelegant requirement. A negative result (multi-camera training degrades performance relative to camera-specific models) would quantify the cost of generalization and inform whether the CDM approach should remain camera-specific or pursue unified architectures.

Evaluating CDM depth consistency across multiple viewpoints for point cloud fusion. The paper uses depth-only single-view policies explicitly to avoid the calibration complexity of multi-view point cloud fusion (Section 4: "pointclouds fused from multi-view cameras require careful camera calibrations and are more sensitive to irrelevant backgrounds"). But many manipulation systems—particularly those operating in cluttered scenes or requiring occlusion-resistant perception—use two or more cameras to reconstruct a partial 3D scene. A critical unvalidated assumption is that CDM-processed depth is geometrically consistent across views: if two cameras observe the same object from different angles, do the CDM-enhanced depth maps produce point clouds that align when transformed into a common frame? Cross-view inconsistency could arise if CDMs learn view-dependent biases (different corrections for the same surface seen from different angles) or if residual scale errors differ between cameras. A direct experiment would mount two cameras with known extrinsics, capture simultaneous RGB-D frames of a calibration target or textured object, apply the appropriate CDM to each, back-project to point clouds, and measure the alignment error (e.g., Chamfer distance or ICP residual) compared to raw-depth alignment. This would determine whether CDMs can serve as a drop-in replacement for multi-camera fusion pipelines or whether additional cross-view consistency losses are needed during CDM training.

Measuring instance-to-instance robustness of CDM noise models across camera units. The paper trains one noise model per camera model (e.g., one for RealSense D435) and assumes it captures the noise distribution of all D435 units. But manufacturing variance in sensor alignment, IR projector intensity, and firmware calibration could produce unit-specific noise patterns that a model trained on one camera's data does not capture. A systematic study would collect ByteCameraDepth-style data from, say, 5-10 separate D435 units in the same scenes, train CDMs separately on each unit's noise, and measure how much worse a CDM trained on unit A's noise performs when evaluated on unit B's depth, compared to a CDM trained directly on unit B's noise. If cross-unit variance is small (e.g., depth metrics degrade by <10% relative), the per-model approach is validated. If cross-unit variance is large, deployment would require per-unit calibration or fine-tuning, substantially increasing the practical burden. This experiment is enabled directly by the ByteCameraDepth data collection protocol—it simply requires repeating the collection on multiple instances of the same camera model.

Extending CDMs to dynamic scenes and moving cameras. All CDM evaluation in the paper is on static scenes—the Hammer dataset (Section 5.1) consists of still images, and the manipulation experiments use a fixed third-view camera. Real-world manipulation often involves head-mounted cameras (on the robot's wrist or end-effector), moving viewpoints (as the robot navigates), and dynamic scenes (objects being moved, humans in the workspace). These introduce temporal consistency requirements that static-frame CDMs are not designed for: frame-to-frame depth jitter (small random fluctuations in predicted depth for the same surface in consecutive frames) could cause the policy to perceive spurious object motion. A natural stress test would deploy CDMs on a wrist-mounted camera during a manipulation task requiring fine motion (e.g., peg insertion) and measure whether frame-to-frame depth stability is sufficient to avoid introducing high-frequency noise into the policy's action predictions. If jitter is significant, temporal smoothing or training with frame-stacking (multiple consecutive frames as input) could be explored.

---

Practical Applications and Downstream Use Cases

Simulation-based data generation for generalist robot policies. The most immediate practical application is scaling up data collection for behavior cloning and offline reinforcement learning. Generalist robot policies (e.g., Octo by Octo Model Team et al., 2024, π₀ by Black et al., 2024, OpenVLA by Kim et al., 2025) are trained on large mixtures of real and simulated data, but the integration is fragile—real depth data is noisy, simulated depth is clean, and policies must implicitly learn to handle both distributions. CDMs offer a concrete path to making all depth data—whether from real sensors or simulation—live in the same canonical clean depth space, enabling seamless mixing without modality-specific augmentation. For organizations generating thousands to millions of simulated demonstrations (as MimicGen and its variants enable), plugging CDMs into the real-world evaluation pipeline means simulation-trained policies can be validated on real hardware without per-task adaptation, substantially reducing the cost of iterating on task design, reward shaping, and policy architecture. The paper's demonstration that policies trained on ~680-800 simulated demonstrations transfer to 26/30 and 14-22/30 real-world success rates (Table 3) provides a concrete lower bound on what this looks like for long-horizon tabletop tasks.

On-robot depth enhancement for precision manipulation in unstructured environments. The paper validates CDMs on articulated objects (microwave door), transparent surfaces (glass), reflective objects (metal fork), and thin structures (fork tines, plate edges)—exactly the failure cases that make consumer depth cameras unreliable for everyday manipulation. A logistics or service robot operating in a kitchen, laboratory, or retail environment will encounter these object types constantly. Integrating CDMs as a real-time preprocessing module (0.151s latency on a 4090, enabling >6 Hz policy rate per Table 4) means the robot's perception stack receives clean geometry for grasp planning, placement verification, and collision checking without requiring a dedicated perception engineer to hand-tune hole-filling, depth clipping, and temporal filtering parameters per deployment site. For the tested D435 and L515 cameras—two of the most widely used depth sensors in robotics research and prototyping—the paper provides pretrained weights, making this integration feasible today.

Relabeling of existing RGB-D datasets for policy pre-training. Many existing robot manipulation datasets (e.g., BridgeData, RoboNet, large-scale teleoperation collections) include RGB-D streams from consumer depth cameras, but the depth channel is often discarded or aggressively preprocessed due to noise. Applying CDMs retrospectively to these datasets could produce clean depth annotations suitable for pre-training depth-based or multi-modal policies at scale, without requiring new data collection. The value of this depends on CDM generalization to scenes, lighting conditions, and camera units not represented in the ByteCameraDepth training data—the cross-camera generalization results (Table 1) are encouraging, but the cross-unit robustness question remains open. Even so, for datasets collected with the exact camera models covered by the released weights (D435, L515, D405, Azure Kinect, ZED 2i), CDM-based relabeling provides an immediate path to higher-quality depth supervision with no additional hardware cost.