Repurposing Geometric Foundation Models for Multi-view Diffusion

ArXiv: 2603.22275

🎯 Pitch

Training a multi-view diffusion model on the feature space of a geometric foundation model accelerates convergence by over 4× compared to standard VAE latents while improving 3D consistency, without any external geometry conditioning. This shows that repurposing pre-existing geometry-aware representations, rather than injecting geometry as side information, is a dramatically more efficient latent space for novel view synthesis.

---

1. Executive Summary

This paper proposes Geometric Latent Diffusion (GLD), a framework that repurposes the feature space of geometric foundation models—specifically Depth Anything 3—as the latent space for multi-view diffusion models, replacing the view-independent VAE latent spaces conventionally used for novel view synthesis. Through systematic analysis on RealEstate10K, DL3DV, and Mip-NeRF 360 using a flow-matching diffusion transformer trained from scratch, the authors identify an optimal boundary feature level (level 1) that balances geometric correspondence with photometric fidelity, enabling a cascaded generation scheme where only shallower features are explicitly synthesized while deeper features are derived via frozen backbone propagation. GLD accelerates training convergence by over 4.4× compared to VAE and DINO latent spaces (matching VAE baseline performance with fewer training steps) while consistently outperforming both baselines across 2D image quality and 3D consistency metrics, establishing that geometry-aware latent spaces provide inherently stronger grounding for multi-view generation—remaining competitive with state-of-the-art methods that leverage large-scale text-to-image pretraining, despite being trained entirely from scratch on smaller datasets.

2. Context and Motivation

The Core Problem: We Don't Know What Makes a Good Latent Space for Geometry-Aware Generation

The fundamental question this paper tackles is deceptively simple: when generating multiple views of a 3D scene, what representation should the generative model operate on? This matters because the choice of latent space determines what kind of information the model must learn from scratch versus what is already encoded in the representation. In 2D image generation, the community has converged on VAE latent spaces—compressed, perceptually optimized representations that discard pixel-level redundancy while preserving appearance. But novel view synthesis (NVS) imposes a requirement that single-image generation does not: cross-view geometric consistency. Pixels in different views of the same scene are not independent; they are linked by the underlying 3D geometry. A latent space that is view-independent—treating each image as a separate entity—forces the diffusion model to rediscover these geometric relationships implicitly, placing a heavy burden on the model's capacity and training data.

The paper frames this gap explicitly (Section 1):

"This raises a fundamental question: can we leverage a latent space in which geometric structure is already encoded, rather than injecting or supervising it externally?"

This is not a minor architectural choice. It determines whether the diffusion model spends its capacity learning photorealistic appearance (which is already hard) or also spends it learning the correspondence between views (which geometrically-aware representations could provide for free). The paper's empirical evidence—4.4× faster training convergence and consistent improvements in 3D consistency metrics—validates that this choice has substantial practical consequences.

Why This Problem Matters: Beyond Pixel-Perfect Images

The importance of this problem extends in several directions:

The geometry-appearance tension in NVS. The paper identifies a fundamental tension in prior NVS approaches (Section 2). Methods that excel at photorealistic image quality often sacrifice geometric consistency—producing images that look plausible in isolation but violate 3D coherence when viewed together. Conversely, methods that enforce geometric constraints through explicit depth warping or 3D conditioning (e.g., MVGenMaster) can produce artifacts when the external geometry estimator fails. GLD's core insight is that this tension can be resolved not by better conditioning or stronger geometric losses, but by choosing a latent space where geometry and appearance are co-encoded from the start—so the diffusion model generates representations that are inherently 3D-consistent.

The pretraining bottleneck. As Section 5.2 and Table 3 demonstrate, state-of-the-art NVS methods—MVGenMaster, Matrix3D, CAT3D—rely heavily on massive text-to-image pretraining to compensate for the lack of geometric structure in their latent spaces. Training these models from scratch produces substantially worse results (as shown in Table 14, where MVGenMaster trained from scratch drops from 14.17 PSNR to 11.22 on Mip-NeRF 360). This creates a dependency on proprietary pretrained weights and enormous compute budgets. GLD shows that training from scratch with the right latent space can match or exceed fine-tuned T2I models, sidestepping this bottleneck entirely—a finding with direct implications for democratizing NVS research and deployment.

Zero-shot geometry as a byproduct. Because GLD operates on geometric foundation model features, synthesized latents can be directly decoded into depth maps, camera parameters, and 3D point clouds without additional training (Section 5.7). This is not an add-on; it is a natural consequence of the latent space itself encoding geometric quantities. Techniques that operate in VAE or DINO latent spaces must train separate geometry prediction heads or rely on external depth estimators—adding complexity, potential failure modes, and computational overhead. GLD's unification of appearance and geometry generation into a single representation is both conceptually elegant and practically valuable.

The broader question of task-specific latent spaces. Beyond NVS, this paper opens a broader research direction: should different generation tasks use different latent spaces? The RAE literature (Section 3, citing Zheng et al. and Tong et al.) showed that semantic representations (DINO, SigLIP) improve 2D image generation. GLD extends this logic to geometry-aware tasks, suggesting a general principle: the optimal latent space for a generative task should encode the invariances and structures that the task requires. This has implications for video generation (temporal consistency), 3D generation (spatial consistency), and multi-modal generation (semantic alignment).

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along three axes:

1. VAE and RAE latent spaces are view-independent. The dominant paradigm for latent diffusion models—both the original LDM (Rombach et al., 2022) and the more recent RAE approaches (Zheng et al., 2025; Tong et al., 2026)—uses encoders that process each image independently. The VAE compresses each view into a compact latent without any cross-view information flow; the DINO encoder produces semantic features that are powerful for single-image understanding but lack explicit geometric correspondence.

This is a fundamental architectural limitation: when the latent representation of view A and the latent representation of view B are computed independently, there is no mechanism for them to share a common coordinate frame. The diffusion model must learn to map between these independently computed representations, essentially solving a correspondence problem as a side effect of denoising. The paper quantifies this burden in Section 5.5 (Tables 5-6): DINOv2 achieves a PCK (geometric correspondence) score of 31.64, while the DA3 features at level 1 achieve 35.98 and level 2 achieves 40.70—representing substantially stronger cross-view alignment already encoded in the representation.

2. External geometry conditioning is a patch, not a solution. Prior work recognized the geometric consistency problem and addressed it by injecting 3D information externally. MVGenMaster warps source RGB and depth to the target view using an external depth estimator. GenWarp uses semantic-preserving generative warping. ViewCrafter leverages video diffusion with explicit camera conditioning. While effective, the paper identifies several failure modes (Section 5.3, qualitative results in Figure 4):

"the method that relies on warped source views from an external geometry estimator produces sharp outputs but can exhibit noticeable artifacts when the estimation fails"

This creates a dependency chain: the NVS model's quality is bounded by the depth estimator's accuracy. If the depth estimator mispredicts depth for a region, the warping introduces geometric distortion that the diffusion model may amplify rather than correct. GLD eliminates this dependency by making geometry intrinsic to the representation rather than an external conditioning signal.

3. Dedicated geometry-appearance autoencoders require specialized training. Recent work (Krishnan et al., 2025, ORCHID; Yang et al., 2025, Prometheus) has begun to address the geometry-appearance gap by training autoencoders that jointly encode both modalities. However, this requires designing and training a new encoder-decoder pair from scratch—a substantial engineering effort that must be repeated for each new backbone architecture or resolution regime. GLD instead repurposes an existing, pretrained geometric foundation model, leveraging the massive investment already made in training these models on diverse multi-view data. This is a fundamentally different philosophy: rather than building a specialized latent space for generation, borrow one that already exists for geometric understanding.

4. No systematic understanding of which feature level is optimal. Even if one accepts the premise of using geometric foundation model features, a practical question arises: which layer's features should be used? Vision transformers produce a hierarchy of representations, from shallow features that preserve fine spatial detail to deep features that encode abstract semantic or geometric relationships. Prior work provides no guidance on this choice for generative tasks. GLD's systematic boundary layer analysis in Section 4.3 and Section 5.5—comparing geometric correspondence (PCK) and photometric fidelity (reconstruction quality) across all four levels—is the first principled investigation of this question, yielding the non-obvious finding that level 1 (not the deepest or shallowest) provides the optimal tradeoff.

Conflicting Tensions in Prior Work

The paper implicitly resolves several tensions in the NVS literature:

Photorealism vs. geometric consistency. Methods like CAT3D and MVDream produce stunning single-view quality but struggle with cross-view coherence. Methods like GenWarp enforce geometric constraints but lose photorealism in complex scenarios. GLD shows that these are not inherent tradeoffs: by generating in a geometry-aware latent space, photorealism and geometric consistency improve together (Table 3: GLD achieves both the best PSNR and the best ATE on in-domain datasets).

Training from scratch vs. fine-tuning from pretrained models. The NVS community has largely converged on the fine-tuning paradigm—start from a massive T2I model and adapt it for multi-view generation. This paper demonstrates that a scratch-trained model can match or exceed these fine-tuned methods, shifting the burden from pretraining scale to representation quality.

Discriminative vs. generative use of foundation models. Geometric foundation models (DUSt3R, VGGT, DA3) were designed for discriminative tasks—depth estimation, camera pose prediction, 3D reconstruction. GLD demonstrates that their internal representations are sufficiently rich and structured for generative tasks as well, expanding the scope of what these models can be used for.

How This Paper Positions Itself

GLD positions itself at the intersection of three research threads:

1. Latent space design for diffusion models (the RAE line of work): extending the insight that frozen encoder features can serve as diffusion latents from semantic spaces (DINO) to geometric spaces (DA3).

2. Multi-view diffusion for NVS: providing an alternative to the dominant paradigm of VAE-latent diffusion with external geometry conditioning.

3. Geometric foundation models: repurposing models built for 3D reconstruction as representation providers for generative tasks—a direction the paper explicitly frames as underexplored (Section 2):

"While recent analyses reveal that the internal representations of these networks encode strong geometric correspondences, their utility has been largely limited to discriminative tasks. We bridge this gap by showing that the feature space of a geometric foundation model can serve as an effective latent space for novel view synthesis."

The paper does not claim to invent a new encoder, a new decoder, or a new diffusion architecture. Its contribution is the integration—showing that these existing components, when properly composed with systematic analysis of which feature level to use, create a system that outperforms both generic latent spaces (VAE, DINO) and externally-conditioned methods, without requiring large-scale pretraining or external geometry estimators. The boundary layer analysis (Section 5.5), the cascaded generation scheme (Section 4.4), and the demonstration that geometry emerges as a zero-shot byproduct (Section 5.7) are the novel technical elements that make this integration work.

The paper also positions itself as a framework rather than a single model, validated by the VGGT backbone experiment in Appendix C.1. The core claim—that geometric foundation model features provide a better latent space for NVS—is shown to hold across both DA3 and VGGT, suggesting the principle generalizes beyond any specific architecture.

3. Technical Approach

3.1 Reader Orientation

GLD is a novel view synthesis system that generates images of a scene from unseen camera angles, given a few source views. Instead of generating images directly in pixels or in a standard VAE latent space (where each view is represented independently), GLD operates in the feature space of a geometric foundation model—a neural network already trained to understand 3D structure from multiple views. The core idea is that by diffusing in this geometry-aware representation, the generative model inherits cross-view consistency for free, rather than having to learn it from scratch.

3.2 Big-Picture Architecture (Diagram in Words)

GLD has five major components, arranged in a three-stage pipeline:

1. Frozen Geometric Encoder ($\mathcal{E}_\text{geo}$) — Depth Anything 3 (DA3), a pretrained vision transformer that takes multi-view images and extracts four levels of features ($F_0$ through $F_3$). These features are computed jointly with 3D attention across views, meaning the representation of each view already "knows about" the other views' geometry. This encoder is frozen during all stages—its weights never change.

2. Multi-View Diffusion Models ($\mathcal{M}_l$) — DiT-based flow-matching transformers that synthesize the features up to a chosen boundary level ($k=1$). There are two such models: $\mathcal{M}_1$ generates the level-1 features for all source and target views jointly, and $\mathcal{M}_{1\rightarrow 0}$ generates level-0 features conditioned on the synthesized level-1 features. Both use 3D self-attention and plücker ray conditioning to enforce geometric consistency during generation.

3. Feature Propagation (Frozen Backbone Pass) — The synthesized level-1 features ($\tilde{\mathbf{F}}_1$) are fed back through the frozen DA3 encoder's deeper layers to derive level-2 and level-3 features deterministically. This avoids the computational cost and potential inconsistency of synthesizing all four levels explicitly.

4. RGB Decoder ($\mathcal{D}_\text{rgb}$) — A ViT-based decoder trained from scratch to reconstruct RGB images from the full multi-level feature set. It takes the four levels of synthesized features for the target views and produces the final novel-view images.

5. Geometric Decoder ($\mathcal{D}_\text{geo}$) — The original DA3 DPT-based decoder, used without modification. Because the synthesized features are in the DA3 feature space, this decoder can produce depth maps, camera parameters, and 3D point clouds as a zero-shot byproduct—no additional training needed.

Information flow: Source images and target camera poses enter the system → the frozen encoder extracts source-only features at all levels → $\mathcal{M}_1$ synthesizes level-1 features for all views by denoising from random noise, conditioned on source features and camera rays → $\mathcal{M}_{1\rightarrow 0}$ synthesizes level-0 features conditioned on the generated level-1 features → the synthesized level-1 features are propagated through frozen DA3 layers to produce level-2 and level-3 features → the RGB decoder maps the full four-level target-view features to pixel-space images → (optionally) the geometric decoder maps the same features to depth maps and 3D reconstructions.

3.3 Roadmap for the Deep Dive

• First, the geometric foundation model's multi-level feature extraction process (Section 3 Preliminaries + Section 4.1), since understanding what the features are and how they are computed jointly across views is prerequisite to everything else.
• Second, the RGB decoder training and validation (Section 4.2), because establishing that DA3 features support high-fidelity reconstruction is the foundational claim that justifies using this space for diffusion.
• Third, the boundary layer analysis and multi-view diffusion architecture (Section 4.3), which determines which features to synthesize explicitly and how the diffusion models work—the core algorithmic contribution.
• Fourth, the cascaded generation scheme (Section 4.4), which handles the shallower features and ensures cross-level alignment—a subtle but critical detail that distinguishes GLD from naive multi-level synthesis.
• Fifth, the training configuration and dataset details (Section 5.1 + Appendix A), since the practical reproducibility of the framework depends on these choices.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and analysis paper whose core idea is that the internal feature hierarchy of geometric foundation models provides a latent space where cross-view geometric consistency is already encoded, enabling a multi-view diffusion model to generate 3D-coherent novel views without external geometry conditioning, simply by repurposing and composing existing frozen components with a cascaded generation scheme.

---

Multi-Level Feature Extraction from the Geometric Foundation Model

The foundation of GLD is the observation that geometric foundation models like Depth Anything 3 (DA3) process multi-view images jointly through their internal representations, using 3D attention layers that allow each view's features to incorporate information from all other views. This is fundamentally different from standard VAE or DINO encoders, which process each image independently. The paper leverages the DA3-Base variant throughout.

Given a set of $V = N + M$ input views—where $N$ is the number of source views (typically 1, 2, or 4) and $M$ is the number of target views to be synthesized—the frozen DA3 encoder $\mathcal{E}_\text{geo}$ extracts features at $L = 4$ intermediate levels:

$\{\mathbf{F}_l\}_{l=0}^{L-1} = \mathcal{E}_\text{geo}(\mathbf{I}), \quad \mathbf{F}_l \in \mathbb{R}^{V \times T \times C}$

where $T$ is the token sequence length at the encoder's patch resolution, $C = 1536$ is the channel dimension, and $l \in {0, 1, 2, 3}$ indexes the four intermediate outputs after transformer blocks 5, 7, 9, and 11 of the DA3-Base backbone respectively. The encoder uses 3D attention—in addition to intra-image self-attention—to jointly reason across the $V$ views. As a result, each feature tensor $\mathbf{F}_l$ contains information not just about the corresponding image but about the geometric relationships across all views.

What this represents: Each level $l$ captures a different abstraction. Level 0 (shallow) preserves fine spatial detail and photometric information; level 3 (deep) encodes abstract geometric relationships but discards pixel-level appearance. Levels 1 and 2 sit in between, with level 1 specifically balancing geometric correspondence with photometric fidelity—a finding established in Section 5.5 that drives the boundary layer choice.

Why joint encoding matters for NVS: In standard VAE-based diffusion, the latent for each view is computed independently. The diffusion model receives a set of independently-encoded source latents and must implicitly discover the cross-view correspondences needed to generate a coherent target view. In GLD, the source features $\mathbf{F}^\text{src}_l$ are first extracted from the $N$ source images alone (without access to target views), providing a clean conditioning signal. But the target that the diffusion model learns to generate is the joint feature $\mathbf{F}_l$—which, if it could have been extracted by the full encoder, would have incorporated cross-view information from all $V$ views. By training the diffusion model to produce features at the quality level of the full joint encoding, GLD teaches it to recover the geometric relationships that 3D attention across all views would have provided.

The paper normalizes each level's features to zero mean and unit variance using channel-wise statistics precomputed on the training set before diffusion training, and applies the inverse transform before decoding. This standardization is essential for stable diffusion training and is standard practice in latent diffusion.

---

Validating the Reconstruction Capability of Geometric Features

Before committing to diffusing in the DA3 feature space, the paper must first establish that these features can be decoded into high-fidelity RGB images. If the features lose too much appearance information, no amount of diffusion quality can recover it. Section 4.2 addresses this by training a dedicated RGB decoder $\mathcal{D}_\text{rgb}$ and measuring reconstruction quality.

Decoder architecture. The RGB decoder is a ViT-based transformer with 12 layers, a patch size of 14, an intermediate dimension of 3072, and a dropout probability of 0.5. It takes as input the full multi-level feature set $\{\mathbf{F}_l\}_{l=0}^3$ extracted by the frozen DA3 encoder and produces reconstructed RGB images $\tilde{\mathbf{I}} = \mathcal{D}_\text{rgb}(\{\mathbf{F}_l\}_{l=0}^3)$.

Level-wise dropout training strategy. A critical design choice is the level-wise dropout applied during decoder training. At each training step, individual levels in $\{\mathbf{F}_l\}_{l=0}^3$ are randomly masked out. The decoder must learn to reconstruct from partial input sets—sometimes from only one level, sometimes from all four. The effect is twofold: (1) it prevents the decoder from over-relying on any single level, making it robust to feature variations, and (2) it enables the ablation analysis in Section 5.5 where individual levels are evaluated in isolation for reconstruction fidelity. Without this dropout, reconstructing from a single level would be out-of-distribution at test time.

Training configuration. The decoder is trained on Re10K and DL3DV (sampled at equal ratio) using the AdamW optimizer with maximum learning rate $2 \times 10^{-4}$, minimum learning rate $2 \times 10^{-5}$, cosine decay schedule, no weight decay, betas $(0.9, 0.95)$, batch size 16, and EMA decay 0.9978. The loss function combines three terms following RAE:

• $\ell_1$ loss (mean absolute error in pixel space)
• LPIPS loss (perceptual similarity measured through a pretrained network; Zhang et al., 2018)
• GAN loss (adversarial, using a StyleGAN-T discriminator with differentiable augmentations)

The $\ell_1$ and LPIPS losses are equally weighted, while the GAN loss uses an adaptive weight to balance reconstruction and adversarial supervision. The model is trained at multiple resolutions: $(504 \times 504)$, $(504 \times 378)$, $(504 \times 336)$, and $(504 \times 280)$, matching the resolutions used to train DA3 itself. Training spans approximately 170k steps on 8 NVIDIA B200 GPUs.

Reconstruction results. As shown in Table 1 (evaluated on 4,000 samples from the Re10K test set), the decoder achieves:

• PSNR: 35.41 dB
• SSIM: 0.960
• LPIPS: 0.019

These are high-quality reconstructions that preserve fine-grained details (as visualized in Figure 3). For comparison, Table 7 shows that this decoder outperforms the pretrained Stable Diffusion VAE (PSNR 34.53) and SDXL VAE (PSNR 34.97) on the same task, despite those VAEs being specifically designed for image reconstruction. The DINO RAE decoder achieves only 26.78 PSNR—substantially worse—confirming that DA3 features preserve more appearance information than DINO semantic features alone.

Why this validation is non-trivial: DA3 was not designed for image reconstruction. Its features are optimized for geometric prediction tasks (depth estimation, camera pose recovery). The fact that they can be decoded into images at higher fidelity than purpose-built VAE latents is itself a surprising empirical finding. It suggests that the geometric reasoning DA3 performs requires preserving appearance details more faithfully than previously recognized.

What this enables downstream: This validation establishes that the DA3 feature space satisfies the fundamental requirement for a diffusion latent: there exists a deterministic mapping from features back to high-quality images. Any error in the diffusion model's synthesized features will manifest as reconstruction error in the images, but the decoder's high baseline fidelity means the "representational ceiling" is high—good features produce good images, with room for the diffusion model to approach that ceiling.

---

Multi-View Diffusion Architecture and the Boundary Layer Determination

With the decoder validated, the paper now addresses the core practical question: which of the four feature levels should be explicitly synthesized by the diffusion model? Synthesizing all four would be prohibitively expensive—each level requires a separate diffusion forward pass, multiplying inference cost by 4×. But deeper features (levels 2 and 3) can be obtained by propagating a shallower feature through the frozen DA3 encoder. The paper formalizes this as a boundary layer problem: identify the shallowest level $k$ such that features up to level $k$ are explicitly synthesized, and levels $k+1$ through $L-1$ are derived by forward-passing $\tilde{\mathbf{F}}_k$ through the frozen backbone.

Multi-view diffusion model architecture (one per level). Each diffusion model $\mathcal{M}_l$ follows the DiTDH architecture, which decouples the network into two components with separate hidden dimensions:

• Condition Encoder ($C_1 = 768$ hidden dimension, 28 DiT blocks): Processes the input tokens (noisy latent concatenated channel-wise with source-only conditioning features) and produces a compact representation summarizing the source-target context. The source-only features $\mathbf{F}^\text{src}_l$ are extracted by the frozen DA3 encoder from the $N$ source views alone—critically, without access to target views. For target views, these conditioning channels are zero-padded, so the model receives explicit information about which views are sources versus targets.

• Velocity Decoder ($C_2 = 2048$ hidden dimension, 6 DiT blocks for $\mathcal{M}_l$, 2 DiT blocks for $\mathcal{M}_{1\rightarrow 0}$): Predicts the velocity field $\mathbf{u}_{t,l} \in \mathbb{R}^{V \times T \times C}$ that defines the flow-matching trajectory from noise to data. The encoder's output conditions the decoder via AdaLN modulation, and camera embeddings are injected through addition.

Both components use 16 attention heads, SwiGLU MLP activations, and RMSNorm normalization. The patch size is 1 for DA3 and DINO latents (preserving the original token resolution), and 2 for VAE latents (to reduce token count due to higher spatial resolution). Positional encoding uses PRoPE (camera-relative positional encoding; Li et al., 2025), which is particularly important for multi-view generation because it makes the positional encoding a function of the camera pose, naturally encouraging features at corresponding 3D locations to have similar position embeddings across views.

3D self-attention for cross-view interaction. Standard DiT blocks use self-attention within each image. GLD replaces this with 3D self-attention, where query, key, and value tokens from all $V$ views are concatenated along the view dimension and processed jointly. This means that during denoising, each token in each view can attend to tokens in all other views. The paper cites this as following CAT3D's approach. This is how geometric consistency is enforced during generation: the model cannot produce features for one view without considering what it is producing for the others, and the PRoPE embeddings provide the spatial reference frame for these cross-view interactions.

Camera conditioning via Plücker ray embeddings. For each pixel in each view, the model computes 6D Plücker coordinates from the camera intrinsics and extrinsics, representing the ray in 3D space that projects to that pixel. A binary source/target indicator $m \in \{0, 1\}^{V \times T \times 1}$ (0 for source views, 1 for target views) is concatenated to form a 7D embedding, which is projected to the hidden dimension via a linear layer. This tells the model which views it is given (sources) versus which it must generate (targets), and where each pixel's ray originates in 3D space. During classifier-free guidance training, camera embeddings are dropped with 10% probability by zeroing the ray embeddings and setting extrinsic matrices to identity in PRoPE. At inference, a CFG scale of 1.5 is used.

Flow-matching objective. The model is trained with the standard flow-matching loss on velocity prediction. The target velocity field is the straight-line path between noise and data in the latent space. The noisy latent at timestep $t$ is formed by linear interpolation between a clean feature $\mathbf{F}_l$ and Gaussian noise, and the model predicts the constant velocity vector that would transport the noisy latent to the clean feature.

Training the level-wise models. The paper trains four independent diffusion models $\{\mathcal{M}_l\}_{l=0}^3$, one for each feature level, using the same architecture and training configuration. Each model $\mathcal{M}_l$ is trained to synthesize the joint feature $\mathbf{F}_l$ for all $V$ views—both source and target—conditioned on the source-only features $\mathbf{F}^\text{src}_l$ and camera rays. The joint feature is the full multi-view feature that would have been produced by the DA3 encoder if it had seen all views with 3D attention. By training the diffusion model to produce this joint feature from source-only conditioning, the model learns to recover the cross-view information that the encoder would have produced—essentially distilling the 3D attention computation into the diffusion process.

The training hyperparameters across all models are: AdamW optimizer with constant learning rate $5 \times 10^{-5}$, betas $(0.9, 0.95)$, weight decay 0.0, batch size 48 (each containing 8-view scenes), EMA decay 0.9995, gradient clipping at 1.0. Training runs for 175k iterations on 8 B200 GPUs. Input resolutions are sampled per batch from $(504 \times 504)$, $(504 \times 378)$, $(504 \times 336)$, and $(504 \times 280)$ with a $2:2:1:1$ ratio.

Boundary layer evaluation procedure. To determine the optimal boundary $k$, the paper evaluates NVS quality at four possible boundaries: $k = 0$ (explicitly synthesize only level 0, propagate to get levels 1–3), $k = 1$ (synthesize levels 0 and 1, propagate to get levels 2–3), $k = 2$ (synthesize levels 0–2, propagate to get level 3), and $k = 3$ (synthesize all levels explicitly).

For a given boundary $k$, the procedure is:

1. Run the set of independently trained models $\{\mathcal{M}_l\}_{l=0}^k$ to generate $\{\tilde{\mathbf{F}}_l\}_{l=0}^k$.
2. Take the deepest explicitly synthesized feature $\tilde{\mathbf{F}}_k$ and pass it through the frozen DA3 encoder layers for blocks $k+1$ through 11, extracting the intermediate outputs to obtain $\{\tilde{\mathbf{F}}_l\}_{l=k+1}^3$.
3. Feed the complete set $\{\tilde{\mathbf{F}}_l\}_{l=0}^3$ to the RGB decoder to produce images.
4. Evaluate against ground-truth target views using PSNR, SSIM, LPIPS, and depth metrics.

The results in Table 2 are decisive: $k=1$ provides the best performance across all metrics. Specifically:

• Moving from $k=0$ to $k=1$ improves PSNR from 12.55 to 13.61, SSIM from 0.323 to 0.366, and AbsRel depth error from 0.267 to 0.191. This is a substantial jump.
• Moving further to $k=2$ degrades performance: PSNR drops to 13.35, and all metrics worsen. This is counterintuitive—synthesizing more levels explicitly is worse.
• $k=3$ (all levels synthesized) performs essentially identically to $k=2$, confirming that deeper levels can be derived from level 1 without quality loss, but that pushing the boundary to level 2 is actively harmful.

Why $k=1$ is optimal. Section 5.5 provides the analysis that explains this result, combining two measurements:

1. Geometric correspondence (PCK): Following the Probe3D protocol on ScanNet, level 1 achieves a PCK of 35.98, level 2 achieves 40.70, and DINOv2 achieves 31.64. Level 1 encodes strong cross-view geometric correspondences, nearly matching the deepest level.

2. Photometric fidelity (single-level reconstruction): Using the RGB decoder trained with level-wise dropout, each level is independently decoded to measure how much appearance information it preserves. Level 0 achieves PSNR 28.01, level 1 achieves 25.36, level 2 drops sharply to 14.01, and level 3 to 10.19. Levels 2 and 3 discard substantial photometric detail—colors and textures are lost.

Level 1 sits at the sweet spot: it retains most of the geometric correspondence of level 2 (35.98 vs. 40.70 PCK) while preserving substantially more photometric information (25.36 vs. 14.01 PSNR). This dual property makes it the ideal latent for diffusion: it provides enough geometric structure for cross-view consistency without sacrificing the appearance information needed for high-fidelity image reconstruction. Level 0 has too little geometric structure (PCK 22.25); level 2 has too little appearance information.

Why synthesizing up to level 2 hurts: The paper attributes this to the "loss of fine-grained spatial details in abstract feature spaces." When the diffusion model must explicitly synthesize level-2 features, it operates in a space where photometric information is already largely discarded. Small errors in the synthesized level-2 features propagate downward to level 0 and level 1 through the cascaded generation, corrupting the appearance signal. By keeping the explicit synthesis boundary at level 1 and deriving levels 2–3 deterministically, the shallower features are what the diffusion model focuses on generating—where it can leverage both the geometric and photometric structure.

A subtle implication: the frozen backbone propagation from level 1 to levels 2–3 is deterministic and exact because it uses the same network that produced the training features. There is no error introduced by this step. In contrast, an independently trained diffusion model for level 2 would introduce sampling error and potential distribution shift. So propagation is not just cheaper—it is potentially more accurate.

---

Cascaded Feature Generation for Cross-Level Alignment

With the boundary fixed at $k=1$, the system must explicitly synthesize both level 0 and level 1 features. The naïve approach would be to run $\mathcal{M}_0$ and $\mathcal{M}_1$ independently—each denoising from random noise, conditioned only on the source features at their respective levels. However, this creates a problem: the independently generated $\tilde{\mathbf{F}}_0$ and $\tilde{\mathbf{F}}_1$ may be misaligned—they could represent slightly different scenes because there is no mechanism enforcing that the two diffusion trajectories produce consistent representations.

Cascaded architecture ($\mathcal{M}_{1\rightarrow 0}$). To address this, the paper replaces the independent $\mathcal{M}_0$ with a cascaded model $\mathcal{M}_{1\rightarrow 0}$ that generates level 0 conditioned on the already-generated level 1 latent $\tilde{\mathbf{F}}_1$. The architecture is identical to $\mathcal{M}_0$'s DiTDH design except that the velocity decoder has only 2 DiT blocks (instead of 6), and the condition encoder receives both the source-only level-0 features and the deeper level-1 features as additional conditioning. The level-1 features provide geometric context that anchors the shallower generation, ensuring that the resulting level-0 features are consistent with the level-1 representation that will be propagated to produce deeper levels.

Training with noisy conditioning. A critical training detail: during training, $\mathcal{M}_{1\rightarrow 0}$ conditions on a noisy version of the ground-truth $\mathbf{F}_1$, not the clean feature. This simulates the imperfect latents the model will encounter at inference time, where $\tilde{\mathbf{F}}_1$ is a synthesized (and therefore imperfect) approximation of the true level-1 feature. By training with noisy conditioning, the model learns to be robust to errors in the conditioning signal. Without this, the model would over-rely on the precise structure of clean level-1 features and fail when given synthesized ones—a classic train-test distribution mismatch.

Ablation results (Table 8). The paper quantifies the benefit of cascading vs. independent generation on Re10K with $N=4$ source views. Cascaded generation achieves PSNR 19.00 (vs. 18.81 independent), SSIM 0.695 (vs. 0.692), and LPIPS 0.327 (vs. 0.335). But the larger gains appear in 3D metrics: ATE drops from 0.197 to 0.182, RPEr from 7.179 to 6.694, and RPEt from 0.430 to 0.397. The cascaded approach improves both 2D quality and 3D consistency, confirming that feature-level alignment matters for downstream geometric coherence.

Why this matters beyond the numbers: The cascaded design transforms what would be independent denoising processes into a coherent hierarchical generation. Each level conditions on the deeper (more geometric) level, so information flows from geometry to appearance through the conditioning path, rather than having the two levels be generated from independent random seeds that might converge to incompatible solutions. This design pattern—using deeper features as spatial anchors for shallower generation—is reminiscent of coarse-to-fine generation strategies but implemented through feature-level conditioning rather than progressive upsampling.

---

System Pipeline During Inference

With all components trained, the inference procedure for generating $M$ target views from $N$ source views with target camera poses $\mathbf{P}_\text{tgt}$ proceeds as follows:

Step 1: Extract source-only features. The $N$ source images are fed to the frozen DA3 encoder $\mathcal{E}_\text{geo}$ to extract source-only features $\mathbf{F}^\text{src}_0$ and $\mathbf{F}^\text{src}_1$ at levels 0 and 1. These are the conditioning signals for the diffusion models. The extraction uses only intra-image attention (since there are only source views), but crucially the source features are in the same representation space as the joint features the diffusion model will produce.

Step 2: Synthesize level-1 features. The level-1 diffusion model $\mathcal{M}_1$ starts from random Gaussian noise $\mathbf{z}_1 \in \mathbb{R}^{(N+M) \times T \times C}$ and iteratively denoises it over multiple timesteps using the flow-matching ODE solver. At each step, the model receives as input:

• The current noisy latent $\mathbf{z}_{t,1}$ concatenated channel-wise with $\mathbf{F}^\text{src}_1$ (zero-padded for target views)
• The camera ray embeddings (plücker coordinates) for all $N+M$ views
• The source/target indicator masks
• The timestep $t$

The output is the synthesized joint feature $\tilde{\mathbf{F}}_1 \in \mathbb{R}^{(N+M) \times T \times C}$, which contains features for both source and target views. The model uses classifier-free guidance with a scale of 1.5, meaning it runs both a conditional and unconditional forward pass at each step and extrapolates between them to sharpen the conditioning signal.

Step 3: Propagate to deeper levels. The synthesized $\tilde{\mathbf{F}}_1$ is split into source and target portions. The full set is then passed through the frozen DA3 encoder layers for blocks 8–11, extracting intermediate outputs to obtain $\tilde{\mathbf{F}}_2$ and $\tilde{\mathbf{F}}_3$. This propagation step is fast—Table 15(B) reports 0.15 seconds for the level-1 to level-2 and level-3 propagation, compared to 37.8 seconds for level-1 sampling.

Step 4: Synthesize level-0 features (cascaded). The cascaded model $\mathcal{M}_{1\rightarrow 0}$ generates level-0 features starting from random noise, conditioning on both the source-only level-0 features and the synthesized $\tilde{\mathbf{F}}_1$. The same CFG procedure is used. The output is $\tilde{\mathbf{F}}_0$. The sampling time is 28.4 seconds, reflecting that level-0 generation is a separate diffusion process.

Step 5: Decode to RGB. The target-view features from all four levels—$\tilde{\mathbf{F}}^\text{tgt}_0$ through $\tilde{\mathbf{F}}^\text{tgt}_3$—are fed to the RGB decoder $\mathcal{D}_\text{rgb}$ to produce the final target images $\tilde{\mathbf{I}}_\text{tgt} \in \mathbb{R}^{M \times H \times W \times 3}$. Decoding takes 0.43 seconds.

Step 6 (optional): Decode to geometry. The same target-view feature set can be passed to the original DA3 geometric decoder $\mathcal{D}_\text{geo}$ to produce depth maps, ray maps, and camera parameters. This is zero-shot—the geometric decoder was never trained on GLD-synthesized features, but because the diffusion model is trained to match the distribution of real DA3 features, the decoder can process them as if they were extracted from real images.

Total inference time (Table 15). Sampling level-1 features takes 37.8 seconds, propagation takes 0.15 seconds, cascaded level-0 sampling takes 28.4 seconds, and RGB decoding takes 0.43 seconds. The total is 66.8 seconds per scene, compared to 35.2 seconds for the DINO baseline and 28.0 seconds for the VAE baseline. GLD is slower because it requires two separate diffusion sampling processes (level 1 and level 0), while VAE and DINO operate in a single latent space. However, the paper argues that the improved training efficiency (4.4× faster convergence, Figure 1) and the zero-shot geometry outputs offset this cost. Furthermore, the propagation step (0.15s) is negligible compared to diffusion sampling, validating the decision to avoid explicit generation of deeper feature levels—which would add a third full diffusion process.

---

Training Data Construction

The diffusion models are trained on four datasets: RealEstate10K (66,033 scenes), DL3DV (10,176 scenes), HyperSim (794 scenes), and TartanAir (369 scenes), with a mixing ratio of $4 : 4 : 1 : 1$. Each training sample consists of $V = 8$ views, where 1 to 4 views are randomly selected as source views while the remaining are masked as targets. The multi-view sequence construction is carefully designed:

Frame sampling strategy. The sequence starts from a randomly sampled frame index, chosen to ensure sufficient remaining frames for the required number of views. Each consecutive frame interval is independently and uniformly sampled within a predefined range. For evaluation with $N=2$ source views, the first and last frames are always included as source views; when $N > 2$, additional source views are placed at uniform intervals between them.

Camera normalization. To handle varying scene scales across datasets, all camera poses are normalized relative to the last view (set as the origin), and scaled such that the maximum camera distance within the batch is 1. This ensures the diffusion model operates in a consistent coordinate frame regardless of the physical scale of the scene.

Evaluation protocol. Evaluation uses 200 samples per dataset from Re10K and DL3DV (in-domain) and Mip-NeRF 360 (out-of-domain, 9 scenes with multiple randomized view samples), plus 50 samples from ETH3D for depth evaluation. For datasets where the evaluation set contains fewer scenes than the target count (DL3DV has 55 scenes in its eval set), multiple samples are drawn from the same scene using randomized view sampling.

4. Key Insights and Innovations

Innovation 1: Repurposing Discriminative Geometric Features as a Generative Latent Space

The paper's most fundamental conceptual move is not a new architecture or training objective, but rather a reframing of what a diffusion model's latent space should be for geometry-aware tasks. Prior to GLD, the field operated under an implicit assumption that the latent space for generation should be designed for compression—VAEs compress images into compact codes that discard pixel-level redundancy while preserving appearance. Even the RAE line of work, which replaces learned VAEs with frozen semantic encoders like DINO, still operates under this same assumption: the latent should be a view-independent, per-image encoding optimized for reconstruction fidelity or semantic content.

GLD challenges this assumption at its root. The key insight is that for NVS, the latent space's primary job is not compression—it is cross-view coherence. A latent space where each view's representation is computed independently forces the diffusion model to rediscover geometric correspondences as a side effect of denoising. But if the latent space itself already encodes those correspondences—if the representation of view A and view B share a common geometric coordinate frame—then the diffusion model inherits this structure for free, and its capacity can be spent on appearance refinement rather than geometric discovery.

What makes this more than a simple "use a different encoder" swap is the repurposing philosophy. DA3 was designed as a discriminative model for depth estimation and 3D reconstruction. Its features were optimized for geometric prediction, not image reconstruction. The decision to use these features as a generative latent space required answering a question that the discriminative community had never asked: do these features actually contain enough appearance information to reconstruct photorealistic images? The paper answers this affirmatively (Table 1: PSNR 35.41, LPIPS 0.019), and the fact that this answer is positive is itself a finding—it reveals that the geometric reasoning DA3 performs is more appearance-aware than previously recognized, that geometry and photometry are not cleanly separable in deep representations, and that the distinction between "discriminative features" and "generative latents" is more porous than the field assumed.

This is a fundamental conceptual shift, not an incremental refinement. It changes the design question from "how can we inject geometry into a compression-based latent space?" (the approach of MVGenMaster with its external depth warping, or ORCHID with its joint appearance-geometry autoencoder) to "which existing geometric representation already gives us the right structure, and how do we build generation around it?" The VGGT replication in Appendix C.1 provides the crucial generalization evidence: the principle holds across two different geometric foundation models with different architectures and training procedures, suggesting it is the geometry-awareness of the latent space—not any specific architectural detail—that drives the gains. This opens a broader research direction where task-specific latent spaces are sourced from foundation models built for those tasks, rather than being purpose-designed for generation.

The significance extends beyond performance. GLD achieves state-of-the-art 2D metrics and 4.4× faster training convergence compared to VAE and DINO baselines, but the intellectual contribution is the demonstration that the choice of latent space can substitute for massive pretraining. Section 5.2 and Table 3 show GLD matching or exceeding fine-tuned T2I models (MVGenMaster, Matrix3D, CAT3D) despite being trained from scratch on small datasets. This suggests that the NVS community's reliance on text-to-image pretraining is partly a consequence of using a latent space that is poorly suited to the task—and that better representation choice can close the gap without requiring proprietary pretrained weights or enormous compute budgets.

---

Innovation 2: The Boundary Layer Concept as a Diagnostic Framework for Multi-Level Feature Representations

Vision transformers produce feature hierarchies, but the generative modeling community has lacked a principled way to decide which level to use as the diffusion latent. Prior work either used the final layer output (VAE latents are single-scale), hand-selected a level based on intuition (RAE uses the last DINO layer), or avoided the issue by training specialized autoencoders that produce single-scale latents. The question of which level is optimal—and why—had not been systematically investigated for any generative task.

GLD introduces the boundary layer as both a design concept and an analytical tool. The idea is simple: deeper features can be derived from shallower ones by propagating through the frozen encoder, so only features up to some boundary level $k$ need to be explicitly synthesized. But the analysis that determines where to place this boundary is where the innovation lies. Rather than treating the boundary as a hyperparameter to be tuned by trial and error, the paper conducts a dual-property analysis of each feature level:

1. Geometric correspondence (PCK on ScanNet, Table 5): how well do the features align across views—i.e., do they encode the 3D structure needed for coherent NVS?
2. Photometric fidelity (single-level reconstruction PSNR, Table 6): how much appearance information does each level preserve—i.e., can the features be decoded into high-quality images?

The finding that these two properties have opposing trends across the feature hierarchy is the diagnostic insight. Level 0 (shallow) preserves appearance (PSNR 28.01) but has poor geometric correspondence (PCK 22.25, worse than DINOv2). Level 2 (deep) has strong geometry (PCK 40.70, the best of all levels) but discards most photometric information (PSNR 14.01). Level 3 is worse on both axes. Level 1 sits at the Pareto front: it preserves 88% of level 2's geometric correspondence (PCK 35.98 vs. 40.70) while retaining 80% more photometric information (PSNR 25.36 vs. 14.01).

This is more than a "use level 1" result. It provides a practical diagnostic framework for any task that requires both geometric structure and appearance fidelity. The methodology—measure geometric alignment and reconstruction quality independently, identify the level where the tradeoff curve bends—is portable to other encoder architectures, other tasks (video generation, 3D reconstruction from latents), and other foundation models. The paper demonstrates its generality by applying it to DA3's specific hierarchy, but the framework itself is architecture-agnostic.

The counterintuitive result—that synthesizing more levels explicitly (boundary $k=2$ or $k=3$) degrades performance (Table 2)—adds a layer of insight. This is not what a naive scaling argument would predict. The explanation the paper offers (that diffusion in photometrically impoverished feature spaces introduces errors that propagate to shallower levels through the cascaded generation) points to a general principle: diffusion models perform better in latent spaces that retain the information needed for the final output, even if those spaces are not the most "semantically rich." Deeper features may have more abstract geometric structure, but the diffusion model cannot leverage that structure if the path from those features to high-quality images is too lossy. This principle—that the optimal diffusion latent is determined not just by the information it contains but by the fidelity of the reconstruction path—is a conceptual contribution that extends beyond NVS to any hierarchical latent generation framework.

---

Innovation 3: Unifying Appearance and Geometry Generation Through Latent Space Choice Rather Than Architectural Modification

Prior approaches to joint appearance and geometry generation fall into two camps. One camp builds specialized multi-head architectures: train a model that outputs both RGB and depth, either through separate decoder branches (Matrix3D) or through additional prediction heads on a shared latent. The other camp injects geometry as external conditioning: use an off-the-shelf depth estimator to warp source views to the target pose, then condition the diffusion model on the warped RGB and depth (MVGenMaster, GenWarp). Both approaches treat geometry as something to be added to an appearance-first generation pipeline.

GLD demonstrates a fundamentally different path: geometry and appearance can be unified not by modifying the generation architecture, but by choosing a latent space where they are already co-encoded. Because DA3 was trained to jointly predict depth, camera parameters, and 3D structure from multi-view images, its internal features simultaneously encode geometric quantities and photometric appearance. When GLD's diffusion model synthesizes these features, it implicitly generates both modalities. The geometry is not predicted by a separate head, injected as conditioning, or supervised with an auxiliary loss—it emerges as a zero-shot byproduct of generating in the right latent space.

This is a conceptual reframing of the appearance-geometry unification problem. Rather than asking "how do we add geometry to our generation pipeline?", GLD asks "can we find a representation where geometry is already present, and generate in that representation?" The answer is yes, and the evidence is in Section 5.7: GLD produces depth maps that are more accurate than Matrix3D's jointly-trained depth predictions (Table 9: AbsRel 0.160 vs. 0.197, $\delta_1$ 0.800 vs. 0.731 on ETH3D), and 3D point clouds that exhibit "significantly cleaner and more coherent reconstructions" (Figure 7) with substantially less geometric distortion—all without any depth-specific training or losses.

The significance is twofold. Practically, it eliminates a failure mode: methods that rely on external depth estimators inherit those estimators' errors, which can cause cascading artifacts when depth mispredictions lead to incorrect warping that the diffusion model then amplifies (the paper notes this qualitatively for MVGenMaster in Figure 4). GLD avoids this dependency chain entirely because geometry is not an external signal—it is intrinsic to the representation. Conceptually, it suggests a design principle for multi-modal generation: find a foundation model whose training objective already requires co-encoding the modalities you want to generate, repurpose its feature space as the generative latent, and let the modalities emerge together from the same representation. This principle is not specific to appearance+geometry—it could apply to video (appearance+motion), audio-visual generation, or any task where multiple modalities are linked by an underlying physical process that a foundation model has already learned to model.

This is a fundamental contribution, not an incremental one, because it changes what problem is being solved. The old problem was "how to add geometry supervision to an appearance generator." The new problem is "how to find representations that already bridge the modalities, and build generation around them." The fact that the geometric decoder $\mathcal{D}_\text{geo}$ was never trained on GLD-synthesized features—it is the original DA3 decoder, used without modification—drives this home: the geometry is not something the system learned to produce; it is something the latent space inherently supports.

---

Innovation 4: The 4.4× Training Acceleration as Evidence That Latent Space Quality Substitutes for Model Capacity and Data Scale

Training efficiency is often reported as a secondary benefit, but in GLD it serves as diagnostic evidence for a broader claim: that the right latent space reduces the difficulty of the learning problem itself, not just the quality of the final output. The 4.4× training acceleration (Figure 1c) means that GLD reaches the same performance as the VAE baseline in fewer than one-quarter of the training steps. This is not a wall-clock speedup from optimized kernels or reduced parameter count—it is a learning efficiency gain, indicating that the diffusion model's optimization landscape is fundamentally better-conditioned in the DA3 feature space.

Why would this be? The paper's analysis in Section 5.5 and Appendix D.1 provides mechanistic evidence. When diffusion models are trained in the VAE or DINO latent spaces, they must discover cross-view correspondences from scratch—the latent representations of different views are computed independently and share no common geometric reference frame. The diffusion model must learn to map between these unaligned representations, which requires building internal representations of 3D geometry and camera transforms within its attention layers. This is a hard learning problem that requires substantial data and optimization steps.

In the DA3 feature space, the correspondences are pre-aligned by the frozen encoder's 3D attention. The diffusion model starts from a representation where tokens at corresponding 3D locations already have similar feature vectors across views (as quantified by the PCK scores in Table 5). The learning problem shifts from discovering correspondences to refining them—a dramatically easier optimization problem. The cross-view correspondence analysis in Appendix D.1 (Figure 14) confirms this: the DA3-trained diffusion model exhibits stronger internal correspondence across nearly all layers, with the largest margin in the decoder blocks where generation is actually performed. The geometry is not something the model has to build; it is something the latent space provides, and the model's attention layers can focus on appearance and detail refinement.

This has implications beyond NVS. It suggests a general principle for generative modeling: the quality of the latent space can substitute for model capacity, data scale, and training duration. A smaller model trained on less data in a well-structured latent space can match or exceed a larger model trained on more data in a poorly-structured latent space. This principle is the generative analog of the representation learning insight that features matter more than model architecture for many discriminative tasks—but applied to the generative domain where it had not been systematically demonstrated. The fact that GLD remains competitive with fine-tuned T2I models despite training from scratch is not just a performance result; it is evidence that the NVS field has been over-relying on pretraining scale to compensate for suboptimal latent space choices.

This innovation is primarily an empirical discovery with theoretical implications. The 4.4× figure itself is specific to the VAE baseline and the DA3 encoder, but the diagnostic methodology—measuring training convergence speed as a proxy for the difficulty of the learning problem—is general. Future work on latent space design for generation can adopt this methodology: compare convergence rates across candidate latent spaces to identify which representations make the generative task easiest, rather than relying solely on final-generation metrics.

5. Experimental Analysis

Evaluation Methodology

• Dataset. GLD is trained from scratch on four datasets: RealEstate10K (Re10K, 66,033 scenes), DL3DV (10,176 scenes), HyperSim (794 scenes), and TartanAir (369 scenes), mixed at a 4:4:1:1 ratio. Each training sample consists of V = 8 views, with 1 to 4 randomly selected as source views and the remainder masked as targets. For evaluation, the paper uses two in-domain benchmarks (Re10K and DL3DV, which overlap with the training distribution) and one out-of-domain benchmark (Mip-NeRF 360, to test generalization to unseen scene types), with 200 samples per dataset. An additional 50 samples from ETH3D are used for depth evaluation. For datasets with limited evaluation scenes (DL3DV has 55 scenes in its eval set; Mip-NeRF 360 has 9), multiple samples are drawn from the same scene using randomized view sampling.

• Base model(s). The primary backbone is Depth Anything 3 (DA3) Base, a geometric foundation model that extracts L = 4 levels of multi-view features using a ViT encoder with 3D attention. This model was not designed for image generation—it was trained for dense 3D reconstruction from arbitrary unposed views. The paper also validates with VGGT (another geometric foundation model) as an alternative backbone in Appendix C.1. The diffusion model itself is a scratch-trained DiTDH architecture (28 encoder blocks at hidden dimension 768, 6 decoder blocks at hidden dimension 2048 for level-wise models, 2 decoder blocks for the cascaded model). The RGB decoder is a ViT-based transformer with 12 layers, patch size 14, hidden dimension 3072, trained from scratch.

• Metrics. The paper evaluates along two orthogonal axes:

  • 2D image fidelity: PSNR, SSIM, and LPIPS, measuring per-pixel accuracy, structural similarity, and perceptual quality of generated target views against ground truth.
  • 3D geometric consistency: (1) Camera estimation errors—Absolute Trajectory Error (ATE), Relative Pose Error for rotation (RPEr) and translation (RPEt), extracted from generated views using an external camera estimator (VGGT), which directly measures how accurately the generated images adhere to the target pose conditioning; (2) Reprojection error, which measures spatial re-alignment accuracy of reconstructed 3D points; (3) MEt3R, which evaluates multi-view consistency using projected feature similarity across views. For depth evaluation, standard metrics are used: AbsRel (absolute relative error), SqRel (squared relative error), and δ_1 (fraction of pixels with relative error < 1.25). For feature correspondence analysis, PCK (Percentage of Correct Keypoints) measures the fraction of matched points within a distance threshold on ScanNet.

• Baselines. The paper compares against two categories:

  • Latent space baselines (all trained from scratch using the same DiTDH architecture as GLD for the same 175k iterations):
    • VAE: Stable Diffusion's encoder-decoder, representing the standard latent space for diffusion models.
    • DINO: DINOv2 ViT-B/14 with registers as the encoder, with a decoder trained from scratch—representing semantic representation spaces following the RAE line of work.
  • State-of-the-art NVS methods (typically fine-tuned from large-scale text-to-image pretrained models):
    • MVGenMaster (Cao et al., 2025): Uses external depth estimation to warp source RGB and depth as conditioning.
    • Matrix3D (Lu et al., 2025): Jointly generates RGB and depth.
    • CAMEO (Kwon et al., 2025): Correspondence-attention alignment for multi-view diffusion.
    • NVComposer (Li et al., 2025): Boosts NVS with multiple sparse and unposed images.
    • CAT3D† (Gao et al., 2024): A reproduced version using the model and checkpoint from CAMEO, since the official CAT3D implementation is unavailable.

• Generation budget / compute accounting. Training efficiency is compared by measuring accuracy at equivalent training steps (Figure 1c): GLD reaches the VAE baseline's performance at approximately 40k steps, while the VAE baseline requires roughly 175k steps—a 4.4× training acceleration. For inference, latency is reported in seconds per scene (Table 15): GLD takes 66.8 seconds (37.8s for level-1 sampling, 0.15s for propagation, 28.4s for cascaded level-0 sampling, 0.43s for RGB decoding), compared to 35.2s for DINO and 28.0s for VAE. The paper notes that GLD is slower at inference due to requiring two separate diffusion sampling processes (level 1 and level 0), but argues this is offset by training efficiency and zero-shot geometry outputs.

• Cross-validation / statistical protocol. No explicit cross-validation or statistical significance testing is reported. Evaluation uses fixed test splits: 200 samples from Re10K, DL3DV, and Mip-NeRF 360, and 50 samples from ETH3D. The boundary layer analysis in Table 2 uses a comparative evaluation across four configurations on the full test set. Ablation studies vary source view counts (N=1, 2, 4) and report metrics without confidence intervals.

Main Quantitative Results

Reconstruction Fidelity of DA3 Features

Before any diffusion training, the paper must first establish that DA3 features are decodable into high-quality images. Table 1 reports the RGB decoder's reconstruction fidelity on 4,000 samples from the Re10K test set:

• PSNR: 35.41 dB
• SSIM: 0.960
• LPIPS: 0.019

These numbers are remarkably strong. Table 7 contextualizes them against standard autoencoders: the pretrained Stable Diffusion VAE achieves 34.53 PSNR/0.939 SSIM/0.028 LPIPS, and the SDXL VAE achieves 34.97/0.945/0.029. The DA3 decoder outperforms both despite DA3 never being designed for image reconstruction. The DINO RAE decoder achieves only 26.78 PSNR/0.830 SSIM/0.148 LPIPS—substantially worse, confirming that DINO semantic features discard significant appearance information that DA3 features preserve.

Figure 3 provides qualitative confirmation: reconstructed images closely match the originals with fine-grained details preserved. This is the foundational result that justifies the entire GLD framework: if DA3 features could not be decoded accurately, no amount of diffusion quality could compensate.

Boundary Layer Determination

Table 2 reports NVS quality as a function of the synthesis boundary k. Four independently trained diffusion models {M_0, M_1, M_2, M_3} are evaluated with different boundaries, propagating deeper features through the frozen DA3 encoder:

The finding is clear and non-monotonic: moving the boundary from level 0 to level 1 improves all metrics substantially (PSNR +1.06, AbsRel −0.076, δ < 1.25 +0.103). But moving further to level 2 degrades performance across all metrics—synthesizing more levels explicitly is actively harmful. The paper attributes this to the loss of fine-grained spatial detail in deeper features (confirmed by the single-level reconstruction analysis in Table 6: level 2 achieves only 14.01 PSNR vs. level 1's 25.36), which introduces errors that propagate to shallower levels through the cascaded generation.

A subtle point: k=3 (synthesizing all four levels explicitly) performs essentially identically to k=2. This means levels 2 and 3 can be derived from level 1 via propagation without quality loss, but that diffusion cannot improve on this deterministic derivation—the frozen backbone's computation is more accurate than a separately trained diffusion model's sampling. This validates the core design choice: only level 1 needs explicit synthesis; deeper features are better propagated than generated.

Comparison Against Latent Space Baselines

Table 3 presents the main quantitative comparison on in-domain (Re10K, DL3DV) and out-of-domain (Mip-NeRF 360) benchmarks with N=2 source views.

On Re10K (in-domain):

• GLD achieves PSNR 16.362, SSIM 0.630, LPIPS 0.431—the best across all methods including the fine-tuned T2I baselines.
• Compared to the VAE baseline (PSNR 15.656, SSIM 0.606, LPIPS 0.456), GLD improves PSNR by +0.706 and LPIPS by −0.025.
• Compared to the DINO baseline (PSNR 15.638, SSIM 0.601, LPIPS 0.448), GLD improves PSNR by +0.724 and SSIM by +0.029.
• The 3D metrics show the most dramatic gaps: GLD achieves ATE 0.211 vs. VAE's 0.278 (1.3× better) and DINO's 0.345 (1.6× better). RPEr is 7.07 vs. VAE's 8.68 (1.2× better) and DINO's 15.59 (2.2× better).

On DL3DV (in-domain):

• GLD achieves PSNR 15.499, SSIM 0.468, LPIPS 0.438—again the best among all methods.
• The VAE baseline achieves PSNR 14.725, SSIM 0.446, LPIPS 0.476. GLD's advantage narrows slightly but remains consistent.
• The 3D advantage is larger: ATE 0.209 vs. VAE's 0.589 (2.8× better) and DINO's 0.546 (2.6× better). RPEr is 5.75 vs. VAE's 15.00 (2.6× better) and DINO's 13.12 (2.3× better).
• MEt3R (0.378 vs. VAE's 0.407 vs. DINO's 0.410) and reprojection error (0.612 vs. VAE's 0.674 vs. DINO's 0.708) show consistent improvement.

On Mip-NeRF 360 (out-of-domain):

• GLD achieves PSNR 14.542, SSIM 0.288, LPIPS 0.504—competitive with the best fine-tuned methods (MVGenMaster: 14.170/0.304/0.511) and better than all scratch-trained baselines (VAE: 13.942/0.274/0.548; DINO: 13.718/0.267/0.542).
• The 3D advantage persists: ATE 0.589 vs. VAE's 1.221 (2.1× better) and DINO's 0.949 (1.6× better). RPEr 15.97 vs. VAE's 35.34 (2.2× better) and DINO's 27.57 (1.7× better).
• Reprojection error (0.630) and MEt3R (0.406) are the best among all methods including fine-tuned baselines, suggesting GLD's geometric consistency generalizes even when the content domain shifts (scene-level training → object-centric evaluation).

Comparison against fine-tuned T2I models. GLD surpasses all fine-tuned baselines across all 2D metrics on both in-domain benchmarks, despite being trained from scratch on smaller datasets. On Re10K, GLD's PSNR 16.362 beats MVGenMaster's 15.226, Matrix3D's 14.490, CAMEO's 13.800, and NVComposer's 11.140. On DL3DV, the margin is similar: GLD 15.499 vs. MVGenMaster 14.565, Matrix3D 13.330, CAMEO 12.320. For 3D metrics, GLD achieves significantly lower ATE than all fine-tuned methods on both in-domain datasets (e.g., Re10K ATE 0.211 vs. MVGenMaster 0.282, Matrix3D 0.413, CAMEO 0.446). On the out-of-domain Mip-NeRF 360, GLD remains competitive with MVGenMaster (the strongest fine-tuned baseline) despite the domain shift, and achieves the best reprojection error (0.630 vs. MVGenMaster's 0.676).

Comparison Against Scratch-Trained MVGenMaster

Table 14 provides a controlled comparison: MVGenMaster trained from scratch (same training setup as GLD) vs. GLD on Mip-NeRF 360:

The gap is dramatic: scratch-trained MVGenMaster collapses to PSNR 11.217 (vs. 14.170 fine-tuned), with ATE exploding to 1.804 (vs. 0.320 fine-tuned). GLD, trained from scratch, achieves PSNR 14.542—better than fine-tuned MVGenMaster—and ATE 0.589. This demonstrates that GLD's geometric latent space is not just helpful for scratch-trained models; it specifically addresses the fragility that makes methods like MVGenMaster heavily dependent on T2I pretraining.

Depth and 3D Reconstruction Quality

Table 9 compares GLD's zero-shot depth predictions (from the original DA3 geometric decoder applied to synthesized features) against Matrix3D's jointly-trained depth outputs on ETH3D:

GLD produces more accurate depth across all metrics, with AbsRel 0.160 vs. Matrix3D's 0.197 (19% reduction in relative error) and δ₁ 0.800 vs. 0.731 (9.4% more pixels within the 25% error threshold). This is notable because Matrix3D's depth prediction is jointly trained with its RGB output—it has explicit depth supervision—while GLD's depth emerges zero-shot from the geometric decoder applied to synthesized features.

Figures 6 and 7 provide qualitative 3D reconstruction comparisons. GLD's point clouds, unprojected from synthesized RGB-D using the generated depth and ray maps, exhibit consistent 3D geometry across diverse camera trajectories. In contrast, Matrix3D produces "noticeable geometric distortions." Specific failure cases for Matrix3D in Figure 7 include duplicated objects and misaligned structural elements, while GLD yields "significantly cleaner and more coherent reconstructions."

Training Efficiency

Figure 1c reports training convergence curves: GLD achieves the same PSNR at approximately 40k training steps that the VAE baseline requires 175k steps to reach—a 4.4× acceleration. The curve continues to improve beyond this point, suggesting the efficiency gain is not just early-stage but persists throughout training. The DINO baseline shows intermediate convergence behavior.

Varying Number of Source Views

Table 4 evaluates robustness across N = 1, 2, 4 source views on Re10K and DL3DV:

N=1 (single source view):

• On Re10K: GLD achieves PSNR 13.50 vs. VAE's 12.99 and DINO's 13.49. SSIM is 0.552 (GLD) vs. 0.519 (VAE) vs. 0.543 (DINO). The 3D advantage is substantial: ATE 0.267 vs. 0.371 (VAE) and 0.442 (DINO).
• On DL3DV: GLD achieves PSNR 13.03 vs. VAE's 12.47 and DINO's 12.80. ATE 0.237 vs. 0.880 (VAE, 3.7× better) and 0.743 (DINO, 3.1× better).

N=4 (four source views):

• On Re10K: GLD achieves PSNR 19.00 vs. VAE's 18.68 and DINO's 17.73. 3D improvements narrow but persist: ATE 0.182 vs. 0.200 (VAE) and 0.221 (DINO).
• On DL3DV: GLD achieves PSNR 17.09 vs. VAE's 16.37 and DINO's 15.44. ATE 0.143 vs. 0.294 (VAE, 2.1× better) and 0.274 (DINO, 1.9× better).

The key trend: the margin of improvement in 3D metrics grows as source views decrease. With N=1 on DL3DV, GLD achieves over 3× lower ATE and RPEr compared to both baselines; with N=4, the gap is approximately 2×. This suggests that the geometric priors in the DA3 feature space become particularly valuable when fewer visual cues are available—precisely the regime where discovering cross-view correspondences from independently-encoded latents is hardest.

Ablation Studies and Robustness Checks

Cascaded vs. independent level-0 generation (Table 8). On Re10K with N=4 source views, replacing the cascaded M_{1→0} with independent M_0 and M_1 degrades all metrics: PSNR drops from 19.00 to 18.81, SSIM from 0.695 to 0.692, and LPIPS increases from 0.327 to 0.335. 3D metrics show larger relative drops: ATE increases from 0.182 to 0.197 (8.2% worse), RPEr from 6.694 to 7.179 (7.2% worse), RPEt from 0.397 to 0.430 (8.3% worse). This confirms that feature-level misalignment from independent generation harms geometric consistency more than pixel-level fidelity.

VGGT backbone (Table 13, Appendix C.1). Replacing DA3 with VGGT as the geometric foundation model:

• On Re10K: GLD w/ VGGT achieves PSNR 16.17 (vs. DA3's 16.36), ATE 0.216 (vs. 0.211). Both outperform VAE (15.66, 0.278) and DINO (15.64, 0.345).
• On DL3DV: GLD w/ VGGT achieves PSNR 15.25 (vs. DA3's 15.50), ATE 0.188 (vs. 0.209). Again, both substantially outperform VAE (14.73, 0.589) and DINO (14.35, 0.546).
• On Mip-NeRF 360: GLD w/ VGGT achieves PSNR 13.57, ATE 0.596—slightly weaker than DA3 (14.54, 0.589) but still dramatically better than VAE (13.94, 1.221) and DINO (13.72, 0.949).

The consistent pattern—VGGT backbone outperforms VAE and DINO, but slightly underperforms DA3—demonstrates that the core principle (geometric latent spaces improve NVS) generalizes across geometric foundation model architectures. The performance difference between backbones likely reflects DA3-specific properties (training data, architecture details) rather than a failure of the general principle.

Feature correspondence in trained diffusion models (Appendix D.1, Figure 14). Measuring cross-view correspondence (PCK) in the internal 3D attention maps of trained diffusion models reveals: the DA3-latent model exhibits stronger correspondence across nearly all layers compared to VAE-latent and DINO-latent models, with the largest margin in the velocity decoder blocks. Importantly, correspondence is essentially absent in the condition encoder for all three latent spaces and emerges only in the velocity decoder—suggesting that cross-view geometric reasoning is primarily performed by the decoder during denoising, not encoded by the conditioning pathway. The DA3 latent space makes it easier for the decoder to establish this correspondence, consistent with the training acceleration claim.

Single-level reconstruction analysis (Table 6, Section 5.5). Using the level-wise dropout-trained decoder to reconstruct from individual feature levels quantifies photometric information content per level:

• Level 0: PSNR 28.01, SSIM 0.922, LPIPS 0.138
• Level 1: PSNR 25.36, SSIM 0.873, LPIPS 0.138
• Level 2: PSNR 14.01, SSIM 0.627, LPIPS 0.491
• Level 3: PSNR 10.19, SSIM 0.508, LPIPS 0.768

The sharp drop between level 1 and level 2 (PSNR −11.35, LPIPS +0.353) quantifies the "photometric cliff" that makes level 2 unsuitable as a synthesis boundary. Figure 5 visualizes this: level 2 and 3 reconstructions show noticeable color loss and smoothed textures, while levels 0 and 1 preserve detailed appearance.

Geometric correspondence per level (Table 5, Section 5.5). PCK on ScanNet reveals the opposite trend:

• Level 0: 22.25
• Level 1: 35.98
• Level 2: 40.70
• Level 3: 20.98
• DINOv2: 31.64

Level 1 achieves 88% of level 2's geometric correspondence (35.98 vs. 40.70) while preserving substantially more photometric information. Level 3 surprisingly underperforms level 2, likely because the deepest features become too abstract for precise spatial correspondence. DINOv2 (31.64) sits between level 0 and level 1, confirming that semantic representations encode more geometric structure than shallow DA3 features but less than mid-level geometric features.

Scratch-trained MVGenMaster comparison (Table 14). The negative result for MVGenMaster trained from scratch is itself informative: without T2I pretraining, MVGenMaster's warping-based approach becomes fragile, with ATE exploding from 0.320 to 1.804. This provides evidence that the reliance on external depth estimators creates a brittle dependency that pretraining can mask but not eliminate.

Critical Assessment

Does the 4.4× training acceleration claim hold up?

The paper reports that GLD "accelerating training by more than 4.4× compared to the VAE latent space" (Abstract) and Figure 1c visually shows GLD reaching the VAE baseline's final PSNR at approximately 40k steps while the VAE baseline requires 175k steps. This is reasonable but has an important caveat: the comparison is against a VAE baseline trained with the same DiTDH architecture, not against the full VAE-latent diffusion pipeline as deployed in prior NVS work (which might include different architectures, training recipes, or data). The 4.4× figure should be understood as "GLD with our architecture converges 4.4× faster than the same architecture in VAE latent space," not "GLD is 4.4× faster than any VAE-based NVS method." The paper does not report training time in GPU-hours or FLOPs, only in steps, which is acceptable if per-step costs are comparable (the VAE baseline uses patch size 2 vs. DA3's patch size 1, so token counts differ, but both use the same DiTDH design).

Does GLD genuinely outperform state-of-the-art fine-tuned methods?

The paper's headline claim—"GLD remains competitive with state-of-the-art methods that leverage large-scale text-to-image pretraining"—is well-supported by Table 3. GLD achieves the best PSNR, SSIM, and LPIPS on both in-domain benchmarks, and achieves the best ATE on all three benchmarks. However, several nuances matter:

1. The fine-tuned baselines are not equally strong on all metrics. MVGenMaster, for instance, uses external depth warping and achieves stronger 3D metrics than other fine-tuned methods. On Re10K, MVGenMaster's ATE is 0.282 vs. GLD's 0.211—a clear win for GLD. But on Mip-NeRF 360, MVGenMaster achieves RPEr 10.92 vs. GLD's 15.97, meaning the fine-tuned model maintains better rotation accuracy out-of-domain. GLD's advantage is most consistent on in-domain data; out-of-domain, it is competitive but not uniformly superior.

2. Training data mismatch. GLD is trained on four scene-level datasets (Re10K, DL3DV, HyperSim, TartanAir). Some fine-tuned baselines (MVGenMaster, Matrix3D, NVComposer) incorporate object-centric data (Objaverse, CO3D) during fine-tuning, which may explain their relatively stronger performance on Mip-NeRF 360 (which contains object-centric scenes). The paper acknowledges this (Section 5.2: "GLD is trained exclusively on scene-level data, whereas competing baselines incorporate object-centric datasets"). This makes GLD's out-of-domain performance more impressive—it generalizes without seeing object-centric data—but also means the comparison slightly favors GLD on scene-level in-domain benchmarks where its training distribution matches.

3. CAT3D† is a reproduction, not the original. The official CAT3D implementation was unavailable, so the paper uses a reproduction from CAMEO. If the reproduction underperforms the original, GLD's advantage over CAT3D specifically may be overstated.

Is the boundary layer analysis robust?

Table 2 uses a single evaluation (200 Re10K samples at N=2 source views, with independently trained level-wise models) to determine k=1 as optimal. The finding that level 1 provides the best tradeoff is well-justified by the complementary analyses in Tables 5 and 6 (geometric correspondence and photometric fidelity). However, the boundary evaluation has several limitations:

1. The independently trained level-wise models may not represent the cascaded system's behavior. The boundary analysis uses four separately trained M_l models, but the final GLD system uses M_1 and a cascaded M_{1→0} trained with noisy level-1 conditioning. The optimal boundary might shift if evaluated with the actual cascaded architecture, since the cascaded model's level-0 generation quality depends on level-1 quality differently than independent models.

2. Single dataset, single backbone. The boundary is determined on DA3 features from Re10K. Whether k=1 remains optimal for VGGT or DL3DV is not evaluated. The PCK/photometry analysis in Tables 5 and 6 is also DA3-specific.

3. No continuous boundary analysis. The paper evaluates four discrete boundaries (0, 1, 2, 3). A finer-grained analysis (e.g., testing intermediate levels if the backbone provides more than 4 feature maps) is not possible with DA3's architecture but would strengthen the claim that level 1 is genuinely the optimal tradeoff point rather than simply the best among four options.

Do the 3D metrics genuinely measure geometric consistency?

The paper uses ATE, RPEr, and RPEt from an external camera estimator (VGGT) applied to the generated views. This is a reasonable proxy: if the generated views are geometrically consistent with the target camera poses, a camera estimator should recover those poses accurately. However, this metric measures condition fidelity (did the model generate the view it was asked to generate?) rather than intrinsic 3D consistency (would any two generated views be consistent with each other, even if not exactly at the specified poses?). A model could achieve good ATE by memorizing typical view transitions from the training data, without truly understanding the underlying 3D geometry. The reprojection error and MEt3R metrics partially address this by measuring pairwise consistency between generated views without reference to ground-truth poses, and GLD's advantage persists on these metrics—which is reassuring.

What experiments are missing?

1. Per-scene breakdown by difficulty. The paper reports aggregate metrics across 200 samples per dataset. It would be informative to see whether GLD's advantage over VAE/DINO baselines is uniform across scene types (indoor vs. outdoor, small vs. large baselines, texture-rich vs. texture-poor). The qualitative results suggest GLD handles large viewpoint changes particularly well (Figure 4), but this is not quantified.

2. Ablation on 3D attention. The paper uses 3D self-attention in the diffusion model, following CAT3D. How much of GLD's geometric consistency comes from the DA3 latent space vs. the 3D attention mechanism? An ablation training GLD without 3D attention would disentangle these contributions. If most of the gain persists without 3D attention, that strengthens the claim that the latent space itself is the primary driver.

3. Ablation on PRoPE. Similarly, the camera-relative positional encoding (PRoPE) could be contributing to geometric consistency independently of the latent space. Training GLD with standard sinusoidal positional encoding would isolate the latent space effect.

4. Comparison against stronger RAE baselines. The paper uses DINOv2 ViT-B/14 as the semantic RAE baseline. Newer and stronger semantic encoders exist (DINOv3, SigLIP 2). A DINOv3 baseline would test whether the geometric-vs-semantic distinction holds with state-of-the-art semantic representations, or whether sufficiently powerful semantic features can match geometric features.

5. Inference cost amortization. The paper acknowledges GLD is slower at inference (66.8s vs. 28.0s for VAE) but argues training efficiency offsets this. For latency-sensitive applications, this tradeoff could be unfavorable. The paper does not explore whether inference can be accelerated (e.g., fewer sampling steps, distillation) without sacrificing the geometric consistency gains.

6. Diversity and stochasticity analysis. NVS is fundamentally a one-to-many problem for occluded regions. The paper evaluates only reconstruction metrics (PSNR, SSIM) against a single ground-truth target view, which penalizes diverse but valid samples. Generating multiple samples and measuring diversity metrics (or FID against a distribution of valid views) would test whether GLD maintains generation diversity or collapses to mode-averaged outputs.

Does the 4.4× training acceleration actually matter in practice?

The training acceleration claim is technically valid but its practical significance depends on context. If the total training cost is dominated by data loading, preprocessing, or the frozen encoder's forward passes (which must be run for both GLD and the VAE baseline during data preparation), then the diffusion model's step count may not be the bottleneck. The paper does not report total wall-clock training time or FLOPs, only convergence in steps—which is standard for such comparisons but limits the operational meaning of the 4.4× figure. That said, the convergence speed serves primarily as diagnostic evidence for the paper's central thesis (that geometric latent spaces make the learning problem easier), and for that purpose, step-based comparison is sufficient.

6. Limitations and Trade-offs

6.1 Inference Latency: GLD Is 2.4× Slower Than VAE Baselines at Generation Time

The headlining speed advantage of GLD is training convergence — 4.4× faster than VAE and DINO baselines in steps-to-match-performance. However, the paper openly acknowledges that this advantage does not carry over to inference, where wall-clock latency matters for deployment. Table 15 reports per-scene generation times: GLD requires 66.8 seconds, compared to 28.0 seconds for the VAE baseline and 35.2 seconds for DINO — a 2.4× increase over VAE. The breakdown in Table 15(B) reveals why: GLD performs two sequential diffusion sampling processes — 37.8 seconds for the level-1 sampling via $\mathcal{M}_1$, then another 28.4 seconds for the cascaded level-0 sampling via $\mathcal{M}_{1 \rightarrow 0}$. Neither process can start before the previous one finishes; they form a serial dependency chain, while VAE operates in a single latent space requiring only one diffusion pass.

The consequence is practical: for latency-sensitive applications — interactive novel view synthesis, real-time rendering, or any deployment where users wait for output — GLD's serial two-stage generation makes it substantially less attractive than VAE-based alternatives, regardless of its training efficiency or geometric consistency advantages. The paper does not explore whether inference can be accelerated without sacrificing quality (e.g., through fewer sampling steps, distillation, or parallelizing the two stages). This is an unreconciled tradeoff: the representation that enables better geometric consistency and faster training also imposes a structural latency penalty that the current architecture cannot circumvent.

Evidence in the paper: Table 15 explicitly compares total inference latency across methods; the breakdown in Table 15(B) reveals the serial nature of the bottleneck.

Mitigation status: Not addressed. The authors note the propagation step (0.15s) is negligible and validate their decision to avoid a third explicit synthesis stage, but they do not propose any mechanism to reduce the two remaining diffusion passes or reorder them for parallelism. The paper frames the tradeoff implicitly — training speed and zero-shot geometry outputs offset inference cost — but does not resolve it for latency-critical settings.

---

6.2 Difficulty Estimation Cost Is Not Accounted for — But the Analogue Here Is Dependence on a Specific Frozen Foundation Model

This limitation is structural rather than computational. GLD's entire framework depends on a single geometric foundation model (primarily Depth Anything 3, with VGGT explored in Appendix C.1) whose architecture, training data, and feature hierarchy define the properties — both strengths and weaknesses — of the latent space. The paper demonstrates that DA3 works and VGGT also works, but provides no analysis of what happens if neither is available, if a user wants to use a different geometric backbone, or if the backbone's training data distribution does not match the deployment domain.

The consequence is a hard dependency. The boundary layer selection (k = 1), the specific PCK and photometric fidelity tradeoffs (Tables 5-6), the reconstruction decoder architecture, and even the number of feature levels (L = 4) are all tied to the specific ViT architecture and training procedure of DA3. A practitioner who wants to deploy GLD must first have access to a compatible geometric foundation model — and if that model is updated, retrained, or deprecated, the entire pipeline may need to be re-validated. Furthermore, the paper does not characterize whether the strong cross-view correspondences measured in Table 5 degrade when the foundation model encounters scenes or camera configurations outside its training distribution. If DA3's 3D attention produces noisy or misaligned features for certain scene types, those errors will propagate through the diffusion model and the deterministic backbone propagation — there is no fallback mechanism.

Evidence in the paper: The VGGT experiment in Appendix C.1 provides the only generalization evidence across backbones. The paper does not evaluate GLD with non-geometric foundation models (e.g., would a video model's spatio-temporal features work?), leaving the boundary of the "geometric latent space" principle untested.

Mitigation status: Partial — the VGGT replication demonstrates the principle generalizes to at least one alternative backbone. However, the paper does not provide guidance on how to validate a new backbone (what subset of Tables 5-6 analysis to repeat), how to anticipate sensitivity to backbone updates, or how to design a system that is robust to backbone degradation. Section 4.2 notes that the geometric encoder is frozen, but does not discuss failure modes of this freezing choice.

---

6.3 Domain Specialization: Training on Scene-Level Data Limits Generalization to Object-Centric Views

The paper acknowledges explicitly (Section 5.2) that GLD "is trained exclusively on scene-level data, whereas competing baselines incorporate object-centric datasets during fine-tuning." This is both a transparency admission and a capability boundary. The four training datasets — RealEstate10K, DL3DV, HyperSim, TartanAir — all consist of indoor and outdoor scenes with relatively structured camera trajectories and multi-view coverage. Object-centric datasets like Objaverse or CO3D, which feature isolated objects with 360-degree viewpoints, are excluded entirely from GLD's training.

The consequence is visible in the out-of-domain results on Mip-NeRF 360 (Table 3). While GLD still outperforms scratch-trained baselines, its margin over fine-tuned T2I models narrows. On Re10K (in-domain), GLD achieves ATE 0.211 vs. MVGenMaster's 0.282 — a clear advantage. On Mip-NeRF 360 (out-of-domain, containing object-centric scenes), GLD's ATE is 0.589 vs. MVGenMaster's 0.320 — the fine-tuned model, which saw object-centric data during training, is now better on this metric. The same pattern appears for rotation error: RPEr 15.97 (GLD) vs. 10.92 (MVGenMaster). The geometric latent space helps, but it does not fully compensate for the domain gap between scene-level training and object-centric evaluation. A practitioner wanting to generate novel views of isolated objects — a common use case in e-commerce, 3D asset creation, or product visualization — would likely find GLD's performance below methods that explicitly train on object data.

Evidence in the paper: Section 5.2 explicitly states the scene-level training limitation; Table 3 (Mip-NeRF 360) and Table 14 quantify the out-of-domain performance gap relative to fine-tuned methods.

Mitigation status: Acknowledged but not addressed. The paper does not experiment with adding object-centric data to GLD's training or evaluate what modifications would be needed to make the framework effective for objects. Given that geometric foundation models like DA3 are trained on both scene and object data, this is likely a fixable limitation — the bottleneck is training compute and dataset curation, not a fundamental architectural constraint.

---

6.4 The Single Benchmark, Single Backbone Analysis Limits Claims of Generality

All primary experiments use the MATH benchmark as the evaluation... [correction: this is a different paper — re-anchor to the actual domain]. All primary experiments establish the optimality of boundary level k = 1 using a single backbone (DA3-Base) on three scene-level benchmarks (Re10K, DL3DV, Mip-NeRF 360). The training data covers four datasets, but the evaluation is confined to real-world indoor and outdoor scenes. The paper does not evaluate on:

• Object-centric benchmarks (CO3D, Objaverse renders, GSO)
• Synthetic scenes with controlled geometry complexity
• Dynamic scenes or videos
• Tasks beyond novel view synthesis that require geometric latent spaces (e.g., 3D reconstruction from generated latents, multi-view stereo, depth estimation benchmarks beyond ETH3D)

The consequence is that the paper's central claim — "geometric foundation model features provide a better latent space for geometry-aware generation" — is supported for scene-level NVS with DA3-Base and VGGT, but its generality to other geometric tasks, other backbone sizes (DA3-Large, DA3-Huge), or other data domains is unproven. The boundary layer analysis (Section 5.5) relies on DA3's specific 4-level feature hierarchy; other geometric models may have 3, 5, or more levels with different tradeoff curves, and k = 1 may not remain optimal. A practitioner trying to adapt GLD to a new domain would need to replicate the entire boundary layer analysis (training four independent diffusion models, measuring PCK and single-level reconstruction fidelity) without knowing whether the finding will transfer.

Evidence in the paper: Section 5.1 lists the evaluation datasets (Re10K, DL3DV, Mip-NeRF 360, ETH3D for depth); all are scene-level. The VGGT experiment (Appendix C.1) provides cross-backbone evidence, but the benchmarks remain the same. The paper does not claim generality beyond what is tested.

Mitigation status: The VGGT replication partly addresses backbone-specificity concerns, but domain-specificity is unaddressed. The paper frames itself as "encouraging further investigation into task-specific latent space design" (Section 6), implicitly acknowledging that its specific findings are a starting point rather than a complete characterization.

---

6.5 The Cascaded Generation Introduces a Serial Dependency That Compounds Error Across Levels

The cascaded design (Section 4.4) is both a key contribution and a structural vulnerability. The model $\mathcal{M}_{1 \rightarrow 0}$ is trained to generate level-0 features conditioned on level-1 features, using noisy ground-truth level-1 conditioning during training to simulate inference conditions. This is a standard domain randomization strategy, but it has a subtle failure mode: errors in $\tilde{\mathbf{F}}_1$ can be amplified rather than corrected by the cascaded model. If the level-1 synthesis produces a feature that misrepresents the geometry — say, placing a wall at the wrong depth — then $\mathcal{M}_{1 \rightarrow 0}$ will generate level-0 features consistent with that incorrect geometry. The RGB decoder will then produce an image that looks photorealistic (because the level-0 features have high photometric fidelity) but is geometrically wrong. There is no feedback loop or consistency check that could detect and correct such errors after level-1 synthesis is complete.

The paper provides evidence for this concern in two places. First, Table 8 shows that independent generation (where level 0 is not conditioned on level 1) degrades 3D metrics more than 2D metrics (ATE increases by 8.2% vs. PSNR drop of 0.55%), suggesting that cascading ties the geometric accuracy of level 0 to the geometric accuracy of level 1. If level 1 is correct, this is beneficial; if level 1 is wrong, the error propagates. Second, the Appendix D.1 analysis (Figure 14) shows that cross-view correspondence in the diffusion model's internal attention maps is strongest in the velocity decoder's intermediate blocks, not at the output. This means the model is building geometric understanding during denoising, but the final synthesized features are a point estimate with no associated uncertainty — the cascaded model cannot know which regions of $\tilde{\mathbf{F}}_1$ are reliable versus uncertain.

Consequence: GLD's outputs can be visually convincing but geometrically inconsistent in regimes where level-1 synthesis struggles. The paper's qualitative results (Figure 4) show some artifacts in challenging cases, but these are not systematically categorized as level-1 failures vs. level-0 failures vs. decoder failures. A practitioner cannot easily diagnose whether a bad output stems from a bad level-1 prediction (which might be addressed by improving $\mathcal{M}_1$) or a bad cascaded refinement (which might be addressed by training $\mathcal{M}_{1 \rightarrow 0}$ with more aggressive noise).

Evidence in the paper: Table 8 quantifies the cascaded vs. independent gap; Appendix D.1 Figure 14 shows correspondence emerging mid-decoder; the noisy conditioning training strategy is described in Section 4.4 but not ablated (e.g., what happens with clean conditioning vs. different noise levels?).

Mitigation status: Partial. The noisy conditioning during $\mathcal{M}_{1 \rightarrow 0}$ training is the primary mitigation, but its effectiveness is not evaluated in isolation. The paper does not explore alternatives such as iterative refinement between levels (allowing level 0 synthesis to "correct" level 1), uncertainty estimation, or consistency checks between synthesized features and propagated features.

---

6.6 Unresolved Tension Between the Cascaded Design and the Independent Level-Wise Training Paradigm

A methodological subtlety cuts across the boundary layer determination (Section 4.3) and the final cascaded system (Section 4.4): the boundary layer analysis in Table 2 uses four independently trained diffusion models $\{\mathcal{M}_l\}_{l=0}^3$, but the deployed GLD system uses $\mathcal{M}_1$ (unchanged) and $\mathcal{M}_{1 \rightarrow 0}$ (trained with noisy level-1 conditioning). These are not the same models. $\mathcal{M}_0$ in the boundary analysis was trained to generate level-0 features from source-only conditioning with no awareness of level-1 features; $\mathcal{M}_{1 \rightarrow 0}$ is trained to generate level-0 features conditioned on level-1 features (which may be noisy). There is no guarantee that the optimal boundary identified with independent models remains optimal — or even valid — when evaluated with the cascaded architecture.

The consequence is an internal inconsistency in the paper's methodology. The boundary selection that motivates the entire cascaded design (k = 1 is optimal) is based on a configuration that is not the same as the deployed system. The paper never reports an ablation where the boundary is re-evaluated using the actual cascaded architecture — e.g., training $\mathcal{M}_{2 \rightarrow 1 \rightarrow 0}$ (cascaded from level 2 down) and comparing it against the level-1 boundary system. It is possible that deeper boundaries, which were rejected based on independent model evaluation, would perform differently — perhaps even better — with a properly cascaded architecture that mitigates the photometric information loss through careful conditioning. This gap does not invalidate the paper's core claims (the independent-model evaluation is internally valid for comparing latent spaces, and the cascaded system outperforms baselines), but it weakens the optimality claim for k = 1 specifically. A practitioner who follows the paper's methodology exactly — train independent models, evaluate boundaries, fix k = 1 — is implementing a pipeline that was optimized with a proxy, not with the actual cascaded architecture they will deploy.

Evidence in the paper: Section 4.3 describes training four independent models $\{\mathcal{M}_l\}_{l=0}^3$ for the boundary evaluation; Section 4.4 then replaces $\mathcal{M}_0$ with $\mathcal{M}_{1 \rightarrow 0}$ without re-running the boundary analysis. The ablations in Table 8 only compare cascaded vs. independent at the already-chosen boundary (k = 1), not at other boundaries.

Mitigation status: Not addressed. The paper implicitly assumes that the boundary ranking from independent models transfers to the cascaded setting, but this assumption is neither stated nor tested. This is a methodological gap rather than a practical failure — the system still works well — but it means that future work adopting GLD's framework should re-validate the boundary choice with the actual cascaded architecture rather than treating k = 1 as universally optimal.

7. Implications and Future Directions

How This Work Changes the Landscape

GLD reframes the novel view synthesis problem from "how do we inject geometry into a compression-based generative pipeline?" to "which existing representation already encodes the geometric structure we need, and how do we build generation around it?" This is not an incremental refinement of the VAE-latent diffusion paradigm—it is a categorical shift in what the latent space is expected to provide. The paper's evidence that this shift matters is concrete: the right latent space accelerates training by 4.4× (Figure 1c), improves 3D consistency by up to 2.8× on ATE (Table 3, DL3DV), and enables zero-shot geometry decoding that outperforms jointly-trained depth predictors (Table 9, AbsRel 0.160 vs. Matrix3D's 0.197). These are not marginal gains from a better architecture or loss function—they emerge from changing what representation the diffusion model operates on.

The conceptual contribution is that GLD resolves a tension the NVS community has lived with since the introduction of multi-view diffusion: photorealism and geometric consistency are not competing objectives that must be traded off through careful loss balancing or external conditioning. They can be unified not by modifying the generation process, but by choosing a latent space where they are co-encoded from the start. Prior work addressed the geometry gap through two strategies—external warping of depth-conditioned features (MVGenMaster, GenWarp, ViewCrafter) or joint training of appearance and geometry decoders (Matrix3D, ORCHID). Both approaches treat geometry as something to be added to an appearance-first pipeline, creating dependency chains (the warping approach relies on depth estimator accuracy; the joint-training approach requires specialized architectures and multi-task losses). GLD demonstrates that neither strategy is necessary if the latent space itself carries geometric structure. The paper's VGGT replication (Appendix C.1, Table 13) confirms this is not an artifact of DA3's specific architecture—it is a property of geometric foundation model features broadly.

This reframing has several downstream effects on the research landscape:

1. The fine-tuning-from-T2I paradigm is called into question—not as uniformly wrong, but as potentially unnecessary. Table 3 shows GLD matching or exceeding fine-tuned T2I models (MVGenMaster, Matrix3D) on in-domain benchmarks while training from scratch on smaller datasets. Table 14 makes this point sharper: MVGenMaster trained from scratch collapses to PSNR 11.217 (vs. 14.170 fine-tuned), while GLD trained from scratch achieves 14.542 on the same out-of-domain benchmark. This suggests that the NVS field's reliance on text-to-image pretraining is partly a consequence of using a latent space (VAE) that is poorly suited to the task—the pretrained weights compensate for the representation's lack of geometric structure. GLD demonstrates that this compensation is not the only path: a geometry-aware latent space can substitute for pretraining scale, making high-quality NVS accessible to groups without access to proprietary T2I model weights or the compute to fine-tune them.

2. The boundary between discriminative and generative feature spaces becomes porous. DA3 was designed and trained for 3D reconstruction—a discriminative task. The fact that its internal features support high-fidelity image reconstruction at PSNR 35.41 (Table 1, exceeding the Stable Diffusion VAE's 34.53) and can be diffused to generate novel views with state-of-the-art quality means that the distinction between "features for understanding" and "features for generation" is not fundamental. This opens a new design methodology: for any generation task that requires task-specific invariances (temporal consistency for video, spatial consistency for 3D, semantic alignment for text-to-image), look for a foundation model whose training objective already requires encoding those invariances, validate its features' reconstruction fidelity, and build generation in that space. The paper does not claim this methodology is universal, but it provides a worked example and the diagnostic tools (PCK for geometric correspondence, single-level reconstruction for photometric fidelity) to apply it elsewhere.

3. The "optimal latent space" question is reframed from compression efficiency to task-specific structure. The dominant narrative in latent diffusion has been that the latent space should compress images while preserving appearance—a legacy of the VAE's origins in perceptual compression. GLD demonstrates that for multi-view generation, the latent space's primary job is not compression (DA3 features have dimension 1536 per token at patch resolution, substantially larger than the VAE's latent channels) but cross-view alignment. The PCK analysis in Table 5 shows that this alignment varies dramatically across feature levels (22.25 at level 0, 40.70 at level 2), and that the optimal choice (level 1, PCK 35.98) is not the most compressed or the most semantically abstract level, but the one that balances geometric correspondence with photometric fidelity. This suggests a general principle: the optimal latent for a conditional generation task should encode the conditioning-relevant structure (here, cross-view geometric correspondence) while preserving enough output-relevant information (here, photometric detail) for faithful decoding. The paper's dual-property analysis (Tables 5-6) provides a template for operationalizing this principle.

4. Research directions become more attractive or less attractive in light of these results. The paper makes geometry injection through external conditioning (warping, depth estimators) less attractive as a primary research direction—GLD achieves better geometric consistency without any external geometry at inference time, and the qualitative results (Figure 4) show that external depth estimators introduce failure modes (warping artifacts) that the geometric latent space avoids. The paper makes architectural innovations for multi-view attention no less important—GLD still uses 3D self-attention and PRoPE—but shifts the emphasis from "how can attention layers discover geometry?" to "given that the latent already encodes geometry, how can attention layers refine it?" The cross-view correspondence analysis in Appendix D.1 (Figure 14) directly supports this: the DA3-latent model achieves stronger internal correspondence across all layers, but the correspondence still emerges primarily in the velocity decoder rather than being trivially present in the encoder. The problem becomes easier but does not vanish.

The paper also makes latent space design for specific tasks a more attractive general research direction. If the RAE line of work showed that semantic representations improve 2D image generation, GLD shows that geometric representations improve 3D-aware generation. The natural extension is to other task-structure pairs: optical flow features for video generation, depth-disentangled features for view synthesis under lighting changes, multi-modal features for audio-visual generation. The methodology—identify a foundation model whose training objective already encodes the target structure, validate reconstruction fidelity, determine the optimal feature level through dual-property analysis—is directly portable.

---

Follow-Up Research This Work Enables

Disentangling the contributions of the geometric latent space vs. 3D attention vs. PRoPE. GLD uses all three mechanisms—DA3 features, 3D self-attention in the diffusion model, and camera-relative positional encoding (PRoPE). The paper does not ablate any of them against the others. A controlled experiment would train four variants of the diffusion model: (1) DA3 latent + 3D attention + PRoPE (the full GLD system), (2) DA3 latent + standard self-attention + PRoPE (isolating 3D attention), (3) DA3 latent + 3D attention + sinusoidal positional encoding (isolating PRoPE), and (4) DA3 latent + standard self-attention + sinusoidal positional encoding (removing both geometric inductive biases from the diffusion model). If variant (4) still substantially outperforms the VAE baseline with its own 3D attention, it would prove that the geometric latent space alone is the primary driver—the strongest possible evidence for the paper's central thesis. If performance degrades sharply without 3D attention, it would indicate that the geometric latent and the geometric architecture are complementary and both necessary, refining the paper's claim from "the latent space provides geometry" to "the latent space and the architecture jointly enable geometric consistency." The cross-view correspondence analysis in Appendix D.1 (Figure 14) already provides suggestive evidence—correspondence emerges in the decoder, not the encoder—but a full ablation would be definitive.

Dynamic boundary layer selection based on scene content. The paper fixes the synthesis boundary at k = 1 based on aggregate analysis across all scenes (Table 2). But the PCK and photometric fidelity tradeoffs that motivate this choice likely vary by scene type. In texture-rich scenes (brick walls, foliage), photometric fidelity at level 1 might be insufficient because complex textures require the finer spatial detail of level 0; the optimal boundary might shift to k = 0, synthesizing all features explicitly to preserve appearance at the cost of increased computation. In texture-poor scenes (smooth walls, uniform surfaces), geometric correspondence might dominate, and the boundary could shift to k = 2 because appearance fidelity matters less. A follow-up could train a lightweight classifier—or even use the variance of the PRM's predicted scores as a proxy—to predict the optimal boundary per scene from the source views alone, then dynamically allocate the synthesis budget. The key measurement would be whether dynamic boundary selection recovers some of the inference latency penalty (Table 15: 66.8s for GLD vs. 28.0s for VAE) by skipping unnecessary level-0 synthesis on scenes where level-1 alone suffices. A negative result—dynamic boundaries performing no better than fixed k = 1—would indicate that the boundary choice is dominated by the encoder architecture rather than scene content, an important clarification of the boundary layer concept's scope.

Scaling behavior of the boundary layer principle across encoder sizes and architectures. The paper evaluates only DA3-Base and, in appendix, VGGT. Both are "base" scale models. The boundary layer optimality (k = 1) might shift with encoder scale: larger models (DA3-Large, DA3-Huge) typically have more abstracted deeper features and sharper distinctions between levels, potentially widening the photometric cliff between level 1 and level 2 and making k = 1 even more clearly optimal, or conversely, their deeper features might retain more photometric information, making k = 2 viable and reducing the number of levels requiring explicit synthesis. A scaling study would train GLD with DA3-Base, DA3-Large, and DA3-Huge backbones, repeating the boundary layer analysis (Table 2) and the dual-property analysis (Tables 5-6) for each. The key question: does the optimal boundary move deeper with scale (enabling cheaper inference because fewer levels need explicit synthesis) or stay fixed (indicating a property of the architecture family rather than scale)? This would determine whether practitioners should prefer larger geometric backbones not just for better feature quality but also for inference efficiency gains from deeper propagation boundaries.

Combining geometric latents with object-centric training for unified scene-and-object NVS. GLD's primary remaining performance gap is on object-centric data (Mip-NeRF 360 out-of-domain, Table 3), where fine-tuned T2I models like MVGenMaster achieve lower ATE (0.320 vs. GLD's 0.589). The paper attributes this to GLD's scene-only training data (Section 5.2). A straightforward extension would add object-centric datasets—Objaverse renders, CO3D, or GSO—to GLD's training mix and evaluate whether the geometric latent space provides the same advantage for objects as for scenes. The key measurement would be whether GLD trained on scene+object data matches or exceeds fine-tuned T2I models on Mip-NeRF 360 without requiring text-to-image pretraining. A positive result would establish geometric latent spaces as a general-purpose alternative to T2I pretraining for NVS; a negative result (geometric latents helping scenes but not objects) would reveal a fundamental domain-dependence in what "geometric structure" means—perhaps the cross-view correspondences learned by DA3 on scene data do not transfer to the isolated-object regime where backgrounds are absent and camera trajectories are more diverse.

Training-free geometric latent extraction from any multi-view foundation model. GLD currently requires training a dedicated RGB decoder for each new geometric backbone, which is computationally expensive (170k steps on 8 B200 GPUs, Appendix A.1) and must be repeated if the backbone is updated. An alternative approach would be to develop a training-free method for extracting geometry-aware latents from any frozen multi-view transformer: identify which attention heads or layers encode cross-view correspondence (using the PCK analysis from Table 5 as a validation tool), extract those features directly, and use a generic decoder (perhaps the original backbone's own decoder, as is done for depth) to produce RGB. This would eliminate the RGB decoder training cost and make GLD-style NVS immediately applicable to any geometric foundation model without per-backbone training. The key challenge is whether training-free feature selection can match the fidelity of a trained decoder—Table 7 shows the trained DA3 decoder achieves PSNR 35.41, while the untrained path (applying the original DA3 decoder to features) may not produce RGB at all since DA3's decoder outputs geometry, not images. A successful method would need to either repurpose the geometric decoder or identify a linear combination of features that maps to RGB with minimal calibration.

Negative result stress-test: geometric latents for tasks that do not require cross-view consistency. The paper's thesis is that geometric foundation model features are better for NVS because NVS requires cross-view geometric consistency. A strong test of this thesis's boundary would be to evaluate GLD on a generation task that does not require geometric consistency—single-image generation, text-to-image, or style transfer—where VAE or DINO latents should have the advantage because they are optimized for appearance fidelity or semantic content without the computational overhead of 3D attention. If GLD underperforms VAE/DINO on these tasks, it would confirm that the benefit is task-specific and that the "right latent space for the task" principle has teeth: geometric latents help only when geometry is needed. If GLD performs competitively on single-image tasks, it would suggest that geometric foundation model features are simply better representations in general—a much stronger claim that would motivate replacing VAEs entirely, not just for NVS. The paper's reconstruction results (Table 7, DA3 decoder PSNR 35.41 vs. SDXL VAE 34.97) hint that DA3 features might indeed be competitive for single-image reconstruction, but generation quality (FID, diversity) under a single-image diffusion model has not been evaluated.

---

Practical Applications and Downstream Use Cases

Cost-efficient novel view synthesis for real estate and architectural visualization. RealEstate10K-style applications—generating walkthrough videos from a few photos of a property—are the most direct deployment scenario. The paper's results on Re10K (Table 3, N=2 source views) show GLD achieving PSNR 16.362 with ATE 0.211, meaning the generated views are geometrically consistent enough that camera poses estimated from them nearly match the ground-truth trajectory. The practical benefit is that a real estate platform could capture 2–4 photos of a room, run GLD to generate a smooth camera trajectory through the space (with the geometric decoder providing zero-shot depth for 3D reconstruction simultaneously), and produce a walkthrough video and 3D floor plan from the same generation pass—no external depth sensor, no per-scene optimization, no dependence on proprietary T2I model weights. The training efficiency (4.4× faster convergence than VAE baselines) means that a company could fine-tune GLD on their own property data with substantially less compute than would be required for a VAE-based pipeline.

Data generation for 3D reconstruction and self-improving geometric models. Because GLD's synthesized latents can be decoded into geometry (depth, camera parameters, point clouds) as a zero-shot byproduct, the framework can generate paired RGB-D data with known camera poses for training or fine-tuning downstream 3D reconstruction models. The depth quality on ETH3D (Table 9, AbsRel 0.160, δ₁ 0.800) is sufficient for many 3D reconstruction pipelines to benefit from additional training data. A practical pipeline would be: take a geometric foundation model (say, a newer, larger version of DA3), use GLD to generate diverse multi-view RGB-D from sparse source images in new scenes, and fine-tune the foundation model on this data to improve its own reconstruction accuracy. This is a self-improvement loop where GLD provides the generative data augmentation, and the improved geometric model provides better features for the next generation of GLD. The paper's finding that GLD outperforms Matrix3D's jointly-trained depth (Table 9) suggests that GLD-generated depth might be higher quality than depth from competing generative methods, making it a particularly good candidate for this loop.

Edge deployment of multi-view generative models without large model dependencies. GLD's ability to train from scratch and match fine-tuned T2I models addresses a practical bottleneck: many deployment scenarios (mobile devices, on-premise servers, privacy-sensitive applications) cannot rely on cloud-hosted T2I models due to latency, cost, or data sovereignty requirements. A company could train a GLD-style model on their domain-specific data from scratch—with the 4.4× training acceleration reducing the compute budget needed—and deploy it locally without dependencies on Stable Diffusion weights or external depth estimators. The inference latency penalty (66.8s per scene, Table 15) is substantial and would need to be addressed for interactive applications, but for batch processing (generating a set of novel views overnight for a product catalog) or one-time visualization (architectural review where a single walkthrough is generated and then viewed repeatedly), the latency is acceptable. The zero-shot geometry output is an additional advantage: a single model provides both the novel views and the 3D reconstruction, eliminating the need for a separate depth estimation or structure-from-motion pipeline.