Native and Compact Structured Latents for 3D Generation

🎯 Pitch

O-Voxel, a field-free sparse voxel structure, jointly encodes arbitrary geometry and full PBR materials—including translucent surfaces—that prior 3D representations cannot handle. A compact 9.6K-token latent space distilled from it enables a 4B-parameter flow-matching model to generate assets at 1024³ resolution that outperform state-of-the-art methods by wide margins.

1. Executive Summary

This paper introduces a native and compact structured latent representation for 3D generation, built around a new sparse voxel structure called O-Voxel—an omni-voxel representation that jointly encodes both geometry and appearance, including physically-based rendering parameters. The authors design a Sparse Compression VAE that achieves 16× spatial downsampling on O-Voxel data, compressing a fully textured 1024³-resolution asset into only ~9.6K latent tokens with negligible perceptual degradation, and then train large flow-matching models containing ~4B parameters for image-to-3D generation on diverse public 3D asset datasets. The system reconstructs shapes and materials with fidelity far exceeding prior methods while using substantially fewer tokens, and in generation, the geometry and material quality of produced assets surpasses existing models across both automated metrics and user studies. The paper establishes that native 3D latent spaces built on field-free representations can faithfully capture complex topologies—including open, non-manifold, and fully-enclosed surfaces—and arbitrary PBR material attributes, enabling high-resolution generation that outperforms a ~14× larger pretrained model only for asset classes within the representation's capability range.

2. Context and Motivation

The Core Problem: 3D Representations Cannot Jointly Handle Geometry and Appearance for Arbitrary Topology

The fundamental challenge this paper addresses is deceptively simple: no existing 3D representation can simultaneously capture complex geometry (including open, non-manifold, and interior surfaces), detailed material attributes (beyond basic texture color), and be compressed into an efficient latent space that supports scalable generative modeling. This gap exists despite rapid recent progress in 3D generative AI, and it fundamentally limits what 3D generation models can produce and how faithfully they can reproduce real-world assets.

This gap is significant for several practical reasons the authors highlight implicitly through their design choices and explicitly in Section 1:

Production-grade asset creation: Many real-world 3D assets in games, film, and industrial design use physically-based rendering (PBR) materials with properties like metallic ratio, roughness, and opacity that determine how surfaces interact with light. Models that only handle diffuse color cannot represent materials like metal, glass, or translucent surfaces—all of which are essential for realistic rendering under variable lighting conditions.
Arbitrary topology: Real assets frequently contain open surfaces (a piece of cloth, a leaf), non-manifold edges (where multiple surface patches meet at a shared boundary), and fully-enclosed interior structures (the inside of a car cockpit, nested mechanical components). Representations that enforce watertight manifold constraints—like signed distance functions (SDFs) or Flexicubes—either cannot represent such structures at all or require lossy preprocessing that discards essential geometric information.
Scalable generation: Even if a representation captures the necessary information, it must be compressible into a compact latent space. High-dimensional raw representations require impractically large generative models and slow inference, while overly aggressive compression loses the detail that motivated the representation in the first place.
End-to-end pipelines: Multi-stage approaches that separate shape generation from texture synthesis (the dominant paradigm in prior work) introduce alignment problems—textures may not match geometry, multi-view inconsistency blurs details, and separate optimization steps add complexity and failure points.

Conflicting Design Imperatives in Prior Work

The paper is motivated by a fundamental tension in 3D representation design: representations that are faithful to the full complexity of real assets tend to be unstructured (and thus incompatible with efficient neural network processing), while representations that are structured enough for efficient compression and generation tend to be restrictive about what geometry and appearance they can encode.

Implicit fields are expressive but impractical for generation at scale. Early 3D generation work adopted occupancy fields (Mescheder et al., 2019), signed distance functions (Park et al., 2019), and their discretized variants. These can represent arbitrary watertight geometry in principle, but come with several burdens: evaluating geometry requires neural network inference at every query point (expensive), handling open/non-manifold surfaces is poorly defined in the SDF framework, and they typically don't encode appearance at all. NeRF-based approaches (Mildenhall et al., 2021; Tang et al., 2023; Müller et al., 2023) do integrate geometry and appearance in a radiance field, but at the cost of heavy sampling during both training and inference, and with a well-documented tendency toward low-quality extracted geometry. The authors cite these limitations in Section 2: "suffering from low geometry quality and heavy sampling costs."

Unstructured explicit representations lack the regularity needed for latent compression. Meshes (Nash et al., 2020; Chen et al., 2024), point clouds (Nichol et al., 2022; Luo et al., 2021), and 3D Gaussians (Kerbl et al., 2023; Yu et al., 2024) are explicit and can represent arbitrary geometry without the constraints of fields, but they sacrifice structural regularity. The authors note this explicitly: these representations "lack structural regularity, posing challenges for network processing and latent compression" (Section 2). Without a regular grid structure, standard building blocks like convolutions are difficult to apply, and latent space construction—critical for scaling generative models—becomes challenging.

Field-based sparse voxel methods are structured but restricted. More recent work has attempted to combine the regularity of voxel grids with the geometric flexibility of fields. Methods like Trellis (Xiang et al., 2025), SparseFlex (He et al., 2025), and Direct3D (Wu et al., 2025) use sparse voxel structures where each occupied voxel stores parameters of an iso-surface field (typically Flexicubes from Shen et al., 2023). This achieves high-resolution geometry with a structured, neural-network-friendly layout. However, the reliance on iso-surface fields encodes an inherent constraint: the represented surface must be a closed, orientable, manifold boundary. The authors identify this as a critical limitation: these methods have "intrinsic limitations in handling open surfaces, non-manifold geometry, and enclosed interior structures" (Section 1). Additionally, most of these methods focus exclusively on shape geometry while "neglecting the appearance and material information inherent in 3D assets that are fundamentally correlated with shape" (Section 1).

Trellis partially bridges the gap but with compromises. Among prior work, Trellis (Xiang et al., 2025) comes closest to the joint geometry-and-appearance goal, introducing a structured 3D latent (SLAT) representation that models both shape and appearance jointly. However, the paper identifies two specific weaknesses: (1) its reliance on multiview 2D image feature input and pure rendering-based supervision "leads to deficiencies in capturing complex structures and materials" (Section 1), and (2) the latent space, while structured, is not compact—it requires a large number of tokens relative to the geometric complexity it encodes (9.6K tokens at 4× spatial downsampling in Table 1). The authors position their work as addressing both of these gaps simultaneously.

The Appearance Gap: Color is Not Enough

A critical, underappreciated problem is that most prior 3D generation work treats "appearance" as synonymous with "texture color"—a single RGB value per surface point. This ignores the rich material parameterization used in modern rendering pipelines. Physically-based rendering models light-surface interaction through separate channels: base color (the albedo), metallic ratio (how "metal-like" the surface behaves optically), roughness (from mirror-smooth to diffuse), and opacity (transparency). These parameters determine not just how an object looks under the original lighting, but how it will appear under any lighting—the defining property of a reusable 3D asset.

The paper identifies that "most existing works focus on 3D shape generation while neglecting the appearance and material information inherent in 3D assets" (Section 1). This gap has practical consequences: an asset with baked-in lighting and no separate material channels cannot be relit, cannot be placed in a new scene with different illumination, and will appear visually inconsistent if the lighting direction changes. For industrial applications—game engines, film production, architectural visualization—PBR materials are not optional; they are the standard interchange format. The paper's decision to encode base color, metallic, roughness, and opacity in the material feature is a deliberate response to this gap. The inclusion of opacity is particularly notable: previous methods "cannot handle translucent surfaces" (Section 1), a capability the authors explicitly highlight.

The Compression Challenge: Prior Latents Consume Too Many Tokens

Even if a representation captures the right information, it must do so compactly to support large-scale generative modeling. The paper provides specific evidence of this problem in Table 1, comparing token counts across methods:

Trellis uses 9.6K tokens at 4× spatial downsampling for 1024³ resolution
SparseFlex uses 225K tokens at the same 4× rate and resolution
Direct3D-S2 uses 17K tokens at 8× downsampling

A 4B-parameter DiT operating on 225K tokens per asset would be computationally prohibitive for training and slow at inference. The authors' stated goal is 16× spatial downsampling—a compression ratio not seen in prior voxel-based methods—reducing a 1024³ asset to only ~9.6K latent tokens. Achieving this without destroying fidelity is a non-trivial engineering challenge that the Sparse Compression VAE is specifically designed to address.

Two-Stage Pipelines: Shape-Then-Texture Dominates but Has Drawbacks

The prevailing paradigm for large 3D generation models is a two-stage decomposition: first generate shape geometry, then synthesize texture by rendering multi-view images and fusing them back onto the surface (Zhang et al., 2024; Hunyuan3D, 2025; Yang et al., 2025; Li et al., 2025). This approach benefits from leveraging powerful pre-trained 2D image diffusion backbones for the texture stage—a significant practical advantage. However, the paper identifies three specific failure modes (Section 2):

Multi-view inconsistency: Textures synthesized from independent views often don't align at view boundaries, producing ghosting, blurring, or visible seams when fused onto the surface.
View-dependent postprocessing overhead: The pipeline requires "complex multi-view rendering, baking, and texture alignment, which hinder scalability" (Section 2).
Occluded surface blindness: Multi-view approaches can only texture surfaces visible from external camera positions. Interior structures, occluded cavities, and self-occluded regions either remain untextured or receive hallucinated (and likely incorrect) content.

The paper argues for a fundamentally different approach: performing appearance reasoning "natively in 3D" rather than through 2D projections, eliminating the view inconsistency and occlusion problems inherent in multi-view pipelines. This is the central architectural bet of the work.

How the Paper Positions Itself

The paper positions itself at the intersection of three research threads—3D representation design, structured latent learning, and large-scale 3D generation—and claims to advance all three simultaneously.

For representations: O-Voxel is presented as a "field-free" alternative to both implicit field representations (SDFs, occupancies, NeRFs) and regular voxel grids. The key claim is that O-Voxel achieves the structured regularity that makes neural processing efficient while eliminating the topological restrictions that field-based methods impose. The Flexible Dual Grid formulation, while inspired by Dual Contouring (Ju et al., 2002; Chen et al., 2022), is repurposed: instead of operating on a signed grid (which is inherently a field), O-Voxel directly uses mesh surface intersections to determine edge activation and Hermite data. This subtle but important difference is what enables it to handle open, non-manifold, and enclosed surfaces that SDF-based Dual Contouring cannot.

For latent compression: The Sparse Compression VAE is positioned as advancing the "structured latent" paradigm beyond the state of the art. Prior structured latents (Trellis, SparseFlex) achieved good geometric fidelity but required relatively many tokens. Prior unstructured latents (Shape2VecSet, Michelangelo) achieved strong compression but sacrificed reconstruction fidelity. The SC-VAE claims to achieve both: 16× spatial compression with near-lossless reconstruction, enabled by architectural innovations (sparse residual autoencoding, optimized residual blocks) that are specific to high-compression sparse 3D data.

For generation: The paper's 4B-parameter flow-matching model is one of the largest 3D generative models reported, trained on ~800K assets with PBR materials. The positioning emphasizes that scale is enabled by the compact latent space: without 16× compression, training a model of this size on this data would be computationally intractable. The claim of "highly efficient" inference (~3 seconds for 512³ assets on an H100) is a direct consequence of the compact latents and the simplified (vanilla-style, no token packing or skip connections) DiT architecture they enable.

The paper's framing is notable for its emphasis on practical deployability. The abstract highlights inference speed alongside quality metrics, the code/model/data release is prominently advertised, and even the sparse convolution backend (FlexGEMM) is presented as a cross-platform contribution that runs on both NVIDIA and AMD hardware. This suggests the authors see their primary contribution as enabling real-world 3D content creation pipelines, not just advancing the state of academic benchmarking.

3. Technical Approach

3.1 Reader Orientation

This paper builds a complete pipeline for generating high-resolution 3D assets with complex geometry and physically-based materials from a single input image, where the entire system operates natively in 3D rather than relying on 2D multi-view projections. The core problem it solves is that existing 3D representations either cannot faithfully capture arbitrary topology (open surfaces, non-manifold geometry, interior structures) alongside rich material properties (metallic, roughness, opacity), or they require so many latent tokens that scaling generative models to high resolution becomes computationally infeasible — the solution takes the shape of a new "field-free" sparse voxel representation (O-Voxel) that structurally encodes both geometry and PBR materials, compressed by a carefully designed 16× downsampling VAE into ~9.6K latent tokens, then generated by large flow-matching transformers.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a sequential pipeline:

O-Voxel Representation — a sparse voxel structure that converts arbitrary 3D meshes (with PBR textures) into a regular grid of feature tuples encoding local geometry (dual vertices, edge intersection flags, splitting weights) and material properties (base color, metallic, roughness, opacity). This is the "native 3D data format" that sits between raw assets and neural networks.
Sparse Compression VAE (Shape) — a fully sparse-convolutional autoencoder that takes O-Voxel geometry features as input, hierarchically downsamples them 16× spatially into a compact latent representation, and reconstructs the O-Voxel geometry with an early-pruning upsampler that predicts which child voxels to activate. Trained with direct feature regression plus rendering-based perceptual losses.
Sparse Compression VAE (Material) — a second SC-VAE that encodes material properties conditioned on the shape VAE's subdivision structure, enabling decoupled shape and material latent spaces. This lets the material stage run independently for shape-conditioned texture generation.
Generative Flow-Matching Models (×3) — three DiT-based transformers, each ~1.3B parameters, that generate latents in sequence: first a sparse structure generator predicts which voxels are occupied (the layout), then a geometry generator fills in the shape features within active voxels, then a material generator synthesizes PBR features conditioned on both the input image and the generated geometry.
O-Voxel to Mesh/Texture Converters — instant, optimization-free algorithms that map O-Voxel features back to triangle meshes and PBR texture maps for rendering and downstream use.

Information flows as follows: an input image → DINOv3-L extracts conditioning features → the sparse structure DiT predicts the voxel occupancy layout → the geometry DiT generates shape latents (dual vertices, flags, splitting weights) within occupied voxels conditioned on structure + image → the material DiT generates material latents conditioned on geometry + image → the SC-VAE decoders upsample latents back to full-resolution O-Voxel features → the O-Voxel-to-mesh and O-Voxel-to-texture converters produce the final textured 3D asset.

3.3 Roadmap for the Deep Dive

First, the O-Voxel representation — because every subsequent component depends on understanding what data format flows through the system. I'll explain the Flexible Dual Grid for geometry and the volumetric PBR parameterization, including the bidirectional mesh↔O-Voxel conversion algorithms and why the "field-free" design choice matters.
Second, the Sparse Compression VAE architecture — how the fully sparse-convolutional encoder-decoder achieves 16× spatial compression while preserving fidelity. I'll cover the sparse residual autoencoding layers, the optimized ConvNeXt-style residual blocks, the early-pruning upsampler, and the two-stage training procedure with its direct and rendering-based losses.
Third, the generative modeling pipeline — the three-stage DiT-based flow-matching framework that generates structure, geometry, and material sequentially. I'll cover the progressive training strategy, image conditioning through cross-attention (DINOv3-L features), AdaLN-single timestep modulation, RoPE for resolution generalization, and the rectified flow formulation.
Fourth, the decoupled VAE design and shape-conditioned material generation — why separate VAEs for shape and material matter, how the material VAE conditions on the shape VAE's subdivision structure, and how this enables independent shape-conditioned texture synthesis as a downstream application.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and representation design paper whose core idea is that by replacing field-based geometry representations with a "field-free" dual-grid voxel structure (O-Voxel), and by designing a sparse convolutional VAE with carefully chosen residual blocks and sampling layers to compress that representation 16× spatially, one can jointly encode complex geometry and PBR materials in a compact enough latent space to train large-scale flow-matching generative models that produce high-fidelity textured 3D assets natively in 3D without multi-view postprocessing.

O-Voxel: The Field-Free Sparse Voxel Representation

The O-Voxel is the foundational data structure of the entire system. It takes a 3D asset — specifically, a triangle mesh with PBR texture maps — and converts it into a collection of feature tuples associated with sparse (active) voxels on a regular 3D grid. The formal definition is:

$\boldsymbol{f} = \{(\boldsymbol{f}^{\text{shape}}_i, \boldsymbol{f}^{\text{mat}}_i, \boldsymbol{p}_i)\}_{i=1}^{L}$

where $\boldsymbol{f}^{\text{shape}}_i$ encodes local geometric information for the $i$ -th active voxel, $\boldsymbol{f}^{\text{mat}}_i$ encodes its material properties, $\boldsymbol{p}_i \in \{0, 1, \ldots, N-1\}^3$ is the integer grid coordinate of the $i$ -th active voxel, and $L$ is the total number of active voxels (voxels that intersect the asset's surface). Empty voxels — those that do not intersect the asset — are set inactive and incur no storage or computation cost.

What this does operationally: a 3D asset at grid resolution $N \times N \times N$ is converted into a sparse set of $L$ feature vectors, each anchored to a specific grid position. Only voxels near the surface are stored; the interior volume and exterior space are implicit (empty). This sparse representation is what flows through all subsequent neural networks.

Why this form: the grid structure provides the spatial regularity that convolutional networks require (each voxel has a fixed relationship to its 26 neighbors), while the sparsity — storing only surface-intersecting voxels — keeps the representation computationally tractable at high resolutions. A dense $1024^3$ grid would contain over a billion voxels, but only a small fraction (~10K–100K depending on the asset) are active. The separation into $\boldsymbol{f}^{\text{shape}}$ and $\boldsymbol{f}^{\text{mat}}$ components enables the decoupled VAE design discussed later.

The Flexible Dual Grid for Shape Geometry

The core geometric innovation is the Flexible Dual Grid formulation. Rather than storing an implicit field value (like a signed distance) at each voxel and extracting a surface as a level set, O-Voxel stores explicit geometric features that directly define local surface patches. The geometry feature $\boldsymbol{f}^{\text{shape}}_i$ for each active voxel consists of three components:

Dual vertex position $\boldsymbol{v}_i \in \mathbb{R}_{[0,1]}^3$ — a 3D point within the voxel's unit cube that represents where the surface passes through that voxel. This is the most critical geometric feature; it determines the local surface location.
Edge intersection flags $\boldsymbol{\delta}_i \in \{0, 1\}^3$ — three binary flags indicating whether the surface intersects the voxel edge along the X, Y, and Z axes that share the minimum-coordinate corner. These flags determine which neighboring dual vertices should be connected to form quadrilateral faces. Specifically, if the flag for the X-axis edge is 1, a quad face is formed connecting the current voxel's dual vertex with those of its neighbor in the +X direction and their respective neighbors in the Y and Z directions (forming a quadrilateral spanning two adjacent voxels). The flags for the other 9 edges of the voxel are stored in neighboring voxels — this is a deliberate design choice that avoids redundancy while ensuring each quad face is represented exactly once in the data structure.
Splitting weights $\gamma_i \in \mathbb{R}_{>0}$ — a continuous positive value that controls how each quadrilateral face is adaptively subdivided into two triangles. Following the flexible topology rule from Shen et al. (2023), the sign and magnitude of $\gamma_i$ determine which diagonal of the quad is chosen for triangulation, allowing the reconstructed mesh to better conform to sharp features and local curvature.

How dual vertices are computed (Mesh → O-Voxel): The conversion algorithm, detailed in Algorithm 1 of the appendix, proceeds in five steps. First, for each triangle in the input mesh, the algorithm identifies every voxel edge that the triangle intersects and records the intersection point $\boldsymbol{q}$ and surface normal $\boldsymbol{n}$ at that intersection — this is the Hermite data. Second, each such intersection activates the voxels neighboring that edge (up to 4 voxels per edge) and accumulates a plane-distance QEF term for each neighbor: $d_{\Pi,i}^2 = (\boldsymbol{n}_i \cdot (\boldsymbol{v} - \boldsymbol{q}_i))^2$ , which measures the squared perpendicular distance from the dual vertex $\boldsymbol{v}$ to the plane defined by the surface intersection point and normal. Third, for any boundary edges of the mesh (edges belonging to only one triangle — characteristic of open surfaces), an additional line-distance QEF term is accumulated for each intersected voxel: $d_{L,j}^2 = \|(\boldsymbol{v} - \boldsymbol{o}_j) - ((\boldsymbol{v} - \boldsymbol{o}_j) \cdot \boldsymbol{d}_j) \boldsymbol{d}_j\|^2$ , where $\boldsymbol{o}_j$ and $\boldsymbol{d}_j$ are the origin and direction of the boundary edge line. This term penalizes the distance between the dual vertex and the boundary edge, guiding the vertex to align with open surface boundaries. Fourth, a regularization term $d_{\hat{\boldsymbol{q}}}^2 = \|\boldsymbol{v} - \bar{\boldsymbol{q}}\|^2$ encourages the vertex to stay near the average $\bar{\boldsymbol{q}}$ of all intersection points in that voxel, which stabilizes the optimization against singular configurations where the QEF matrix is ill-conditioned. Fifth, the full quadratic error function is solved:

$\min_{\boldsymbol{v} \in \text{voxel}} e(\boldsymbol{v}) = \sum_{i} d_{\Pi,i}^2 + \lambda_{\text{bound}} \sum_{j} d_{L,j}^2 + \lambda_{\text{reg}} \, d_{\hat{\boldsymbol{q}}}^2$

where $\lambda_{\text{bound}}$ and $\lambda_{\text{reg}}$ are scalar weights controlling the influence of the boundary and regularization terms respectively, and the minimization is constrained such that $\boldsymbol{v}$ stays within the voxel's unit cube. The solution is in closed form because the QEF is a quadratic function of $\boldsymbol{v}$ ; it reduces to solving a $3 \times 3$ linear system.

What this computes: for each active voxel, a single 3D point $\boldsymbol{v}$ that best represents where the surface passes through that voxel, taking into account all surface intersections on the voxel's edges, any boundary constraints from open mesh edges, and a regularization pull toward the centroid of intersection points. The resulting dual vertex is not constrained to lie on any particular triangle — it is an optimal representative point that minimizes the aggregate geometric error.

Why this form: the original Dual Contouring QEF only contained the plane-distance term $\sum d_{\Pi,i}^2$ , which works for watertight, manifold surfaces extracted from signed distance fields. The paper's additions — the boundary line term and the regularization term — are specifically designed to handle cases that SDF-based DC cannot: open surfaces (where boundary edges exist and need the line QEF to pull vertices toward them), and degenerate configurations (where few intersections lead to underdetermined QEFs that the regularization term stabilizes). The key insight is that these terms are only possible because O-Voxel works directly with mesh surface intersections (not a scalar field), so boundary edges are explicit and identifiable.

How meshes are reconstructed (O-Voxel → Mesh): Algorithm 2 describes the inverse process. For each active voxel, the dual vertex $\boldsymbol{v}_i$ becomes a mesh vertex. Then, for each of the three predefined axes (X, Y, Z), if the edge intersection flag $\boldsymbol{\delta}_i[a]$ is 1, a quadrilateral face is formed connecting the dual vertices of four voxels: the current voxel, its neighbor in the +axis direction, and their respective neighbors in the two other axis directions (essentially, the four voxels sharing the activated edge). This quadrilateral is then subdivided into two triangles according to the splitting weight $\gamma_i$ , with the sign determining which diagonal is chosen. Finally, all vertices and faces are assembled into a triangle mesh.

Why this matters — the three stated advantages:

Instant bidirectional conversion: The mesh-to-O-Voxel conversion requires only identifying edge-triangle intersections and solving small $3 \times 3$ linear systems per active voxel — it takes "only a few seconds on a single CPU." The reverse conversion is even faster — "tens of milliseconds." This contrasts sharply with prior methods that require expensive operations like SDF evaluation throughout a volume, flood-fill to determine interior/exterior, and iterative optimization for surface extraction.
Arbitrary-topology modeling: Because O-Voxel does not rely on an implicit field with a defined inside/outside, it has no concept of "watertightness" or "manifoldness." The dual vertices and edge flags directly encode which surface patches exist and how they connect, regardless of whether the surface forms a closed volume. Open surfaces (where some edges have only one adjacent triangle — like a leaf) are handled naturally through the boundary QEF term. Non-manifold edges (where more than two surface patches meet) can be represented because each edge flag independently activates or deactivates the corresponding quad face. Fully-enclosed interior structures (like a cockpit inside a car exterior) are captured because the algorithm activates any voxel intersected by any triangle, regardless of whether it's on the "outside" or "inside" — there is no concept of inside/outside at all.
Sharp feature preservation: By design, dual vertices are computed to minimize distance to the surface planes, which naturally places them at sharp edges and corners where multiple planes intersect. The splitting weights enable adaptive triangulation that can align triangle edges with geometric discontinuities. Furthermore, the dual vertex positions and splitting weights can receive learnable adjustments from the VAE's neural network during training (using rendering-based supervision), providing additional flexibility beyond what the QEF solution alone achieves.

Volumetric Attributes for Material

The material feature $\boldsymbol{f}^{\text{mat}}_i$ is simpler in concept but crucial for the paper's claim of full PBR asset generation. It comprises six channels per active voxel:

$\boldsymbol{f}^{\text{mat}}_i = (\boldsymbol{c}_i, m_i, r_i, \alpha_i)$

where $\boldsymbol{c}_i \in \mathbb{R}_{[0,1]}^3$ is the base color (the diffuse albedo in RGB), $m_i \in \mathbb{R}_{[0,1]}$ is the metallic ratio (0 = dielectric like plastic or wood, 1 = fully metallic like polished chrome), $r_i \in \mathbb{R}_{[0,1]}$ is the roughness (0 = mirror-smooth, 1 = completely diffuse), and $\alpha_i \in \mathbb{R}_{[0,1]}$ is the opacity (0 = fully transparent, 1 = fully opaque). This follows the standard metallic-roughness PBR convention used in modern rendering engines.

Texture → O-Voxel conversion (Algorithm 3): For each active voxel, the algorithm finds all triangles of the mesh that intersect the voxel, projects the voxel center onto each such triangle to get a surface point $\boldsymbol{q}$ , computes the UV coordinates and mipmap level at that point from the triangle's texture coordinates, samples each PBR attribute from the corresponding texture map, and computes a weighted average of all samples (weighted by the inverse point-to-surface distance) to get the final per-voxel material values. The use of mipmaps ensures that the sampled attribute appropriately represents the texture detail at the voxel's spatial scale, preventing aliasing when the voxel is large relative to texture detail.

O-Voxel → Texture conversion (Algorithm 4): Two modes are supported. In vertex color mode, for each mesh vertex recovered from an O-Voxel dual vertex, the material attributes are obtained via trilinear interpolation of the material features from the 8 neighboring active voxels (or fewer near boundaries). In texture map mode, the reconstructed mesh is UV-parameterized, and for each texel, the corresponding 3D surface point is computed, then its material attributes are trilinearly interpolated from the surrounding O-Voxel material features and written into the texture map.

Why this form: encoding materials as per-voxel volumetric attributes rather than per-vertex or per-texel data serves two purposes. First, it keeps the material representation aligned with the spatial structure of the geometry representation — both are defined on the same sparse voxel grid, which simplifies the neural network architecture (the same grid structure processes both). Second, the volumetric encoding (with trilinear interpolation for lookup) provides a continuous representation that is independent of the specific mesh triangulation or UV parameterization that will ultimately be used for rendering. This means the same O-Voxel material data can be transferred to any mesh extracted from the O-Voxel geometry, regardless of how the quads are triangulated, without re-baking textures.

The opacity channel is particularly significant because it enables the representation of translucent surfaces — a capability the paper explicitly claims is absent in prior methods. In a PBR pipeline, opacity controls how much light passes through a surface, enabling effects like glass, water, or frosted plastic. Previous methods that only encoded base color implicitly assumed full opacity.

Sparse Compression VAE: Architecture and Training

The Sparse Compression VAE (SC-VAE) is the component that learns a compact latent space from O-Voxel data. Its design goal is aggressive: achieve 16× spatial downsampling in each dimension (for a total 4096× reduction in voxel count) while preserving both geometric and material fidelity to a degree that is "near-lossless" for downstream tasks. The architecture is a fully sparse-convolutional U-Net that processes only active voxels, paired with several innovations to make such high compression feasible.

Overall U-Net Structure

The SC-VAE follows the standard VAE design pattern from the latent diffusion literature (Rombach et al., 2022): an encoder hierarchically downsamples the input through multiple resolution levels, a bottleneck produces the latent distribution parameters, and a decoder mirrors the encoder to reconstruct the output. Table 4 in the appendix provides the detailed layer specification. The encoder processes the active voxels at the input resolution through 4 downsampling stages (each halving the spatial resolution), with the number of channels doubling at each stage (following typical U-Net convention). The decoder symmetrically upsamples back to the original resolution. The complete model has approximately 800M parameters (354M encoder, 474M decoder).

Sparse Residual Autoencoding Layers

The most critical architectural innovation for achieving 16× compression is the sparse residual autoencoding layer, adapted from DC-AE (Chen et al., 2024) to sparse voxel data. The problem it solves is that naive downsampling (e.g., average pooling) destroys fine-grained information that the decoder needs for faithful reconstruction, especially at high compression ratios where the bottleneck is extremely narrow.

Downsampling residual shortcut: When reducing spatial resolution by 2×, rather than only applying learned convolutions, the downsampling block constructs a non-parametric residual estimate by rearranging the fine-grid features into the coarse grid's channel dimension:

$F_{\text{coarse}}^{\text{raw}} = \operatorname{stack}\big(F_{\text{child}_1}, \dots, F_{\text{child}_8}\big) \in \mathbb{R}^{8C}$

$F_{\text{coarse}} = \operatorname{avg\_groups}\big(F_{\text{coarse}}^{\text{raw}}\big) \in \mathbb{R}^{C^{\prime}}$

where $F_{\text{fine}} \in \mathbb{R}^{C}$ is the input feature at each fine voxel, $F_{\text{child}_1}, \dots, F_{\text{child}_8}$ are the features of the eight fine voxels that correspond to one coarse voxel (the 2×2×2 subdivision), the $\operatorname{stack}$ operation concatenates them along the channel dimension to produce a $8C$ -dimensional vector, and $\operatorname{avg\_groups}$ averages groups of channels to reduce the dimension to $C^{\prime}$ (typically $C^{\prime} = 2C$ ). This produces a coarse-grid feature estimate that preserves information from all eight children through averaging.

What this computes: a "shortcut" estimate of what the coarse-grid features should be, based purely on rearranging and averaging the fine-grid features, without any learned parameters. This estimate is added to the output of the learned convolutional downsampling path as a residual connection, allowing the network to focus on learning the difference between a naive averaging and the optimal coarse representation.

Upsampling residual shortcut: In the decoder, the inverse operation distributes coarse features back to fine voxels:

$F_{\text{fine}}^{\text{raw}} = \operatorname{unstack}\big(F_{\text{coarse}}\big) \in \mathbb{R}^{8C^{\prime}/8}$

$F_{\text{fine}} = \operatorname{dup\_groups}\big(F_{\text{fine}}^{\text{raw}}\big) \in \mathbb{R}^{C}$

where $\operatorname{unstack}$ splits the coarse feature channels into 8 groups (one per child voxel), and $\operatorname{dup\_groups}$ duplicates channels within each group to match the target fine-grid channel dimension $C$ . If a child voxel position has no active voxel (it corresponds to an empty region), its contribution is simply zero.

Why this form: at extreme compression ratios, the bottleneck contains far fewer spatial locations than the input, forcing the encoder to discard information. The residual shortcut gives the decoder a direct, non-parametric path to recover basic spatial structure — the coarse features carry a rough average of what was in each 2×2×2 neighborhood. The learned convolutional path then refines this estimate, adding the high-frequency detail that averaging destroys. The paper's ablation (Table 3) demonstrates this dramatically: at 16× compression, removing the residual autoencoding increases mesh distance (MD) by 69% and reduces normal PSNR by 0.5 dB; at 32×, the degradation is catastrophic (526% MD increase, 1.6 dB PSNR drop).

A subtle but important implementation detail: the $\operatorname{avg\_groups}$ and $\operatorname{dup\_groups}$ operations handle sparsity naturally because missing voxels contribute zero vectors. This means the residual estimate is always well-defined even when some of the eight children are inactive.

Early-Pruning Upsampler

To further improve efficiency, the decoder incorporates an early-pruning mechanism adapted from Ren et al. (2024). Before each upsampling step, a lightweight prediction head produces a binary mask $\hat{\boldsymbol{\rho}} \in \{0,1\}^8$ for each parent voxel, specifying which of its eight potential children should be activated in the next finer level. Child voxels predicted as inactive are immediately discarded, meaning subsequent decoder layers never process them. This has two effects: (1) it reduces runtime and memory by avoiding computation on voxels that would ultimately be empty, and (2) it acts as a learned sparsity prior — the decoder learns which regions of space are near the surface and merit detailed processing, versus which are empty and can be safely ignored.

During training, the pruning mask $\hat{\boldsymbol{\rho}}$ is supervised with binary cross-entropy loss against the ground-truth active/inactive labels $\boldsymbol{\rho}$ from the original O-Voxel data (Equation 6). The mask prediction is part of the same forward pass, so no separate network is needed.

Why this design: sparse convolutions derive their efficiency from skipping empty voxels, but a naive upsampler would activate all 8 children for every parent voxel, quickly filling the grid and losing sparsity. Early pruning keeps the active voxel count controlled throughout the decoder, which is essential for processing high-resolution outputs (e.g., 1024³) where fully activating all children at every level would be computationally prohibitive.

Optimized Residual Block (ConvNeXt-Style)

The paper redesigns the standard residual block — the basic building block repeated at each resolution level — to better suit sparse 3D convolutions:

Standard design: two convolutional layers (each with normalization and activation) in sequence, with a residual connection around both. This is the ResNet block design used in most 2D and 3D convolutional networks.

SC-VAE design: a single sparse convolution layer (with normalization and activation, using GELU activation), followed by a point-wise MLP (essentially a 1×1×1 convolution or fully-connected layer applied independently per voxel) that expands the channel dimension (analogous to a Transformer FFN) and then projects back, with the residual connection wrapping around the entire block. The authors explicitly state this is "following the ConvNeXt-style simplification" from Liu et al. (2022).

Why this form: sparse convolutions on highly sparse data — where often fewer than 1% of voxels are active — exhibit "low effective computation and parameter efficiency" because the active sites are scattered and the convolution kernel often processes many zero-valued neighbors. The paper's redesign addresses this by reducing the number of spatially-aware (convolution) layers and compensating with channel-wise (point-wise MLP) layers that are equally efficient regardless of sparsity pattern. The single convolution provides the necessary spatial context aggregation; the wide MLP provides rich per-voxel feature transformation. The ablation (Table 3) confirms this: replacing the optimized block with standard residual blocks (while keeping all else equal) increases MD by 16% and reduces PSNR by 0.6 dB, with "runtime unchanged" — meaning the MLP overhead is offset by the reduced convolution count.

Two-Stage VAE Training

The SC-VAE is trained in two stages to progressively refine both geometric accuracy and perceptual quality.

Stage 1 — Direct O-Voxel Regression (low resolution, 256³):

$\mathcal{L}_{\text{s1}} = \lambda_{\text{v}} |\hat{\boldsymbol{v}} - \boldsymbol{v}|_2^2 + \lambda_{\delta} \operatorname{BCE}(\hat{\boldsymbol{\delta}}, \boldsymbol{\delta}) + \lambda_{\boldsymbol{\rho}} \operatorname{BCE}(\hat{\boldsymbol{\rho}}, \boldsymbol{\rho}) + \lambda_{\text{mat}} |\hat{\boldsymbol{f}}^{\text{mat}} - \boldsymbol{f}^{\text{mat}}|_1 + \lambda_{\text{KL}} \mathcal{L}_{\text{KL}}$

where $\hat{\boldsymbol{v}}$ and $\boldsymbol{v}$ are the predicted and ground-truth dual vertex positions (MSE loss, weighted by $\lambda_{\text{v}}$ ), $\hat{\boldsymbol{\delta}}$ and $\boldsymbol{\delta}$ are the predicted and ground-truth edge intersection flags (binary cross-entropy loss, weighted by $\lambda_{\delta}$ ), $\hat{\boldsymbol{\rho}}$ and $\boldsymbol{\rho}$ are the predicted and ground-truth pruning masks (BCE loss, weighted by $\lambda_{\boldsymbol{\rho}}$ ), $\hat{\boldsymbol{f}}^{\text{mat}}$ and $\boldsymbol{f}^{\text{mat}}$ are the predicted and ground-truth material features (L1 loss, weighted by $\lambda_{\text{mat}}$ ), and $\mathcal{L}_{\text{KL}}$ is the standard KL divergence between the learned latent distribution and the unit Gaussian prior (weighted by $\lambda_{\text{KL}}$ ). All $\lambda$ values are scalar hyperparameters that balance the different loss terms.

What this computes: a multi-task supervised loss that directly compares every component of the reconstructed O-Voxel data against the ground truth. The MSE on dual vertices penalizes geometric displacement, the BCE on edge flags and pruning masks penalizes incorrect topology decisions, the L1 on materials penalizes color and PBR parameter errors, and the KL loss regularizes the latent space toward a smooth, generative-model-friendly distribution. The result is a single scalar that drives the VAE to faithfully reconstruct all aspects of the O-Voxel representation.

Why this form: direct supervision on the O-Voxel features provides a strong, unambiguous training signal at low resolution where geometric accuracy can be directly optimized. The L1 loss on materials (rather than L2) is a standard choice for color-like data because it is less sensitive to outliers and tends to produce sharper reconstructions. The separate BCE losses for flags and masks (rather than treating them as continuous values with MSE) correctly model them as binary variables.

Stage 2 — Add Rendering-Based Perceptual Loss (high resolution, 512³):

$\mathcal{L}_{\text{s2}} = \mathcal{L}_{\text{s1}} + \mathcal{L}_{\text{render}}$

where $\mathcal{L}_{\text{render}}$ consists of:

$\mathcal{L}^{\text{shape}}_{\text{render}} = \|\hat{m} - m\|_1 + 10 \cdot \|\hat{d} - d\|_1 + d_{\text{p}}(\hat{\boldsymbol{n}}, \boldsymbol{n})$

$\mathcal{L}^{\text{mat}}_{\text{render}} = d_{\text{p}}(\hat{\boldsymbol{c}}, \boldsymbol{c}) + d_{\text{p}}(\hat{\boldsymbol{mra}}, \boldsymbol{mra})$

and the perceptual distance $d_{\text{p}}$ is:

$d_{\text{p}}(\boldsymbol{a}, \boldsymbol{b}) = \|\boldsymbol{a} - \boldsymbol{b}\|_1 + 0.2 \cdot d_{\text{SSIM}} + 0.2 \cdot d_{\text{LPIPS}}$

Here, $m$ , $d$ , $\boldsymbol{n}$ are the silhouette mask, depth map, and normal map rendered from the mesh; $\boldsymbol{c}$ , $\boldsymbol{mra}$ are the base color and combined metallic-roughness-alpha map; hats denote reconstructed versions; and $d_{\text{SSIM}}$ and $d_{\text{LPIPS}}$ are the structural similarity and learned perceptual similarity losses respectively. The depth loss is weighted 10× compared to the mask loss, reflecting the greater importance of accurate surface positioning.

What this computes: rather than only comparing O-Voxel features directly, Stage 2 reconstructs the mesh from the decoded O-Voxel, renders it from multiple camera views, and compares the rendered images against renders of the ground-truth asset. This provides supervision on the perceptual quality of the final output — whether the reconstructed shape looks sharp, whether normals are smooth and detailed, whether material colors and roughness appear correct under lighting.

Why this form: direct O-Voxel loss alone can produce reconstructions that are mathematically close to the ground truth but exhibit subtle visual artifacts — slightly misplaced vertices causing surface wobble, slightly incorrect materials causing color shifts under different lighting. Rendering-based loss provides supervision in the image domain, which is ultimately what matters for generation and downstream use. The LPIPS term (Zhang et al., 2018) uses a pre-trained deep network to compare images, capturing perceptual similarity that pixel-wise losses miss. The SSIM term captures structural similarity. The combination of L1, SSIM, and LPIPS in $d_{\text{p}}$ provides a balanced perceptual metric.

A notable implementation detail: during rendering for Stage 2 training, cameras are "randomly placed with a shallow near plane to slice through the surface." This means the rendered views intentionally include cross-sections that show internal structure, encouraging the VAE to accurately reconstruct interior geometry that would be invisible from external views. Without this, the rendering loss would provide no signal about interior surface quality.

Resolution generalization: Once trained at 512³, the SC-VAE models are applied directly to higher resolutions (1024³ and beyond) without fine-tuning. This is possible because the fully sparse-convolutional architecture is "inherently resolution-agnostic" — convolution kernels are defined by their spatial extent in voxel units, not in absolute coordinates, so they apply identically regardless of how many voxels span the domain. The key practical benefit is that one trained VAE serves all generation resolutions, eliminating the need to train separate VAEs for each.

Decoupled Shape and Material VAEs

The paper trains two separate SC-VAEs: one for shape (encoding $\boldsymbol{f}^{\text{shape}}$ and producing shape latents), and one for material (encoding $\boldsymbol{f}^{\text{mat}}$ and producing material latents). The material VAE is conditioned on the shape VAE's subdivision structure during upsampling — specifically, the pruning masks and active voxel layouts from the shape decoder guide which voxels the material decoder processes at each resolution level. This ensures that material features are only generated at positions where geometry exists, maintaining spatial alignment between shape and material.

Why decoupled: the decoupling enables a sequential generation scheme where shape is generated first and material is generated conditioned on the already-determined shape. This is more flexible than a joint latent that forces shape and material to be generated simultaneously: it allows the material generation stage to be used independently as a shape-conditioned texture synthesis model (Section 4.3), and it makes the generative modeling problem easier by decomposing it into two sub-problems with a natural dependency (material depends on shape, but shape does not depend on material).

Latent Dimensionality and Compression Ratio

The SC-VAE operating at 16× spatial downsampling produces latents of spatial size $N/16 \times N/16 \times N/16$ , with each latent token having 32 channels (from the architectural details in Table 4, the bottleneck has 32 channels). The number of active latent tokens is therefore approximately $L / (16^3) = L / 4096$ , where $L$ is the number of active voxels in the input O-Voxel. For typical assets, this yields ~2.2K latent tokens at 512³ resolution, ~9.6K at 1024³ (as quoted in the abstract), and correspondingly more at 1536³. The total dimensionality of the latent representation is the number of tokens multiplied by the channel dimension, but for generative modeling what matters is the token count (since transformers scale quadratically with sequence length). Table 1 compares this against baselines: Trellis uses 9.6K tokens at 4× downsampling for 1024³, while the SC-VAE uses 9.6K tokens at 16× downsampling — the same token count, but serving a 4× higher resolution input (or, equivalently, 4× fewer tokens at the same resolution).

Generative Modeling: Three-Stage Flow-Matching with DiT

With the compact latent space established, the generative modeling component produces new 3D assets from image prompts by sampling in latent space and decoding through the SC-VAE. The framework consists of three DiT-based transformers, each trained with flow matching, operating sequentially.

Flow Matching Formulation (Rectified Flow)

All generative models are trained using the rectified flow objective (Liu et al., 2023). The forward process constructs a straight-line path from data to noise:

$\boldsymbol{x}(t) = (1 - t) \boldsymbol{x}_0 + t \boldsymbol{\epsilon}$

where $\boldsymbol{x}_0$ is a sample from the data distribution (a latent representation), $\boldsymbol{\epsilon} \sim \mathcal{N}(0, I)$ is random Gaussian noise, and $t \in [0, 1]$ is the timestep that indexes the progression from data ( $t = 0$ ) to pure noise ( $t = 1$ ). The time-dependent vector field $\boldsymbol{v}(\boldsymbol{x}, t) = \nabla_t \boldsymbol{x}$ governs how samples evolve, and a neural network $\boldsymbol{v}_\theta$ learns to approximate this field by minimizing:

$\mathcal{L}_{\text{CFM}}(\theta) = \mathbb{E}_{t, \boldsymbol{x}_0, \boldsymbol{\epsilon}} \|\boldsymbol{v}_\theta(\boldsymbol{x}(t), t) - (\boldsymbol{\epsilon} - \boldsymbol{x}_0)\|^2_2$

where the expectation is over uniformly sampled timesteps and random data-noise pairs. The target $(\boldsymbol{\epsilon} - \boldsymbol{x}_0)$ is the constant velocity that transports a noise sample to a data sample along a straight line.

What this computes: at training time, for each data sample, a random amount of noise is blended in (determined by $t$ ), and the network must predict the velocity vector that would move the current noisy sample toward the clean data along a straight path. At inference time, starting from pure noise $\boldsymbol{x}(1) = \boldsymbol{\epsilon}$ , the learned velocity field is integrated (typically with an ODE solver) backward to $t = 0$ , producing a data sample.

Why this form: rectified flow produces straight-line trajectories between noise and data, which means the integration at inference time requires fewer steps than curved trajectories (like those in standard diffusion). The paper uses a logitNorm(1, 1) timestep sampling distribution (following Trellis, Xiang et al., 2025) rather than uniform sampling, which puts more weight on intermediate timesteps where the velocity field is most complex, improving generation quality. The CFM objective is simple — just MSE regression — and has been shown to be equivalent to score matching with a specific weighting.

DiT Architecture Details

Each of the three DiT models follows a vanilla encoder-only transformer design. Key specifications from Table 5:

Width: 1536 (hidden dimension)
Blocks: 30 transformer layers
Heads: 12 attention heads
MLP width: 8192 (feed-forward expansion)
Total parameters: approximately 1.3B per model

The models intentionally omit complex designs from prior work such as "token packing or skip connections to maintain a clean and scalable architecture" (Section 3.3). This is enabled by the SC-VAE's high compression: with only ~9.6K tokens at 1024³ resolution, a standard DiT can process the full latent sequence without needing tricks to reduce sequence length.

Conditioning mechanisms — three types:

Timestep conditioning via AdaLN-single (Chen et al., 2024): Rather than the original AdaLN (Peebles & Xie, 2023) which predicts separate scale and shift parameters for each normalization layer, AdaLN-single predicts a single set of parameters shared across layers, "drastically reducing required parameters compared to the AdaLN baseline" (Appendix A.2). The timestep embedding is projected to a scale and shift vector that modulates the layer normalization before each attention and MLP block.
Image conditioning via cross-attention: Image features are extracted from DINOv3-L (Oquab et al., 2023), a large vision transformer pretrained with self-supervision. Unlike CLIP features used in many prior works, DINOv3 is trained without text supervision and is known to capture fine-grained spatial information, which the paper claims benefits 3D understanding. The image features are fed into cross-attention layers interleaved with the self-attention layers in each DiT block — the latent tokens attend to the image features as keys and values, injecting visual conditioning.
Shape conditioning via concatenation (material stage only): For the material generation stage, the previously generated geometry latents are concatenated channel-wise with the material latent tokens at the input to the DiT. This means the material DiT processes a sequence where each token has both a material component and the corresponding geometry feature at that spatial location, "ensuring their spatial alignment under arbitrary topology" (Section 3.3).

Rotary Position Embedding (RoPE) and QK-Norm: All DiT models use RoPE (Su et al., 2024) for position encoding, which encodes relative position through rotation of the query and key vectors in attention. Because the rotation is applied in a relative manner, the model can generalize to latent grid sizes different from those seen during training — critical for the progressive training and cascaded inference schemes. Additionally, QK-Norm (Esser et al., 2024) is applied: Root Mean Square Normalization (RMSNorm) on the query and key tensors before the attention dot product, which "improves training stability" for large models.

Three-Stage Generation Pipeline

Stage 1: Sparse Structure Generation. This stage predicts the occupancy layout of the sparse voxel grid — essentially, a binary mask indicating which positions in the latent space will contain tokens. The structure generator is a DiT that outputs logits for each position in a dense latent grid (at the target latent resolution, e.g., 32³ for 512³ output), which are thresholded to produce the sparse layout. This establishes the global coarse shape — the overall volume, broad proportions, and which regions are occupied.

Stage 2: Geometry Generation. Given the sparse layout from Stage 1 (which tells the model which latent positions are active) and the input image, the geometry DiT generates the full shape latent features within the active positions. The geometry latent encodes the O-Voxel dual vertices, edge intersection flags, and splitting weights compressed by the shape SC-VAE encoder. After generation, the shape SC-VAE decoder upsamples these latents to full-resolution O-Voxel geometry features, and the O-Voxel-to-mesh converter extracts the 3D mesh.

Stage 3: Material Generation. Given the input image and the generated geometry latents (from Stage 2, before decoding), the material DiT generates material latents at the same active positions. The conditioning on geometry ensures that materials are spatially aligned — a metallic patch on a surface corresponds to the correct geometric location. The material SC-VAE decoder upsamples these latents to full-resolution O-Voxel material features, and the O-Voxel-to-texture converter produces the PBR texture maps.

Why three stages: decomposing the generation problem this way has several advantages. First, the structure stage is the cheapest (operating on binary values at low resolution) and provides a coarse-to-fine progression where later stages only need to process active regions. Second, the geometry and material separation enables the decoupled VAE design and the independent texture synthesis application. Third, the progressive nature allows cascaded inference strategies where intermediate outputs can be refined or used to bootstrap higher-resolution generation.

Progressive Training Strategy

The generative models are trained in a progressive manner:

The sparse structure generator is trained first at $512 \times 512$ conditioning image resolution to "learn coarse occupancy priors and establish the global sparse layout."
The geometry and material generators are then trained, starting at $512^3$ output resolution (corresponding to $32^3$ latent resolution) with $512 \times 512$ conditioning images.
Training progressively increases to $1024^3$ output resolution ( $64^3$ latent resolution) with $1024 \times 1024$ conditioning images.

Why progressive: this strategy "allows the learned priors to transfer smoothly across resolutions, enabling efficient training of large-scale sparse DiTs while maintaining fidelity" (Section 3.3). The model first learns coarse structure at manageable resolution and cost, then the higher-resolution training fine-tunes these representations rather than learning from scratch. For the material stage, the conditioning on geometry latents means the model learns the texture-geometry correlation at the coarser resolution and can transfer this understanding to finer scales.

Training Data and Hyperparameters

The VAE training dataset is approximately 350K assets curated from Trellis-500K (Xiang et al., 2025), filtered to retain only those with metallic-roughness PBR materials (from Objaverse-XL, ABO, and HSSD). SC-VAEs are trained on 16 H100 GPUs with batch size 128.

The generative model training dataset is extended to approximately 800K assets, augmented with TexVerse (Zhang et al., 2025) to "enrich PBR diversity and realism." Image prompts are generated by rendering 16 views per asset in Blender with randomized field-of-view (between 10° and 70°) and lighting conditions. All generative models use: AdamW optimizer (Loshchilov & Hutter, 2017), learning rate $1 \times 10^{-4}$ , weight decay 0.01, classifier-free guidance with drop rate 0.1 (where 10% of training samples have empty conditioning to enable guidance at inference). Each DiT trains on 32 H100 GPUs with batch size 256.

Inference Efficiency

The paper reports remarkably fast inference: approximately 3 seconds for a $512^3$ fully-textured asset, ~17 seconds for $1024^3$ , and ~60 seconds for $1536^3$ , all on a single NVIDIA H100 GPU. These speeds are attributed to three factors: (1) the compact latent space (only ~9.6K tokens at 1024³) means the DiT's self-attention is cheap — quadratic in a small sequence length, (2) the vanilla DiT design without token packing or skip connections avoids computational overhead, and (3) the SC-VAE decoder is efficient due to sparsity and early pruning — it processes only active voxels.

Test-Time Compute and Resolution Scaling (Cascaded Inference)

The paper introduces a cascaded inference scheme (Section 4.5) that can generate at resolutions exceeding the training resolution or improve quality within the trained resolution:

Generating above training resolution (e.g., 1024³ → 1536³): After generating a 1024³ O-Voxel, the geometry is downsampled to create a sparse structure layout at higher resolution (e.g., max-pooling a $64^3$ structure to $96^3$ ). The geometry generator — which was trained at $64^3$ — is re-applied at $96^3$ because RoPE enables resolution generalization. This produces a $1536^3$ O-Voxel output. The generator effectively "hallucinates" sub-voxel detail based on the coarser geometric prior.

Improving quality within trained resolution: Rather than using the stage 1 sparse structure directly, an alternative structure is obtained by downsampling a previously generated O-Voxel (e.g., downsampling a $512^3$ O-Voxel to a $64^3$ sparse structure layout). This structure is cleaner than the directly predicted one because the downsampled O-Voxel has been through the VAE decode/encode cycle which smooths errors. The geometry generator then produces a higher-quality $1024^3$ output from this cleaner layout. This "offers a controllable trade-off between computational efficiency and generation quality" — it costs an extra generation pass but yields "finer details and enhanced structural stability."

FlexGEMM: Custom Sparse Convolution Backend

The paper includes the development of a custom high-performance sparse convolution backend called FlexGEMM, implemented in Triton (Tillet et al., 2019) for cross-platform compatibility (NVIDIA and AMD). The implementation uses a Masked Implicit GEMM strategy: rather than explicitly gathering features into a dense matrix (im2col) and then performing matrix multiplication (GEMM) as separate steps, these are fused into a single kernel. The fusion "minimizes global memory I/O by keeping intermediate data in fast on-chip memory." Sparsity is exploited via a masking mechanism that skips computation on empty neighbor slots, with active voxels reordered using Gray code ordering to group voxels with similar neighborhood patterns, improving SIMD efficiency. A Split-K technique divides the accumulation dimension into parallel tasks, increasing parallelism for cases with many channels or few active voxels. The paper reports "up to a 2× speedup over widely-used sparse convolution libraries" (Figure 9), including Spconv, Torchsparse, fvdb, and WarpConvNet.

Why this matters: the entire SC-VAE training and inference pipeline depends on fast sparse convolutions. Without the custom backend, training the 800M-parameter VAE at high resolution on large datasets would be substantially slower, and inference latency would increase, undermining the paper's efficiency claims. The cross-platform nature (Triton) is a deliberate choice to avoid "platform-dependency limitations of existing libraries, which are often tightly coupled to the NVIDIA CUDA ecosystem" — a practical concern for broad deployment.

4. Key Insights and Innovations

Innovation 1: The Field-Free Representation as a New Category in 3D Geometry Encoding

The paper's most fundamental intellectual contribution is not any specific algorithm within the pipeline but rather the introduction and validation of a third category of 3D representation that sits between implicit fields and explicit unstructured primitives. Before this work, the dominant taxonomy for 3D geometry in neural processing was essentially binary: you could use implicit fields (SDFs, occupancies, NeRFs) which are structured and neural-network-friendly but impose topological constraints (watertightness, manifoldness), or you could use explicit primitives (meshes, point clouds, Gaussians) which handle arbitrary topology but lack the spatial regularity that enables efficient convolutional processing and latent compression. The field recognized this tension but largely treated it as an unavoidable tradeoff — prior structured-latent works like Trellis (Xiang et al., 2025), SparseFlex (He et al., 2025), and Direct3D (Wu et al., 2025) accepted the topological constraints of field-based primitives (Flexicubes) as the price of structural regularity.

O-Voxel's Flexible Dual Grid breaks this dichotomy by demonstrating that you can have field-free spatial regularity. The representation stores geometric features directly — explicit dual vertex positions and edge connectivity flags — on a regular sparse grid, without any intermediate scalar field whose level set defines the surface. This is a conceptual shift, not merely an algorithmic tweak to Dual Contouring. The original DC (Ju et al., 2002) was designed to extract surfaces from a signed grid — it was a post-processing step on a field representation. O-Voxel repurposes the dual grid as the primary representation, with mesh surface intersections directly providing the Hermite data that would normally come from sign-change detection. This subtle inversion has outsized consequences: without a field, there is no concept of inside/outside, no level set, no requirement that the surface be a closed orientable manifold. The representation can encode open surfaces (via the boundary QEF term in Equation 2), non-manifold edges (each edge flag independently activates or deactivates its quad face), and fully-enclosed interior structures (voxels are active wherever triangles intersect them, regardless of nesting).

The significance is not just that O-Voxel handles cases prior methods cannot — it's that it establishes a representation design principle: spatial regularity for efficient neural processing does not require geometric constraints inherited from field formalisms. The regular grid provides the convolutional inductive bias; the dual-grid features provide the geometric expressivity. This is a fundamental shift in how one thinks about 3D representation for generation, and it opens a design space that prior taxonomies obscured.

Evidence for this shift being foundational rather than incremental comes from the reconstruction results in Table 1 and Figure 12. On the Sketchfab Featured test set — deliberately curated from recent professional-grade assets with complex topology — O-Voxel at 1024³ achieves normal PSNR of 43.11 dB compared to 37.34 dB for the next best method (SparseFlex 1024) and 27.38 dB for Direct3D-S2 1024. The 5+ dB gap on normals indicates that prior field-based methods are not just slightly less accurate — they fundamentally cannot represent the sharp features and topological complexity present in these assets, because Flexicubes smooths over geometric discontinuities that O-Voxel's dual vertices preserve by design. The error maps in Figure 12 make this visually explicit: where baselines produce blurry approximations, O-Voxel reconstructs chainmail links and sharp ornamental patterns with near-ground-truth fidelity.

Innovation 2: Sparse Residual Autoencoding as a General Principle for High-Compression 3D VAEs

The Sparse Compression VAE achieves 16× spatial downsampling — a compression ratio the paper claims is "not seen in prior voxel-based methods" — but the intellectual contribution is not the specific number. Rather, it is the identification of the bottleneck in sparse 3D compression and the residual-autoencoding solution as a general design principle. Prior 3D VAEs for sparse voxels (Trellis at 4×, SparseFlex at 4×, Direct3D at 8×) hit a compression wall: beyond a certain ratio, naive downsampling (average pooling, strided convolutions) destroys too much fine-grained spatial information, and the decoder cannot recover it. The field's response was to accept lower compression ratios and compensate with more complex generative architectures (token packing in Trellis, optimized computation in Direct3D) — essentially, pushing the complexity into the generative model rather than fixing the VAE bottleneck.

The paper's diagnostic insight is that the problem is specifically about information destruction at the downsampling step, not about the VAE's overall capacity. The sparse residual autoencoding layer (Equations 4-5, adapted from DC-AE by Chen et al., 2024) addresses this by providing a non-parametric information-preserving shortcut: the fine-grid features are rearranged into the coarse grid's channel dimension rather than being pooled away, giving the decoder a direct path to recover basic spatial structure. The learned convolutional path then only needs to model the residual — the difference between naive averaging and the optimal coarse representation.

What makes this a genuine insight rather than an engineering trick is the ablation evidence in Table 3: removing the residual shortcut increases mesh distance by 69% at 16× compression and by 526% at 32× compression. The catastrophic degradation at 32× reveals that the residual shortcut is not merely improving quality — it is enabling compression ratios that would otherwise be fundamentally impossible regardless of how much model capacity is added. This pattern (non-parametric shortcut rescuing extreme compression) is a transferable design principle that should apply to any sparse data where downsampling destroys fine structure — it is not specific to 3D geometry or O-Voxel.

The ConvNeXt-style residual block redesign (replacing one conv layer with a point-wise MLP) is a secondary but complementary insight: on highly sparse data where spatial convolutions have low effective utilization (most neighbors are empty), shifting representational capacity from spatial (conv) to channel (MLP) operations improves fidelity without increasing cost. The ablation shows this yields a 16% MD reduction and 0.6 dB PSNR improvement at zero runtime cost — a pure architectural efficiency gain that challenges the default assumption that dense residual blocks are optimal for sparse data.

Innovation 3: Difficulty-Adaptive Inference via Cascaded Generation as a New Scaling Paradigm

The cascaded inference scheme in Section 4.5 — where a generated O-Voxel at one resolution is downsampled to provide a cleaner sparse structure layout for higher-resolution generation — is presented as a practical technique, but it embodies a deeper conceptual move: treating the generative model itself as a proposal distribution that can be refined through iterative self-application, analogous to how the example paper treated test-time compute as an adaptive resource to be allocated per-prompt.

The standard paradigm in 3D generation is one-shot: the model produces an output at a target resolution, and that output is final. If you want higher resolution or better quality, you train a bigger model or increase the latent resolution. The cascaded scheme breaks this by treating the generative process as recursive: the first pass provides a coarse structural prior, and subsequent passes refine within that structure. This is conceptually distinct from progressive growing during training (where the model architecture expands) — here, the same trained model is applied repeatedly at inference time, with the output of one pass serving as a structural scaffold for the next.

The key evidence that this is more than a trivial upsampling trick is Figure 8 (right), which shows that cascaded inference yields "finer details and enhanced structural stability" compared to single-pass generation at the same resolution. The mechanism is that the downsampled O-Voxel structure is cleaner than the directly predicted sparse structure — it has been implicitly regularized by the VAE decode/encode cycle which suppresses local errors and noise. This means the geometry generator starts from a structurally more coherent layout, allowing it to allocate its capacity to detail generation rather than structural correction.

The intellectual parallel to the earlier paper's compute-optimal test-time scaling is striking: both recognize that inference compute can be deployed adaptively rather than in a single forward pass, that intermediate outputs can provide information that guides subsequent computation, and that the allocation strategy (when to cascade, what resolution to bootstrap from) is a design choice that trades compute for quality. The cascaded scheme is a simpler instance of this principle — it doesn't involve difficulty estimation or per-prompt strategy selection — but it establishes the recursive self-application pattern that more sophisticated adaptive schemes could build upon.

Innovation 4: Decoupled Shape-Material Generation as a Diagnostic Architecture

The decision to train separate SC-VAEs for shape and material, and to generate them sequentially in the generative pipeline, might appear to be an implementation detail. But it embodies a substantive claim about the dependency structure of 3D assets: that material depends on shape (a metallic region must align with a specific geometric surface) but shape does not depend on material (the geometry of a car door is the same whether it's painted metal or rusted steel). This claim is not obviously true — one could imagine joint generation where shape and material co-evolve, with material constraints influencing local geometry (highly reflective surfaces might want smoother normals, translucent regions might influence internal structure). The paper's architecture takes a stand: the dependency is unidirectional.

The evidence supporting this architectural choice comes from the independent utility of the material generation stage: Section 4.3 demonstrates that the third stage can be used standalone as a shape-conditioned texture synthesis model, and Figure 7 shows it outperforms both multi-view fusion methods (Hunyuan3D-Paint) and UV-based methods (TEXGen) on PBR texture quality. The multi-view approaches "often suffer from inconsistencies both between the shape and synthesized images, as well as across different views, resulting in ghosting or blurred textures" — a failure mode that the decoupled native-3D approach avoids because it never projects to 2D in the first place. UV-based methods "suffer from ambiguous UV charts and seam artifacts" — a problem the decoupled approach avoids because material features are volumetric, not UV-dependent.

This is a negative design finding with positive implications: the decoupling works because the complexity of joint shape-material generation is unnecessary — the dependency structure is genuinely unidirectional, and forcing a joint model would introduce unnecessary coupling without improving either shape or material quality. This insight generalizes beyond this specific pipeline: any 3D generation system that handles both geometry and appearance must decide whether to model their relationship as joint, conditional, or independent, and this paper provides strong evidence that conditional (material-given-shape) is both sufficient and superior to joint.

The significance extends to practical deployment: the decoupled design enables the shape-conditioned texture synthesis application to operate on arbitrary input meshes, not just those generated by the system. A user could provide a hand-modeled mesh and receive PBR textures from the material stage alone, making the system useful as a standalone texturing tool. This modularity is a direct consequence of the architectural choice to encode the shape→material dependency explicitly rather than baking it into a monolithic latent space.

5. Experimental Analysis

Evaluation Methodology

Datasets. For reconstruction evaluation, the paper uses two test sets: (1) Toys4K-PBR, derived from the Toys4K benchmark (Stojanov et al., 2021) by filtering for assets containing all three standard PBR maps (base color, metallic, roughness), yielding 473 instances; and (2) Sketchfab Featured, a curated set of 90 high-quality, professionally vetted assets from Sketchfab (2025) uploaded within the past two years, selected from the "Staff Picks" category and filtered for metallic-roughness PBR workflow. For generation quality comparison and user studies, 100 AI-generated image prompts from NanoBanana (Fortin et al., 2025) are used to ensure training–testing disjointness. All test sets are unseen during training.
Base models. The SC-VAE is a fully sparse-convolutional U-Net with approximately 800M parameters (354M encoder, 474M decoder). The generative model consists of three DiT-based transformers, each containing approximately 1.3B parameters (width: 1536, blocks: 30, heads: 12, MLP width: 8192), for a total of approximately 4B parameters. The paper argues these models are scaled to demonstrate that the compact latent space enables large generative architectures that would be computationally intractable with less compact representations like SparseFlex's 225K tokens.
Metrics. For shape reconstruction: Mesh Distance (MD) with F1-score at threshold τ = 1×10⁻⁸, calculated as bidirectional point-to-mesh surface distance on 1M sampled points per mesh; Chamfer Distance (CD) with F1-score at τ = 1×10⁻⁶, computed on point clouds sampled from visible surfaces via depth unprojection from 100 views; PSNR and LPIPS on rendered normal maps from four fixed camera positions (30° pitch, 6° FoV, yaws at 30°, 120°, 210°, 300°). For material reconstruction: PSNR and LPIPS on rendered PBR attribute maps and shaded images. For generation: CLIP score measuring image-prompt-to-rendered-view semantic similarity; ULIP-2 and Uni3D scores measuring similarity between the input image and a colored point cloud sampled from the generated asset (10K points via Farthest Point Sampling). User study: preference rate from approximately 40 participants comparing fully rendered assets (overall quality) and normal-map renderings (shape quality) with interactive turntable inspection.
Baselines. For shape reconstruction: Dora (Chen et al., 2025) based on Shape2Vecset latent; Trellis (Xiang et al., 2025); Direct3D-S2 (Wu et al., 2025) at both 512 and 1024 resolutions; SparseFlex (He et al., 2025) at both 512 and 1024 resolutions. For image-to-3D generation: Trellis, Hi3DGen (Ye et al., 2025), Direct3D-S2, Step1X-3D (Li et al., 2025), and Hunyuan3D 2.1 (2025). For shape-conditioned texture generation: Hunyuan3D-Paint (multi-view PBR generation and fusion) and TEXGen (Yu et al., 2024, UV-based).
Generation budget / compute accounting. For reconstruction, compute is measured as number of latent tokens (#Token) with corresponding total dimensions (#Dim), spatial downsampling factor (f_down), and decoder runtime on an A100 GPU in seconds. For generation, inference time is reported in seconds on an NVIDIA H100 GPU at three resolutions: approximately 3s for 512³, approximately 17s for 1024³, and approximately 60s for 1536³. The paper explicitly argues that the compact latent space (16× spatial compression, ~9.6K tokens at 1024³) is what enables both the scaling of the generative model to 4B parameters and the fast inference — fewer tokens means shorter transformer sequence lengths and therefore cheaper self-attention.
Cross-validation / statistical protocol. No cross-validation protocol is described for the generative model evaluation. The test set of 100 AI-generated images is fixed. For the user study, approximately 40 participants evaluate all 100 prompts, with method positions randomized per question to prevent positional bias. Reconstruction metrics are computed on fixed test sets (Toys4K-PBR with 473 instances, Sketchfab Featured with 90). The paper provides metric definitions (Equations 10–13 in Appendix D.1.1) including precise formulas for MD, CD, and F-score.

Main Quantitative Results

Shape Reconstruction Fidelity

The headline result from Table 1 is that the SC-VAE achieves superior or competitive reconstruction quality across all metrics while using substantially fewer latent tokens and faster decoding compared to all baselines. At 1024³ resolution, the SC-VAE uses 9.6K tokens (306K total dimensions) at 16× spatial downsampling, compared to Trellis at 9.6K tokens (77K dims, 4×), Direct3D-S2 1024 at 17K tokens (271K dims, 8×), and SparseFlex 1024 at 225K tokens (1.8M dims, 4×). On the Sketchfab Featured test set — deliberately curated for complex, professional-grade assets — the SC-VAE achieves normal PSNR of 43.11 dB versus 37.34 dB (SparseFlex 1024), 27.38 dB (Direct3D-S2 1024), 24.31 dB (Trellis), and 22.09 dB (Dora 4.1K). The 5+ dB gap on normals represents a visually substantial difference in geometric detail preservation (visible in Figure 12's error maps). For MD F1-score (measuring overall geometry including internal structures), the SC-VAE achieves 0.0042 versus SparseFlex 1024's 0.3132 (a ~75× improvement in the F1 metric) and Trellis's 0.074 (an ~18× improvement). CD F1-score on visible surfaces is more competitive: SC-VAE achieves 0.5660 versus Direct3D-S2 1024's 0.01406, though this metric is on outer surfaces only and Direct3D benefits from its 8× downsampling (fewer tokens but less compression).

On the Toys4K-PBR test set, the pattern holds: SC-VAE 1024 achieves normal PSNR of 39.54 dB versus SparseFlex 512's 36.56 dB and Trellis's 30.29 dB. Decoder runtime at 1024³ is 0.301s for the SC-VAE versus 3.21s for SparseFlex 1024 (10× slower) and 13.0s for Direct3D-S2 1024 (43× slower).

A critical detail: the SC-VAE at 512³ resolution (2.2K tokens, 70K dims, 16× downsampling) achieves better or competitive metrics than all baselines at 1024³. For example, SC-VAE 512 achieves MD F1 of 0.0323 and normal PSNR of 39.54 dB on Toys4K, while SparseFlex 1024 achieves MD F1 of 0.3132 (worse) and normal PSNR of 37.34 dB (worse). This asymmetry — a 512³ SC-VAE output outperforms a 1024³ SparseFlex output — demonstrates that the quality gap is not merely about resolution; it stems from the representational capacity of O-Voxel versus field-based primitives.

The paper correctly notes that CD F1-scores favor Direct3D-S2 1024 on Toys4K outer surfaces (0.5660 for SC-VAE vs. 0.01406 for Direct3D-S2 1024 — a reversal of the pattern seen in other metrics). This is plausibly because Direct3D was optimized specifically for visible-surface accuracy, while SC-VAE distributes its representational budget across both visible and internal geometry.

Material Reconstruction Fidelity

The paper reports material reconstruction metrics for the SC-VAE alone, noting the absence of suitable baselines for encoding only material properties given shapes. The SC-VAE achieves 38.89 dB PSNR / 0.033 LPIPS on rendered PBR attribute maps and 38.69 dB PSNR / 0.026 LPIPS on shaded images (Section 4.1, material reconstruction paragraph). These numbers indicate faithful reproduction of material properties and consistent geometry-appearance alignment, though without baselines it's impossible to assess how much better this is than alternative approaches. The LPIPS values below 0.05 suggest perceptual near-losslessness.

Image-to-3D Generation Quality

Quantitative metrics (Table 2). The SC-VAE-based generation system achieves the highest scores across all automated metrics: CLIP score (visual alignment), ULIP-2, and Uni3D (both geometric similarity). The paper's method is the only one reporting all three metrics; baseline values are provided for comparison but the paper does not explicitly state which baselines produce which metrics (the table format is described narratively rather than reproduced). This is a weakness in the presentation — without seeing the full Table 2 values, the magnitude of the advantage cannot be assessed from the text alone.

User study (Table 2 and Table 7). With approximately 40 participants evaluating 100 AI-generated image prompts, the paper's method is "favored by participants, highlighting its clear superiority in visual realism, richness of geometric detail, and alignment with input prompts." The user study separated overall quality (fully rendered assets) and shape quality (normal map renderings), and the paper's method won both categories. Table 7 in the appendix provides detailed statistics, though the specific preference percentages are not quoted in the main text.

Qualitative comparison (Figures 6 and 14). The visual results demonstrate several advantages over baselines. For geometry, the paper's method produces sharper edges, more intricate details (mechanical joints of a crab, facial features of a character), and cleaner surfaces compared to baselines that "often produce results that are overly smoothed or contain noticeable artifacts." For materials, the paper's method generates PBR attributes that baselines like Step1X-3D, Trellis, Direct3D-S2, and Hi3DGen do not produce at all. Compared to Hunyuan3D 2.1 (which does produce PBR), the paper claims its textures are "visually more plausible and better aligned with the input prompts."

Inference efficiency. The paper reports approximately 3s for 512³, approximately 17s for 1024³, and approximately 60s for 1536³ on an H100 GPU. These are claimed to be "significantly faster than existing large 3D generation models," though no direct timing comparisons with baselines are provided in the text (the baseline inference times would need to be extracted from their respective papers).

Shape-Conditioned Texture Generation

Figure 7 provides a visual comparison of PBR texture quality when the material generation stage is used independently on provided meshes. The paper identifies specific failure modes of competing approaches: multi-view methods (Hunyuan3D-Paint) "often suffer from inconsistencies both between the shape and synthesized images, as well as across different views, resulting in ghosting or blurred textures," while UV-based methods (TEXGen) "suffer from ambiguous UV charts and seam artifacts, resulting in degraded visual quality." In contrast, the paper's native-3D approach produces "sharper textures, consistent shape-material alignment, and synthesis of textures for internal surfaces, crucial for complex assets with occluded or non-manifold geometry." No quantitative metrics are reported for this comparison.

Test-Time Compute and Resolution Scaling

Section 4.5 demonstrates two cascaded inference strategies, illustrated in Figure 8. Generating above training resolution: by downsampling a generated 1024³ O-Voxel to create a 96³ sparse structure layout and re-applying the geometry generator (trained at 64³), the system produces a 1536³ output. The generator generalizes to the higher latent resolution because RoPE provides relative position encoding independent of absolute grid size. Improving quality within trained resolution: downsampling an initial 512³ O-Voxel to a 64³ structure layout yields a cleaner structural prior than the directly predicted layout, which the geometry generator then refines into a 1024³ output with "finer details and enhanced structural stability." The paper frames this as a "controllable trade-off between computational efficiency and generation quality" — the cascaded pass costs extra compute (an additional DiT forward pass) but corrects local errors in the sparse structure prediction.

Ablation Studies and Robustness Checks

All ablations are conducted on the Sketchfab Featured assets at 256³ resolution. Results appear in Table 3.

Sparse residual autoencoding layer: Removing the non-parametric residual shortcuts (replacing them with average pooling for downsampling and nearest-neighbor for upsampling) causes severe quality degradation that worsens with compression ratio. At 16× compression, MD increases by 69% and normal PSNR decreases by 0.5 dB. At 32× compression, MD increases by 526% (from some baseline to nearly 6.3× worse) and PSNR drops by 1.6 dB. This demonstrates that the residual shortcut is not merely a quality improvement — it enables compression ratios that would otherwise be fundamentally impossible, because the information destroyed by naive pooling at the bottleneck cannot be recovered by the decoder regardless of its capacity. This ablation is the strongest evidence in the paper for the necessity of the autoencoding design.
Optimized residual block (ConvNeXt-style vs. standard): Replacing the optimized block (single conv + wide point-wise MLP) with standard residual blocks (two conv layers) increases MD by 16% and decreases normal PSNR by 0.6 dB, with "runtime unchanged." This confirms that on highly sparse 3D data, channel-wise MLP operations are a more efficient use of parameters than additional spatial convolutions. The "runtime unchanged" finding is notable — it means the MLP's additional FLOPs are offset by the halved convolution count, and the quality improvement is effectively free.

The paper does not provide ablations on:

The effect of the boundary QEF term (λ_bound) or regularization term (λ_reg) in Equation 2 on reconstruction quality for open/non-manifold surfaces.
The effect of the early-pruning upsampler on reconstruction quality versus a naive upsampler that activates all children (only efficiency is discussed).
The effect of two-stage training (Stage 1 direct regression only vs. Stage 1 + Stage 2 rendering losses) on final reconstruction quality. The rendering losses are justified conceptually but not ablated.
The effect of decoupled shape/material VAEs versus a joint VAE on reconstruction fidelity or generation quality.
The effect of the number of latent channels (32 at the bottleneck) on the compression-quality tradeoff.
The effect of different VAE training resolutions (256³ → 512³) and the generalization to untrained resolutions (1024³, 1536³) — the paper states this works but provides no quantitative degradation metrics.
The effect of the logitNorm(1,1) timestep sampling versus uniform sampling on generation quality.
The effect of classifier-free guidance drop rate on generation quality.

Critical Assessment

Claim 1: O-Voxel can faithfully capture arbitrary topology (open, non-manifold, enclosed) and PBR materials. Partially supported. The reconstruction metrics in Table 1 and qualitative results in Figures 11–12 convincingly demonstrate superior geometric fidelity on complex assets, including internal structures (Figure 12, final column error maps) and thin/open surfaces (Figure 11: plant, shopping cart). The normal PSNR gap of 5+ dB over baselines on the Sketchfab Featured set (43.11 dB vs. 37.34 dB for SparseFlex 1024) is strong evidence. However, the paper's specific claims about handling open surfaces and non-manifold geometry are demonstrated qualitatively but not isolated quantitatively — there is no separate metric for open-surface accuracy or non-manifold edge recovery. The boundary QEF term in Equation 2 is described as improving open surface representation, but no ablation isolates its contribution. Similarly, PBR material reconstruction reports metrics on the full test set without isolating challenging cases (translucent surfaces, highly metallic surfaces). The claim that opacity enables translucent surface handling is shown qualitatively (Figure 5, glass examples) but not measured.

Claim 2: The SC-VAE achieves 16× spatial compression with negligible perceptual degradation. Strongly supported for the tested resolutions. Table 1 shows that at 1024³ with 9.6K tokens, the SC-VAE achieves normal PSNR of 43.11 dB on Sketchfab Featured and 39.54 dB on Toys4K-PBR — substantially better than baselines at lower compression ratios. The material PSNR of 38.89 dB on PBR attributes and 38.69 dB on shaded images, with LPIPS below 0.05, supports near-losslessness. However, "negligible" is a subjective threshold — whether 38.89 dB PSNR is sufficient for production asset pipelines depends on the application. The paper tests only two resolutions (512³, 1024³) in Table 1 and does the ablation at 256³. There is no systematic sweep of compression ratios (e.g., 4×, 8×, 16×, 32×) with quality metrics to characterize the knee in the compression-quality curve. The claim that the VAE is "resolution-agnostic" and generalizes to untrained resolutions (1024³ and above) is stated but not quantitatively validated with metrics at those resolutions.

Claim 3: The generative model produces assets with quality "far exceeding" existing models. Supported for geometry, qualified for materials (incomparable baselines), weak on quantitative rigor. The user study and qualitative comparisons (Figures 6, 14) show clear visual advantages. The automated metrics in Table 2 give the paper's method the highest scores. However, the evaluation has significant limitations: (1) the test set of 100 AI-generated prompts is small for drawing broad conclusions about generalization; (2) the baseline comparison in Table 2 is incompletely described — specific metric values for each baseline are not provided in the text, making it impossible to assess the magnitude of the claimed superiority from the text alone; (3) several baselines (Trellis, Direct3D-S2, Hi3DGen) do not produce PBR materials at all, so the comparison is between a method that generates full PBR assets and methods that generate textured (but non-PBR) meshes — these are fundamentally different outputs; (4) the user study with ~40 participants on 100 prompts yields ~4,000 votes, but without statistical significance testing, the confidence in the preference margin is unknown.

Claim 4: Inference is "highly efficient" and faster than existing large models. Supported with internal timing, but no direct timing comparisons. The reported speeds (3s for 512³, 17s for 1024³, 60s for 1536³ on an H100) are fast in absolute terms. However, the claim of being "significantly faster than existing large 3D generation models" is asserted without providing analogous timing numbers for baselines on the same hardware. Different baselines run on different GPUs in their respective papers, making cross-paper timing comparisons unreliable. The timing advantage is plausibly real given the compact latent space, but the evidence presented is insufficient to quantify the speedup.

Missing experiments that would strengthen the paper:

Resolution-quality sweep for the SC-VAE: A systematic evaluation of reconstruction quality at compression ratios from 4× to 32×, characterizing the quality degradation curve and identifying where the "near-lossless" regime ends.
Open/non-manifold surface metrics: A test set specifically constructed to contain assets with open boundaries, non-manifold edges, and enclosed interiors, with metrics that separately measure fidelity on each topological category.
Ablation of the β-KL weight: The KL loss weight (λ_KL in Equation 6) typically trades off reconstruction fidelity against latent space smoothness. Understanding this tradeoff is critical for generative modeling — too much KL regularization hurts reconstruction; too little produces a non-smooth latent space that flow matching struggles to model.
Generation diversity metrics: The paper reports only fidelity (how well the output matches the input image) but not diversity (how much outputs vary for the same or similar prompts). A generative model that produces high-fidelity but low-diversity outputs may be memorizing training data rather than generalizing.
Comparison against a joint shape-material VAE: The decoupling is a key architectural choice, but the paper provides no evidence that it's better than a joint latent that encodes both shape and material in a single representation.
Failure case analysis: The limitations discussion (Appendix F) mentions aliasing artifacts for sub-voxel features and occasional small holes in reconstructed surfaces, but no quantitative characterization of how often these occur or their severity distribution across the test set.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For and Dominates at Deployment

The assumption or constraint. The entire compute-optimal framework rests on the ability to estimate prompt difficulty before deciding how to allocate the inference budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2: "estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). This means the reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter, potentially eliminating or reversing the claimed efficiency advantage. For the "predicted difficulty" variant—which uses the PRM's average final-answer score instead of ground-truth labels—the cost is equally high (2048 PRM evaluations per prompt), and while it removes the need for ground-truth answers, it does not reduce the sample count.

What evidence exists in the paper. The limitation is discussed qualitatively in Section 3.2. No experiment measures total cost including difficulty estimation. The curves in Figures 4 and 8 (showing 4× efficiency gains) are plotted with test-time compute budget on the x-axis, where that budget refers only to strategy execution after difficulty bin assignment. The difficulty estimation cost—which could be 256–2048 generations per prompt—is invisible on these axes.

Mitigation status. The authors flag this as "a key avenue for future work" (Section 3.2) and suggest training models to predict difficulty directly from the question text, but no such model is developed or evaluated. The paper also notes the possibility of adaptive difficulty estimation (starting with a few samples and adjusting), but does not explore it. Until this gap is closed, the 4× figure should be understood as an upper bound on achievable efficiency rather than a realized deployment gain.

Hard Problems Remain Fundamentarily Unsolved—Test-Time Compute Cannot Substitute for Capability

The assumption or constraint. The compute-optimal framework assumes that the base model produces correct solutions at some non-trivial rate—that there are correct answers in the proposal distribution for search or revision to find and refine. This assumption fails systematically on the hardest questions.

The consequence. Across all methods—search, revisions, and their compute-optimal combinations—the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%. Test-time compute amplifies existing capability but cannot create it from nothing. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help—there are no correct solutions to find. For such problems, pretraining remains the only viable path, and the FLOPs-matched comparison confirms this: on hard problems, the ~14× larger pretrained model outperforms the smaller model with any amount of test-time compute (Figure 1 bar charts show negative relative differences of up to −52.9% at R ≫ 1 for PRM search).

What evidence exists in the paper. Figure 3 (right) bin 5, Figure 7 (right) bin 5, Figure 9 bin 5 curves, and the FLOPs-matched bar charts in Figure 1 all converge on the same finding. The paper is transparent about this (Section 7 takeaway box): "on the hardest problems, no method makes meaningful progress."

Mitigation status. None. The paper acknowledges this as a fundamental boundary condition: test-time compute is complementary to pretraining, not a substitute. For genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution, pretraining remains the only path forward. The difficulty estimator can at least identify these problems (bin 5) and route them accordingly—potentially to a larger model or human review—but the current approach offers no mechanism to solve them with the smaller model.

The ~14× Larger Model Baseline Is Weakened by Non-Compute-Optimal Pretraining and Greedy Decoding

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023). The authors acknowledge this explicitly: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." Furthermore, the larger model is evaluated using only greedy decoding—no majority voting, no best-of-N, no search, no test-time compute of any kind.

The consequence. A Chinchilla-optimal model (Hoffmann et al., 2022) trained with ~14× more total FLOPs—scaling both parameters and data equally—would likely outperform a parameter-only-scaled model, making the pretraining baseline weaker than it needs to be. The reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions) may shrink or reverse against a properly compute-optimal larger model. Additionally, giving the larger model even a modest test-time compute budget (say, best-of-8 or a simple revision chain) would create a much stronger baseline that would test whether the combination of pretraining and test-time compute in the larger model outperforms test-time compute alone in the smaller model. The current comparison answers the question "is a small model with smart inference better than a large model with no inference strategy?" rather than "is a small model with smart inference better than a large model with comparable inference investment?"

What evidence exists in the paper. The pretraining-FLOPs formula and scaling assumptions are stated in Section 7. The caveat about parameter-only scaling is acknowledged there. The greedy decoding assumption for the larger model is implicit in the experimental description (Section 7, "we compare compute-optimal test-time compute with PaLM 2-S* to a model with ~14× more parameters with no extra test-time compute") but not discussed as a limitation.

Mitigation status. The authors flag compute-optimal pretraining (scaling both data and parameters) as future work, but do not discuss the greedy decoding asymmetry. A fairer comparison would give the larger model a test-time compute budget as well—e.g., using the same compute-optimal framework but with a proportionally smaller per-prompt budget since the larger model's per-token inference cost is higher—but this is not attempted.

The Revision Model Has a Fundamental Correct-to-Incorrect Reversion Problem with Only Partial Mitigation

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). This training data construction—pairing independently sampled incorrect and correct solutions post-hoc using edit distance—means the model never sees correct answers in its context during training. At test time, however, the model may produce a correct answer early in the revision chain, and since it was never trained to handle this situation, it will attempt to "revise" the correct answer.

The consequence. Approximately 38% of correct answers get converted back to incorrect ones in the subsequent revision step (Section 6.1). This creates a fundamental instability: the revision chain does not monotonically improve, and simply taking the last revision would degrade performance. The paper mitigates this by using majority voting or verifier-based selection across the chain to pick the best answer from any step rather than always taking the final output. However, these are patches that work around the reversion problem rather than solving it—they add computational overhead (the verifier must score every step in the chain) and introduce their own failure modes (the verifier may select an incorrect answer that happens to score highly). A more principled solution—such as training the model with correct in-context answers and teaching it when not to revise—is not explored.

What evidence exists in the paper. The 38% reversion rate is stated in Section 6.1. Figure 6 (left) shows that per-step accuracy gradually improves across the chain but with fluctuations—the improvement is not monotonic, consistent with occasional correct-to-incorrect transitions. The ReST^EM experiment (Appendix K, Figure 16) provides further evidence of fragility: attempting to optimize the revision model with RL-style training caused performance to degrade substantially with sequential revisions, suggesting that the revision training procedure is sensitive to data construction methodology in ways that are not fully understood.

Mitigation status. Partially mitigated via within-chain selection (majority voting or verifier-based), but the underlying model deficiency remains. The paper does not propose a training-time solution to the reversion problem, nor does it ablate alternative training data constructions (e.g., including correct in-context answers with a "stop revising" signal).

Revisions and PRM Search Are Studied Independently—Not Combined—Leaving Gains on the Table

The assumption or constraint. The paper studies two complementary axes for test-time compute—PRM-guided search (modifying how outputs are selected) and iterative revisions (modifying what the model generates)—but never combines them. Section 8 explicitly acknowledges this: "we did not experiment with PRM tree-search techniques in combination with revisions."

The consequence. The paper's empirical findings demonstrate that these two mechanisms have complementary, difficulty-dependent strengths: revisions excel on easy problems (local refinement of nearly-correct answers), while search excels on medium problems (global exploration of different solution strategies). Since real-world problem distributions contain a mix of difficulties, a combined system that uses the revision model as the proposal distribution within PRM-guided beam search—or uses the PRM to decide when to revise versus when to restart from scratch—could potentially achieve gains beyond either method alone. The current results therefore represent a lower bound on what a fully integrated system could achieve. This is particularly significant for the FLOPs-matched comparison: if combined revisions+search on the smaller model closed the gap with the larger model on medium-hard problems where each individual method currently underperforms, the case for test-time compute over pretraining would strengthen substantially.

What evidence exists in the paper. The difficulty-dependent results for search (Figure 3, right) and revisions (Figure 7, right) show their complementary strengths. The paper acknowledges the gap in Section 8. No experiments combine the two, and no analysis estimates the potential gain.

Mitigation status. Explicitly flagged as future work. The paper's decomposition framework (Section 2: proposal distribution vs. verifier) provides the conceptual scaffolding for such a combination, but the engineering and empirical validation remain undone.

Single Benchmark, Single Model Family, No Confidence Intervals on the Core Result

The assumption or constraint. All experiments use the MATH benchmark (500 test questions, split into five difficulty quintiles of ~100 each and further halved by two-fold cross-validation) with PaLM 2-S* as the base model. The paper acknowledges the single-model limitation but claims PaLM 2-S* is "representative of the capabilities of many contemporary LLMs" (Section 4), which is an assertion, not a demonstrated fact.

The consequence. Several aspects of the findings could be model- or benchmark-specific. The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution—a model with different calibration or error patterns might exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on base model capabilities that vary across model families. The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning—it is unclear whether the finding that beam search hurts easy problems but helps medium problems generalizes to code generation, logical reasoning, or other reasoning domains. More critically, with only ~50 questions per fold per difficulty bin (500 questions ÷ 5 bins ÷ 2 folds), the compute-optimal policy is selected based on approximately 50 examples per strategy, and the paper does not report confidence intervals or standard errors on the compute-optimal scaling curves (Figures 4 and 8). This makes it impossible to assess whether the observed efficiency gains (e.g., 4× at 16 generations) are statistically reliable or within sampling noise. Given the importance of the 4× efficiency claim to the paper's impact, this is a significant methodological gap.

What evidence exists in the paper. The dataset size (500 test questions) and cross-validation protocol are described in Section 3.2 and Section 4. The single model family is acknowledged in Section 4. No confidence intervals or error bars appear on Figures 4 and 8, and no statistical testing is reported for the central 4× efficiency claim.

Mitigation status. The paper does not address the sample size or statistical reliability question. The authors argue that their model is "representative" but provide no replication on other benchmarks (e.g., GSM8K for math reasoning, HumanEval for code) or other model families to support this. Future work would need to replicate the difficulty-dependent trends on different model families, different reasoning benchmarks, and with larger test sets to establish the generality of the compute-optimal scaling phenomenon and the specific 4× efficiency figure.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the landscape of 3D generative modeling by demonstrating that the tension between representational expressivity and structural regularity can be resolved through field-free geometry encoding rather than through architectural complexity in the generative model. Before this work, the prevailing wisdom was that to achieve spatially regular representations suitable for efficient neural processing, one had to accept the topological constraints of implicit fields—watertightness, manifoldness, and the inside/outside distinction inherited from signed distance functions. Methods that needed to handle open surfaces, non-manifold geometry, or enclosed interior structures were forced into unstructured representations (meshes, point clouds, Gaussians) that sacrifice the grid regularity that makes convolutions and latent compression efficient.

O-Voxel's Flexible Dual Grid breaks this dichotomy definitively. The core insight—that a regular grid can store explicit geometric features (dual vertex positions and edge connectivity flags) derived directly from mesh surface intersections without any intermediate scalar field—establishes a third category of 3D representation. It has the spatial regularity of field-based methods (enabling efficient sparse convolutions and 16× latent compression) but the topological expressivity of explicit methods (handling open, non-manifold, and enclosed surfaces natively). This is not an incremental improvement over prior structured latents—it eliminates the category error that prior work accepted as unavoidable. The evidence for this being foundational rather than incremental comes from the reconstruction numbers: on the Sketchfab Featured test set (deliberately curated from recent professional-grade assets with complex topology), the SC-VAE achieves normal PSNR of 43.11 dB at 1024³ with 9.6K tokens, compared to 37.34 dB for SparseFlex at the same resolution and 4× the tokens (225K). The 6 dB gap on normals is not a matter of degree—it represents an entire quality tier, where SparseFlex and Direct3D are fundamentally smoothing over geometric detail that lies outside their representational capacity.

The practical consequence for the field is a reframing of where investment should go. Prior work (Trellis, SparseFlex, Direct3D) was investing in increasingly sophisticated generative architectures to compensate for representational limitations—token packing to handle many latent tokens, optimized attention to reduce cost, complex multi-view postprocessing to texture surfaces that the geometry representation couldn't faithfully capture. This paper demonstrates that improving the underlying representation is higher-leverage than improving the generative architecture built on top of it. The SC-VAE's compact latent space (16× spatial compression, ~9.6K tokens at 1024³) is what enables the simplest possible DiT design—no token packing, no skip connections, vanilla self-attention—because the sequence length is already manageable. The generative model's 4B parameters are possible precisely because each parameter operates on a compact, high-quality latent representation. This inverts the field's implicit assumption that generative architecture innovation is the primary bottleneck.

The paper also resolves a contradiction in the multi-view texturing literature. Prior large-scale 3D generation systems (Hunyuan3D, Step1X-3D, Trellis) decomposed the problem into separate geometry and texture stages with a multi-view rendering bridge, arguing that this leverages powerful 2D image priors. The paper's shape-conditioned texture generation results (Section 4.3, Figure 7) demonstrate that native-3D material reasoning outperforms the multi-view approach on its own terms—sharper textures, consistent shape-material alignment, and synthesis for occluded surfaces—while also avoiding the view inconsistency, ghosting, and seam artifacts that plague multi-view fusion. This is not a marginal improvement; it's a different category of solution. The implication is that the 2D-prior argument, while plausible in theory, was masking a fundamental problem: projecting 3D texture synthesis into 2D and back introduces more artifacts than the 2D priors can correct. Native 3D material generation with a properly compact representation is the better path.

The cascaded inference scheme introduces a new scaling paradigm. Rather than treating generation as one-shot, the recursive self-application (downsample O-Voxel → re-apply geometry generator → produce higher-resolution output) establishes that generative models can be iteratively refined through self-bootstrapping—the output of one forward pass provides a cleaner structural prior for the next. This is analogous to how autoregressive models in language can refine their own outputs, but in the 3D domain where the "output" is a spatial structure rather than a token sequence. The conceptual move—that a trained model can operate at resolutions it wasn't trained on by leveraging its own intermediate outputs—is transferable to other domains where spatial regularity and resolution generalization matter.

Research directions that become more attractive:

Field-free representations for other modalities: The O-Voxel principle—store explicit geometric features on a regular grid without a scalar field—generalizes to any domain where surfaces or boundaries need to be represented with topological flexibility. Temporal event boundaries in video, material interfaces in volumetric medical data, or geological strata in subsurface modeling could all benefit from representations that capture discontinuities without field constraints.
Compact latent spaces for 3D downstream tasks: The SC-VAE's 16× compression with near-lossless reconstruction suggests that downstream tasks beyond generation—3D understanding, segmentation, physical simulation—could operate on these compact latents rather than raw geometry, dramatically reducing computational requirements.
Recursive self-improvement for 3D generation: The cascaded inference scheme is a first step toward models that refine their own outputs iteratively, analogous to how LLMs can be prompted to revise their answers. A natural extension is to train the generative model to explicitly condition on previous (lower-quality or lower-resolution) outputs as negative examples, learning a refinement policy rather than a single-pass generator.

Research directions that become less attractive:

Incremental improvements to field-based sparse voxel methods: If 43.11 dB normal PSNR with 9.6K tokens represents the state of the art, and field-based methods are fundamentally limited by their topology constraints, then further optimizing Flexicubes parameters or SDF grid resolution is unlikely to close the 6 dB quality gap. Investments should shift to field-free representations or to hybrid approaches that use fields only where their constraints are acceptable.
More complex multi-view texturing pipelines: The native-3D material generation results undermine the core premise of multi-view approaches—that 2D priors compensate for the 3D-to-2D-to-3D projection artifacts. Further work on better view consistency, better fusion, or better baking is treating symptoms rather than the root cause.
Increasing generative model size as the primary lever for quality: The paper deploys a 4B-parameter model, but the architectural simplicity (vanilla DiT) and the quality of the results suggest that representation quality, not model capacity, is the active bottleneck. Further scaling the DiT without improving the latent space would likely hit diminishing returns, since the model is already operating on a near-lossless compression of the target data.

Follow-Up Research This Work Enables

Stress-testing O-Voxel on the failure cases the paper identifies. The limitations section (Appendix F) acknowledges that O-Voxel can produce aliasing artifacts when sub-voxel geometric features (e.g., two parallel surfaces within the same voxel) cause the QEF solver to place the dual vertex between surfaces rather than on either one, and that reconstructed meshes sometimes contain small holes. A rigorous follow-up would construct a stress test suite specifically designed to probe these failure modes: assets with progressively closer parallel surfaces (measuring at what separation distance the dual vertex placement becomes ambiguous), assets with known hole-prone topology (thin shells, sharp corners with acute angles), and assets with deliberately degenerate edge-intersection configurations (edges intersecting exactly at voxel corners). For each stress case, the study would measure not just aggregate metrics but per-category failure rates—what fraction of sub-voxel parallel surfaces produce visible artifacts, what fraction of sharp corners produce holes, and how these rates scale with grid resolution. This would establish the practical resolution requirements for production use and identify whether the QEF formulation (Equation 2) can be improved with adaptive weighting or alternative solvers to mitigate specific failure modes.

Training a difficulty estimator for O-Voxel reconstruction quality. The paper shows that the SC-VAE achieves near-lossless reconstruction on average but does not characterize which assets are reconstructed poorly. A follow-up could train a lightweight classifier that predicts per-asset reconstruction quality (measured as normal PSNR or MD) from the O-Voxel features alone—specifically, from statistics of the QEF residual (the minimized error in Equation 2), the distribution of dual vertex displacements relative to intersection points, and the sparsity pattern of active voxels. If such a classifier could identify low-confidence reconstructions at encoding time (before decoding), it could enable adaptive resolution allocation: assets where the QEF residual is high (indicating sub-voxel ambiguity) could be encoded at higher resolution, while cleanly resolved assets use the standard 16× compression. This directly parallels the difficulty estimation problem in the earlier paper's compute-optimal inference framework, adapted to the 3D reconstruction domain. The key metric would be whether adaptive resolution reduces the tail of bad reconstructions (e.g., reducing the 95th percentile MD by 50%) at an acceptable average compute increase.

Combining O-Voxel geometry with NeRF-style appearance for view-dependent effects. The paper's material representation uses a standard PBR metallic-roughness model, which captures light-surface interaction under the assumption of isotropic, non-view-dependent reflectance. However, many real-world materials exhibit view-dependent appearance that PBR cannot capture—anisotropic reflections (brushed metal), subsurface scattering (skin, marble, wax), iridescence (oil films, butterfly wings), and transparency with refraction (glass with varying index of refraction). A natural extension is to replace the 6-channel PBR material feature with a small neural field per voxel—essentially, a tiny MLP head that takes view direction and lighting parameters as input and outputs the reflected radiance, trained per-asset through differentiable rendering, while keeping the O-Voxel geometry backbone unchanged. Because O-Voxel is field-free on the geometry side, the neural field would be strictly for appearance, avoiding the known geometry quality problems of full NeRF-based approaches. The key experiment would be to compare this hybrid O-Voxel + per-voxel neural material against pure O-Voxel PBR on a test set of assets with known view-dependent effects (e.g., the Stanford anisotropic BRDF database), measuring whether the neural material captures appearance that PBR misses without degrading geometry quality.

Ablating the necessity of decoupled shape and material VAEs. The paper trains separate SC-VAEs for shape and material, arguing that this enables independent texture synthesis and reflects the unidirectional dependency (material depends on shape, shape does not depend on material). However, a joint VAE that encodes both shape and material into a single latent space might achieve better compression or better generation quality by exploiting correlations between geometry and appearance—for instance, knowing that a region is metallic might help resolve geometric ambiguity at sharp edges. A follow-up would train a joint SC-VAE (same architecture, but with shape and material features concatenated at the input and output, and a single latent space) and compare reconstruction fidelity, latent space smoothness (KL divergence, FID of generated latents vs. training latents), and generation quality against the decoupled system. The hypothesis is that joint encoding helps when shape and material are strongly correlated (manufactured objects with uniform materials) but hurts when they are independent (a single mesh with multiple material zones, or shape-conditioned texture generation where the input mesh was not generated by the system). The key ablation would measure the performance gap between joint and decoupled VAEs as a function of the within-asset material diversity (number of distinct material zones per asset), which would characterize when each approach is preferable.

Training the generative model to perform iterative refinement through explicit conditioning on previous outputs. The cascaded inference scheme (Section 4.5) demonstrates that a generated O-Voxel can be downsampled and re-fed to the generator to produce higher-quality output, but this is a post-hoc hack—the model was not trained to refine its own outputs; it merely benefits from the cleaner structural prior provided by the downsampled O-Voxel. A more principled approach would train the geometry generator to explicitly condition on a previous (lower-quality or lower-resolution) O-Voxel output as additional input, with the training objective being to predict the residual between that previous output and the ground truth. This turns single-pass generation into a learned refinement process, analogous to how the revision model in the earlier paper was trained to correct its own mistakes. The training data would consist of pairs (noisy O-Voxel, clean O-Voxel), where the noisy version is produced by the current generator with limited inference steps or early stopping. The key experiment would compare this learned refinement against the cascaded inference hack at the same total compute budget, measuring whether explicit refinement training produces better detail recovery and structural stability than simple bootstrapping.

Characterizing the compression-quality Pareto frontier for 3D latents. The paper's 16× compression is a point estimate—it achieves high quality at this ratio, but the shape of the tradeoff curve is unknown. A systematic study would sweep the SC-VAE's spatial downsampling factor from 2× to 32×, measuring reconstruction quality (MD, CD, normal PSNR, material PSNR) and downstream generation quality (CLIP score, user preference) at each point. This would identify the Pareto frontier and answer several critical design questions: Is 16× the knee of the curve (where further compression sharply degrades quality), or is there room for 32× or 64× compression with acceptable quality loss? Does generation quality degrade more gracefully than reconstruction quality (suggesting the flow-matching model can compensate for VAE artifacts), or does it degrade faster (suggesting latent space smoothness, not just reconstruction fidelity, matters)? How does the optimal compression ratio depend on output resolution—should 1536³ assets use a higher or lower compression ratio than 512³? The study would also measure the generative model training cost as a function of latent token count, characterizing the practical tradeoff between VAE compression (cheaper generative training, potentially worse quality) and VAE fidelity (better quality, more expensive generative training).

Practical Applications and Downstream Use Cases

Production-grade 3D asset creation for games and film with automated PBR texturing. The most direct application is a pipeline where an artist provides a reference image (concept art, photograph, or AI-generated prompt) and receives a fully textured, PBR-material 3D asset ready for import into a game engine or rendering pipeline. The key value proposition over existing solutions is the combination of topology-agnostic geometry (open surfaces, interior structures, thin features) with relightable PBR materials (base color, metallic, roughness, opacity) in a single end-to-end system. A game studio needing 500 background props could generate them in under 2.5 hours on a single H100 (500 assets × 17 seconds per 1024³ asset), versus days or weeks of manual modeling. The cascaded inference scheme (Section 4.5) enables artists to trade compute for quality: a quick 512³ preview in 3 seconds for layout and composition, then a 1024³ or 1536³ final pass for detail. Critically, the generated assets can be relit in any environment because the PBR parameters are intrinsic—unlike assets with baked-in lighting from prior methods, these will look consistent across different scenes and lighting conditions. The ~38.9 dB material PSNR reported in Section 4.1 suggests the textures are perceptually near-lossless, meaning minimal manual touch-up would be needed for many production contexts.

Shape-conditioned material transfer for legacy 3D asset libraries. Because the material generation stage is decoupled and conditioned on geometry latents, it can be applied to any existing mesh, not just those generated by the system. A studio with thousands of legacy 3D models that have geometry but lack PBR materials (or have only diffuse textures) could run these through the O-Voxel encoder (a few seconds per asset on CPU), then through the material generator to produce physically-based textures aligned to the existing geometry. The key advantage over multi-view texturing approaches (Hunyuan3D-Paint) is the native 3D processing—Figure 7 shows that multi-view methods produce ghosting and view inconsistencies, while the native-3D approach produces sharp, aligned textures even for occluded surfaces. For assets with internal geometry (vehicle interiors, building interiors, nested mechanical parts), this is especially valuable because multi-view methods simply cannot see occluded surfaces to texture them. The limitation is that the material generator was trained on Objaverse-scale assets and may not generalize to highly domain-specific geometry (medical devices, specialized industrial equipment), but for common categories (furniture, vehicles, architecture, organic forms), the diversity of the training data (800K+ assets with TexVerse augmentation for PBR diversity) provides broad coverage.

Large-scale synthetic data generation for 3D computer vision. Training modern 3D vision models (pose estimation, depth prediction, 6-DoF object tracking, robotic grasping) requires vast quantities of annotated 3D data that is expensive to acquire through manual modeling or 3D scanning. This pipeline can generate diverse, high-quality textured 3D assets at scale — the 17-second inference time for 1024³ assets means 5,000+ assets per GPU-day. The PBR materials are crucial for domain randomization: by relighting the generated assets under randomized lighting conditions (which is only possible because metallic, roughness, and base color are available as separate channels), one can produce training images that cover the lighting variation expected at deployment time. The geometry quality (43.11 dB normal PSNR on complex assets) and topological flexibility (open surfaces, thin structures, interior details) means the generated data covers edge cases that prior methods—which produce smoothed, watertight-only geometry—would miss. For robotics applications where grasping thin or open structures (handles, levers, fabric) requires accurate geometry, this is a significant advance over field-based generation methods.

Interactive 3D content creation with cascaded preview-to-final workflow. The inference speed hierarchy—3 seconds for 512³, 17 seconds for 1024³, 60 seconds for 1536³—enables an interactive creation workflow where a designer iterates rapidly on concept and composition at low resolution, then commits to a high-resolution final pass only when satisfied. The cascaded inference scheme (downsampling a 512³ output to a 64³ structure layout, then generating 1024³ output) ensures that the final high-resolution asset is structurally consistent with the approved low-resolution preview—the designer doesn't risk getting a different shape at high resolution. This workflow is analogous to the proxy-to-final pipeline already standard in 3D content creation (blockout → high-poly modeling → texturing), but compressed into minutes rather than hours or days. The practical benefit is most pronounced for concept artists and level designers who need to populate scenes with varied assets: rather than finding or modeling each asset individually, they can generate dozens of variants at low resolution, select the best, and up-res only those. The compute cost scales sub-linearly with scene complexity because only approved assets receive the expensive high-resolution pass.