Vision2025~101 minarXiv:2511.16317

NaTex: Seamless Texture Generation as Latent Color Diffusion

score 2Zeqiang Lai, Yunfei Zhao, Zibo Zhao +5Tencent Hunyuan↑ 16★ 125Nov 20, 2025cs.CVhttps://github.com/Zeqiang-Lai/NaTex

🎯 Pitch

NaTex generates 3D textures at native resolution by treating them as dense color point clouds, completely bypassing the multi-view image baking that plagues existing methods with seams, misalignment, and occlusion errors. The framework introduces a geometry-aware VAE that compresses color point clouds over 80× and a diffusion transformer with native 3D positional control, outperforming all prior texturing pipelines in both fidelity and cross-view coherence.

1. Executive Summary

NaTex introduces a latent color diffusion framework that generates textures natively in 3D space by modeling texture as a dense color point cloud—predicting RGB values directly at 3D coordinates rather than relying on multi-view image baking. The system comprises a geometry-aware color point cloud VAE (achieving over 80× compression via cross-attention-based latent set encoding with a dedicated geometry branch for fine-grained shape guidance) and a multi-control diffusion transformer featuring native geometry control (RoPE positional embeddings paired with geometry latents concatenated channel-wise to color tokens). Tested against Paint3D, TexGen, Hunyuan3D-2, RomanTex, and MaterialMVP on a shared test set, NaTex-2B consistently outperforms prior methods across c-FID, LPIPS, CMMD, and CLIP-I metrics, establishing that direct 3D-native generation fundamentally avoids the occlusion inpainting, boundary misalignment, and cross-view inconsistency that cascade from 2D-lifting pipelines—while demonstrating that the approach generalizes training-free to part segmentation and with simple fine-tuning to material generation and texture refinement, provided the base geometry supplies sufficient surface sampling density for the point cloud representation.

2. Context and Motivation

The Core Problem: Textures Generated from 2D Views Cannot Achieve Seamless 3D Consistency

The fundamental problem NaTex addresses is deceptively simple to state but extraordinarily difficult to solve: how do you automatically generate a high-quality texture for a given 3D mesh such that the texture is perfectly aligned to geometric boundaries, free of visible seams, and consistent across the entire surface—including regions that would be occluded from any single camera viewpoint? This matters because textured 3D assets are the raw material of modern computer graphics: every video game character, every visual effects shot, every virtual try-on experience, and every immersive VR environment requires textured 3D models. Yet manual texturing remains an artisanal bottleneck—the paper explicitly describes it as "a profound bottleneck—an artisanal process that is both time-consuming and requires deep expertise" (Section 1). Automating this process would democratize 3D content creation and dramatically accelerate production pipelines across the entertainment, e-commerce, and simulation industries.

The paper identifies that the dominant paradigm—multi-view diffusion (MVD) texturing—has become nearly universal across both research systems (Hunyuan3D, CLAY, MaterialMVP, MVAdapter) and commercial products (Tripo, Rodin, Meshy, HiTem3D). Yet this paradigm carries three fundamental, irreducible failure modes that NaTex argues cannot be fully solved within the 2D-lifting framework—only worked around with increasingly complex heuristics.

Three Irreducible Failure Modes of Multi-View Texturing

The paper structures its critique around three specific pathologies, illustrated in Figure 2 and detailed in Section 1. Understanding each is essential to appreciating why the authors believe a paradigm shift is necessary, not merely incremental improvement.

1. Occlusion inpainting is fundamentally unavoidable. In any multi-view texturing pipeline, the first step is to render the input geometry from multiple camera viewpoints and generate 2D images (via a geometry-conditioned diffusion model) that align with those views. These images are then "baked" back onto the 3D surface via deterministic backprojection using the known camera parameters. The problem is that any finite set of camera views will miss some portion of the surface—consider the interior of a concave bowl, the underside of a character's arm, or the space between adjacent parts on a mechanical assembly. These occluded regions receive zero pixel coverage during backprojection, producing holes in the texture that must be filled by some post-hoc inpainting process.

The paper argues this is not a solvable engineering problem but a fundamental geometric limitation of the paradigm itself: "occlusion regions are an inevitable aspect of multi-view texturing; no matter the approach, they cannot be entirely avoided" (Section 1). Existing methods apply 2D inpainting or interpolation as a patch, but these operate without full 3D context—an inpainting algorithm filling a hole on a UV map has no direct awareness of the 3D surface geometry surrounding that hole, leading to visible seams and inconsistent content where the inpainted region meets the backprojected region.

2. Pixel-precise boundary alignment is nearly impossible from projected 2D signals. For a texture to look convincing, features must cleanly terminate at geometric boundaries—the color of a chair's seat should stop exactly at the edge where the seat meets the leg, a character's shirt pattern should not bleed onto the skin, and decorative trim should follow the exact contour of a 3D surface. Multi-view methods attempt to achieve this alignment by conditioning the 2D image generator on per-view geometric signals like normal maps or depth maps. However, the paper identifies two compounding sources of error:

Latent-space diffusion inherently introduces small positional errors. The denoising process operates in a compressed latent space, not pixel space, and the subsequent decoding step cannot guarantee exact boundary reproduction. These errors are typically imperceptible in flat 2D images but become glaring when backprojected onto a 3D surface where neighboring texture regions map to geometrically distant surface points (across a sharp edge, for instance).
2D geometric conditioning is informationally impoverished. A normal map rendered from a single viewpoint discards all structural information about occluded surfaces and foreshortens geometric details that are oblique to the camera. The authors state that "2D normal control is often insufficiently precise to handle fine-grained details" (Section 1). What looks like proper alignment in the 2D rendered view may correspond to millimeter-scale misregistration on the 3D surface.

3. Multi-view consistency is prohibitively expensive and remains unsolved. Even the most advanced video generation models (the paper cites HunyuanVideo and Wan) "struggle to achieve satisfactory results" in maintaining perfect consistency across views. The problem is that each view is generated with some degree of independence from the others—even with cross-view attention mechanisms—and the small variations in color intensity, content placement, and lighting interpretation across views accumulate during the baking process. When two views contribute overlapping pixel coverage to the same surface region (which is necessary for coverage), their slightly inconsistent predictions must be blended. The result is blurring, ghosting, or visible seam lines where the blend transition occurs.

The paper frames this as a cascading error problem: errors in each stage (view generation, backprojection, blending, inpainting) compound rather than cancel, and "these problems are broadly existing in 2D lifting methods... which are largely inevitable and stem from the cascading errors in modality changes" (Section 1). Each modality transition (3D → 2D projection → latent encoding → diffusion → decoding → 2D → 3D backprojection) introduces its own error distribution, and the pipeline amplifies rather than attenuates these errors.

Why Prior 3D-Native Attempts Haven't Solved the Problem

The idea of generating textures directly in 3D is not new. The paper acknowledges several prior attempts at native 3D texture generation (Section 2.2), but identifies specific limitations that prevented them from achieving production-quality results:

Proxy representations introduce their own artifacts. Early GAN-based methods (Texturify, Oechsle et al. 2019) operated on face colors or learned texture fields, but GAN training on 3D data is notoriously unstable and limited in diversity. More recent approaches have used various intermediate representations:

UV-map methods (TexGen, TexGarment, Paint3D) unwrap the 3D surface onto a 2D plane and generate textures in that flattened space. But UV unwrapping is itself a lossy, distortion-introducing process—seams must be cut, stretching is inevitable, and the resulting 2D image lacks the structured spatial coherence of the original 3D surface. The authors note that UV-space methods "rely on UV quality" and are sensitive to the specific unwrapping algorithm chosen.
Gaussian Splatting (TexGaussian) represents texture as colored 3D Gaussians, but this representation was designed for novel-view synthesis, not texture generation—controlling the density and placement of Gaussians to faithfully represent surface color at arbitrary resolution is challenging.
Octree representations (TexOct) compress the color field hierarchically, but octree traversal introduces computational overhead and the voxel-grid structure can produce visible quantization artifacts.
Triplane representations (UniTEX) compress 3D information into three orthogonal 2D feature planes, but the paper notes UniTEX is "limited at low resolution due to the complexity of triplane representation"—aggressive compression loses high-frequency texture detail.

The paper positions these attempts as having the right motivation (3D-native generation) but the wrong execution strategy—they used representations that were "data inefficient" and introduced their own "cascade errors." None of them, critically, leveraged the latent diffusion paradigm that has proven so successful in image, video, and 3D shape generation.

The unexplored opportunity: latent diffusion for color fields. The paper's key conceptual move is to observe that while latent diffusion models (LDMs) have become the dominant approach for image generation (Stable Diffusion, Flux), video generation (Wan, HunyuanVideo), and 3D shape generation (CLAY, Hunyuan3D), no prior work has applied this paradigm to texture generation—that is, to the problem of generating a color field conditioned on a fixed input geometry. The authors explicitly state this gap: "NaTex directly predicts RGB color for given 3D coordinates via a latent diffusion approach, a paradigm that has shown remarkable effectiveness in image, video, and 3D shape generation, yet unexplored for texture generation" (Section 1).

Why is this gap significant? Because existing 3D diffusion models (like CLAY or Hunyuan3D) jointly generate geometry and appearance, which means the geometric representation competes with the appearance representation for model capacity. By assuming the geometry is given (from an artist's model, a 3D scanner, or a separate geometry generator) and focusing the entire model capacity on texture, NaTex can allocate far more representational power to color detail, surface patterning, and material properties. This is a "separation of concerns" argument: geometry generation and texture generation are different problems with different requirements (geometry needs structural priors, texture needs fine detail and color fidelity), and a dedicated texture model can specialize.

How NaTex Positions Itself Relative to Existing Work

The paper's positioning can be understood along three axes:

Against MVD texturing: a paradigm critique, not an incremental improvement. NaTex is not proposing a better inpainting algorithm, a more consistent multi-view generator, or a smarter blending scheme. It argues that the entire 2D-lifting paradigm is fundamentally limited and that "native" 3D generation avoids these problems by construction. When textures are generated directly as a function $f(\mathbf{x}) = \mathbf{c}$ for 3D coordinates $\mathbf{x}$ , there are no occluded regions (the model can be queried at any surface point), no backprojection errors (there is no projection step), and no cross-view inconsistency (there is a single coherent color field, not multiple independently generated views). Section 1 frames this as changing textures from "second-class citizens" (derived products of 2D processes) to "first-class citizens" of the 3D representation.

Against prior 3D-native methods: a representation and scaling critique. The paper argues that earlier native methods chose representations (UV maps, octrees, triplanes, Gaussians) that were not well-suited to the scalability demands of modern generative modeling. The color point cloud representation, by contrast, is compatible with the VecSet/transformer architecture that has scaled successfully to billions of parameters in geometry generation (Hunyuan3D). The paper positions its approach as bringing "the full promise of 3D-native texture generation" into the scalable deep learning era, enabled by: (1) an efficient VAE achieving over 80× compression (making diffusion on point clouds computationally feasible), (2) a tight geometric conditioning mechanism (the geometry-aware dual-branch VAE), and (3) a flexible multi-control DiT that can be repurposed for downstream tasks.

For the broader field: a framework, not a point solution. The paper emphasizes extensibility throughout—the VAE can encode any "RGB-like" field (albedo, roughness/metallic as pseudo-color channels, part segmentation labels as color values), the DiT accepts multiple control modalities (image, geometry, color), and the inference process supports flexible token counts and sampling steps without retraining. This positions NaTex as a platform for 3D appearance generation rather than a texture-specific tool, with early evidence provided for material generation, texture refinement, part segmentation, and part texturing (Section 3.3, Figure 5).

The Specific Gap NaTex Fills

To summarize the gap: prior to NaTex, there existed:

MVD-based texturing pipelines that leverage powerful 2D image priors but suffer from irreducible occlusion, alignment, and consistency problems due to the 3D → 2D → 3D modality transitions.
Early 3D-native methods (GANs, UV-space models, octree/GS/triplane representations) that avoid the MVD problems in principle but produce lower-quality results due to representation limitations and do not scale to the data and model sizes that modern diffusion requires.
Joint geometry-and-appearance 3D diffusion models (CLAY, Hunyuan3D) that generate both shape and texture but allocate only a fraction of their capacity to appearance and cannot accept arbitrary input geometry as a condition for texture generation.

What was missing—and what NaTex provides—is a system that (a) generates texture natively in 3D (avoiding the MVD failure modes), (b) uses the scalable latent diffusion paradigm (enabling high-quality, diverse generation), (c) treats geometry as a fixed input condition (specializing model capacity on appearance), and (d) is architecturally extensible to materials, semantics, and refinement (providing a platform rather than a point tool). The paper's contribution is not any single architectural innovation in isolation but rather the demonstration that these pieces compose into a working system that substantially outperforms both MVD and prior native approaches, with the latent color diffusion paradigm serving as the unifying conceptual framework.

3. Technical Approach

3.1 Reader Orientation

NaTex is a latent diffusion model that generates a texture for a given 3D mesh by directly predicting RGB color values at any 3D coordinate on the mesh surface—think of it as learning a function color = f(x, y, z) that, when queried at every point on the surface, paints the entire object. The system solves the fundamental problem that multi-view-based texturing pipelines (which generate 2D images from different camera angles and "bake" them onto the 3D surface) inevitably produce misaligned, inconsistent, or incomplete textures because they must handle occluded regions, cross-view blending, and the information loss from projecting 3D geometry into 2D views—NaTex avoids all of these by working entirely in 3D, treating texture as a dense color point cloud and generating it with a compression-then-diffusion pipeline adapted from successful image and shape generation architectures.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components arranged in a standard latent diffusion pipeline:

Color Point Cloud Sampling — takes a textured 3D mesh (during training) or an untextured mesh with UV coordinates (during inference) and produces a dense set of 3D points, each carrying position, normal, and (when available) RGB color. This is the raw representation that the rest of the system operates on.
Geometry-Aware Color VAE — compresses the dense point cloud into a much smaller set of "latent tokens" (achieving over 80× compression) while simultaneously extracting geometric features through a parallel geometry encoder branch. The geometry branch produces a second set of latent tokens that encode surface shape information at the same spatial locations as the color tokens. During training, the VAE learns to reconstruct the original colors from the compressed latents; during inference, only the geometry branch is needed to encode the input mesh, providing shape conditioning to the generator.
Multi-Control Color DiT — a diffusion transformer that generates the color latent set from pure noise, conditioned on three types of control signals: (a) geometry control via positional embeddings (RoPE) on the latent token coordinates plus channel-wise concatenation of the geometry latent set from the VAE, (b) image control via DINOv2 features extracted from a reference image, and (c) color control (optional) via a previously generated or initial texture's latent set, used for refinement and material generation tasks. The DiT operates with a flow-matching objective, learning to predict the velocity field that transports noise to the target color latent distribution.
Color Field Decoder — takes the generated (or reconstructed) color latent set and queries it at arbitrary 3D coordinates using cross-attention, producing RGB values at those points. This decoder can output per-vertex colors, per-face colors, or a full UV texture map by sampling the corresponding 3D positions from the UV parameterization.

Information flows as follows during texture generation (inference): an untextured mesh enters → surface points are sampled with their positions and normals → the geometry branch of the VAE encodes these into geometry latent tokens → these tokens, along with their 3D positions (for RoPE) and an input image (for DINOv2 features), condition the DiT → the DiT denoises random latent tokens into a color latent set → the color decoder queries this latent set at every desired surface point → the resulting RGB values form the final texture. During VAE training, the flow is simpler: a textured mesh is sampled → both geometry and color branches encode their respective point cloud subsets → the decoder reconstructs colors → losses compare reconstruction quality.

3.3 Roadmap for the Deep Dive

First, the color point cloud representation and VAE architecture, because the VAE's design—particularly the tight coupling between geometry and color branches—is the foundational mechanism that enables all downstream generation, and understanding the compression scheme is prerequisite to understanding the DiT's latent space.
Second, the loss functions and training objectives for the VAE (KL divergence, color regression with near-surface supervision, truncated UDF loss), because these define what the latent space preserves and how geometric awareness is baked into the representation.
Third, the multi-control DiT architecture, covering each control mechanism (image, geometry, color) in sequence, because the DiT is where generation actually happens and the native geometry control design is the paper's core architectural contribution for enabling precise texture-geometry alignment.
Fourth, the training and inference strategies for the DiT (flow matching, classifier-free guidance, illumination-invariant loss, flexible token counts and sampling steps), because these operational details determine the model's generation quality, speed, and flexibility across applications.
Fifth, the application-specific adaptations (material generation, texture refinement, part segmentation, part texturing), because demonstrating how the same architecture extends to these tasks validates the framework's extensibility claim and reveals the practical implications of the design choices made in the core components.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and architecture paper whose core idea is that 3D texture generation can be formulated as a standard latent diffusion problem by choosing the right representation (dense color point cloud) and the right conditioning mechanism (native 3D geometry control via co-designed VAE and DiT), eliminating the cascade of errors that plague 2D-lifting pipelines.

Color Point Cloud Representation and the Motivation for Compression

The fundamental object that NaTex operates on is a dense color point cloud sampled from a textured 3D mesh. Formally, the input to the VAE is a set of points $\mathbf{P}_c \in \mathbb{R}^{N \times 9}$ , where $N$ is the number of sampled points (during VAE training, up to 6,144 tokens' worth of points, though the exact point count can vary) and each point carries 9 values: 3 for RGB color, 3 for 3D position $(x, y, z)$ , and 3 for surface normal $(n_x, n_y, n_z)$ . The goal of the VAE is to learn a continuous color field $f(\mathbf{x}) = \mathbf{c}$ that maps any 3D coordinate $\mathbf{x} \in \mathbb{R}^3$ to its color $\mathbf{c} \in \mathbb{R}^3$ , enabling reconstruction at arbitrary resolutions—including at UV coordinate positions or per-face centers that were not in the original training samples.

Why a point cloud rather than a UV map, voxel grid, or triplane? The paper's answer has several layers. First, a point cloud is a geometry-native representation: each point carries its exact 3D position and normal, preserving the spatial structure of the surface without the distortion introduced by UV unwrapping (where a 3D surface must be cut and flattened, inevitably stretching some regions and creating seams). Second, unlike voxel grids, point clouds are sparse—they only store information where the surface exists, avoiding wasted computation on empty space, which matters because texture detail is concentrated on the surface. Third, unlike triplanes (which compress 3D information into three 2D feature planes and lose high-frequency detail as acknowledged by the UniTEX limitations cited in Section 2.2), point clouds with positional encoding can represent arbitrarily fine spatial detail because the point density can be increased without changing the architecture. Fourth, and perhaps most importantly, the point cloud representation is compatible with the VecSet/transformer architecture that has scaled successfully in 3D shape generation (3DShape2VecSet, Hunyuan3D-VAE)—the VAE uses cross-attention between learned queries and input points, an operation whose complexity scales with the number of queries times the number of points, not quadratically with resolution.

However, operating directly on dense point clouds in a diffusion model would be computationally prohibitive. A typical textured mesh sampled at sufficient density for high-quality reconstruction might require tens or hundreds of thousands of points—far too many for a transformer-based diffusion model where self-attention complexity scales quadratically with sequence length. This is where the VAE compression becomes essential: the paper targets a latent representation of up to 6,144 tokens (during training), which at the input point densities used represents over 80× compression. The VAE's job is to compress the dense color information into a compact set of latent vectors that the DiT can efficiently process, while preserving enough detail for high-quality reconstruction.

Geometry-Aware Color VAE: Architecture and Dual-Branch Design

The VAE architecture is inspired by 3DShape2VecSet but adapted in several critical ways for the color domain. The core mechanism is a cross-attention-based encoder-decoder that uses a set of learned query vectors to attend to the input point cloud, producing a much smaller set of output latent tokens.

The encoder produces a set of latent tokens from a point cloud. Formally, given an input point cloud with $N$ points, the encoder uses a fixed set of $M$ learnable query vectors (where $M \ll N$ ; the maximum during training is $M = 6144$ ) that cross-attend to the input points through multiple transformer layers. Each query vector starts as a learned embedding and, after attending to the input points, becomes a latent token that summarizes the color and geometry information from the points most relevant to its learned "receptive field." The decoder operates in reverse: given the $M$ latent tokens, it uses cross-attention where arbitrary query 3D coordinates attend to the latent tokens, producing RGB predictions at those coordinates. This design means the decoder can be queried at any point, including points not seen during training—during inference, the model can produce colors at per-vertex positions, face centers, or UV-mapped coordinates by simply feeding those coordinates as queries to the decoder.

A critical architectural innovation: the geometry-aware dual-branch design. Prior work like 3DShape2VecSet was designed for shape autoencoding on geometry point clouds (positions only). NaTex needs to encode color point clouds, but also needs to provide geometric conditioning to the downstream DiT. The paper's solution is to add a parallel geometry branch to the VAE that is tightly coupled with the color branch. Specifically:

The geometry encoder takes only positions and normals from the input point cloud (not colors) and produces a geometry latent set—a set of latent tokens that encode local surface shape. These geometry queries are "point queries" randomly sampled from the color point cloud, meaning each geometry latent token is explicitly associated with a specific 3D position on the surface. This explicit positional association is what enables the native geometry control in the DiT (discussed below).
The color encoder takes positions, normals, and colors, and produces the color latent set. Crucially, the geometry latent set is used as queries to guide the color encoder—in the cross-attention layers of the color encoder, the queries come from the geometry latent set rather than being independently learned. This means the color latents are "anchored" to the geometry latents: they represent color information at the same spatial locations as the geometry tokens.
Both encoder branches share the same backbone architecture as Hunyuan3D-VAE, which consists of multiple layers of cross-attention (where queries attend to input points) and self-attention (where tokens attend to each other). The decoder only needs the color branch—it takes the color latent set and answers color queries.

Why this dual-branch design matters for downstream generation. The paper calls this "co-designing the VAE–DiT architecture" (Section 3.1). The core insight is that if the color and geometry latents are spatially aligned (each geometry token and its corresponding color token represent the same surface region), then the DiT can apply pointwise geometric conditioning by simply concatenating geometry tokens to color tokens along the channel dimension. This is far more precise than alternative approaches where geometry conditioning is provided as a separate, potentially misaligned embedding. The paper's ablation (Figure 9) directly tests this: replacing the tightly-coupled geometry branch with a standalone shape VAE (like Hunyuan3D-VAE trained independently on geometry) produces worse texture-geometry alignment, visible as color "diffusion" (bleeding) across geometric boundaries like the chair back. The tight coupling ensures that the geometry guidance is specific to exactly the surface region being colored.

An important note on query ordering and the VecSet modification. The authors state that while they retain the cross-attention compression mechanism from 3DShape2VecSet, their latent set is ordered—unlike the original VecSet formulation which was permutation-invariant. The reason is that the point queries are "known and sampled from the input geometry at test time," which makes pointwise geometry conditioning possible. If the latents had no fixed spatial correspondence (as in a true set), you could not meaningfully apply positional embeddings or concatenate geometry tokens to specific color tokens. This design choice trades permutation invariance (which is useful for unordered point clouds) for spatial structure (which is essential for geometric conditioning).

VAE Training: Loss Functions and Their Justifications

The VAE is trained with three loss terms, each serving a distinct purpose. The overall objective is:

$\mathcal{L} = \lambda_{\text{KL}}\mathcal{L}_{\text{KL}} + \lambda_{\text{color}}\mathcal{L}_{\text{color}} + \lambda_{\text{udf}}\mathcal{L}_{\text{UDF}}$

where $\lambda_{\text{KL}}$ , $\lambda_{\text{color}}$ , and $\lambda_{\text{udf}}$ are scalar weights controlling the relative importance of each term (exact values are not specified in the paper, but they are hyperparameters tuned during training).

What it computes: a weighted sum of three distinct losses. The KL term regularizes the latent space toward a standard Gaussian prior (necessary for the diffusion model to operate on a well-behaved distribution). The color term measures reconstruction accuracy—how closely the decoded colors match the ground-truth colors at queried points. The UDF term encourages the model to learn the underlying surface geometry from the point cloud, which helps the decoder distinguish between on-surface regions (where color prediction matters) and off-surface regions (where queries might fall due to sampling noise or during UV-based inference).

Why this form: each term addresses a specific failure mode. Without the KL term, the VAE could learn an arbitrary latent distribution that the DiT would struggle to model—the Gaussian prior ensures the diffusion process has a well-defined target distribution. Without the color term, there is obviously no signal for texture reconstruction. Without the UDF term, the decoder might produce plausible colors even far from the surface (since it has no geometric awareness of where the surface actually is), which would cause artifacts when queries are made at coordinates near but not exactly on the surface (e.g., at UV-mapped positions that don't perfectly match the training point distribution).

The color regression loss. The paper's treatment of the color loss is particularly detailed because it involves a non-obvious design choice that materially affects reconstruction quality. The loss supervises the model at two types of query points:

On-surface points: points that lie exactly on the mesh surface, sampled from the input point cloud. These are the primary training signal—the model must learn to reproduce the ground-truth color at every surface location.
Near-surface points: points generated by taking on-surface points and randomly offsetting them along their normal direction by a distance within a threshold $\gamma$ . The paper states this threshold exists but does not provide a specific numerical value. The purpose is to teach the decoder that colors should be consistent in a thin shell around the surface, not just at exact surface coordinates.

Why near-surface supervision matters. During inference, when generating a UV texture map, the model queries 3D coordinates derived from the UV parameterization. These coordinates may not exactly match the surface positions used during training due to floating-point precision, UV seam handling, or minor discrepancies between the training mesh and the inference mesh. If the decoder were trained only on exact surface points, it might produce arbitrary (and likely wrong) colors for coordinates that are even slightly off the surface. By training it to produce the same color within a small tolerance band around the surface, the model learns a smoother color field that is robust to small query perturbations. This is analogous to how signed distance functions are often supervised not just at the zero level set but in a band around it, ensuring smooth gradient behavior.

The truncated UDF loss. The Unsigned Distance Function (UDF) represents the distance from any 3D point to the nearest surface point. Unlike a Signed Distance Function (SDF), which distinguishes inside from outside (requiring a watertight mesh with consistent orientation), the UDF is always non-negative and works with arbitrary meshes—including those with holes, non-manifold edges, or inconsistent normals. The paper uses a truncated version:

$o(\mathbf{x}) = \begin{cases} 1, & \text{if } udf(\mathbf{x}) > s \\ \frac{udf(\mathbf{x})}{s}, & \text{if } udf(\mathbf{x}) \leq s \end{cases}$

where $udf(\mathbf{x})$ is the unsigned distance from point $\mathbf{x}$ to the nearest surface, and $s$ is a truncation threshold (not explicitly specified in the paper, but conceptually it defines the width of the "near-surface band").

What it computes: a normalized distance value between 0 and 1. Points exactly on the surface have UDF = 0, giving $o(\mathbf{x}) = 0$ . Points within the truncation band ( $udf(\mathbf{x}) \leq s$ ) get a linearly scaled value between 0 and 1. Points beyond the truncation distance get clamped to $o(\mathbf{x}) = 1$ . The model is trained to predict this $o(\mathbf{x})$ value at query points, effectively learning to distinguish on-surface from off-surface regions.

Why this form: the truncation is necessary because far-away points (deep inside or far outside the object) provide no useful information about the surface geometry—they are all simply "far." Without truncation, the model would waste capacity learning to predict large distance values precisely. The linear scaling within the band provides a smooth gradient that helps the model learn the surface location. The paper notes that "correlating the color point cloud with a watertight mesh (required for standard SDF loss) is non-trivial" (Section 3.1, Equation 1 discussion)—many 3D assets in training datasets are not watertight, so using a UDF rather than SDF is a practical necessity for working with real-world 3D content. The UDF also naturally handles thin structures and overlapping geometry that would confuse an SDF.

KL divergence term. This is the standard VAE regularizer that encourages the latent distribution to match a unit Gaussian prior $\mathcal{N}(0, I)$ . The encoder outputs mean and log-variance parameters, and the KL divergence penalizes deviation from the prior. This term is necessary for the subsequent diffusion model: the DiT learns to denoise samples drawn from $\mathcal{N}(0, I)$ into structured latent tokens, and this only works if the VAE's latent distribution is approximately Gaussian.

Training data construction for the VAE. During training, point clouds are sampled from textured meshes using Blender (Appendix A). The paper renders 24 views uniformly around the object with random elevation angles in the range of $45^\circ$ to $-30^\circ$ , using randomly selected illumination environments (point lights, area lights, HDRI maps). The color point cloud is then sampled from the rendered and ground-truth textured mesh. The explicit mention of multiple lighting conditions during data preparation is important: it means the VAE sees textures under varying illumination, which helps it learn to separate albedo (the intrinsic surface color) from lighting effects—a capability that supports the material generation application where albedo must be disentangled from specular highlights and shadows.

Multi-Control Color DiT: Architecture Overview

The diffusion transformer (DiT) is responsible for generating the color latent set. It follows the rectified flow diffusion transformer architecture (inspired by Flux) with specific adaptations for 3D texture generation. At a high level, the DiT takes a set of noisy color latent tokens and iteratively denoises them into clean color latents, conditioned on three types of signals: the input geometry, a reference image, and (optionally) an initial texture. The denoised color latents are then decoded by the VAE's color decoder to produce the final texture.

Why rectified flow (flow matching) rather than DDPM? The paper does not explicitly justify this choice, but rectified flow has become the dominant paradigm in recent large-scale generative models (Stable Diffusion 3, Flux) because it enables straighter sampling trajectories—the model learns to predict a constant-velocity path from noise to data, which allows fewer sampling steps at inference time. The paper exploits this property: all reported results use only 5 diffusion steps (Appendix A), and the model even achieves "one-step generation capability" (Section 4.2, Figure 10) despite not being distilled, thanks to "the strong conditioning." This would be substantially harder with a standard DDPM objective that learns curved trajectories.

The base DiT architecture. The paper uses a transformer backbone where the noisy color latents form the input token sequence. Each latent token is associated with a 3D position (the position of the point query it corresponds to in the VAE). Self-attention layers allow tokens to interact globally, which is crucial for texture coherence—a token representing a surface point on the front of an object can attend to tokens on the back, enabling the model to maintain consistent color palettes and patterns across the entire surface without needing an explicit consistency module (as MVD methods require).

Multi-Control Color DiT: Image Control

Image conditioning provides the semantic and stylistic guidance—given a reference image of an object, the model generates a texture that matches its appearance. The paper uses DINOv2-Giant as the image encoder, following the approach of Hunyuan3D-2.

Why DINOv2 rather than CLIP? DINOv2 produces dense spatial features (patch-level embeddings) that preserve spatial information about where visual features appear in the image, whereas CLIP produces a single global embedding vector that discards spatial layout. For texture generation, knowing where a stripe, logo, or color transition appears on the reference object is crucial for placing the corresponding texture feature on the correct part of the 3D mesh. DINOv2's patch features provide this spatial grounding.

Specific implementation details (Section 3.2, "Image Control" paragraph). The paper uses embedding from the last hidden layer without the class token—this means they take the per-patch embeddings (a sequence of feature vectors, one per image patch) and discard the global classification token. Unlike Hunyuan3D-2, which processes images at resolution 518, NaTex scales the input to 1022 resolution, because "higher-resolution conditioning helps for capturing fine-grained details" (Section 3.2). To manage token length (since 1022² patches would produce an impractically long sequence at standard DINOv2 patch size), the paper retains the original aspect ratio by cropping the object from the 1022-resolution input image using a binary mask—the image is cropped tightly around the object, reducing the number of patches to only those containing the object, not the background. No additional positional embedding is used for image tokens, as the authors find that "the position information encoded in Dino is sufficient" (Section 3.2)—DINOv2's training objective (self-distillation with no labels) incidentally preserves enough positional information in its patch features for the DiT to infer spatial layout without explicit position codes.

The DINOv2 patch tokens are concatenated with the noisy color latent tokens (and geometry tokens, and optionally color control tokens) to form the full input sequence to the DiT's self-attention layers. This means every latent token can attend to every image patch, allowing global reasoning between the reference image and the 3D surface.

Multi-Control Color DiT: Native Geometry Control (Core Innovation)

Geometry conditioning is the mechanism that ensures the generated texture aligns with the input mesh's surface details—the stripes on a shirt follow the contours of the sleeves, the eyes on a character land on the face not the ear, and decorative trim traces the exact edge of a geometric feature. The paper proposes native geometry control which comprises two complementary mechanisms:

1. RoPE (Rotary Position Embedding) based on latent token positions. Recall that each latent token in the color set is associated with a specific 3D position—the position of the point query that generated it in the VAE encoder. The paper applies RoPE to the noisy color latent tokens using these 3D positions. RoPE encodes positional information by rotating the query and key vectors in self-attention by an angle proportional to the position difference, so tokens that are spatially close on the 3D surface have similar rotations and attend more strongly to each other.

What RoPE provides geometrically. This is the "coarse structural guidance" (Section 3.2, "Geometry Control" paragraph). RoPE gives the DiT an inductive bias toward spatial locality: a latent token representing a point on the left sleeve of a shirt will naturally attend more to other tokens on the left sleeve (which have similar 3D positions) than to tokens on the right sleeve or the collar. This helps the model learn that texture patterns should be continuous across nearby surface regions and respect the 3D topology of the object—a stripe that wraps around a cylindrical arm should continue around the circumference, not suddenly jump.

Why RoPE positional encoding matters. Without it, the DiT would have to learn spatial relationships purely from the ordering of tokens in the sequence (which is arbitrary, since point clouds have no natural ordering) or from the geometry latent concatenation (discussed next). RoPE injects an explicit geometric prior that makes the learning problem substantially easier. The ablation in Figure 9 confirms this: removing RoPE from each color token degrades texture-image alignment, visible in features like "the stripes on the awning of the house and the colors of the traffic light" (Section 4.2).

2. Geometry latent set concatenation. This is the second and more fine-grained geometry control mechanism. The geometry branch of the VAE produces a geometry latent set during both training and inference. During inference, the untextured input mesh is sampled to produce a geometry point cloud (positions and normals only), which is encoded by the geometry encoder into a set of geometry latent tokens. These geometry tokens are spatially aligned with the color tokens because the geometry latent set was used as queries during color encoding—each geometry token and its corresponding color token represent the same surface location.

The paper concatenates the geometry latent tokens with the noisy color latent tokens along the channel dimension, not the sequence dimension. This means the sequence length stays the same, but each token now carries both geometry information (from the fixed geometry latents) and color information (from the noisy, evolving color latents). The concatenation is: [color_latent_i || geometry_latent_i] for each position $i$ , then projected back to the model's hidden dimension through a linear layer.

Why this concatenation is the key to native control. This is the core insight of the paper's architecture. Because the tokens are pointwise aligned, the DiT can learn precise correspondences between local surface geometry and local texture. When processing a token representing a point near a sharp edge on the mesh, the concatenated geometry latent provides detailed information about that edge's curvature, orientation, and neighboring surface topology, allowing the DiT to place texture boundaries exactly at geometric boundaries. This is fundamentally different from MVD methods where geometry is provided as a 2D normal map—the normal map loses precision due to projection, cannot represent occluded geometry, and does not provide a direct 1-to-1 correspondence with the color prediction. The ablation in Figure 9 confirms: replacing the tightly-coupled geometry branch with a standalone shape VAE produces worse texture-geometry alignment, with "color sometimes diffus[ing], as seen in the chair back" (Section 4.2).

The combination of RoPE (coarse structural guidance) and geometry latent concatenation (fine-grained, pointwise guidance) is what the paper means by "native geometry control"—the model receives geometric information that is complete (no occlusion), precise (per-point rather than per-view), and structurally aligned with the color representation, enabling the exact boundary alignment visible in Figure 7 where NaTex perfectly aligns textures to geometry while competing methods show misaligned stars and buttons.

Multi-Control Color DiT: Color Control (Optional Prior Texture)

The DiT supports an additional conditioning signal: a color control latent set obtained by encoding an existing texture through the VAE. This is concatenated with the noisy color latents and geometry latents along the channel dimension, producing a three-way concatenation: [noisy_color || geometry || prior_color].

Why color control exists and what it enables. This mechanism creates a "neural refiner" capability. When conditioning on an initial texture (which might be incomplete due to occlusion, misaligned due to MVD errors, or low-resolution), the DiT learns to produce an improved version that corrects these defects while preserving the intended appearance. The paper demonstrates three use cases:

Texture refinement/inpainting (Section 4.1, Figure 8): Given a texture with occluded regions (holes from the backprojection process), the model conditioned on that partial texture generates a complete texture that naturally fills the holes—the comparison in Figure 8 shows NaTex producing "cleaner and better-aligned textures in the occluded regions" versus OpenCV interpolation, visible in the zoomed-in window of the house.
Material generation (Appendix B): Given a textured mesh with albedo (base color), a second DiT conditioned on this albedo texture generates roughness and metallic maps. The albedo provides the color control signal that tells the material generator what the surface looks like, so it can infer appropriate material properties (shiny metal for metallic surfaces, rough matte for fabric, etc.).
Part segmentation with initial color guidance (Section 3.3): The color control mechanism can be adapted to condition on a 2D segmentation mask, producing a 3D part segmentation that respects the mask's boundaries.

Training with color control. During training, color control is applied by encoding the ground-truth texture, partially corrupting it (to simulate the types of defects the refiner should correct), and using the corrupted version as the color control signal while training the DiT to recover the original. The paper does not specify the exact corruption procedure, but the principle is standard for refinement models: the model learns a conditional distribution over clean textures given noisy or incomplete ones.

DiT Training: Flow Matching, Classifier-Free Guidance, and Illumination Invariance

Flow matching objective. The DiT is trained with a flow matching loss rather than the standard DDPM noise prediction loss. In flow matching, the model learns to predict a velocity field $\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c})$ that transports samples from a simple base distribution (Gaussian noise at $t = 0$ ) to the data distribution (VAE latents at $t = 1$ ). The training loss is:

$\mathcal{L}_{\text{FM}} = \|\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c}) - (\mathbf{z}_1 - \mathbf{z}_0)\|_2^2$

where $\mathbf{z}_t = (1 - t)\mathbf{z}_0 + t\mathbf{z}_1$ is the linear interpolation between noise $\mathbf{z}_0 \sim \mathcal{N}(0, I)$ and data $\mathbf{z}_1$ (the target color latents), $t$ is the time step, $\mathbf{c}$ represents all conditioning signals, and $(\mathbf{z}_1 - \mathbf{z}_0)$ is the ground-truth velocity (constant along the straight path from noise to data).

What it computes: at each training step, a random $t$ is sampled, a noise sample $\mathbf{z}_0$ and a data sample $\mathbf{z}_1$ are drawn, their linear interpolation $\mathbf{z}_t$ is computed, and the model predicts the velocity that would move from $\mathbf{z}_t$ toward $\mathbf{z}_1$ . The loss penalizes the squared L2 distance between the predicted velocity and the true constant velocity.

Why flow matching: the straight-line paths enable fewer sampling steps—the model only needs to approximate the straight trajectory, not a curved one. This is why the paper achieves good results with only 5 sampling steps and even one-step generation (Figure 10). The linear interpolation schedule $\mathbf{z}_t = (1-t)\mathbf{z}_0 + t\mathbf{z}_1$ is the simplest possible conditional probability path and has become the standard in recent diffusion models.

Classifier-free guidance (CFG). The paper uses CFG with a guidance scale of 2.0 and replaces conditioning embeddings with zero embeddings at a 10% probability during training (Appendix A). During inference, the guided prediction is:

$\mathbf{v}_{\text{guided}} = \mathbf{v}_\theta(\mathbf{z}_t, t, \emptyset) + w \cdot (\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c}) - \mathbf{v}_\theta(\mathbf{z}_t, t, \emptyset))$

where $w = 2.0$ is the guidance scale, $\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c})$ is the conditional prediction, and $\mathbf{v}_\theta(\mathbf{z}_t, t, \emptyset)$ is the unconditional prediction (with zeroed-out conditioning). This sharpens the conditional distribution, trading diversity for fidelity to the conditioning signals.

Illumination-invariant loss for albedo generation. For albedo (base color without lighting) texture generation, the paper introduces an additional loss term that encourages the model to produce lighting-independent predictions. The hybrid loss is:

$\mathcal{L} = \|\epsilon_{\text{pred}} - \epsilon_{\text{gt}}\|_2^2 + \gamma \|\epsilon_{\text{pred}} - \epsilon_{\text{pred2}}\|_2^2$

where $\epsilon_{\text{pred}}$ is the model's velocity prediction for an input image under one illumination condition, $\epsilon_{\text{pred2}}$ is the prediction for the same object under a different illumination, $\epsilon_{\text{gt}}$ is the ground-truth velocity (which is illumination-independent for albedo), and $\gamma = 5$ is a weight hyperparameter.

What it computes and why: the first term is the standard flow matching loss. The second term penalizes the model if its predictions differ when the same object is shown under different lighting conditions. If the model were picking up on illumination cues (shadows, specular highlights) to inform its texture prediction, these cues would differ between the two inputs, producing different predictions. By penalizing this difference, the loss forces the model to ignore illumination and focus on the intrinsic surface properties (albedo). This is following the approach of MaterialMVP, cited in the paper.

The paper notes that the illumination-invariant loss is "introduced once pretraining converges" (Appendix A)—the model is first trained with only the standard flow matching loss, and once it has learned the basic texture generation task, the illumination-invariant loss is added to refine albedo separation. This staged training prevents the illumination-invariant loss from interfering with early learning when the model hasn't yet developed the capability to distinguish texture from lighting.

VAE: Encoding Any Field as Color

A distinctive feature of the NaTex framework is its treatment of the VAE as a universal "color" encoder—any modality that can be represented as RGB-like values can be encoded and generated using the same architecture with no modification. The paper gives two specific examples:

Material properties as pseudo-color. Physically Based Rendering (PBR) materials typically require multiple texture maps: albedo (base color, 3 channels), roughness (how matte or glossy, 1 channel), and metallic (metal vs. non-metal, 1 channel). NaTex represents roughness and metallic by packing them into an RGB image: roughness in the red channel, metallic in the green channel, and the blue channel set to zero. This "pseudo-color" image is then treated identically to a regular RGB texture—sampled as a color point cloud, encoded by the VAE, and generated by the DiT conditioned on the albedo texture as color control. The decoder naturally produces three values per query, which are interpreted as roughness, metallic, and a padding zero. This is elegant because it requires zero architectural changes to support material generation—the VAE and DiT don't need to know whether the three channels represent RGB color or material properties; they just learn to generate them from the data distribution.

Part segmentation as color. Similarly, discrete part labels (e.g., "this region is the left sleeve, this region is the torso") can be mapped to RGB color values—each part gets a distinct color, turning the segmentation into a color texture. The model can then be conditioned on a 2D segmentation mask (from SAM, for instance) and generate the corresponding 3D part-colored texture. The paper shows this works both training-free (by simply feeding the mask as the image condition, Section 3.3) and with fine-tuning (Appendix B, Figure 15) for improved accuracy on complex structures.

Why this universality matters. It means the same trained VAE and DiT backbone can serve as a platform for multiple appearance-related tasks without architectural forks. The VAE learns general-purpose compression of surface-attached "color" fields, and the DiT learns general-purpose conditional generation in that latent space, with the meaning of "color" determined by the training data distribution. This extensibility is a significant practical advantage over task-specific architectures that would need to be redesigned and retrained for each new modality.

Inference Flexibility: Token Scaling and One-Step Generation

The paper highlights an important property of the architecture: the VAE and DiT support variable token counts and variable sampling steps at inference time without retraining or distillation.

Token scaling. The VAE is trained with up to 6,144 tokens, but during inference, more tokens can be used by simply increasing the number of point queries. Figure 10 demonstrates this: increasing the token count improves generation quality and alignment (visible in the windows of the generated house), because more tokens means higher spatial resolution—each token represents a smaller surface region, enabling finer detail. This works because the VAE encoder uses cross-attention where a fixed set of learned query vectors attend to the input point cloud; at inference, the number of queries can simply be increased (each query is a learned embedding that can attend to any set of input points), and the DiT can process the longer sequence because transformers naturally handle variable-length inputs. The computational cost scales with the token count (self-attention is quadratic), but no architectural changes are needed.

One-step generation. Figure 10 further shows that the model achieves "one-step generation capability" despite not being distilled (unlike consistency models or progressive distillation approaches that require specialized training). The paper attributes this to "the strong conditioning"—because the DiT receives rich, spatially precise conditioning signals (RoPE + geometry latents + image features + optional color control), the unconditional velocity prediction (used in CFG) is already quite informative, and the guided prediction with $w = 2$ can make a large correction in a single step. This is a practical advantage for applications like texture refinement where speed matters, though the paper notes that 5 steps are used for all reported results, suggesting the quality does improve with more steps.

Output Modalities: UV Texture Maps, Per-Vertex Colors, Per-Face Colors

The inference pipeline supports multiple output formats by querying the decoded color field at different sets of 3D coordinates:

UV texture map: For each texel (pixel) in the UV texture image, the corresponding 3D position on the mesh surface is computed via the UV parameterization, and the decoder is queried at that position to produce the RGB value. This is the most common output format for integration with standard rendering pipelines.
Per-vertex colors: The decoder is queried at each vertex position of the mesh, producing a color for that vertex. Between vertices, colors are interpolated by the rendering engine.
Per-face colors: The decoder is queried at each face center (the centroid of the triangular face), producing one color per face. This is useful for applications where face-level granularity is sufficient.

The key point is that the decoder is a continuous function $f(\mathbf{x}) = \mathbf{c}$ , so it can be evaluated at any set of 3D coordinates—the output format is a post-processing choice, not a constraint of the generation process. This is possible because the VAE decoder uses cross-attention: arbitrary query coordinates attend to the latent tokens, producing predictions at those coordinates. This contrasts with methods that generate textures in a fixed-resolution UV map (where changing resolution requires retraining or resampling artifacts) or as per-face colors with a fixed mesh topology.

Summary of Design Choices and Their Justifications

Color point cloud representation over UV maps/triplanes/octrees: preserves 3D spatial structure without distortion, compatible with scalable transformer architectures, and enables the decoder to be queried at arbitrary resolution—avoiding the distortion, resolution constraints, and computational inefficiency of alternatives.
Geometry-aware dual-branch VAE with geometry latents as color encoder queries: produces spatially aligned geometry and color latent sets, enabling pointwise concatenation in the DiT for precise texture-geometry alignment—a tighter integration than using a separate, independently trained shape VAE.
Truncated UDF loss with near-surface color supervision: handles non-watertight meshes (common in real-world 3D assets) and ensures robust color prediction for query coordinates slightly off the surface—necessary for practical UV-map generation where coordinate precision varies.
RoPE on 3D positions plus geometry latent concatenation for native geometry control: provides both coarse structural priors (via RoPE's spatial attention bias) and fine-grained per-point guidance (via channel-wise concatenation of spatially aligned geometry features) that together enable exact boundary alignment without requiring explicit consistency modules.
Flow matching over DDPM: enables efficient sampling (5 steps, down to 1 step) due to straight trajectories, critical for practical deployment where inference speed matters.
DINOv2-Giant at 1022 resolution with object cropping over CLIP or lower resolution: preserves spatial information in image features for precise feature-to-surface correspondence, and higher resolution captures fine detail—justified by the design goal of matching fine-grained texture patterns from reference images to specific 3D surface regions.
Multi-control concatenation along channel dimension: preserves sequence length while allowing the DiT's self-attention to jointly reason over all conditioning signals at every token position, which is computationally more efficient than cross-attention-based conditioning and enables the strong conditioning that supports one-step generation.
Universal "color as any field" encoding: enables material generation, part segmentation, and other appearance-related tasks with zero architectural changes—a practical demonstration of the framework's extensibility that reduces engineering overhead for new applications.

4. Key Insights and Innovations

Innovation 1: 3D-Native Texture Generation as a Latent Diffusion Problem—A Paradigm Shift, Not a Pipeline Patch

The paper's most fundamental intellectual contribution is the reframing of texture generation from a 2D-lifting problem (generate images, bake them onto 3D) to a 3D-native latent diffusion problem (compress a color point cloud, generate it in latent space, decode to arbitrary surface queries). This is not an incremental improvement over multi-view diffusion (MVD) texturing—it is a paradigm shift that rejects the premise on which all prior state-of-the-art methods were built.

What the field did before. The dominant assumption, shared across virtually all production texturing pipelines (Hunyuan3D, CLAY, Paint3D, MaterialMVP, as well as commercial products like Tripo and Rodin), was that generating 2D images from multiple camera views and baking them onto a 3D surface was the natural—perhaps inevitable—approach to automated texture generation. This assumption was driven by a practical fact: 2D image generative models (Stable Diffusion, Flux) were mature and powerful, while 3D generative models were nascent and limited. The MVD paradigm was a way to leverage 2D progress for a 3D problem. But the paper's critical insight is that this paradigm surfaces fundamental geometric problems that upgrading the 2D components cannot solve—occlusion regions are irreducible (no finite view set covers every surface point of an arbitrary mesh), latent-space diffusion errors at pixel boundaries become irrevocable when backprojected to 3D discontinuities, and cross-view consistency is a combinatorially expensive problem that video models still struggle with at scale.

What NaTex does differently at the conceptual level. The paper demonstrates that texture generation can be formulated as a color field generation problem f(x) = c for 3D coordinates x, built entirely within the standard latent diffusion paradigm that already dominates image, video, and 3D shape generation. This reframing is more than a representational choice—it changes what failure modes are possible. When textures are generated directly in 3D, there is no backprojection step, so there cannot be backprojection errors. There are no camera viewpoints, so there cannot be occlusion holes needing inpainting. There is a single unified color field rather than multiple independently generated views, so cross-view consistency is not a problem to solve—it is inherent to the representation.

Why this matters conceptually. The paper is making a strong theoretical claim disguised as an architectural paper: the 2D-lifting approach to 3D texturing is structurally flawed, not merely immature. The problems that MVD pipelines struggle with are not bugs that better 2D generators, smarter blending, or more sophisticated inpainting will fix—they are consequences of the 3D-to-2D-to-3D modality transitions themselves. Each transition introduces an information-theoretic bottleneck (projection loses occlusion data, latent encoding loses spatial precision, decoding adds hallucinated detail) and the cascade of these errors through the pipeline is irreducible. The paper's core intellectual move is recognizing that the only way to avoid these cascade errors is to eliminate the modality transitions entirely—generate texture in the same space where texture lives.

Evidence anchoring. The paper provides both quantitative and qualitative evidence that this is a paradigm shift with practical consequences, not merely a philosophical position. Table 2 shows NaTex consistently outperforming five prior methods across c-FID, LPIPS, CMMD, and CLIP-I metrics. More tellingly, Figure 7 reveals that even commercial MVD-based systems (Rodin-Gen2, Tripo 3.0) produce visible boundary misalignment (misaligned stars and buttons on a character) that NaTex completely avoids—and this is on an object without occluded regions, demonstrating that the problem is not just occlusion but the fundamental precision loss of the 2D-to-3D mapping. The paper also notes that NaTex's paradigm is "unexplored for texture generation" (Section 1), making this not an improvement over a known approach but the first demonstration that a known paradigm (latent diffusion) applies to a previously unsolved formulation of the problem (native 3D color field generation).

Innovation 2: The Co-Designed VAE–DiT Architecture with Spatially Aligned Geometry and Color Latents as a New Conditioning Primitive for 3D Generation

Beyond the paradigm choice, the paper's most original architectural contribution is the concept of native geometry control via spatially aligned, paired latent tokens—a conditioning mechanism that provides per-point geometric guidance by design rather than by learning. This is a fundamentally different approach to geometry conditioning than anything in prior 3D generation work.

What the field did before. In multi-view texturing, geometry conditioning is provided through 2D projections—normal maps, depth maps, or position maps rendered from each camera viewpoint. These are view-fragmented: each conditioning signal sees only the part of the geometry visible from one angle, with no information about occluded surfaces. They are also informationally impoverished: foreshortening compresses geometric detail at oblique angles, and the precise 3D location of a texture feature boundary cannot be uniquely determined from a 2D normal map alone (since the depth dimension is collapsed). In prior 3D-native methods that used triplanes, octrees, or Gaussian splats, geometry conditioning was either implicit (the representation itself encoded geometry, leaving less capacity for texture) or provided through separate, independently trained shape encoders that produced a global or coarse geometric embedding—not per-point aligned features.

What NaTex does differently at the conceptual level. The key idea is to co-design the VAE and DiT such that geometry and color latent tokens are structurally aligned at identical spatial positions, then use this alignment as the basis for conditioning. Specifically: the geometry branch of the VAE encodes the input mesh into a set of geometry latent tokens at known 3D positions; the color branch uses these geometry tokens as queries, producing color tokens at the same positions; the DiT receives these geometry tokens concatenated channel-wise to the (noisy) color tokens, providing per-point geometric guidance that is exact, complete (no occlusion gaps), and spatially precise.

This is conceptually different from all prior approaches because it transforms geometry conditioning from a separate, potentially misaligned signal into an intrinsic structural property of the latent representation itself. The geometry tokens are not an external input that the DiT must learn to correlate with color tokens—they are anchored to the same spatial coordinates, so the correspondence is guaranteed by the architecture rather than learned from data. This is analogous to how image diffusion models use pixel-coordinate positional encodings (which are also guaranteed-correspondence structural priors), but extended to irregular 3D surfaces.

Why it matters beyond raw performance. The paper's ablation (Figure 9) tests this claim directly by replacing the tightly-coupled geometry branch with a standalone ShapeVAE—and finds that texture diffuses across geometric boundaries. This negative result is as important as the positive one: it demonstrates that the spatial alignment is not a minor optimization but a necessary condition for the precise boundary alignment that distinguishes NaTex from MVD methods. The field now has a new conditioning primitive to explore: paired, spatially aligned latent sets where one encodes structure and the other encodes appearance. This could extend beyond texture to other problems where a fixed geometry must guide the generation of a surface-attached signal (material properties, semantic labels, displacement maps, ambient occlusion).

Evidence anchoring. Figure 9 provides direct visual evidence: removing RoPE or replacing the tight geometry embedding with a standalone shape VAE both degrade alignment, with the embedding swap causing "color sometimes diffus[ing], as seen in the chair back." Figure 7 provides the strongest comparative evidence—the boundary alignment of NaTex textures compared to both open-source and commercial MVD methods—showing that this design achieves what prior methods cannot.

Innovation 3: The Universal Color Field Abstraction—A Single Architecture for Appearance, Materials, and Semantics

The third distinctive contribution is the paper's demonstration that treating diverse surface properties—albedo, roughness/metallic, part segmentation labels—as instances of a single "color field" abstraction allows a unified VAE-DiT architecture to serve as a platform for multiple appearance-related tasks without architectural modification. This is a conceptual unification, not merely a software engineering convenience.

What the field did before. Texture generation, material generation, and part segmentation were treated as separate problems with separate architectures. MVD-based texture pipelines used one set of models; material generation (e.g., MaterialMVP) used specialized extensions; part segmentation used entirely different approaches (point-based classifiers, mesh-based graph networks, or 2D segmentation projected to 3D). Each required domain-specific design, training data, and engineering.

What NaTex shows at the conceptual level. The paper demonstrates that if you can map a surface property to an RGB-like representation, you can use the identical VAE to compress it, the identical DiT to generate it, and the identical decoder to query it. Roughness and metallic become two channels in a pseudo-color image. Part labels become distinct RGB colors per part. This works because the VAE architecture is representation-agnostic—it compresses any dense point cloud signal into a latent set, regardless of what the signal's channels semantically represent. The DiT architecture is similarly agnostic to the semantic meaning of the channels it generates. The only thing that changes between tasks is the training data distribution.

This is significant because it reframes the architecture from a texture generator to a surface-attached signal generator. The color field f(x) = c can be understood as a general-purpose surface function, where c can represent any property that varies continuously or discretely across the surface. The paper provides early but compelling evidence for this unification: material generation (Appendix B, Figures 11–14), part segmentation (Appendix B, Figure 15), and part texturing (Appendix B, Figures 16–17) all use the same NaTex-2B model with at most fine-tuning, not architectural changes.

Why this constitutes an innovation rather than an obvious consequence. The non-obvious part is that this unification actually works at production quality. It's easy to say "just treat material as color"—but making that produce high-quality roughness and metallic maps that hold up under varying lighting conditions (as shown in Figure 12) requires that the VAE's latent space genuinely captures the statistical structure of diverse surface signals, and that the DiT's conditioning mechanisms are flexible enough to control generation across these modalities. The detailed results—material alignment in Figure 13 showing NaTex producing "more accurate and better-aligned materials compared to prior approaches"—provide the empirical validation that the abstraction is not merely elegant but practically effective.

Evidence anchoring. The paper devotes substantial Appendix B material to demonstrating this extensibility. Figure 13 directly compares NaTex's material pipeline against a conventional MVD-based material pipeline, showing visible improvements in alignment and accuracy. Figure 15 shows fine-tuned part segmentation handling complex structures with "well-aligned boundaries." These are not toy demonstrations—they show the same architecture producing competitive or superior results across multiple distinct tasks, validating the abstraction's practical utility.

Innovation 4: The Diagnostic Identification of Multi-View Cascade Errors as an Irreducible Systemic Failure, Not a Collection of Fixable Bugs

While this is an analytical rather than a methodological contribution, the paper's systematic characterization of why MVD texturing is structurally limited—and its use of this diagnosis to motivate a paradigm shift—represents a genuine intellectual contribution that the field can use to evaluate future approaches.

What the field did before. Prior work generally treated MVD texturing problems as engineering challenges to be incrementally solved: better inpainting algorithms would fix occlusion holes, better multi-view consistency modules would fix seams, better geometric conditioning would fix alignment. Each paper in the MVD lineage proposed improvements to one component of the pipeline, with the implicit assumption that the paradigm was fundamentally sound and the problems were resolvable through better components.

What NaTex argues at the conceptual level. The paper's diagnosis, laid out in Section 1 and Figure 2, is that the MVD paradigm suffers from cascade errors in modality changes—errors introduced at each 3D-to-2D-to-3D transition that compound multiplicatively rather than additively. Crucially, the paper argues that three of these error sources are irreducible: occlusion holes are a geometric inevitability of finite view sampling, latent-space boundary imprecision is inherent to compression-based diffusion, and cross-view consistency requires solving a problem (perfect multi-view generation) that even state-of-the-art video models have not solved. The argument is not that these problems are hard—it's that they are consequences of the paradigm's structure, and no amount of component-level improvement can eliminate them entirely.

This diagnosis matters because it provides a decision criterion for the field: if you need perfect boundary alignment (for mechanical CAD models), zero occlusion artifacts (for objects with concavities or internal structure), or guaranteed consistency (for textures that must look identical from all angles), then the 2D-lifting paradigm is the wrong approach regardless of how good the 2D components become. This is a stronger claim than "our method is better than theirs"—it is a claim about the asymptotic limits of the competing paradigm.

Evidence that this diagnosis is empirically grounded rather than rhetorical. The paper's evidence comes from the failure cases of MVD methods, not just NaTex's successes. Figure 7 shows commercial systems (which presumably use the best available 2D generators, inpainting, and blending) still producing visible misalignment on objects without occlusion—meaning the problem is not just occlusion but the fundamental precision limit of the 2D-to-3D mapping. Figure 8 shows NaTex's refinement mode correcting precisely the types of artifacts (occlusion in windows) that conventional inpainting fails to fix. The performance gap in Table 2—with NaTex outperforming all MVD-based methods on all metrics—is consistent with the diagnosis that the limiting factor is the paradigm, not the implementation quality within that paradigm.

Why this is an innovation despite being analytical. The field lacks clear frameworks for evaluating when a paradigm is worth improving versus when it's worth abandoning. This paper provides one such framework, grounded in the concept of irreducible modality-transition errors, and validates it with comparative evidence. Future work on texture generation can use this framework to decide whether to invest in better 2D lifting or pivot to 3D-native approaches.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses a test set following the evaluation protocol from MaterialMVP (He et al., 2025), as stated in Section 4.1: "Following the evaluation protocol from MaterialMVP... we use the same test set." The exact size of this test set is not specified in the paper beyond being "the same" as MaterialMVP's. Training data is not benchmarked—it consists of textured 3D meshes rendered in Blender with 24 views per object, random elevations from 45° to −30°, and random illumination environments (point lights, area lights, HDRI maps), as described in Appendix A. The object categories and dataset scale are not enumerated—the paper provides no count of training assets, licensing details, or data source provenance beyond the Blender-based rendering procedure.
Base model. The primary model is NaTex-2B, consisting of a 300M-parameter color VAE and a 1.9B-parameter color DiT (Appendix A). No ablation of model scale is performed—neither a smaller variant (to test whether the architectural innovations matter at reduced capacity) nor a larger one (to test scaling trends). The choice of ~2B total parameters appears to be a fixed engineering decision without explicit justification relative to alternatives. The VAE is trained with a maximum of 6,144 latent tokens. All reported results use 5 diffusion steps and a classifier-free guidance scale of 2.0 (Appendix A).
Metrics. The paper employs four metrics for texture generation quality, all computed on rendered 2D views of the textured 3D assets (Table 2): c-FID (CLIP-based Fréchet Inception Distance), measuring distributional distance between generated and reference textures in CLIP feature space; LPIPS (Learned Perceptual Image Patch Similarity, Zhang et al., 2018), measuring perceptual similarity between corresponding rendered views; CMMD (CLIP Maximum-Mean Discrepancy), another distributional metric in CLIP space; and CLIP-I (CLIP-Image Similarity), measuring cosine similarity between CLIP embeddings of generated and reference images—higher is better for this metric, lower for the others. For VAE reconstruction evaluation (Table 1), the paper additionally uses PSNR (Peak Signal-to-Noise Ratio, computed directly on the color point cloud), PSNR* and SSIM* (Wang et al., 2004, computed on rendered 2D images from six orthogonal views), and LPIPS* (computed on the same rendered views). The asterisk notation indicates metrics "calculated on the six orthogonal rendered views" rather than directly in 3D.
Baselines. The paper compares against five methods (Table 2, Figure 7): Paint3D (Zeng et al., 2024), TexGen (Yu et al., 2024), Hunyuan3D-2 (Zhao et al., 2025), RomanTex (Feng et al., 2025), and MaterialMVP (He et al., 2025). These span the range from open-source research systems to production-scale models. Additionally, Figure 7 includes visual comparisons with two closed-source commercial systems: Rodin-Gen2 and Tripo 3.0. For the refinement/inpainting comparison (Figure 8), the baseline is conventional OpenCV interpolation-based inpainting. For the VAE reconstruction comparison (Table 1), the baseline is implicitly the VAE itself at different latent sizes—no comparison against alternative compression schemes (e.g., UV-map-based autoencoding, octree-based compression, triplane VAE) is provided.
Generation budget / compute accounting. The paper does not define or use a standardized generation budget in the manner common in inference-scaling work (e.g., "number of forward passes" or "total FLOPs"). Instead, comparisons are made at the model level—all methods generate textures for the same test prompts, and metrics are computed on the outputs. The diffusion sampling budget is controlled via two parameters: number of latent tokens (maximum 6,144 during training, with token scaling during inference as shown in Figure 10) and number of diffusion steps (5 steps for all reported results, with one-step generation demonstrated in Figure 10). No comparison is made between methods at equalized inference time or FLOPs—the computational cost of running NaTex vs. Paint3D vs. Hunyuan3D-2 is not reported, making it impossible to determine whether NaTex's quality improvements come at higher computational cost or represent genuine efficiency gains.
Cross-validation / statistical protocol. The paper reports no cross-validation, no confidence intervals, no statistical significance tests, and no error bars on any quantitative result. Table 2 presents single numbers per method per metric without variance estimates. The test set protocol is "following MaterialMVP"—meaning a fixed test split, used once. The ablation in Figure 9 is qualitative (visual comparison of three conditioning variants on specific examples), not quantitative. This is a significant methodological gap: with no reported variance, the reader cannot distinguish meaningful differences from sampling noise in the metric computations, particularly given that the test set size is not disclosed.

Main Quantitative Results

Texture Reconstruction (VAE Evaluation)

The paper evaluates the VAE's reconstruction fidelity as a function of latent size, reported in Table 1 and visualized in Figure 6.

Headline results. The reconstruction quality improves monotonically with the number of latent tokens, despite the VAE being trained with a maximum of 6,144 tokens—the model generalizes to higher token counts at inference. The paper states: "Although our model is trained with a maximum of 6144 tokens, the reconstruction quality improves as the latent size increases" (Section 4.1). Specific numbers are presented in Table 1, which the paper includes as a visual element in the HTML but does not transcribe in the text. The exact PSNR, PSNR*, SSIM*, and LPIPS* values at each token count must be read from the table figure.

What this demonstrates. The VAE's decoder uses cross-attention between arbitrary query coordinates and the latent token set, meaning the spatial resolution of the output is determined by the number of queries, not by the number of latent tokens at encoding time. More latent tokens provide higher-capacity encoding of the color field (each token represents a smaller surface region), and the decoder can exploit this additional capacity at inference without architectural changes. Figure 6 provides a visual side-by-side showing progressively sharper reconstruction as token count increases—though specific token counts for each column in Figure 6 are not labeled in the paper text.

Critical omissions. No comparison is provided against alternative compression schemes—we cannot assess whether the VecSet-based VAE achieves better rate-distortion than a UV-map-based VAE (as in TexGarment), a triplane-based encoder, or an octree-based compression. The "over 80× compression" claim (Section 1, Section 3.1) is stated without defining the uncompressed baseline precisely—80× relative to what point density? The input point cloud size (N in the notation P_c ∈ R^{N×9}) is never specified, making the compression ratio unverifiable.

Texture Generation Quality (Comparison Against Prior Methods)

The paper evaluates generation quality against five prior methods on the MaterialMVP test set, reported in Table 2 and Figure 7.

Headline results. NaTex "consistently outperforms the others" across all four metrics (c-FID, LPIPS, CMMD, CLIP-I), according to Section 4.1. The exact numerical values are in Table 2 (a figure in the HTML). The paper frames this as evidence that 3D-native generation avoids the artifacts that limit MVD-based approaches. Crucially, the comparison in Figure 7 includes commercial systems (Rodin-Gen2, Tripo 3.0) and shows these systems "struggle to align textures along geometry boundaries, while our method achieves near-perfect alignment" (Section 4.1). Specific failure cases highlighted: "misaligned stars and buttons" on a character model—notably, a case "without occlusion regions," demonstrating that the problem is boundary alignment precision, not just occlusion inpainting.

Critical contextualization of the comparison. Several factors complicate the interpretation of these results:

Geometry mismatch. Commercial systems (Rodin-Gen2, Tripo 3.0) generate textures for their own internally generated geometry, while the open-source comparison (in Figure 7) standardizes geometry to Hunyuan3D 2.5 for all methods. The commercial comparison thus conflates geometry quality with texture quality—differences in geometric detail, mesh topology, or UV parameterization between the commercial systems' outputs and NaTex's outputs could produce perceptual differences misattributed to texture generation alone.
Metrics are 2D-projection-based. All four metrics are computed on rendered 2D views, not directly in 3D. This means they capture how the textured object looks from sampled camera angles, but may not capture 3D-specific artifacts (seams visible only from certain angles, texture stretching in UV-space, color discontinuities at mesh boundaries in regions not captured by the sampled views). For a paper whose central claim is superiority in 3D-native consistency, metrics that directly measure 3D properties (e.g., texture seam error, color variance across mesh edges, occlusion-region quality as judged by a 3D-aware scorer) would provide stronger evidence.
Missing baseline. The paper does not compare against a simple but important baseline: NaTex with the geometry branch replaced by an independent shape encoder (the ablation from Figure 9), evaluated quantitatively on the full test set. If this degraded variant still outperforms MVD methods, the paradigm choice alone—not the specific geometric conditioning design—would explain the gains. If it does not, the tightly-coupled geometry VAE is essential. Either result would be informative, but neither is reported quantitatively.

Texture Refinement / Inpainting (Qualitative Comparison)

Figure 8 compares NaTex's refinement mode (using color control, described in Section 3.2) against conventional OpenCV interpolation-based inpainting on a house model with an occluded window region.

Headline result."Our method generates cleaner and better-aligned textures in the occluded regions (see the zoomed-in window of the house)" (Section 4.1). The visual difference is presented qualitatively in Figure 8, with two rendered views showing the occlusion hole in the multi-view images (left) and the inpainted result comparison (presumably right, though figure structure is unclear from text).

Critical absence. This comparison has no quantitative component. There is no metric reported for inpainting quality—no FID, LPIPS, or even user study on the inpainted regions. The baseline (OpenCV interpolation) is the weakest possible inpainting baseline—no comparison is made against learning-based 2D inpainting (e.g., LaMa, Stable Diffusion inpainting), against multi-view-aware inpainting that uses geometry information, or against MVD-based refinement approaches. The claim that NaTex serves as an effective refiner is supported by one qualitative example, which is insufficient to establish general efficacy.

Ablation Studies and Robustness Checks

Geometry conditioning mechanisms (Figure 9): The paper ablates two aspects of the native geometry control: (1) removing RoPE positional encoding from color tokens, and (2) replacing the tightly-coupled geometry branch (trained jointly with the color VAE) with an embedding from a standalone ShapeVAE (Hunyuan3D-VAE, trained independently). The findings: "both conditions improve image-texture alignment, such as the stripes on the awning of the house and the colors of the traffic light. Additionally, the shape embedding influences texture-geometry alignment. Without the tight embedding, color sometimes diffuses, as seen in the chair back" (Section 4.2). This ablation is purely qualitative—three visual examples are shown, but no quantitative metric is reported on any test set split, making it impossible to assess the magnitude of degradation or whether it generalizes beyond the displayed examples. The finding that both RoPE and tight geometry embedding matter is expected given the architecture's design rationale, but the absence of quantitative measurement means the effect size is unknown.
Inference token scaling (Figure 10): The paper demonstrates that increasing the number of latent tokens at inference time (beyond the 6,144 training maximum) improves quality and alignment: "both generation quality and alignment improve gradually as the token length increases (see windows)" (Section 4.2). This is a robustness check on the VAE's cross-attention decoder—confirming it generalizes to higher capacity than seen during training. Figure 10 also demonstrates one-step generation capability despite no distillation. Both results are qualitative visual comparisons, with no quantitative metrics reported.
One-step generation (Figure 10): The model achieves "one-step generation capability" without distillation, attributed to "the strong conditioning" (Section 4.2). This is a surprising result for a flow-matching model—it means the conditional velocity prediction with CFG is sufficiently accurate that a single Euler step approximates the full trajectory. However, the paper does not quantify the quality gap between 1-step and 5-step generation—no metrics, no user study, only qualitative examples in Figure 10. The claim of practical one-step capability is therefore supported by existence (it sometimes works) but not characterized (how often, how much quality degrades).
Material generation quality (Figures 11–14, Figure 13): The material generation pipeline (using color control conditioned on albedo, with roughness and metallic encoded as pseudo-color channels) produces "well-aligned and coherent roughness and metallic maps" (Appendix B). Figure 13 compares NaTex's material pipeline against "a conventional MVD-based material pipeline" and shows NaTex producing "more accurate and better-aligned materials." No quantitative metrics are reported for material generation—the comparison is entirely qualitative. Specific failure modes of the MVD material baseline are not enumerated beyond the general "struggle with alignment and sometimes misinterpret material properties" observation (Appendix B).
Part segmentation quality (Figure 15): The fine-tuned part segmentation model produces "highly accurate [results] even on complex cases" with "well-aligned boundaries" (Appendix B). Again, no quantitative metrics—no IoU, no per-part accuracy, no comparison against dedicated 3D part segmentation methods.

What is conspicuously absent from the ablation suite:

No ablation of VAE design choices. The geometry-aware dual-branch design is compared against a standalone ShapeVAE (Figure 9), but there is no ablation of alternative VAE architectures: a single-branch VAE that encodes both geometry and color jointly (without separate branches), a UV-map-based VAE (to validate the claim that point clouds are superior to UV-space), or an octree-based encoder (to validate the claim of superiority over TexOct's approach).
No ablation of DiT architecture scale or depth. The paper trains one model at one scale (1.9B DiT, 300M VAE). There is no evidence about whether the architecture's benefits would persist at smaller scale (important for practical deployment) or improve further at larger scale (important for understanding whether the paradigm is scaling-limited).
No ablation of conditioning mechanisms independently. The image control (DINOv2), geometry control (RoPE + geometry latents), and color control are ablated only in the specific combination shown in Figure 9. There is no ablation removing image control entirely (to measure how much the image contributes vs. geometric priors), no ablation testing different image encoders (CLIP vs. DINOv2 at matched resolution), and no ablation of the 1022 resolution choice for DINOv2.
No ablation of training data scale or composition. The model is trained on an unspecified amount of data with an unspecified distribution of object categories. There is no experiment varying training data volume to assess whether the architectural advantages are robust to data constraints, and no experiment testing generalization to object categories held out from training.
No robustness check on mesh quality or UV parameterization. The model's performance is evaluated on meshes from Hunyuan3D 2.5, which produces high-quality, watertight geometry with (presumably) clean UV maps. There is no evaluation on artist-created meshes with non-watertight geometry, degenerate UV islands, or inconsistent normal orientation—precisely the types of real-world meshes that the UDF loss was designed to handle. The claim that the UDF loss is necessary because "correlating the color point cloud with a watertight mesh... is non-trivial" (Section 3.1) implies robustness to non-watertight geometry, but this robustness is never tested.

Critical Assessment

Does the evidence support the claim that NaTex "significantly outperforms previous methods in texture coherence and alignment"?

The quantitative evidence in Table 2 shows NaTex outperforming five prior methods on four metrics—this is the paper's strongest empirical result and does support a claim of improved 2D rendering quality of the textured output. However, several qualifications apply:

The metrics measure 2D projection quality, not 3D texture quality. c-FID, LPIPS, CMMD, and CLIP-I are all computed on rendered 2D views. A method that produces perfectly consistent 3D textures (NaTex's claimed advantage) and a method that produces slightly inconsistent but visually similar textures from specific camera angles could achieve similar scores. The metrics cannot distinguish "this texture is seam-free in 3D" from "this texture looks good from the sampled viewpoints but has hidden seams." Given that NaTex's central claim is about 3D-native consistency, the absence of any 3D-specific metric—texture seam error measured along UV boundaries, color variance across mesh edges, occlusion-region quality evaluated directly on the 3D surface, or a multi-view consistency score computed from renderings at dense, uniformly sampled viewpoints—is a significant gap between the claims and the evidence.

The commercial system comparison is confounded by geometry. Figure 7 shows NaTex against Rodin-Gen2 and Tripo 3.0, but these systems use their own geometry while NaTex uses Hunyuan3D 2.5 geometry. Differences in geometric fidelity, mesh tessellation, and UV parameterization between these geometry sources could cause or amplify the texture quality differences visible in the figure. A fairer comparison would hold geometry constant—either by baking all methods' textures onto a shared mesh, or by comparing only against open-source methods where geometry can be controlled (as done for Paint3D, TexGen, etc. in Table 2).

The test set size is undisclosed. Without knowing whether the test set contains 50, 500, or 5,000 assets, and without any reported variance measures, the reliability of the metric differences in Table 2 cannot be assessed. A 0.01 difference in LPIPS on a 50-asset test set could easily be noise; on a 5,000-asset set, it would be more meaningful. The paper's choice to not report test set size or variance is a substantial methodological weakness.

Does the evidence support the claim that NaTex avoids the "inherent limitations of the MVD pipeline" (occlusion, alignment, consistency)?

For occlusion inpainting: The evidence is limited. Figure 8 provides one qualitative example (a house with an occluded window) showing NaTex's refinement mode outperforming OpenCV interpolation. This is a single example with a trivial baseline (interpolation is the weakest possible inpainting method). No comparison is made against learning-based 2D inpainting, multi-view-aware inpainting, or the inpainting modules used by the MVD baselines (Paint3D, Hunyuan3D-2). The claim that occlusion is "fundamentally unavoidable" in MVD pipelines (Section 1) is a geometric argument that does not require empirical validation, but the claim that NaTex solves this problem requires evidence that NaTex's occlusion handling is better than the best available MVD inpainting—which the paper does not provide.

For boundary alignment: The evidence is stronger qualitatively (Figure 7, the misaligned stars and buttons) but unquantified. There is no metric measuring boundary alignment—no edge-based metric comparing texture edges to geometric edges, no Chamfer distance between texture feature boundaries and mesh feature lines, no user study asking raters to judge alignment quality. The ablation in Figure 9 shows that the geometric conditioning design matters for alignment on specific examples, but a small set of qualitative examples does not establish that the alignment problem is "solved" in a general sense.

For cross-view consistency: This is essentially unmeasured. Cross-view consistency would require evaluating whether the texture looks the same from different viewpoints—a metric comparing renderings from different angles of the same surface region, or measuring color variance across overlapping camera-projections of the same surface point. None of the reported metrics capture this property. The paper's argument that consistency is "inherent to the representation" (there is a single color field, not multiple views) is a structural claim, not an empirical one—it is true by construction that a single-valued function cannot be inconsistent with itself, but this does not guarantee that the generated color field is consistent in the perceptual sense (e.g., that a pattern wrapping around a cylinder aligns where it meets itself, which depends on the model's ability to generate globally consistent features from local conditioning).

Does the evidence support the claim of "strong generalization capabilities" to downstream applications?

For material generation: The evidence is qualitative only (Figures 11–14, one comparative figure vs. MVD material pipeline in Figure 13). The paper claims NaTex produces "more accurate and better-aligned materials compared to prior approaches" but provides no metric—no PSNR, SSIM, or LPIPS on roughness/metalness maps against ground truth; no rendering quality metric (e.g., perceptual similarity of relit renderings under novel illumination compared to ground-truth relit renderings); no user study. The visual results look plausible but "plausible" is not the same as "quantifiably better than prior work."

For part segmentation: Figure 15 shows "highly accurate" results on complex cases after fine-tuning, but again no IoU, no per-part accuracy, no comparison against SAM-3D or other 3D segmentation baselines. The training-free variant is mentioned (Section 3.3) but not shown in figures—we see only the fine-tuned results. Without a baseline comparison, the claim of generalization to segmentation is only an existence proof (the model can be made to do it), not evidence of competitive performance.

For texture refinement: Figure 8 provides one qualitative comparison against OpenCV interpolation. The paper states refinement can be done "in just five steps without any distillation, making it extremely fast and efficient"—but provides no timing comparison, no FLOPs comparison, and no quantitative quality metrics at different step counts. The claim of speed and efficiency is entirely unsubstantiated.

What experiments would have strengthened the paper?

Quantitative boundary alignment metrics. A metric measuring how well texture edges align with geometric edges—e.g., the distance between Canny edges in the rendered texture and silhouette edges of the geometry—would directly measure the paper's central claimed advantage. Without such a metric, the alignment claims rest entirely on a handful of qualitative examples.

A 3D consistency metric. Computing the variance of decoded color for the same surface point when that point is queried from different latent token positions, or measuring color discontinuities across UV seams, would directly test whether the single-color-field representation actually produces perceptually consistent textures in practice.

Ablation with standardized geometry. Comparing NaTex against MVD methods where all methods texture the identical mesh (the Hunyuan3D 2.5 geometry) and render with identical lighting would remove the geometry confound from the commercial system comparison. This is partially done for the open-source baselines in Table 2, but the Figure 7 commercial comparison should be repeated with standardized geometry, or the geometry confound should be explicitly acknowledged and qualified.

Data scale ablation. Training NaTex on 10%, 50%, and 100% of the training data and measuring the performance gap against MVD baselines at each data scale would reveal whether NaTex's advantage requires large-scale 3D data (which is harder to obtain than 2D image data) or persists in data-constrained regimes. This is critical for assessing the practical deployability of 3D-native approaches vs. MVD approaches that can leverage massive pre-trained 2D models.

Inference cost comparison. Reporting wall-clock time and/or FLOPs for NaTex at typical generation settings (5 steps, standard token count) compared to MVD baselines (which require multiple diffusion forward passes for multiple views, plus backprojection and blending) would contextualize the quality improvements. If NaTex achieves better quality at lower cost, that strengthens the paradigm argument. If it achieves better quality at higher cost, the practical recommendation is more nuanced.

Statistical reporting. Reporting means and standard deviations across multiple runs, or confidence intervals on metric differences, would allow readers to assess whether the reported improvements are statistically reliable given the (undisclosed) test set size. The current reporting of single numbers without variance makes the quantitative comparisons uninterpretable beyond ordinal ranking.

Summary of evidential strength and weakness

The paper's core contribution—that 3D-native texture generation via latent color diffusion is feasible and can produce compelling results—is clearly demonstrated through the visual results in Figures 1, 5, 7, 11–14, and 16–18. These visuals are genuinely impressive and make a strong existence proof.

The paper's stronger comparative claim—that NaTex consistently outperforms prior methods on texture quality—is supported by Table 2 for the specific metrics and test set used, but is weakened by undisclosed test set size, absence of variance measures, and reliance on 2D-projection metrics that do not directly measure the 3D properties (consistency, alignment, occlusion quality) that the paper argues are NaTex's key advantages.

The paper's strongest theoretical claims—that MVD pipelines suffer from irreducible modality-transition errors that 3D-native generation structurally avoids—are supported by geometric reasoning and by the qualitative evidence of MVD artifacts (Figure 7, Figure 8), but are not directly tested in a way that isolates the paradigm choice from implementation quality. A head-to-head comparison at equalized compute budget, with progressively improved inpainting and consistency modules on the MVD side, would be needed to establish that the MVD ceiling is fundamental rather than merely current.

The extensibility claims (material generation, segmentation, refinement) are supported by existence proofs (qualitative results exist) but not by comparative evidence (no quantitative metrics against dedicated baselines for any downstream task). This is appropriate for a paper that labels these as "applications" and "primary verifications" rather than core contributions, but readers should understand that the evidence for these extensions is preliminary, not conclusive.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Accounted for in the Headline Efficiency Numbers

The assumption or constraint. The paper presents NaTex as a paradigm that "advances texture generation, producing more seamless results" (Appendix C), but the computational cost of the native 3D generation process relative to the MVD pipelines it critiques is never quantified. This is not an acknowledged limitation with a quoted admission—it is an omission. The paper reports no FLOPs counts, no wall-clock timings, no GPU-hours for training or inference, and no comparison of NaTex's inference cost against the multi-view generation + backprojection + blending pipeline used by competing methods.

The consequence. A practitioner evaluating whether to adopt NaTex over an MVD-based alternative cannot make an informed cost-benefit decision. The quality improvements in Table 2 and Figure 7 may come at a substantially higher computational cost—NaTex requires a 1.9B-parameter DiT forward pass (5 steps) plus VAE encoding and decoding, while MVD methods distribute their computation across multiple 2D diffusion forward passes (one per view). If NaTex's single forward pass through a 1.9B transformer plus a 300M VAE is more expensive than, say, six forward passes through a smaller 2D diffusion model with efficient attention, the quality improvements may not justify the cost in latency-sensitive or high-throughput deployments. The one-step generation capability (Figure 10) is demonstrated qualitatively but without timing data, leaving unclear whether one-step NaTex is faster or slower than MVD methods.

What evidence exists in the paper. None. The paper contains exactly zero quantitative information about computational cost: no parameter counts for competing methods (so the 1.9B DiT + 300M VAE cannot be compared against baseline model sizes), no inference time measurements, no training compute budget, no FLOPs analysis, and no cost-efficiency metrics (e.g., LPIPS per FLOP or CLIP-I per GPU-second). The phrase "extremely fast and efficient" is used for texture refinement (Section 3.3, Appendix B) without any supporting numbers.

Mitigation status. Not addressed. The paper does not acknowledge this gap, propose a cost model, or suggest that future work should include efficiency comparisons. This is a significant practical limitation for anyone evaluating deployment feasibility—the quality improvements are demonstrated but their cost is unknown.

All Quantitative Evaluation Uses 2D Projection Metrics That Do Not Measure the Claimed 3D Advantages

The assumption or constraint. The paper's core argument is that NaTex avoids three specific 3D artifacts that plague MVD methods: occlusion inpainting errors, texture-geometry boundary misalignment, and cross-view inconsistency (Section 1, Figure 2). Yet every quantitative metric in Table 2 (c-FID, LPIPS, CMMD, CLIP-I) and Table 1 (PSNR*, SSIM*, LPIPS*) is computed on rendered 2D projections of the textured mesh. None of these metrics directly measures any of the 3D properties the paper claims to improve.

The consequence. The quantitative evidence is structurally incapable of distinguishing between: (a) a texture that is genuinely seam-free and perfectly aligned in 3D (NaTex's claimed advantage), and (b) a texture that looks good from the specific rendered viewpoints used for metric computation but has hidden seams, alignment errors, or occlusion artifacts visible from other angles. The paper provides qualitative visual evidence of MVD alignment failures (Figure 7, misaligned stars and buttons) but does not quantify these failures or NaTex's improvement over them. For occlusion specifically, the single qualitative comparison (Figure 8) uses OpenCV interpolation as the baseline—the weakest possible inpainting method—and provides no quantitative metric. The structural advantage of 3D-native generation (single coherent color field, no viewpoint fragmentation) is true by construction, but the empirical claim that this structural advantage translates to perceptually meaningful improvements in 3D consistency is not directly tested by any reported metric.

What evidence exists in the paper. The gap between claims and metrics is visible by inspection. Section 4.1 states the evaluation uses metrics "calculated by rendering the reconstructed textured meshes into 2D images" (PSNR*, SSIM*, LPIPS*) and metrics from the MaterialMVP protocol (c-FID, CMMD, CLIP-I), all of which operate on 2D renderings. The paper never defines or computes a 3D-native metric such as: texture seam error along UV boundaries, color variance across mesh edges when viewed from densely sampled viewpoints, occlusion-region reconstruction quality measured directly on the 3D surface, or a multi-view consistency score computed from renderings at viewpoints not used during metric evaluation.

Mitigation status. Not addressed. The paper does not acknowledge this gap between its 3D-oriented claims and its 2D-projection metrics, nor does it propose or compute any 3D-consistency metric. The qualitative figures (7, 8, 9) provide existence proofs of alignment improvements in specific cases but cannot establish the generality or magnitude of the advantage across the test set.

The Test Set Protocol Is Underspecified, Making the Quantitative Results Unverifiable and Potentially Unreliable

The assumption or constraint. The paper evaluates NaTex against five prior methods "following the evaluation protocol from MaterialMVP" and using "the same test set" (Section 4.1). However, critical details of this protocol are never specified: the test set size is not disclosed, no variance measures (standard deviations, confidence intervals) are reported for any metric in Table 2, and no statistical significance testing is performed. The paper also provides no information about how the competing methods' outputs were obtained—were official pretrained models used with default settings? Were models retrained or fine-tuned for the comparison?

The consequence. A practitioner reading Table 2 cannot determine whether the reported metric differences are meaningful. If the test set contains 50 assets and LPIPS varies by ±0.02 across runs, a 0.01 difference is indistinguishable from noise. If it contains 5,000 assets, the same difference might be highly reliable. Without test set size and variance, the quantitative comparison is reduced to an ordinal ranking ("NaTex gets the best numbers") without any measure of confidence. This is particularly problematic because the paper's central claim—that NaTex "significantly outperforms previous methods" (Abstract)—depends on these numbers. A reviewer or practitioner attempting to reproduce the evaluation would not know how many assets to test on, what variance to expect, or whether their reproduced numbers fall within a reasonable range of the reported values.

What evidence exists in the paper. The methodological gap is visible by what is absent from Section 4.1 and Table 2: no "N = " anywhere, no ± values, no mention of standard deviation or confidence intervals, no description of how many runs per method were performed, and no discussion of statistical protocol. The MaterialMVP protocol is cited as the authority, but the paper's description of what "following" this protocol means is limited to "we use the same test set and four metrics" (Section 4.1). Whether MaterialMVP's protocol included variance reporting and the paper omitted it, or whether MaterialMVP itself did not report variance, is unclear from the text.

Mitigation status. Not addressed. The paper shows no awareness of this as a limitation. This is a significant methodological weakness that prevents any confident conclusion about the statistical reliability of the quantitative results, independent of the quality of the visual results.

Generalization Beyond the Training Data Distribution and Model Scale Is Unexplored

The assumption or constraint. The paper trains and evaluates exactly one model at one scale (NaTex-2B: 300M VAE + 1.9B DiT) on one dataset of unspecified size and composition. There is no experiment varying the training data scale (e.g., 10%, 50%, 100%), no experiment testing generalization to object categories held out from training, no experiment varying model size (e.g., a smaller NaTex-500M or larger NaTex-5B), and no evaluation on out-of-distribution mesh quality—all test geometry comes from Hunyuan3D 2.5, which produces high-quality, watertight meshes.

The consequence. Several important practical questions are unanswerable:

Data efficiency: 3D-native texture generation requires 3D training data (textured meshes), which is substantially scarcer and more expensive to produce than the 2D image data that MVD methods can leverage through pre-trained image diffusion models. If NaTex's advantage depends on large-scale 3D data that most practitioners cannot obtain, the practical recommendation is less clear than the paper suggests. Conversely, if NaTex remains competitive with less data, that would strengthen the paradigm argument—but neither case is tested.
Object category generalization: If the training data skews toward certain object categories (characters, vehicles, furniture), does NaTex generalize to unseen categories, or does it produce degraded textures? MVD methods that leverage general-purpose image diffusion models may have broader generalization by virtue of their 2D pre-training, but this hypothesis cannot be evaluated from the paper's experiments.
Mesh quality robustness: The paper justifies the truncated UDF loss by noting that "correlating the color point cloud with a watertight mesh (required for standard SDF loss) is non-trivial" (Section 3.1), implying that the UDF handles non-watertight geometry. However, all test geometry comes from Hunyuan3D 2.5, which generates watertight, high-quality meshes. Whether NaTex degrades gracefully on artist-created meshes with holes, non-manifold edges, inconsistent normals, or degenerate UV islands is untested. A practitioner working with real-world 3D assets (scanned meshes, CAD exports, user-generated content) cannot assume robustness from the current evidence.
Scaling behavior: The paper presents one model scale as a point solution. Without scaling experiments, there is no evidence that the design would continue to improve with larger models (important for practitioners deciding whether to invest in scaling) or that it works at smaller scales (important for practitioners with compute constraints). The claim that latent diffusion is a "scalable" paradigm for texture generation (Section 1) is supported by analogy to image/video/shape generation, not by direct evidence in the texture domain.

What evidence exists in the paper. The limitation is visible by the absence of the relevant experiments. Appendix A states training hyperparameters (learning rate schedule, batch size 256, CFG probability 10%) but does not specify training data volume or composition. Appendix C acknowledges some limitations—"the reconstruction quality of the VAE could be improved to support higher-resolution textures. Data curation should be enhanced for material generation"—but these are framed as future improvements, not as gaps in the current evaluation of generalization. No experiment tests data scaling, category generalization, model scaling, or mesh quality robustness.

Mitigation status. The paper does not acknowledge generalization as a limitation or suggest future work on it. The absence of data scale, category split, and mesh quality experiments is a significant gap for a paper that proposes a new paradigm and claims strong practical advantages over established approaches.

The assumption or constraint. The paper presents NaTex as "a unified framework that is broadly applicable across diverse 3D tasks" (Section 3.3) and dedicates substantial space to demonstrating texture refinement, material generation, part segmentation, and part texturing (Section 3.3, Appendix B, Figures 5, 8, 11–18). However, every one of these demonstrations is purely qualitative—no quantitative metric is reported for any downstream task.

The consequence. The extensibility claims remain at the level of "this is possible" rather than "this is competitive with dedicated methods." Specifically:

Texture refinement (Figure 8, Appendix B, Figure 18): The baseline is OpenCV interpolation—literally the most basic possible inpainting method. No comparison is made against learning-based 2D inpainting (LaMa, Stable Diffusion inpainting), multi-view-aware inpainting, or the inpainting modules used by MVD texturing pipelines. The claim that refinement is "extremely fast and efficient" (Section 3.3) is unquantified—no timing data is provided.
Material generation (Figures 11–14, Figure 13): The comparison against "a conventional MVD-based material pipeline" (Appendix B) is purely qualitative and the MVD baseline is not named or described in sufficient detail to assess its strength. No metrics (PSNR, SSIM, LPIPS on roughness/metalness maps; perceptual quality of relit renderings) are reported. The claim of producing "more accurate and better-aligned materials" (Appendix B) is supported by qualitative visual inspection of unspecified examples.
Part segmentation (Figure 15): The fine-tuned model produces "highly accurate [results] even on complex cases" (Appendix B) but no IoU, per-part accuracy, or comparison against dedicated 3D segmentation methods (e.g., PartNet-based classifiers, point-cloud segmentation networks adapted to mesh surfaces) is provided. The training-free variant is mentioned (Section 3.3) but its outputs are not shown in any figure—only fine-tuned results are displayed.

What evidence exists in the paper. The gap is visible by what is absent from every application section: no tables of quantitative results, no baselines beyond trivial ones (OpenCV for inpainting, an unnamed MVD pipeline for materials), and no metrics defined for the downstream tasks. The paper labels these as "primary verifications" (Section 1, contribution list item 4: "adapted to previously mentioned applications for primary verifications"), which accurately describes the evidence level—these are demonstrations that the architecture can be adapted, not evidence that it should be preferred over specialized approaches.

Mitigation status. The paper acknowledges one specific downstream limitation in Appendix C: "Part segmentation could be refined to reduce ambiguity and improve granularity. New methods are needed to handle closed surfaces in adjacent parts for part texturing." However, the broader gap—that none of the downstream applications is evaluated quantitatively or against strong baselines—is not acknowledged. A practitioner interested in using NaTex for material generation, refinement, or segmentation cannot assess from this paper whether the approach is competitive with existing dedicated tools, or whether it merely demonstrates feasibility.

The VAE Reconstruction Quality Is Not Benchmarked Against Alternative Compression Schemes

The assumption or constraint. The paper designs a custom geometry-aware color VAE based on the VecSet architecture and claims it achieves "over 80× compression" (Section 1, Section 3.1). However, the paper provides no comparison against alternative compression approaches—no UV-map-based VAE (as used in TexGarment), no octree-based compression (as used in TexOct), no triplane-based encoding (as used in UniTEX), and no standard image VAE applied to unwrapped UV textures.

The consequence. The reader cannot assess whether the VecSet-based VAE is a genuinely superior compression scheme for color point clouds, or merely an adequate one chosen for architectural compatibility with the DiT. If a simpler UV-map-based VAE achieved comparable reconstruction quality at similar compression ratios, the paper's argument that color point clouds are a "more structured and coherent representation, better suited for generative modeling" (Section 3.1) would be weakened. Conversely, if the VecSet VAE substantially outperforms alternatives, that would strengthen the paradigm argument. Table 1 shows NaTex's VAE reconstruction improves with token count, but provides no external reference point. The "over 80× compression" ratio is stated without defining the uncompressed baseline point density—without knowing N (the number of input points), the compression ratio cannot be independently verified.

What evidence exists in the paper. The only comparison in the VAE evaluation (Section 4.1, Table 1) is internal—different latent sizes of the same architecture. Figure 6 shows qualitative reconstruction results at different settings but does not compare against alternative compression methods. The VAE architecture choice is motivated by prior success on geometry (3DShape2VecSet, Hunyuan3D-VAE) rather than by evidence that it is optimal for color fields.

Mitigation status. Not addressed. The paper states that "alternative designs are possible" and leaves "their exploration to future work" (Section 3.1), but this acknowledgment is about the VAE architecture family, not about the absence of compression benchmarks. A practitioner implementing a texture generation system has no evidence from this paper about whether the VecSet VAE's complexity (cross-attention, dual-branch design, UDF loss) is justified by reconstruction quality improvements over simpler alternatives.

7. Implications and Future Directions

How This Work Changes the Landscape

NaTex represents a paradigm reframing, not an incremental improvement. The paper does not propose a better inpainting module for multi-view texturing pipelines, a more consistent cross-view attention mechanism, or a smarter blending scheme. It argues—and demonstrates with empirical evidence—that the entire 2D-lifting approach to texture generation is structurally limited, and that reformulating texture generation as a 3D-native latent color diffusion problem eliminates those limitations by construction rather than patching them.

The conceptual shift: from 2D-derived to 3D-native appearance generation. Prior to NaTex, the dominant assumption across both research and commercial systems was that generating 2D images from multiple viewpoints and baking them onto 3D surfaces was the natural—perhaps the only scalable—approach to automated texture generation. This assumption was pragmatic: 2D image diffusion models were mature and powerful, while 3D generative models were nascent. NaTex demonstrates that this assumption was contingent, not necessary. By modeling texture as a dense color point cloud and applying the standard latent diffusion paradigm (VAE compression + DiT generation), the paper shows that native 3D texture generation is not only possible but produces measurably better results on the axes that matter most: boundary alignment, occlusion handling, and global consistency.

This reframing changes the research agenda. The question shifts from "how do we make 2D-lifting pipelines work better?" to "what other 3D appearance tasks can be reformulated as native color field generation?" The paper provides early evidence that the answer includes material generation, part segmentation, and texture refinement—all within the same architecture, with at most fine-tuning. This suggests that 3D-native appearance generation may be a unifying framework for tasks previously treated as separate problems with separate architectures.

Why this is a paradigm shift rather than a point solution. The paper's contribution is not primarily the specific VAE architecture or the specific DiT conditioning scheme—those are implementation choices that may be improved upon. The contribution is the demonstration that the latent color diffusion paradigm works for texture generation at production quality, establishing a new baseline approach that future work can build on, extend, and refine. This is analogous to how the original latent diffusion model paper (Rombach et al., 2022) did not propose the optimal image generation architecture but established the paradigm within which subsequent innovations (SDXL, Flux, etc.) developed. NaTex does for 3D texture generation what LDM did for image generation: it shows that the problem fits the paradigm, and that the paradigm shift yields qualitative and quantitative improvements over the prior dominant approach.

Reconciling prior contradictions. The paper resolves an implicit tension in the 3D generation literature. On one hand, joint geometry-and-appearance generation models (CLAY, Hunyuan3D) showed that diffusion models could produce textured 3D assets, but quality was limited because geometry and appearance competed for model capacity. On the other hand, dedicated texturing pipelines (Paint3D, TexGen, Hunyuan3D-2) achieved higher texture quality using 2D priors, but suffered from occlusion, alignment, and consistency artifacts that no amount of pipeline engineering could fully eliminate. NaTex shows that this was a false dichotomy: by separating geometry generation (which can use a dedicated model like Hunyuan3D 2.5) from texture generation (which uses NaTex), and by generating texture natively in 3D rather than projecting from 2D, it is possible to achieve both high quality and structural consistency. The paper does not merely outperform prior methods on metrics—it demonstrates that the tradeoff between quality and consistency was an artifact of the 2D-lifting paradigm, not a fundamental constraint.

Research directions that become more attractive. The paper's success strongly suggests that 3D-native approaches should be the default for any surface-attached signal generation task where geometric precision matters. This includes displacement map generation (where exact alignment to geometric features is critical), ambient occlusion baking (where the relationship between surface points depends on global 3D geometry), and semantic labeling of 3D surfaces (where labels must respect geometric boundaries). Research on improving MVD pipelines—better cross-view attention, more sophisticated inpainting, smarter blending—becomes less attractive relative to research on improving 3D-native generation, because the paper provides evidence that the MVD ceiling is structural rather than temporary.

Research directions that become less attractive. The paper's diagnosis of MVD cascade errors as irreducible (Section 1) implies that engineering effort spent on component-level improvements to 2D-lifting pipelines—better normal-conditioned diffusion, more views, fancier inpainting—is asymptotically bounded. While such improvements may still yield practical gains for deployed systems that cannot be immediately replaced, the paper's evidence suggests that fundamental progress on texture quality requires abandoning the 2D-lifting paradigm rather than perfecting it. This reorients the field's investment toward 3D-native representations, VAE design for surface fields, and geometry-conditioned diffusion architectures.

Follow-Up Research This Work Enables

Direct 3D consistency metrics for texture evaluation. The paper's central claims—superior occlusion handling, boundary alignment, and cross-view consistency—are evaluated primarily through 2D projection metrics (c-FID, LPIPS, CMMD, CLIP-I) that cannot directly measure these 3D properties. A critical follow-up is to design and validate metrics that quantify the specific advantages NaTex claims: (a) texture seam error measured as color discrepancy along UV seam boundaries in 3D space (comparing the color of adjacent texels that map to neighboring 3D positions across a UV cut), (b) geometric boundary alignment computed as the Chamfer distance between Canny edges in the rendered texture and geometric edge features of the mesh, and (c) multi-view consistency measured as the variance of rendered color for the same surface point when viewed from N densely sampled random camera angles. Computing these metrics on NaTex and on the MVD baselines from Table 2 (Paint3D, TexGen, Hunyuan3D-2, RomanTex, MaterialMVP) on the same test set would directly test whether NaTex's structural advantages translate to measurable 3D consistency improvements, and would provide the field with standardized evaluation tools currently absent from the texture generation literature.

Data scaling laws for 3D-native texture generation. The paper trains exactly one model (NaTex-2B) on an unspecified volume of training data. A critical open question is whether 3D-native texture generation is more or less data-efficient than MVD approaches that leverage 2D pre-training. A follow-up study should train NaTex variants at multiple data scales (e.g., 1K, 10K, 100K, 1M textured assets) and measure performance on a held-out test set covering diverse object categories. The key comparison is against MVD baselines at matched data scales—do the MVD methods' 2D pre-training advantages dominate when 3D data is scarce, and do NaTex's structural advantages dominate as 3D data becomes abundant? This experiment would provide practitioners with a clear decision criterion: if you have fewer than X thousand textured 3D assets, use MVD; if more, use NaTex. It would also test the paper's implicit claim that 3D-native generation is a "scalable" paradigm by establishing whether performance improves predictably with data volume.

Mesh quality robustness benchmarking. The paper justifies its truncated UDF loss by noting that real-world 3D assets are often not watertight, but all test geometry comes from Hunyuan3D 2.5, which generates clean, watertight meshes. A stress-test experiment should evaluate NaTex on a curated benchmark of artist-created and 3D-scanned meshes with known defects: non-watertight geometry (holes, missing faces), inconsistent normal orientation, degenerate UV islands, and non-manifold edges. Compare NaTex's texture quality (using the 3D consistency metrics proposed above) against MVD baselines on each defect category. The hypothesis from the paper's design rationale is that NaTex should be more robust because the UDF handles non-watertight geometry and the point cloud representation does not depend on UV quality—but this must be empirically validated before practitioners working with real-world assets can adopt the approach with confidence. A negative result (NaTex degrades sharply on certain defect types) would identify specific mesh preprocessing requirements that users must satisfy.

Joint geometry-texture VAE for end-to-end appearance generation. The current NaTex architecture assumes geometry is given and fixed—the geometry branch encodes it, but no gradients flow back to geometry from the texture generation process. An important extension is to train a joint geometry-texture VAE where the geometry and color branches share a common latent space (rather than separate geometry and color latents), and where the DiT generates both geometry and texture latents jointly. This would enable texture generation to influence geometry refinement—for example, if the generated texture suggests a surface should have a sharp edge (a stripe boundary), the joint model could sharpen the geometry at that location. The experiment would compare joint generation against the current two-stage approach (geometry from Hunyuan3D 2.5, then texture from NaTex) on metrics measuring geometric-texture coherence: how well do texture features align with geometric features when both are generated together? The paper's architecture makes this extension tractable because the cross-attention VAE can naturally handle joint encoding—the challenge is designing a DiT that generates aligned geometry and color latents simultaneously while preserving the pointwise correspondence that enables native geometry control.

Scaling model size to test the ceiling of native texture quality. The paper trains NaTex-2B as a single point on the model-size curve. A scaling study training NaTex variants at 500M, 2B, 5B, and (if feasible) 20B parameters (keeping the VAE at 300M, scaling only the DiT) would establish whether the quality improvements over MVD methods grow or shrink with model capacity. If the gap widens with scale—NaTex-20B improves more relative to NaTex-2B than an equivalently scaled MVD method improves—that provides strong evidence that 3D-native generation is the right asymptotic bet. If the gap narrows (MVD methods catch up at larger scales because 2D image models also scale), the practical recommendation depends on the scale at which MVD parity occurs and whether that scale is economically feasible. This experiment would also test the paper's analogy to image LDMs: does texture quality improve with DiT scale following similar power-law trends as image generation quality?

Training-free and few-shot generalization across object categories and artistic styles. The paper demonstrates generalization through applications (material generation, part segmentation), but these use the same model trained once on the full dataset. A targeted generalization experiment should test NaTex's zero-shot performance on held-out object categories (e.g., train on furniture and vehicles, test on characters and buildings) and on held-out artistic styles (e.g., train on photorealistic textures, test on stylized/cartoon textures). The key question is whether the geometry conditioning (RoPE + geometry latents) provides strong enough structural priors that the model generalizes to unseen shapes, or whether it overfits to the training geometry distribution. Compare against MVD methods that leverage general-purpose image diffusion models with broad visual priors—the MVD methods might have an advantage in style generalization (because their 2D backbones were trained on diverse artistic images) while NaTex might have an advantage in geometric generalization (because its conditioning is inherently 3D-aware). Understanding this tradeoff would guide practitioners choosing between approaches for specific deployment scenarios.

Practical Applications and Downstream Use Cases

Automated texturing in 3D asset production pipelines. The most immediate application is replacing MVD-based texturing in production workflows that generate 3D assets from images or text prompts. Current commercial systems (Tripo, Rodin, Meshy) use MVD texturing and exhibit the boundary misalignment artifacts visible in Figure 7—misaligned decorative elements, bleeding colors across geometric edges, inconsistent appearance across views. NaTex can be dropped into these pipelines as a replacement texturing module: the geometry generator (e.g., Hunyuan3D 2.5) produces the mesh, and NaTex textures it from the same input image. The paper's quantitative results in Table 2 (NaTex outperforming five prior methods across c-FID, LPIPS, CMMD, and CLIP-I) and the qualitative comparison in Figure 7 (showing NaTex avoiding the misalignment that Rodin-Gen2 and Tripo 3.0 exhibit) provide direct evidence of production-quality improvement. The practical benefit is reduced manual cleanup time—artists currently fix misaligned textures and inpaint occlusion holes by hand, and NaTex's native 3D generation avoids these artifacts at generation time.

High-quality material generation for physically based rendering. NaTex's material generation pipeline (Appendix B, Figures 11–14) produces roughness and metallic maps that are inherently aligned to the albedo texture and to the input geometry—a property that MVD-based material pipelines struggle with because each material channel must be independently generated and aligned. In a production PBR workflow, an artist provides a reference image and a base mesh; NaTex generates the albedo, roughness, and metallic maps as aligned, consistent outputs that can be directly used in a renderer. Figure 13 shows NaTex producing "more accurate and better-aligned materials compared to prior approaches," and Figure 12 demonstrates the materials holding up under different lighting conditions—a critical requirement for assets that will be rendered in varied environments. The practical benefit is reducing the multi-pass manual workflow where artists generate albedo, then separately author roughness and metallic, then manually align them and verify lighting consistency.

Texture refinement as a post-processing step for any texturing pipeline. NaTex's color control mechanism (Section 3.2) enables a refinement mode where an initial texture (from any source—MVD pipelines, manual authoring, photogrammetry) is improved by the model. The paper demonstrates this correcting occlusion holes (Figure 8) and projection errors (Appendix B, Figure 18) in just 5 diffusion steps. This makes NaTex-Refiner a practical drop-in post-processing module: take the output of an existing MVD pipeline (which may have occlusion holes, minor misalignments, or blending artifacts), feed it to NaTex with the reference image and geometry, and receive a cleaned-up texture. The 5-step inference (with one-step generation demonstrated in Figure 10 as possible, though quality tradeoffs are unquantified) makes this efficient enough for iterative artist workflows where preview speed matters. The practical benefit is salvaging the substantial investment in existing MVD infrastructure while immediately improving output quality—NaTex refinement can be added to any pipeline without replacing the core texturing approach.

Automated part segmentation and texturing for interactive 3D editing. NaTex's part segmentation capability (Section 3.3, Appendix B, Figure 15) enables automated labeling of 3D mesh components—given a reference image and a 2D segmentation mask (from SAM, for instance), NaTex produces a 3D-consistent part-labeled texture aligned to the geometry. For 3D editing applications (customizing a character's clothing, swapping materials on specific car parts, recoloring furniture components), this provides automatic part identification as a prerequisite to per-part operations. The paper shows this works training-free for simple cases and with fine-tuning for complex structures (Figure 15), meaning a base model could provide immediate utility with the option to specialize. NaTex's part texturing capability (Figures 16–17) further enables per-part texture generation that correctly handles occluded regions between adjacent parts—a known pain point in MVD pipelines where interior surfaces between parts are in shadow or occluded from all camera views. The practical benefit is enabling automated asset customization workflows (e.g., "change the sofa fabric to leather but keep the wooden legs") that currently require manual mesh segmentation and per-part texture authoring.

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Instruction-Guided Lesion Segmentation for Chest X-rays with Automatically Generated Large-Scale Dataset2025· vision← in-domain predecessor
- Segment Anything2023· vision← LLM-judged similar
- Depth Anything 3: Recovering the Visual Space from Any Views2025· vision← in-domain predecessor
Repurposing Geometric Foundation Models for Multi-view Diffusion2026· vision← similar topic
- SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video Editing2026· vision← in-domain predecessor
- LoGeR: Long-Context Geometric Reconstruction with Hybrid Memory2026· vision← similar topic
- DA-Flow: Degradation-Aware Optical Flow Estimation with Diffusion Models2026· vision← similar topic
- Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer2025· vision← similar topic
- Scalable Diffusion Models with Transformers2022· vision← similar topic
- A Global Geometric Framework for Nonlinear Dimensionality Reduction2000· vision← similar topic
SphereDiff: Tuning-free 360° Static and Dynamic Panorama Generation via Spherical Latent Representation2025· vision← similar topic
- SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration2025· vision← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

HunyuanVideo 1.5 Technical Report2025· vision→ in-domain predecessor
- TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times2025· inference-optimization→ LLM-judged similar
- PixelDiT: Pixel Diffusion Transformers for Image Generation2025· vision→ in-domain predecessor
- LongCat-Image Technical Report2025· vision→ similar topic
SphereDiff: Tuning-free 360° Static and Dynamic Panorama Generation via Spherical Latent Representation2025· vision→ similar topic
- MAGREF: Masked Guidance for Any-Reference Video Generation with Subject Disentanglement2025· vision→ in-domain predecessor
- Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance2025· vision→ similar topic
Repurposing Geometric Foundation Models for Multi-view Diffusion2026· vision→ similar topic
- DA-Flow: Degradation-Aware Optical Flow Estimation with Diffusion Models2026· vision→ in-domain predecessor
- Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer2025· vision→ similar topic