ArXiv: 2511.20645

🎯 Pitch

PixelDiT proves that pixel-space diffusion can finally match the quality of latent models, defying years of conventional wisdom that autoencoder compression is essential. By splitting the job into a patch-level DiT for global semantics and a pixel-level DiT with per-pixel modulation for fine textures, it reaches 1.61 FID on ImageNet without any VAE—while side-stepping the lossy reconstruction that silently sabotages latent edits on tiny details.

1. Executive Summary

This paper proposes PixelDiT, a single-stage, fully transformer-based diffusion model that performs end-to-end training and sampling directly in pixel space without the autoencoder used by standard latent diffusion models. The core contribution is a dual-level DiT architecture that disentangles semantic learning from detail refinement — a patch-level DiT captures global layout while a pixel-level DiT with pixel-wise AdaLN modulation (per-pixel conditioning from semantic tokens) and pixel token compaction (temporarily compressing per-pixel tokens for efficient global attention) enables dense per-pixel modeling without prohibitive compute. On ImageNet, PixelDiT-XL achieves 1.61 FID at 256×256 and 1.81 FID at 512×512, surpassing prior pixel-space models and closing much of the quality gap with latent diffusion; extending to text-to-image generation, PixelDiT-T2I attains 0.74 on GenEval and 83.5 on DPG-Bench at 1024² resolution, establishing that pixel-space diffusion can match the fidelity of latent approaches when pixel modeling is explicitly structured — and that avoiding VAE compression yields superior detail preservation for tasks like image editing where lossy reconstruction would otherwise distort fine structures.

2. Context and Motivation

The Core Problem: Pixel-Space Diffusion Remains Impractical Despite Its Promise

This paper confronts a fundamental tension in modern image generation: latent diffusion models (LDMs) are the dominant paradigm because they are computationally efficient, but their reliance on autoencoders introduces structural limitations that cap sample fidelity and hamper certain applications. The central question PixelDiT asks is whether it is possible to reclaim the end-to-end simplicity of pixel-space diffusion — training and sampling directly on raw pixels, without any pretrained compression module — while achieving the image quality and training efficiency that have made latent-space approaches the default choice.

This matters for several concrete reasons that the paper articulates across the introduction and related work:

  • The autoencoder is an architectural dependency, not a conceptual necessity. Latent diffusion works by denoising in a compressed space produced by a separately pretrained variational autoencoder (VAE). The VAE is trained once (with a reconstruction objective) and then frozen during diffusion training. This two-stage pipeline creates a coupling — the diffusion model inherits whatever the autoencoder can and cannot represent. If the VAE discards high-frequency textures, smears fine details, or introduces color shifts during encoding, the diffusion model can never recover them, no matter how many parameters or training steps it receives.

  • Reconstruction fidelity is bounded and misaligned with generation quality. The VAE's training objective (minimizing reconstruction error plus a KL regularization term) is only partially correlated with the downstream goal of producing photorealistic samples. As the paper notes, a "rich line of work improves the autoencoder via better architecture or learning objective, including stronger compression schemes, tokenizers, and analysis of the reconstruction–generation optimization dilemma" (Section 2.1). This is not a solved problem — it is an active area of research precisely because the reconstruction–generation trade-off is fundamental to the two-stage design. PixelDiT's approach sidesteps this entirely: there is no autoencoder, so there is no reconstruction–generation misalignment.

  • Image editing and detail-critical applications suffer most. Figure 1b provides a vivid example: when using FlowEdit to change a bicycle to a motorcycle in a scene with small wall text, Stable Diffusion 3 and FLUX both distort the background text because their VAEs introduce "severe and non-invertible distortions on small details." The VAE reconstruction alone (without any editing) already smears the text. The diffusion model then compounds this error. PixelDiT, operating in pixel space without any VAE, preserves the background text exactly — both in reconstruction and through the editing process. This is a practically significant advantage for any application where unedited regions must remain pixel-perfect, including inpainting, localized editing, and content-aware manipulation.

  • The autoencoder pipeline adds data and compute overhead. Pretraining or co-training a large autoencoder requires additional data and GPU-hours beyond what the diffusion model itself needs. While this overhead is amortized across many training runs (the same VAE can be reused), it represents an extra engineering and computational dependency that pixel-space training eliminates. This is especially relevant for domains where a suitable pretrained autoencoder does not exist (e.g., specialized medical imaging, scientific data, or non-standard color spaces).

The paper frames this as a revisiting of pixel-space diffusion rather than an entirely new direction. Early diffusion models (e.g., ADM, 2021) operated in pixel space and achieved strong results. The shift to latent space was motivated by compute: the quadratic cost of attention in image resolution makes end-to-end pixel-space training at megapixel scales prohibitively expensive. But that shift came with a cost — the VAE bottleneck — that has only become more apparent as diffusion models have matured and users have pushed into detail-sensitive applications.

The Obstacle: Efficient Pixel Modeling Does Not Exist

The paper identifies the core technical obstacle that has prevented pixel-space diffusion from closing the quality gap with latent approaches:

"The core challenge in pixel-space diffusion can be framed as pixel modeling. By pixel modeling, we refer to the mechanism of capturing dense, per-pixel interactions and high-frequency details, which is distinct from the semantic structural generation typically handled by coarse patch tokens. Effectively modeling these per-pixel tokens is crucial for texture fidelity but computationally expensive."

This is a precise diagnosis. The problem is not that pixel-space diffusion is conceptually impossible — ADM proved it works. The problem is that no existing architecture efficiently handles both the semantic structure (long-range dependencies across the image, object layout, global coherence) and the per-pixel detail (texture, edges, high-frequency content) within a single training pipeline. Everything becomes a trade-off.

The paper catalogs the landscape of prior attempts and their specific failure modes:

Trade-off 1: Aggressive patchification kills detail. One line of work (JetFormer, EPG, FARMER, PixNerd) uses large patch sizes (e.g., 16×16 or larger) to keep attention sequence lengths manageable. This is effective for global structure — a 16×16 patch token can represent "this region is mostly sky" — but it fundamentally limits the model's ability to represent sub-patch variations. Two neighboring pixels with different colors inside the same patch are collapsed into a single token vector, and the attention mechanism cannot distinguish them. The paper summarizes this as: "This significantly weakens per-pixel token modeling and hinders the generation of finer visual contents."

Trade-off 2: Fine granularity causes quadratic compute blowup. Another line of work pushes toward near-pixel granularity — small patch sizes (e.g., 1×1 or 2×2) or U-ViT-like architectures that process longer sequences. This preserves detail better, but the global attention must process sequences of length H×WH \times W (or close to it), which incurs O((HW)2)O((HW)^2) complexity. The paper notes this leads to "very long token sequences with quadratic complexity" and "heavy decoder-style stacks, resulting in high training and sampling costs." In practice, a 256×256 image with 1×1 patches produces 65,536 tokens — the attention matrix alone would be ~4.3 billion entries per head, per layer, which is infeasible at scale.

Trade-off 3: Cascaded pipelines add complexity and accumulated errors. Models like Cascaded Diffusion Models (CDM) mitigate the cost by training multiple models at different resolutions (e.g., a low-resolution base model plus a super-resolution upsampler). While this reduces peak memory, it introduces multiple stages of training, multiple models to maintain, and the potential for error accumulation — a mistake in the low-resolution stage propagates and gets amplified by the upsampler. The paper cites this as a mitigation, not a solution, noting that cascaded pipelines "may introduce additional stages and accumulated errors."

Trade-off 4: Convolutional pixel models lack the flexibility of transformers. Some recent pixel-space models (Simple Diffusion, SiD2) use convolutional U-Net architectures rather than transformers. These avoid the quadratic attention cost through local convolutions and strided downsampling, but they sacrifice the flexibility and global receptive field that transformer attention provides. The paper's goal is explicitly a transformer-based pixel model, which inherits the architectural benefits of DiTs (scaling properties, conditioning flexibility, compatibility with multi-modal extensions) but must solve the attention cost problem differently.

The synthesis of these observations is the paper's motivating insight: the obstacle to practical pixel-space diffusion is not any one of these challenges in isolation, but the lack of an architecture that decouples the two tasks — semantic reasoning and pixel refinement — into separate pathways, each optimized for its own computational characteristics. This decoupling is the central architectural idea of PixelDiT, and the paper builds its entire motivation around the claim that prior work attempted to solve both problems with a single mechanism (either coarse patches or fine attention), which inevitably favors one at the expense of the other.

Where Prior Work Falls Short, Specifically

The paper situates itself carefully relative to a dense field of recent pixel-space models. Understanding the specific limitations of each helps clarify what PixelDiT does differently:

Convolutional U-Net approaches (ADM, Simple Diffusion, SiD2): ADM (2021) established that pixel-space diffusion can produce high-quality images using a U-Net backbone with attention at specific resolutions. This was the state of the art before latent diffusion took over. However, U-Nets are architecturally rigid compared to transformers — they are harder to scale, harder to adapt to multi-modal conditioning, and the attention layers are typically limited to low-resolution feature maps to control cost. Simple Diffusion (Hoogeboom et al., 2023) simplified the U-Net design and achieved 2.44 FID on ImageNet 256² at 2B parameters, but this is still substantially behind latent DiTs (e.g., 1.35 FID for LightningDiT-XL at 675M parameters). The paper does not dismiss U-Nets but implicitly argues that the transformer architecture is superior for scaling and multi-modal extension — provided the pixel modeling problem can be solved within it.

Transformer pixel models with aggressive patchification (JetFormer, EPG, FARMER, PixNerd): These models apply transformers directly to pixels but use large patch sizes (≥16) to keep attention affordable. JetFormer (2025) formulates autoregressive generation over raw pixels and text using a 2.8B parameter model but achieves only 6.64 FID on ImageNet 256² — worse than much smaller latent models. EPG-XXL/16 (concurrent work) reaches 1.81 FID with 789M parameters, which is competitive with latent models, but the paper notes that such approaches still struggle with fine texture details because the patch-level representation is the only level of representation. There is no separate mechanism for sub-patch detail refinement.

Fine-grained transformer approaches (PixelFlow, U-ViT variants): PixelFlow (2025) uses a hierarchical flow-based design with 677M parameters and achieves 1.98 FID. It processes features at multiple scales, which helps with detail, but the architecture is fundamentally different from a standard DiT — it uses a Flow-based formulation with specialized multi-scale attention blocks. The paper positions PixelDiT as architecturally simpler and more directly compatible with the DiT/MM-DiT ecosystem that has become standard for latent diffusion.

Neural field and hybrid approaches (PixNerd, JiT): PixNerd (2025) employs lightweight neural field layers for efficient pixel-space diffusion and achieves 1.93 FID at 700M parameters. JiT (concurrent work) demonstrates that plain transformers can efficiently model high-dimensional data by predicting clean images (x0x_0-prediction) rather than noise, reaching 1.82 FID with a 2B-parameter model. Both are strong baselines that the paper directly compares against. PixelDiT's architectural contribution — the dual-level design with explicit pixel modeling — is presented as achieving better results (1.61 FID) with a smaller model (797M parameters) through better-structured computation.

Latent models that try to fix the autoencoder (REPA-E, RAE, LightningDiT, SVG): The paper also acknowledges work that attempts to address the autoencoder problem from within the latent paradigm. REPA-E (2025) jointly tunes the VAE and diffusion transformer to align representations. RAE (2025) replaces the variational bottleneck with a representation autoencoder. LightningDiT (2025) addresses the reconstruction–generation optimization dilemma. These approaches improve latent quality but do not eliminate the autoencoder — they remain two-stage pipelines with a compression module. PixelDiT's approach is orthogonal: rather than improving the autoencoder, remove it.

How PixelDiT Positions Itself

PixelDiT is positioned not as an incremental improvement to latent diffusion, but as a paradigm shift back to pixel space enabled by a specific architectural innovation. The paper explicitly frames this as:

"Single-stage pixel-space diffusion offers a simpler, more elegant, and domain-agnostic paradigm for visual synthesis than the two-stage latent-space approach."

The key claim is that the quality gap between pixel and latent models has persisted not because pixel space is inherently harder to model, but because pixel modeling has been done inefficiently. The paper identifies two root causes (Section 5, Conclusion):

  1. Greater complexity of pixel-space data and noise distributions compared to the smoother, compressed latent space.
  2. Lack of a mature training recipe — including effective architecture, objective, noise scheduling, and optimization design — specifically tuned for pixel-space diffusion transformers.

PixelDiT addresses both. Architecturally, the dual-level design tackles the complexity by factorizing the learning problem: the patch-level pathway handles the "easier" semantic reasoning (for which aggressive patchification is appropriate), and the pixel-level pathway handles the "harder" per-pixel detail refinement (with a specialized architecture that makes this affordable). The training recipe incorporates modern components that have proven effective for latent DiTs — AdaLN-Zero conditioning, RMSNorm, RoPE, logit-normal timestep sampling, representation alignment (REPA), and Rectified Flow — but repurposed for end-to-end pixel training.

The paper also positions itself as a unification of ideas from both pixel and latent literature. The dual-level design echoes the multi-scale processing found in cascaded models (CDM) and hierarchical latent models (Würstchen, DDT), but collapses them into a single end-to-end model. The pixel-wise AdaLN modulation is inspired by conditioning mechanisms in DiTs but extends them from patch-level to per-pixel granularity. The pixel token compaction is a novel compression mechanism that, unlike the VAE bottleneck, is lossless in the sense that high-frequency information is preserved through residual connections and learned expansion layers.

The Stakes: Why This Matters Now

The paper's motivation is strengthened by timing. Several trends make pixel-space diffusion more relevant in 2025–2026 than it was when latent models took over:

  • Image editing and controllable generation are maturing as applications. As diffusion models move beyond "generate a pretty picture" to tasks like FlowEdit, inpainting, and structure-preserving manipulation, the VAE's distortion of unedited regions becomes a practical bottleneck, not a theoretical concern. Figure 1b makes this concrete with a real editing workflow.

  • Transformer architectures for diffusion are stabilizing. The DiT/MM-DiT family (from Peebles & Xie, 2023, through SD3 and FLUX) has become a de facto standard. PixelDiT's architecture is designed to be compatible with this ecosystem — it uses the same DiT blocks, the same AdaLN-Zero conditioning, and the same Rectified Flow formulation — making it a drop-in upgrade path for practitioners already using DiT-based latent models.

  • Training recipes have improved. Techniques like REPA (representation alignment), logit-normal timestep sampling, and improved solvers (FlowDPMSolver) have significantly accelerated latent DiT training. The paper demonstrates that these same techniques transfer to pixel space and are essential for stable, efficient training — one of the key missing pieces that prior pixel-space attempts lacked.

  • The compute gap is narrowing. As GPUs become more powerful and training budgets grow, the compute argument for latent space weakens. PixelDiT-XL uses 311 GFLOPs per forward pass at 256², which is only modestly higher than latent models like DiT-XL (238 GFLOPs) and substantially lower than many prior pixel models (e.g., PixelFlow at 5818 GFLOPs). The paper's FLOPs analysis (Table 15 in the appendix) directly confronts the efficiency argument that has historically favored latent models.

In summary, the paper addresses a problem that is simultaneously old (pixel-space diffusion predates latent diffusion) and newly urgent (VAE limitations are becoming a practical bottleneck for downstream applications). Its contribution is the identification that pixel modeling — the mechanism of capturing dense, per-pixel interactions — is the key missing ingredient, and the demonstration that a specific architectural factorization (dual-level DiT + pixel-wise AdaLN + pixel token compaction) makes this ingredient affordable for the first time in a purely transformer-based model.

3. Technical Approach

3.1 Reader Orientation

PixelDiT is a single-stage, end-to-end diffusion model that generates images by gradually denoising raw pixels, without using a separate autoencoder to compress the image into a smaller latent space first. The system solves the problem that pixel-space diffusion has historically been too computationally expensive for high-quality generation — the core idea is to factor the generation process into two parallel transformer pathways: one that thinks about the image at a coarse, semantic level (where object layouts and global coherence are decided), and another that refines every individual pixel's color and texture (where fine details live), with specialized mechanisms to make this dense per-pixel reasoning affordable.

3.2 Big-Picture Architecture (Diagram in Words)

The PixelDiT system has four major components connected in a single forward pass:

  1. Input Embedding: The noisy image (raw RGB pixels) enters as a 3D tensor $X \in \mathbb{R}^{B \times C \times H \times W}$ and is simultaneously embedded into two representations: coarse patch tokens (groups of $p \times p$ pixels flattened into vectors) and fine pixel tokens (one vector per pixel, kept at low dimension).

  2. Patch-Level DiT Pathway (Semantic Engine): A stack of $N$ augmented DiT blocks processes the patch tokens. Each block applies self-attention with RoPE (Rotary Position Embeddings) and AdaLN-Zero conditioning driven by a global conditioning vector $c$ (combining timestep, class label, and/or text embeddings). This pathway produces semantic tokens $s_N$ — a low-resolution feature map summarizing global layout, object identity, and spatial relationships — that will guide all subsequent pixel-level decisions.

  3. Pixel-Level DiT Pathway (Detail Engine): A stack of $M$ PiT (Pixel Transformer) blocks processes the pixel tokens. Each PiT block has three sub-operations: (a) pixel-wise AdaLN that takes the semantic tokens $s_{\text{cond}}$ and generates distinct modulation parameters for every single pixel, so that updates in "sky" regions follow different rules than updates in "edge" or "texture" regions; (b) pixel token compaction that temporarily squeezes all pixels within each $p \times p$ patch into a single compact token, runs efficient global self-attention across patches, and then expands back to per-pixel resolution; and (c) a per-pixel feedforward network (FFN) with gating.

  4. Output Projection: The final pixel tokens are projected back to RGB values, producing the denoised image prediction (or velocity prediction, in the Rectified Flow formulation).

Information flow: A noisy image enters → pixel tokens and patch tokens are created → the patch-level pathway processes the coarse tokens using global self-attention, producing a semantic description → the pixel-level pathway uses this semantic description to condition per-pixel updates, running efficient attention via compaction → the refined pixel tokens become the output prediction → the process repeats at the next diffusion timestep (or the solver takes the next integration step).

3.3 Roadmap for the Deep Dive

  • First, the Rectified Flow training objective and how it defines what the model learns (velocity prediction in pixel space) — because all architecture decisions serve this objective.
  • Second, the dual-level architecture decomposition — why separating patch-level semantics from pixel-level details is the foundational design choice, and how the two pathways interact.
  • Third, the patch-level DiT pathway in detail — its blocks, conditioning mechanism, and how it produces semantic tokens that will guide the pixel-level pathway.
  • Fourth, the pixel-level pathway and its two core innovations: pixel-wise AdaLN modulation (per-pixel conditioning from semantic context) and pixel token compaction (how efficient global attention is made possible despite processing every pixel).
  • Fifth, the text-to-image extension using MM-DiT blocks — how text conditioning is integrated without changing the pixel-level pathway.
  • Sixth, the representation alignment (REPA) auxiliary loss and why it is essential for training stability.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural innovation paper whose core idea is that pixel-space diffusion becomes practical and competitive when pixel modeling is explicitly structured as a separate, specialized pathway that operates alongside (and is conditioned by) a coarse semantic pathway — rather than trying to handle both semantics and details in a single sequence of tokens.

Rectified Flow Training in Pixel Space

PixelDiT uses the Rectified Flow formulation rather than the standard denoising diffusion probabilistic model (DDPM) or score-based formulation. Rectified Flow defines a straight-line interpolation between the data distribution and a noise distribution, and trains the model to predict the velocity (direction and magnitude of change) along this path.

The training procedure works as follows. For each training sample, an image $x_0$ is drawn from the data distribution (e.g., ImageNet) and noise $\varepsilon$ is drawn from a standard Gaussian $\mathcal{N}(0, I)$ with the same shape as the image. A timestep $t \in [0, 1]$ is sampled — not uniformly, but from a logit-normal distribution (following Esser et al., 2024), which concentrates samples near the middle of the trajectory where the model's predictions are most consequential. The noisy intermediate image is constructed via linear interpolation:

xt=t⋅x0+(1−t)⋅εx_t = t \cdot x_0 + (1 - t) \cdot \varepsilon

where $x_t$ is the corrupted image at time $t$, $x_0$ is the clean image, and $\varepsilon$ is the sampled noise. At $t = 0$, $x_t$ is pure noise; at $t = 1$, $x_t$ is the clean image.

The target velocity $v_t$ that the model should predict is simply the difference between the endpoints:

vt=x0−εv_t = x_0 - \varepsilon

This is the constant "velocity" that would transport the noise sample to the data sample along a straight line. The model $f_\theta$ (our PixelDiT architecture) takes the noisy image $x_t$, the timestep $t$, and optional conditioning $y$ (class label or text), and predicts an estimate of this velocity:

Ldiff=Et,x0,ε[∥fθ(xt,t,y)−vt∥22]\mathcal{L}_{\text{diff}} = \mathbb{E}_{t, x_0, \varepsilon} \left[ \| f_\theta(x_t, t, y) - v_t \|_2^2 \right]

where $\mathbb{E}_{t, x_0, \varepsilon}$ is the expectation over the timestep distribution, the data distribution, and the noise distribution.

What it computes: the mean squared error between the model's predicted velocity and the true velocity, averaged over all timesteps, images, and noise samples. At each training step, a batch of images is corrupted to different degrees $t$, the model predicts the velocity for each, and the L2 distance to the correct velocity is backpropagated.

Why this form: The Rectified Flow objective has two key properties that make it well-suited for pixel-space training. First, it defines a straight-path transport between noise and data, which means the integration path during sampling is simple and requires fewer solver steps than curved paths (the paper uses as few as 25 steps for text-to-image). Second, unlike the standard $\epsilon$-prediction (where the model predicts the noise $\varepsilon$) or $x_0$-prediction (where the model predicts the clean image directly), velocity prediction produces targets of consistent scale across the entire trajectory — $v_t = x_0 - \varepsilon$ has variance that is approximately constant in $t$, whereas $x_0$-prediction produces targets with very different scales at different timesteps (near $t=0$, the model must predict a nearly-clean image from pure noise, which is a much harder regression problem). The appendix (Section G.3) notes that the authors experimented with $x$-prediction and found it did not outperform velocity prediction in their setting, requiring additional tuning of the logit-normal sampling hyperparameters.

During inference (sampling), the model starts from pure noise $x_0 \sim \mathcal{N}(0, I)$ and uses a numerical ODE solver — specifically FlowDPMSolver, a modified DPMSolver++ adapted for the Rectified Flow formulation — to integrate the predicted velocity field from $t=0$ to $t=1$, producing the generated image. The solver treats the model's velocity predictions as evaluations of $dx_t/dt$ and steps forward in time. The paper sweeps the number of solver steps: 100 for ImageNet experiments (where quality is paramount) and 25 for text-to-image generation (where sampling speed matters).

Dual-Level DiT Architecture: The Core Decomposition

The central architectural insight of PixelDiT is that modeling an image in pixel space requires solving two qualitatively different problems, and these should be handled by separate transformer pathways with different computational characteristics. The paper calls this a "dual-level" design.

The first problem is semantic reasoning: understanding what objects are present, where they are located, how they relate to each other, and what the overall composition should be. This requires long-range attention — a patch in the top-left corner needs to "know about" a patch in the bottom-right corner to ensure global coherence — but does not require per-pixel resolution. A 16×16 grid of patches can represent the layout of a scene effectively, and attention over 256 tokens (for a 256×256 image with 16×16 patches) is computationally cheap.

The second problem is texture and detail refinement: making sure edges are sharp, textures are realistic, and high-frequency content is faithfully rendered. This requires per-pixel information — the exact RGB value of each pixel matters — but the spatial context needed for a given pixel is more local. The pixel at position (127, 89) needs to know what the pixels immediately around it are doing to maintain texture consistency, but it does not need to directly attend to a pixel on the opposite corner of the image for texture purposes (that is what the semantic pathway handles).

The dual-level architecture implements this decomposition explicitly:

  • Patch-level pathway: processes $L = (H/p) \times (W/p)$ coarse patch tokens through $N$ DiT blocks with full global self-attention. The patch size $p$ is large (default 16 for XL models), keeping the sequence length short. This pathway's job is semantic — it produces feature maps that describe what each image region contains. The hidden dimension $D$ is large (1152 for XL), giving the semantic pathway substantial representational capacity.

  • Pixel-level pathway: processes $H \times W$ pixel tokens through $M$ PiT blocks. Each pixel is represented with a small hidden dimension $D_{\text{pix}} \ll D$ (16 for all model sizes), so the per-pixel vectors are compact. Global attention would be prohibitive over $H \times W$ tokens, so the PiT blocks use pixel token compaction to temporarily reduce the sequence to $L$ tokens for attention, then expand back. This pathway's job is detail refinement — it receives semantic context from the patch-level pathway (via pixel-wise AdaLN conditioning) and adjusts each pixel's representation accordingly.

Why this decomposition works: The key observation is that the semantic pathway can use large hidden dimensions (expensive per-token computation) because it processes few tokens, while the pixel pathway can use small hidden dimensions (cheap per-token computation) because it processes many tokens. The total FLOPs are balanced: the semantic pathway dominates the per-token cost but on $O(L)$ tokens, while the pixel pathway dominates the token count but with $O(D_{\text{pix}})$ per-token cost. The paper's FLOPs analysis (Table 15) shows that PixelDiT-XL uses only 311 GFLOPs per forward pass at 256² — comparable to latent DiTs (238 GFLOPs for DiT-XL/2) and dramatically less than prior pixel models that tried to do everything in one pathway (e.g., 5818 GFLOPs for PixelFlow-XL/4).

Interaction between pathways. The semantic pathway's output $s_N \in \mathbb{R}^{B \times L \times D}$ is combined with the timestep embedding to form conditioning tokens $s_{\text{cond}} = s_N + t_{\text{emb}}$, where $t_{\text{emb}}$ is the timestep embedding broadcast to match the $L$ spatial positions. These conditioning tokens flow into every PiT block in the pixel-level pathway via pixel-wise AdaLN (detailed below). Crucially, there is no direct attention between the pixel-level pathway and the patch-level pathway — information flows only through the conditioning parameters. This is an intentional design choice that keeps the two pathways architecturally decoupled, allowing each to be optimized independently.

Patch-Level DiT Pathway: Semantic Feature Extraction

The patch-level pathway is architecturally similar to a standard DiT (Peebles & Xie, 2023) with several modern modifications adopted from LightningDiT (Yao et al., 2025).

Patch embedding. The input image $x \in \mathbb{R}^{B \times C \times H \times W}$ (with $C = 3$ for RGB) is divided into non-overlapping patches of size $p \times p$. Each patch is flattened into a vector of length $p^2 \cdot C$ and projected to the model's hidden dimension $D$ via a learned linear layer:

s0=Wpatch⋅xpatchs_0 = W_{\text{patch}} \cdot x_{\text{patch}}

where $W_{\text{patch}} \in \mathbb{R}^{(p^2 C) \times D}$ and $x_{\text{patch}} \in \mathbb{R}^{B \times L \times (p^2 C)}$. The result $s_0 \in \mathbb{R}^{B \times L \times D}$ is the initial sequence of patch tokens, with $L = (H/p) \times (W/p)$.

Global conditioning vector. A separate embedding network produces a single global conditioning vector $c$ that will modulate all subsequent transformer blocks:

c=SiLU(Wt⋅t+Wy⋅y+b)∈RB×1×Dc = \text{SiLU}(W_t \cdot t + W_y \cdot y + b) \in \mathbb{R}^{B \times 1 \times D}

where $t$ is the timestep embedding, $y$ is the class embedding (for class-conditional generation) or text embedding (for text-to-image), $W_t$ and $W_y$ are learned projection matrices, $b$ is a bias, and SiLU is the Sigmoid Linear Unit activation. The vector $c$ has shape $B \times 1 \times D$ — it is one vector per batch element, with no spatial dimension. When used in AdaLN, the spatial dimension is broadcast: the same modulation parameters are applied to all $L$ patch tokens identically. This is in deliberate contrast to the pixel-level pathway, where modulation is per-pixel.

Augmented DiT blocks. Following LightningDiT (Yao et al., 2025), each DiT block replaces the original LayerNorm with RMSNorm (which is faster and more stable at scale) and applies 2D Rotary Position Embeddings (RoPE) in all attention layers. The block structure is:

For block $i$ (from 0 to $N - 1$):

s~i=RMSNorm(si)\tilde{s}_i = \text{RMSNorm}(s_i) si=si+α1(c)⊙Attn(γ1(c)⊙s~i+β1(c);RoPE)s_i = s_i + \alpha_1(c) \odot \text{Attn}\left( \gamma_1(c) \odot \tilde{s}_i + \beta_1(c); \text{RoPE} \right) si+1=si+α2(c)⊙MLP(γ2(c)⊙RMSNorm(si)+β2(c))s_{i+1} = s_i + \alpha_2(c) \odot \text{MLP}\left( \gamma_2(c) \odot \text{RMSNorm}(s_i) + \beta_2(c) \right)

where $\alpha_1, \alpha_2, \gamma_1, \gamma_2, \beta_1, \beta_2$ are learned linear projections of the conditioning vector $c$ that produce per-channel scale, shift, and gating parameters. The $\odot$ denotes element-wise multiplication, broadcast across the token dimension.

What each operation does:

  • $\gamma_1(c) \odot \tilde{s}_i + \beta_1(c)$ scales and shifts the normalized input tokens channel-wise, adjusting which features are emphasized before attention.
  • $\text{Attn}(\cdot; \text{RoPE})$ computes multi-head self-attention with 2D rotary position embeddings — RoPE encodes the $(x, y)$ position of each patch as a rotation applied to the query and key vectors, so the attention score between two patches depends on their relative 2D positions in a rotation-equivariant way.
  • $\alpha_1(c) \odot (\cdots)$ applies a learned channel-wise gate after attention, controlling how much the attention output contributes to the residual stream. This is the "AdaLN-Zero" initialization: $\alpha$ is initialized near zero so that each block initially behaves like an identity function, stabilizing early training.
  • The MLP path mirrors the attention path with separate normalization, modulation, and gating.

After $N$ such blocks, the output $s_N \in \mathbb{R}^{B \times L \times D}$ constitutes the semantic tokens — a feature map at the patch resolution that encodes global context about every image region.

Design notes on global-to-patch broadcasting. The AdaLN parameters are produced from the single vector $c$ and broadcast identically to all $L$ patch tokens. This means every patch receives the same scale, shift, and gating — the modulation is spatially uniform at the patch level. This is appropriate for the semantic pathway: the conditioning signal (timestep, class, text) applies globally to the entire image, and spatial differentiation is handled by the attention mechanism itself (through the interaction of RoPE-encoded queries and keys). The pixel-level pathway, in contrast, will use spatially-varying modulation to convey where different refinement rules should apply.

Pixel-Level DiT Pathway: PiT Blocks and Efficient Pixel Modeling

The pixel-level pathway is where PixelDiT departs most radically from prior architectures. It processes one token per pixel — meaning a 256×256 image produces 65,536 tokens — but keeps this feasible through two innovations: pixel-wise AdaLN (which makes dense conditioning efficient) and pixel token compaction (which makes global attention efficient).

Pixel token embedding. Each image is embedded into per-pixel tokens with a linear layer:

X∈RB×C×H×W→reshape + linearRB×H×W×DpixX \in \mathbb{R}^{B \times C \times H \times W} \xrightarrow{\text{reshape + linear}} \mathbb{R}^{B \times H \times W \times D_{\text{pix}}}

where $D_{\text{pix}}$ is the per-pixel hidden dimension, set to 16 for all model sizes. This is deliberately small — 16 dimensions per pixel is enough to represent local color and texture information, but not enough to encode complex semantics (that is the patch-level pathway's job). The small dimension keeps the memory footprint manageable: storing 65,536 tokens of dimension 16 requires about 1 million floats per image, which is comparable to the memory used by the patch-level pathway.

Spatial organization into patches. For alignment with the patch-level conditioning, the pixel tokens are organized into the same $p \times p$ grid structure. The $H \times W$ tokens are reshaped into $L$ groups of $p^2$ tokens each:

X∈RB×H×W×Dpix→reshapeR(B⋅L)×p2×DpixX \in \mathbb{R}^{B \times H \times W \times D_{\text{pix}}} \xrightarrow{\text{reshape}} \mathbb{R}^{(B \cdot L) \times p^2 \times D_{\text{pix}}}

where $L = (H/p) \times (W/p)$ and the batch and patch dimensions are merged for parallel processing. This means each patch in the semantic grid is associated with a block of $p^2$ pixel tokens that live at the corresponding spatial location. The semantic conditioning token for each patch $s_{\text{cond}} \in \mathbb{R}^{(B \cdot L) \times D}$ will provide context for all $p^2$ pixels within that patch.

PiT block structure. Each PiT block consists of:

  1. Pixel-wise AdaLN modulation: conditioning the pixel tokens using spatially-varying parameters derived from the semantic tokens.
  2. Pixel token compaction: compressing the $p^2$ pixel tokens into a single compact token per patch for attention.
  3. Global self-attention across the $L$ compact tokens.
  4. Pixel token expansion: decompressing the attended compact tokens back to $p^2$ pixel tokens.
  5. Feedforward network (FFN) with pixel-wise gating applied to each pixel independently.

The total depth of the pixel-level pathway is $M$ PiT blocks, with $M = 4$ for the XL variant.

Pixel-Wise AdaLN Modulation

This is the mechanism that connects the semantic understanding from the patch-level pathway to the per-pixel updates in the pixel-level pathway. The core idea is that different pixels need different refinement instructions depending on what semantic region they belong to — a pixel in a "fur" region needs to be refined differently than a pixel in a "sky" region, even within the same image.

Why naive approaches fail. The paper illustrates three possible modulation strategies in Figure 3:

  • (A) Naive global AdaLN: broadcast a single conditioning vector to all pixels, regardless of spatial position. This provides no spatial differentiation — every pixel gets the same scale, shift, and gating. The model has no way to say "refine texture aggressively here but be conservative there."

  • (B) Patch-wise AdaLN: repeat the same modulation parameters for all $p^2$ pixels within each patch. This provides spatial differentiation at the patch level (different patches get different modulations) but cannot capture variations within a patch. If a 16×16 patch straddles an object boundary, all pixels on both sides of the boundary receive identical modulation instructions, which is suboptimal for edge refinement.

  • (C) Pixel-wise AdaLN (PixelDiT's approach): generate distinct modulation parameters for every single pixel, informed by the semantic context of its patch. This enables the model to learn, for example, that pixels near edges should receive stronger detail enhancement than pixels in uniform regions, or that pixels in highly textured areas should be processed differently from pixels in smooth gradients.

How pixel-wise AdaLN is computed. For each of the $L$ patches, we have a semantic conditioning token $s_{\text{cond}} \in \mathbb{R}^{D}$ (one $D$-dimensional vector summarizing the patch's semantic content). This vector is expanded into $p^2$ distinct sets of AdaLN parameters via a learned linear projection:

Θ=Φ(scond)∈R(B⋅L)×p2×6Dpix\Theta = \Phi(s_{\text{cond}}) \in \mathbb{R}^{(B \cdot L) \times p^2 \times 6 D_{\text{pix}}}

where $\Phi: \mathbb{R}^{D} \to \mathbb{R}^{p^2 \cdot 6 D_{\text{pix}}}$ is a learned linear projection (a matrix multiplication), and $6 D_{\text{pix}}$ accounts for six modulation parameter groups, each of dimension $D_{\text{pix}}$: $\beta_1$ (shift before attention), $\gamma_1$ (scale before attention), $\alpha_1$ (gate after attention), $\beta_2$ (shift before FFN), $\gamma_2$ (scale before FFN), and $\alpha_2$ (gate after FFN).

The last dimension is split into these six groups:

(β1,γ1,α1,β2,γ2,α2)∈(R(B⋅L)×p2×Dpix)6(\beta_1, \gamma_1, \alpha_1, \beta_2, \gamma_2, \alpha_2) \in \left( \mathbb{R}^{(B \cdot L) \times p^2 \times D_{\text{pix}}} \right)^6

Each group has shape $(B \cdot L) \times p^2 \times D_{\text{pix}}$ — meaning there is a distinct $D_{\text{pix}}$-dimensional vector of modulation parameters for every one of the $p^2$ pixels in every patch.

How these parameters are applied. Within a PiT block, the pixel tokens $X \in \mathbb{R}^{(B \cdot L) \times p^2 \times D_{\text{pix}}}$ are modulated as follows (before attention):

X~=γ1⊙RMSNorm(X)+β1\tilde{X} = \gamma_1 \odot \text{RMSNorm}(X) + \beta_1

After attention (operating on compacted tokens, described below) and expansion back to pixel resolution:

Xout=X+α1⊙AttnOutputX_{\text{out}} = X + \alpha_1 \odot \text{AttnOutput}

And similarly for the FFN path:

Xout=Xout+α2⊙MLP(γ2⊙RMSNorm(Xout)+β2)X_{\text{out}} = X_{\text{out}} + \alpha_2 \odot \text{MLP}\left(\gamma_2 \odot \text{RMSNorm}(X_{\text{out}}) + \beta_2\right)

All $\odot$ operations are element-wise, with the modulation parameters broadcast across the $D_{\text{pix}}$ channel dimension.

What this enables. Because $\Phi$ is a learned mapping from semantic features to per-pixel modulation parameters, the model can learn spatial modulation patterns. For instance, the semantic token for a patch containing a "cat's eye" might produce higher $\alpha_1$ gate values for pixels at the center of the patch (where fine iris detail matters) and lower values for pixels at the edge (where only fur texture is needed). The semantic token for a "sky" patch might produce uniformly low $\alpha$ values across all pixels, since sky regions need less refinement. These patterns emerge from training and are not hand-designed.

Computational cost. The projection $\Phi$ maps from $D$ (1152 for XL) to $p^2 \cdot 6 D_{\text{pix}}$ (with $p=16$ and $D_{\text{pix}}=16$, this is $256 \cdot 96 = 24576$). The weight matrix has $1152 \times 24576 \approx 28.3$ million parameters — a substantial addition, but the paper argues it is justified by the quality improvement. The appendix (Section G.2) reveals that the authors experimented with lower-parameter alternatives (factorized spatial modulation using a DCT basis, additive conditioning, cross-attention conditioning) and found they did not match the quality of the full pixel-wise AdaLN.

Pixel Token Compaction

Pixel token compaction is the mechanism that makes global attention over all pixels computationally feasible. Without it, attention over $H \times W$ pixel tokens would require $O((HW)^2)$ memory and compute, which is infeasible for all but the smallest images. With $H=W=256$, a naive self-attention would process a 65,536 × 65,536 attention matrix — roughly 4.3 billion entries per attention head, consuming ~17 GB in float32 for a single head.

The core idea. Rather than applying attention directly to the pixel tokens, the PiT block temporarily compresses all $p^2$ pixel tokens within each patch into a single compact token, runs efficient self-attention across the $L = (H/p) \times (W/p)$ compact tokens (where $L$ is small — 256 tokens for a 256² image with $p=16$), and then expands the attended representation back to per-pixel resolution.

Compression operator. The $p^2$ pixel tokens within a patch, each of dimension $D_{\text{pix}}$, are compressed into a single token of dimension $D$:

C:Rp2×Dpix→RD\mathcal{C}: \mathbb{R}^{p^2 \times D_{\text{pix}}} \to \mathbb{R}^{D}

This is implemented as a learned linear map that jointly mixes the spatial and channel dimensions. Specifically, a weight matrix $W_{\mathcal{C}} \in \mathbb{R}^{(p^2 \cdot D_{\text{pix}}) \times D}$ flattens the $p^2 \times D_{\text{pix}}$ input into a vector of length $p^2 \cdot D_{\text{pix}}$ and projects it to $D$ dimensions. With $p=16$, $D_{\text{pix}}=16$, and $D=1152$, this means $16^2 \cdot 16 = 4096$ values are compressed to 1152 dimensions — a compression ratio of about 3.6×.

Global attention. The $L$ compact tokens (one per patch, each of dimension $D$) form a sequence of length $L$. Standard multi-head self-attention is applied to this sequence:

CompactOut=MHSA(CompactIn;RoPE)\text{CompactOut} = \text{MHSA}(\text{CompactIn}; \text{RoPE})

where RoPE provides 2D positional encoding for the compact tokens. This attention has $O(L^2 D)$ complexity rather than $O((HW)^2 D_{\text{pix}})$ — for $H=W=256$ and $p=16$, this is a $(256^2) / (256)^2 = (65536/256) = 256$-fold reduction in the number of attention entries.

Expansion operator. After attention, the compact tokens (now containing global context from all patches) are expanded back to per-pixel resolution:

E:RD→Rp2×Dpix\mathcal{E}: \mathbb{R}^{D} \to \mathbb{R}^{p^2 \times D_{\text{pix}}}

This is the inverse operation: a learned linear map $W_{\mathcal{E}} \in \mathbb{R}^{D \times (p^2 \cdot D_{\text{pix}})}$ that projects each $D$-dimensional compact token to $p^2$ pixel tokens of dimension $D_{\text{pix}}$. The expanded pixel tokens retain the attended global context.

Why this is not a "lossy bottleneck." The paper explicitly contrasts pixel token compaction with the VAE bottleneck in latent diffusion:

"Unlike the lossy bottleneck in VAEs, this mechanism only compresses the representation momentarily for the attention operation. Crucially, this compaction operates purely to reduce the computational overhead of self-attention; it does not compromise fine-grained details, because high-frequency information is preserved through residual connections and learned expansion layers that effectively bypass the pixel-token bottleneck."

The key insight: the original pixel tokens flow through residual connections around the compress-attend-expand block. Even if the compaction loses some fine-grained information during the attention step, the residuals (which are per-pixel and never compressed) carry that information forward. The expansion layer also learns to recover details that the compaction might have abstracted away. Empirically, Table 6 in the paper shows that removing attention from the pixel pathway (keeping only the AdaLN and FFN) degrades performance (gFID increases from 2.36 to 2.56 at 80 epochs), confirming that compact global attention provides meaningful global alignment that purely local processing cannot. Meanwhile, removing the compaction and running full attention over all pixels causes out-of-memory (OOM) — it is computationally infeasible at the training scale used.

Ablation on compaction rate. The appendix (Section C.3, Figure 11) investigates whether less aggressive compaction (keeping 2× or 4× the number of compact tokens per patch) improves quality. Surprisingly, the most aggressive setting (1 token per patch, a $p^2$-fold compression) achieves the best gFID: at 1M training iterations, 1× compaction reaches 1.94 gFID vs. approximately 1.96 for 2× and 1.98 for 4×. The paper hypothesizes that "a longer, redundant token sequence and a larger attention space can be more challenging to optimize and slower to converge," suggesting that the compact representation acts as a beneficial bottleneck that forces the pixel-level pathway to focus on the most important residual information.

Text-to-Image Extension with MM-DiT Blocks

PixelDiT-T2I extends the architecture to text-conditioned generation by incorporating Multi-Modal DiT (MM-DiT) blocks in the patch-level pathway, following the design of Stable Diffusion 3 (Esser et al., 2024). The pixel-level pathway remains architecturally identical to the class-conditioned model — it receives text-relevant context only through the semantic conditioning tokens $s_{\text{cond}}$.

MM-DiT block structure. Each MM-DiT block processes two token streams — image tokens and text tokens — with separate QKV (Query, Key, Value) projections but joint attention. Concretely, for a block with image tokens $s_{\text{img}} \in \mathbb{R}^{B \times L \times D}$ and text tokens $s_{\text{txt}} \in \mathbb{R}^{B \times L_{\text{txt}} \times D}$:

  1. Both streams are normalized (RMSNorm) and modulated via AdaLN with shared conditioning.
  2. QKV projections are computed for each stream independently: $Q_{\text{img}}, K_{\text{img}}, V_{\text{img}}$ from $s_{\text{img}}$ and $Q_{\text{txt}}, K_{\text{txt}}, V_{\text{txt}}$ from $s_{\text{txt}}$.
  3. The queries, keys, and values are concatenated across streams: $Q = [Q_{\text{img}}; Q_{\text{txt}}]$, $K = [K_{\text{img}}; K_{\text{txt}}]$, $V = [V_{\text{img}}; V_{\text{txt}}]$.
  4. Joint multi-head self-attention is computed over the concatenated sequence, allowing image tokens to attend to text tokens and vice versa.
  5. The image portion of the attention output is extracted and used for the image stream; the text portion is used for the text stream.
  6. Each stream has its own FFN with separate AdaLN modulation.

Text encoding. Text prompts are encoded using a frozen Gemma-2 encoder (Gemma Team, 2024), a 2B-parameter language model. Following Sana (Xie et al., 2025), the paper prepends a concise system prompt to the user prompt before encoding. The resulting text token embeddings (one per text token, each of dimension $D_{\text{txt}}$) are projected to the model's hidden dimension $D$ (1536 for T2I) via a learned linear projection, producing $s_{\text{txt}} \in \mathbb{R}^{B \times L_{\text{txt}} \times D}$.

How text information reaches the pixels. Critically, text tokens are not routed directly to the pixel-level pathway. The pixel-level PiT blocks never see text tokens — they only see pixel tokens and the semantic conditioning tokens $s_{\text{cond}}$. The semantic tokens, however, have been produced by MM-DiT blocks where image and text tokens jointly attended to each other. This means the semantic tokens already encode text-relevant information — the patch at the spatial location corresponding to "a red car" carries a representation that encodes "this is a red car" because the MM-DiT attention allowed the image tokens at that location to attend to the text tokens describing the car. The conditioning $s_{\text{cond}}$ therefore conveys text meaning to the pixel-level pathway implicitly, through semantically-informed modulation parameters.

Why this design works. The paper states: "Empirically, we find the semantic tokens from the patch-level pathway are sufficient to convey textual intent to the pixel updates." This is a significant architectural simplification — it means the pixel-level pathway does not need to be redesigned for text conditioning, and the expensive PiT blocks (which process the most tokens) remain unchanged between the class-conditional and text-conditional models. The pixel-level pathway is agnostic to whether the semantic context came from a class label or a text prompt; it simply learns to refine pixels based on the conditioning signal it receives.

Training recipe for T2I. The T2I model is trained in two stages. First, pretraining at 512² resolution on approximately 26M image-text pairs for 400K iterations, using AdamW with learning rate $1 \times 10^{-4}$, batch size 1024, gradient clipping 0.5, and the shifting strategy from SD3 with shift value $\alpha = 3.0$. Second, finetuning at 1024² resolution for an additional 100K iterations, with a lower learning rate of $2 \times 10^{-5}$, batch size 768, higher shift value $\alpha = 4.0$, and tighter gradient clipping at 0.1. The representation alignment loss (REPA) is used during the 512² stage ($\lambda_{\text{repa}} = 0.5$, applied at the 6th patch-level block) but disabled during the 1024² finetuning stage — the paper does not elaborate on why, but this likely reflects the empirical observation that REPA becomes less critical once the semantic pathway has already learned strong representations from the lower-resolution training.

Representation Alignment (REPA) Auxiliary Loss

The paper incorporates an auxiliary loss proposed by Yu et al. (2025) that encourages the patch-level pathway's intermediate features to align with representations from a frozen, pretrained vision encoder.

What REPA does. During training, the activations at a specific depth in the patch-level pathway are compared against features extracted from the same image by a frozen DINOv2 encoder (Oquab et al., 2023). DINOv2 is a self-supervised vision transformer that produces semantically rich features — it understands object identity, part structure, and scene layout without being trained on class labels. The REPA loss penalizes the distance between the model's internal representations and the DINOv2 features, effectively saying: "your semantic understanding should agree with what this strong pretrained vision model sees."

Where REPA is applied. For the XL model, REPA is applied at the 8th block of the patch-level pathway (out of $N = 26$ total blocks). This is a mid-level feature — early enough that the representations are still forming, but late enough that they have accumulated substantial semantic information. For the T2I model, it is applied at the 6th block (out of $N = 14$).

Loss weighting. The total training objective is:

L=Ldiff+λrepa⋅Lrepa\mathcal{L} = \mathcal{L}_{\text{diff}} + \lambda_{\text{repa}} \cdot \mathcal{L}_{\text{repa}}

where $\lambda_{\text{repa}} = 0.5$ for both the XL and T2I models (during the 512² stage). The relatively high weight (0.5 × the primary diffusion loss) reflects how important this auxiliary signal is for training stability.

Why REPA is essential for pixel-space training. The appendix (Section C.2, Table 11) provides a stark ablation: removing REPA from PixelDiT-XL causes FID to degrade from 2.36 to 6.58 at 80 epochs, and from 1.97 to 4.33 at 160 epochs. Inception Score drops from 282.3 to 166.0 at 80 epochs. The appendix (Section G.3) further reveals: "Without this auxiliary objective, the training became unstable and eventually diverged." This strongly suggests that pixel-space diffusion — operating on raw RGB values with their high variance and complex noise distribution — benefits dramatically from having the semantic pathway anchored to a stable, pretrained feature representation. The REPA loss effectively provides a "curriculum" signal: before the model can generate good images, it must first learn to see them in a way that aligns with DINOv2's understanding. This prevents the patch-level pathway from collapsing into degenerate representations during early training, which would produce meaningless conditioning signals for the pixel-level pathway and cause the entire system to diverge.

Implementation detail. The DINOv2 encoder is frozen — no gradients flow through it. Only the PixelDiT parameters are updated based on the REPA loss. The alignment is computed using a simple MSE between the $L_2$-normalized features from DINOv2 and the $L_2$-normalized features from the patch-level pathway at the designated alignment depth.

Summary of Design Choices and Their Justifications

  • Dual-level architecture: separates semantic reasoning (few tokens, large hidden dim) from pixel refinement (many tokens, small hidden dim), so each pathway can be optimized for its own computational characteristics. This is the foundational design choice that makes pixel-space diffusion practical — prior work forced a single pathway to handle both, inevitably favoring one at the expense of the other.

  • Pixel-wise AdaLN over patch-wise broadcast: generates distinct modulation parameters for every pixel, enabling location-specific refinement guided by the semantic context of each patch. This is justified by the quality gap in Table 5: moving from patch-wise to pixel-wise AdaLN improves gFID from 3.50 to 2.36 at 80 epochs. The cost is additional parameters in the projection $\Phi$, but the architecture can absorb this because the pixel-level pathway operates at low hidden dimension.

  • Pixel token compaction over direct pixel attention: reduces attention sequence length from $H \times W$ to $L = (H/p) \times (W/p)$, a $p^2$-fold reduction. This is not a quality compromise — the ablation in Table 6 shows that removing attention from the pixel pathway degrades quality, while the appendix (Figure 11) shows that less aggressive compaction (more tokens) does not help and may hurt convergence. The compaction effectively forces the model to learn what is worth attending to globally versus what can be handled locally.

  • Velocity prediction over noise or $x_0$ prediction: provides targets of consistent scale across the entire diffusion trajectory, simplifying the regression problem and enabling faster convergence. The appendix (Section G.3) notes that $x_0$-prediction was tried and did not outperform velocity prediction, while also requiring more hyperparameter tuning for the timestep sampling distribution.

  • 8-block depth for REPA alignment (XL) over earlier or later alignment: applies the auxiliary loss at a mid-level feature where representations are semantically meaningful but not yet fully specialized to the generation task. Earlier alignment would anchor too-shallow features; later alignment would provide a weaker training signal because the representations are already close to their final form.

  • Frozen Gemma-2 text encoder over trainable text encoding: uses a strong pretrained language model without fine-tuning, consistent with the trend in latent diffusion (SD3, FLUX). The MM-DiT joint attention mechanism is responsible for learning the image-text alignment, not the text encoder.

  • MM-DiT only in patch-level pathway, not pixel-level: keeps the expensive pixel-level pathway (which processes the most tokens) simple and agnostic to the conditioning modality. Text information reaches the pixels through the semantic conditioning tokens, which have been text-aware since the MM-DiT blocks fused text and image features. This is an architectural bet that semantic modulation alone is sufficient to convey text intent to per-pixel refinement — and the T2I results (0.74 GenEval at 1024²) empirically support it.

4. Key Insights and Innovations

Innovation 1: Pixel Modeling as an Explicitly Separable Problem — Not an Afterthought

The field's default assumption has been that pixel-space diffusion's quality gap relative to latent diffusion stems from the inherent difficulty of modeling raw pixels — higher variance, more complex noise distributions, harder optimization. Prior work accepted this as a fundamental limitation and either (a) moved to latent space to avoid it entirely (Rombach et al., 2022; Peebles & Xie, 2023), (b) attempted to brute-force it with larger models and more compute (Simple Diffusion at 2B parameters, SiD2), or (c) worked around it with cascaded pipelines and multi-stage training (CDM; Ho et al., 2022).

PixelDiT's core conceptual move is to reject the premise that pixel space is inherently harder to model. The paper argues instead that the quality gap exists because pixel modeling — the mechanism of capturing dense, per-pixel interactions and high-frequency details — has been conflated with semantic modeling in every prior architecture. When a single transformer pathway processes tokens that must simultaneously encode "this is a dog" and "this pixel's exact RGB value," the architecture faces an impossible trade-off: use large patches (losing detail) or many small patches (losing efficiency). The diagnosis, stated explicitly in Section 1, is that "the obstacle to practical pixel-space diffusion is the lack of an efficient pixel modeling mechanism that can model both global semantics and per-pixel updates" within a single architecture.

What makes this a genuine conceptual innovation rather than an incremental architectural tweak is the articulation of pixel modeling as a distinct, nameable sub-problem with its own computational requirements. Prior work implicitly assumed that a good generative model would handle "all scales" if given enough capacity — the U-Net's multi-scale design, the ViT's patch hierarchy, and cascaded pipelines all reflect variations on this assumption. PixelDiT challenges it directly: semantic reasoning needs long-range attention over few tokens and benefits from large hidden dimensions; pixel refinement needs dense spatial resolution but can function with small hidden dimensions and primarily local context. These are not just different scales of the same operation — they are qualitatively different computational primitives that should not share the same architectural pathway.

The evidence for this diagnosis is not just the final FID numbers but the ablation trajectory in Table 5. The vanilla DiT/16 baseline (which treats pixel-space diffusion exactly like latent diffusion — patchify at 16×16 and run standard DiT blocks) achieves only 9.84 gFID at 80 epochs. Simply adding the dual-level structure with pixel token compaction (before even introducing pixel-wise AdaLN) drops this to 3.50 — more than a 2.8× improvement. This is the architecture learning to separate "what is this region?" from "what is each pixel's exact value?" The further addition of pixel-wise AdaLN (3.50 → 2.36) is substantial but secondary — the fundamental gain comes from the decomposition itself. In other words, the paper demonstrates that the architecture matters more than the model simply being "big enough" for pixel-space diffusion, which is a direct counterargument to the scaling-alone hypothesis that underlies much of the latent diffusion literature.

Innovation 2: Conditioning as a Spatially-Varying Signal — The Pixel-Wise AdaLN Abstraction

Conditioning mechanisms in transformer-based diffusion models have followed a consistent pattern since the original DiT (Peebles & Xie, 2023): a global conditioning vector (encoding timestep, class, and/or text) is projected into AdaLN parameters (scale, shift, gate) and broadcast identically to all tokens. This design assumes that the conditioning signal — "you are at timestep 0.3, generating class 'golden retriever'" — applies uniformly across the entire image. Spatial differentiation is left entirely to the self-attention mechanism: if different regions need different behavior, the attention weights must learn to route information accordingly.

PixelDiT's pixel-wise AdaLN breaks this assumption. The key insight is that conditioning should be a spatially-varying field, not a global constant — and that the semantic pathway's output already contains the information needed to construct this field. The semantic tokens s_cond encode per-patch information like "this region is fur," "this region is sky," "this region contains an edge." By projecting each semantic token into distinct modulation parameters for every pixel within its corresponding patch, the model can learn location-specific refinement policies: pixels in textured regions get amplified gating (passing more detail through the residual), pixels in smooth regions get attenuated gating (preserving the smoothness learned by the semantic pathway), and pixels near boundaries get specialized shift and scale parameters that sharpen edges.

What distinguishes this from prior multi-scale conditioning approaches (e.g., U-Net skip connections, hierarchical feature modulation in cascaded models) is the granularity and learnability. U-Net skip connections provide spatial information at multiple resolutions, but they are additive — they inject features, not behavior policies. Pixel-wise AdaLN modulates how the network processes each pixel, which is a meta-learning signal: the semantic pathway learns to emit instructions ("refine this pixel aggressively, leave that one alone") rather than content. This is a qualitatively different form of information flow between pathways.

The paper provides concrete evidence that this granularity matters through the Figure 3 schematic comparison. Patch-wise AdaLN (variant B), which repeats the same modulation across all p² pixels within a patch, is implicitly tested against the full pixel-wise version — and the gap between 3.50 gFID (with compaction but before pixel-wise AdaLN, which uses patch-level modulation) and 2.36 gFID (with pixel-wise AdaLN) represents the value of per-pixel conditioning specificity. The appendix (Section G.2) further reveals that several cheaper alternatives — additive conditioning, cross-attention conditioning, factorized DCT modulation — were tried and could not match the full pixel-wise approach. This pattern of "cheaper approximations fail" is characteristic of a mechanism that captures something genuine about the problem structure rather than just adding parameters.

Innovation 3: Lossless Compression for Attention — A VAE Counterargument Embedded in Architecture

The VAE in latent diffusion models serves a single purpose: compress the image into a smaller representation so that the diffusion model's attention can run efficiently. This compression is lossy and permanent — once the VAE encoder discards high-frequency information, the diffusion model cannot recover it, and the VAE decoder cannot reconstruct it from the compressed latent code. The field has largely accepted this as a necessary cost of efficiency.

PixelDiT's pixel token compaction proposes a fundamentally different compression philosophy: compress for attention only, preserve through residuals everywhere else. The p² pixel tokens within each patch are squeezed into a single compact token before self-attention (reducing the sequence length by a factor of p², e.g., 256× for p=16), and then expanded back to per-pixel resolution after attention. But — and this is the critical conceptual distinction from a VAE — the original pixel tokens bypass the compression entirely through residual connections. The attention output (which carries global context but at reduced resolution) is added to the uncompressed pixel tokens (which carry full per-pixel information). The expansion layer E is not reconstructing lost detail from a degraded representation; it is distributing globally-contextualized information back to individual pixels, while the residuals preserve everything that was already there.

This is more than an implementation trick. It is a direct architectural argument against the necessity of latent spaces. The paper is making the claim: you do not need a pretrained autoencoder to make attention efficient. You can achieve comparable efficiency (311 GFLOPs for PixelDiT-XL vs. 238 GFLOPs for DiT-XL/2, per Table 15) while preserving exact pixel-level fidelity, simply by structuring the computation so that compression happens only where it is needed (the attention operation) and nowhere else.

The significance of this innovation extends beyond the FLOPs comparison. It reframes the entire debate about pixel vs. latent diffusion. The standard argument for latent space is: "attention over H×W pixels is O((HW)²), which is intractable, so we must compress." PixelDiT's response is: "attention can be made O(L²) where L = (H/p)² by compressing per-patch, and the compression can be lossless with respect to the final output because residuals carry the high-frequency signal." This is not a minor optimization — it is a proof of concept that the VAE bottleneck is architecturally optional, not computationally required.

The appendix ablation on compaction rate (Section C.3, Figure 11) provides an intriguing further insight: less aggressive compaction (keeping 2× or 4× the number of compact tokens per patch) does not improve quality and may slightly degrade it. The 1× compaction (most aggressive, p²-fold reduction) achieves the best convergence. The paper hypothesizes that this is because "a longer, redundant token sequence and a larger attention space can be more challenging to optimize" — but this also suggests something deeper: the compact representation may act as a beneficial information bottleneck that forces the pixel-level pathway to learn which information is worth communicating globally (through attention) versus handling locally (through the per-pixel FFN and residuals). This is an emergent property of the architecture, not a hand-designed constraint, and it parallels findings in other domains where forced compression improves learned representations.

Innovation 4: Empirical Demonstration That Pixel-Space Diffusion Is Architecturally (Not Fundamentally) Limited

The paper's strongest experimental contribution is a counter-demonstration to the prevailing narrative that pixel-space models are inherently inferior to latent models. For years, the field's implicit assumption has been that the FID gap between pixel and latent diffusion — often 2–4 points at comparable model sizes — reflects the genuine difficulty of modeling raw pixels vs. a smooth latent space. PixelDiT-XL achieves 1.61 gFID on ImageNet 256², surpassing not only all prior pixel models but also many strong latent models (e.g., DiT-XL at 2.27, MaskDiT at 2.28, REPA at 1.42 before extended training). On ImageNet 512², it reaches 1.81 gFID — again exceeding prior pixel models and competitive with latent approaches like REPA (2.08).

What makes this more than a leaderboard result is the controlled nature of the comparison. PixelDiT-XL uses 797M parameters and 311 GFLOPs per forward pass — comparable to DiT-XL (675M + 49M VAE decoder parameters, 238 GFLOPs). It is trained for 320 epochs on ImageNet, which is less training than many latent baselines (DiT-XL: 1400 epochs; SiT-XL: 1400 epochs). It uses the same Rectified Flow objective, the same AdaLN-Zero conditioning, and the same REPA auxiliary loss as recent latent models. The architecture is the only major variable, yet it closes the quality gap. This is strong evidence for the paper's central claim: the architecture, not the data space, was the bottleneck.

The text-to-image results reinforce this at scale. PixelDiT-T2I achieves 0.74 on GenEval and 83.5 on DPG-Bench at 1024² resolution with only 1.3B parameters — competitive with or exceeding much larger latent models (SDXL at 2.6B, FLUX-dev at 12B on GenEval). The throughput measurements are particularly telling: at 512², PixelDiT-T2I reaches 1.07 samples/second on an A100, which is faster than the pixel-space baseline PixNerd (1.04) and comparable to latent models like PixArt-α (1.5), despite denoising in raw pixels. At 1024², it achieves 0.33 samples/second — slower than FLUX-schnell (0.5) but within the same order of magnitude, and substantially faster than Hunyuan-DiT (0.05) or FLUX-dev (0.04). This directly refutes the claim that pixel-space models are hopelessly inefficient at high resolutions.

The image editing result (Figure 1b) provides a different kind of evidence: not that pixel models match latent models, but that they exceed them on a practically important dimension. The VAE-reconstructed backgrounds in SD3 and FLUX show visible text distortion even without any editing — the lossy compression has already damaged the image. PixelDiT's pixel-space operation preserves the background text exactly. This is not captured by FID or GenEval scores (which measure overall distributional similarity, not per-sample fidelity), but it matters enormously for downstream applications. The paper is essentially arguing that the standard evaluation benchmarks have been blind to the VAE's failure mode because they measure aggregate statistics rather than instance-level detail preservation.

The paper's self-identified limitations (Section I and the Conclusion) actually strengthen this innovation by clarifying its boundaries. The authors acknowledge that: (1) training pixel-space models with velocity prediction is prone to loss spikes, particularly for deeper architectures; (2) the 1.3B T2I model sometimes struggles with geometrically and texturally complex objects (hands, intricate architecture); and (3) the training recipe (including REPA, which the appendix shows is essential — removing it causes divergence) is still maturing. These are not weaknesses of the argument; they are precise characterizations of where the remaining challenges lie, and they point toward an engineering roadmap (better stabilization techniques, larger models, more data) rather than a conceptual dead end. The paper's contribution is therefore not "pixel-space diffusion is solved" but "pixel-space diffusion is limited by architecture and recipe, not by fundamental computational or statistical constraints" — and the evidence for this distinction is the combination of strong results, controlled comparisons, and honest failure analysis.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses ImageNet-1K (Russakovsky et al., 2015) at 256 × 256 and 512 × 512 resolutions for class-conditioned generation, following the training setup of DiT (Peebles & Xie, 2023). For text-to-image generation, approximately 26M image-text pairs at 1024² resolution with various aspect ratios are collected and used for training. Evaluation for text-to-image uses GenEval (Ghosh et al., 2024, 533 prompts) and DPG-Bench (Hu et al., 2024, 1,065 prompts). Image editing evaluation in Figure 1b uses the FlowEdit dataset (Kulikov et al., 2025) with 281 source-target editing pairs.

  • Base model(s). The paper instantiates three model sizes — PixelDiT-B (184M parameters), PixelDiT-L (569M), and PixelDiT-XL (797M) — with PixelDiT-XL serving as the default for ImageNet experiments. For text-to-image, PixelDiT-T2I uses 1,311M parameters with a hidden dimension of 1536. All models are trained from scratch except the ImageNet 512 × 512 variant, which is fine-tuned from a 320-epoch 256 × 256 checkpoint. The models are chosen to span a range of scales and to enable comparison with prior DiT-based architectures at comparable parameter counts.

  • Metrics. For class-conditioned ImageNet generation, the paper reports gFID (Fréchet Inception Distance), sFID (spatial FID), Inception Score (IS), Precision, and Recall on 50K samples, following the ADM evaluation protocol (Dhariwal & Nichol, 2021). For text-to-image generation, GenEval (overall score and per-category breakdown) and DPG-Bench (overall and per-category) are used. For image editing, MSE and SSIM are computed on background regions outside the editing bounding box (Table 14). GFLOPs per forward pass and throughput (samples/second on a single A100 GPU, fp16) are reported for efficiency comparisons.

  • Baselines. The paper compares against a comprehensive set of prior work spanning both latent diffusion models and pixel-space models. Latent baselines include DiT-XL/2 (Peebles & Xie, 2023), SiT-XL (Ma et al., 2024), MaskDiT (Zheng et al., 2023), REPA (Yu et al., 2025), LightningDiT (Yao et al., 2025), DDT-XL (Wang et al., 2025), RAE-XL (Zheng et al., 2025), and SVG-XL (Shi et al., 2025). Pixel-space baselines include ADM-U/G (Dhariwal & Nichol, 2021), CDM (Ho et al., 2022), RIN (Jabri et al., 2023), VDM++ (Kingma & Gao, 2024), JetFormer (Tschannen et al., 2025), Simple Diffusion (Hoogeboom et al., 2023), FractalMAR-H (Li et al., 2025), FARMER (Zheng et al., 2025), EPG-XXL/16 (Lei et al., 2025), PixelFlow-XL (Chen et al., 2025), PixNerd-XL (Wang et al., 2025), and JiT-G (Li, 2025). Text-to-image baselines include PixArt-α/Σ (Chen et al., 2024), LUMINA-Next (Zhuo et al., 2024), SDXL (Podell et al., 2023), Playground v2.5 (Li et al., 2024), Hunyuan-DiT (Li et al., 2024), DALL·E 3 (OpenAI, 2023), FLUX-dev/schnell (Black Forest Labs, 2024), and pixel-space models PixelFlow and PixNerd.

  • Generation budget / compute accounting. The primary compute metric is GFLOPs per forward pass measured at 256² input resolution, using the convention that one multiply-add equals two FLOPs (Table 15). Throughput is measured in samples per second on a single NVIDIA A100 GPU with fp16 precision (Table 4). Training cost is reported as training epochs or training iterations for different model sizes. For fair comparison, the paper notes that PixelDiT-XL requires 311 GFLOPs per forward pass at 256², which is only modestly higher than latent DiT variants (e.g., DiT-XL/2 at 238 GFLOPs) and substantially lower than prior pixel models (e.g., PixelFlow-XL/4 at 5818 GFLOPs, JiT-G/16 at 766 GFLOPs). The GFLOPs breakdown across resolutions and patch sizes is provided in Table 16.

  • Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. All evaluations are on standard fixed test sets (50K ImageNet samples, 533 GenEval prompts, 1,065 DPG-Bench prompts). For ImageNet, classifier-free guidance (CFG) scales and intervals are selected by sweeping (Table 9): for the 80-epoch checkpoint, guidance scale 3.25 with interval [0.10, 1.00] is used; for the 320-epoch checkpoint, 2.75 with interval [0.10, 0.90] achieves the best gFID of 1.61. Ablation studies (patch size, depth allocation, compaction rate, REPA removal) are evaluated at consistent CFG settings (scale 3.25, interval [0.10, 1.00]) to isolate the effect of architectural changes.

Main Quantitative Results

Class-Conditioned Generation on ImageNet 256 × 256

Table 1 is the central results table for ImageNet 256 × 256. PixelDiT-XL at 320 epochs achieves gFID 1.61, sFID 4.68, IS 292.7, precision 0.78, and recall 0.64. This is the headline result and represents a significant advance over prior pixel-space models. To contextualize:

  • Versus pixel-space models: PixelDiT-XL outperforms the next-best pixel-space models by meaningful margins. EPG-XXL/16 (concurrent work) achieves gFID 1.81 with 789M parameters; JiT-G achieves 1.82 with 2B parameters; PixNerd-XL achieves 1.93 with 700M parameters; PixelFlow-XL achieves 1.98 with 677M parameters. PixelDiT-XL's 1.61 gFID is approximately a 0.2-point improvement over the closest competitor (EPG-XXL/16), which represents roughly a 12% relative reduction in FID.

  • Versus latent-space models: PixelDiT-XL's 1.61 gFID surpasses several well-known latent models despite operating without any autoencoder compression. DiT-XL/2 achieves 2.27 (at 1400 epochs with 675M parameters) — PixelDiT improves on this by 0.66 gFID with 4.4× fewer training epochs (320 vs. 1400). SiT-XL achieves 2.06 (1400 epochs); MaskDiT achieves 2.28 (1600 epochs). More recent latent models have closed the gap: REPA achieves 1.42 (800 epochs), LightningDiT achieves 1.35 (800 epochs), DDT-XL achieves 1.26 (400 epochs), and RAE-XL achieves 1.13 (800 epochs, 839M parameters). PixelDiT-XL does not surpass the very best latent models at this resolution, but it approaches within approximately 0.5 gFID while being architecturally simpler (no VAE) and using substantially less training (320 vs. 400-800 epochs).

  • Convergence speed: Table 1 also shows PixelDiT-XL at only 80 epochs achieving gFID 2.36 and IS 282.3. This is significant because it outperforms many fully-trained pixel-space models (ADM-U: gFID 4.59 at 400 epochs; JetFormer: gFID 6.64 at 2.8B parameters) and is competitive with some latent models. At 320 epochs, continuing training to 800 epochs (Table 17 in the appendix) further improves gFID to 1.54, sFID to 4.49, and IS to 297.0, showing that PixelDiT continues to benefit from extended training.

  • Recall vs. Precision trade-off: PixelDiT-XL achieves recall of 0.64, which is higher than most pixel-space competitors (PixelFlow-XL: 0.60; PixNerd-XL: 0.60; EPG-XXL/16: 0.61) and competitive with latent models. Precision at 0.78 is within the typical range for diffusion models (0.78–0.83). The relatively high recall and moderate precision suggest that PixelDiT generates diverse samples that cover the data distribution well, rather than over-optimizing for a narrow set of high-confidence modes.

Class-Conditioned Generation on ImageNet 512 × 512

Table 2 reports results at 512 × 512 resolution. PixelDiT achieves gFID 1.81, sFID 5.61, IS 278.6, precision 0.78, and recall 0.67. Key comparisons:

  • Versus pixel-space models: PixelDiT outperforms PixNerd-XL (gFID 2.84), EPG-L/32 (2.35), ADM (3.85), RIN (3.95), VDM++ (2.65), and JiT-H (1.94). The 0.13 gFID improvement over JiT-H is notable because JiT-H uses a 2B-parameter model (vs. PixelDiT's 797M after fine-tuning). PixelDiT's recall of 0.67 is the highest among pixel methods at this resolution.

  • Versus latent-space models: PixelDiT's 1.81 gFID surpasses DiT-XL (3.04), SiT-XL (2.62), MaskDiT (2.50), U-ViT-H (4.05), and REPA (2.08). This is one of the paper's strongest results: a pixel-space model outperforms almost all latent diffusion baselines at 512 × 512 except for RAE-XL (1.13, but with 415M additional VAE-like parameters). The paper explicitly notes: "PixelDiT even surpasses latent-space models such as REPA (gFID 2.08) without any pretrained autoencoder" (Section 4.2).

  • Fine-tuning protocol: The 512 × 512 results are obtained by fine-tuning from a 320-epoch 256 × 256 checkpoint for 530 epochs (Table 18 in the appendix). Fine-tuning for only 40 epochs achieves gFID 2.21, indicating that most of the quality improvement comes from extended high-resolution training. The appendix also shows that using a patch size of p=16 consistently outperforms p=32 at 512 × 512 despite the lower compute cost of larger patches (Table 13): at 120 epochs, p=16 achieves gFID 2.23 vs. 3.78 for p=32.

Text-to-Image Generation

Table 4 presents the text-to-image results. PixelDiT-T2I at 512² resolution achieves GenEval 0.78 and DPG 83.7. At 1024² resolution, it achieves GenEval 0.74 and DPG 83.5.

  • At 512² resolution versus pixel-space models: PixelDiT-T2I outperforms PixelFlow (0.60 GenEval, 77.9 DPG) and PixNerd (0.73 GenEval, 80.9 DPG) by substantial margins — 0.18 and 0.05 GenEval improvements respectively. This is the strongest text-to-image result achieved by a pixel-space model at this resolution.

  • At 512² resolution versus latent models: PixelDiT-T2I's 0.78 GenEval surpasses PixArt-α (0.48), PixArt-Σ (0.52), and approaches the best latent models at this resolution (the table does not list 512² results for FLUX or SD3, as their primary evaluations are at 1024²). DPG of 83.7 is competitive with PixArt-Σ (79.5) and exceeds several latent baselines.

  • At 1024² resolution versus latent models: This is the most demanding comparison. PixelDiT-T2I achieves GenEval 0.74, which surpasses PixArt-Σ (0.54), LUMINA-Next (0.46), SDXL (0.55), Playground v2.5 (0.56), Hunyuan-DiT (0.63), DALL·E 3 (0.67), and FLUX-dev (0.67) and FLUX-schnell (0.71). This is a remarkable result: a 1.3B-parameter pixel-space model outperforms a 12B-parameter latent model (FLUX) on GenEval, a benchmark specifically designed to measure text-image alignment. On DPG-Bench, PixelDiT-T2I's 83.5 is slightly behind FLUX-dev (84.0) and FLUX-schnell (84.8), but matches DALL·E 3 (83.5) and exceeds all other listed models.

  • Category-wise analysis: The GenEval category breakdown (Table 10) reveals PixelDiT-T2I's strengths and weaknesses. At 1024², it achieves perfect "Single Object" (1.00), strong "Counting" (0.95), and strong "Color Attribution" (0.68). It is weaker on "Colors" (0.55 vs. FLUX-dev's 0.79) and "Position" (0.88 vs. FLUX-dev's 0.74 — actually a strength here). On DPG-Bench (Table 12), PixelDiT-T2I scores highest on "Other" (89.6) and shows balanced performance across "Entity" (88.6), "Attribute" (87.8), and "Relation" (91.2).

  • Throughput: At 512², PixelDiT-T2I achieves 1.07 samples/second — faster than PixelFlow (0.05) and comparable to PixNerd (1.04) and PixArt-α (1.5). At 1024², it achieves 0.33 samples/second, which is slower than FLUX-schnell (0.5, a distilled model optimized for speed) but substantially faster than FLUX-dev (0.04) and Hunyuan-DiT (0.05). This throughput is achieved with 25 sampling steps using FlowDPMSolver.

Image Editing Detail Preservation

The background preservation experiment (Table 14, Figure 1b) provides quantitative evidence for the qualitative claim that pixel-space models preserve unedited regions better. On 281 FlowEdit samples, PixelDiT achieves MSE 0.001522 and SSIM 0.8628 on background regions, compared to FLUX (MSE 0.009105, SSIM 0.8254) and SD3 (MSE 0.004349, SSIM 0.8400). PixelDiT's MSE is approximately 6.0× lower than FLUX's and 2.9× lower than SD3's, while SSIM is correspondingly higher. This directly quantifies the VAE reconstruction distortion visible in Figure 1b: the lossy encoding-decoding cycle corrupts fine details even in regions that should remain unchanged, and PixelDiT's pixel-space operation avoids this entirely.

Ablation Studies and Robustness Checks

Impact of core architectural components (Table 5). The incremental ablation starting from a vanilla DiT/16 baseline (pixel-space, patch-only) quantifies each component's contribution. The vanilla baseline achieves gFID 9.84 at 80 epochs. Adding RoPE and RMSNorm (modern DiT improvements) improves to 8.53. Adding the dual-level architecture without pixel token compaction causes out-of-memory, demonstrating that compaction is not an optimization but a necessity for training. Adding compaction resolves the memory issue and dramatically improves gFID to 3.50 — a 2.8× improvement over the baseline and a 2.4× improvement over the modernized DiT. Finally, upgrading from patch-wise AdaLN to pixel-wise AdaLN improves gFID from 3.50 to 2.36 at 80 epochs, and to 1.61 at 320 epochs. The labels A–C in Table 5 correspond to the three AdaLN variants illustrated in Figure 3, making the connection between design and empirical impact explicit.

Pixel token compaction analysis (Table 6). Two ablation variants test the role of compaction and attention in the pixel-level pathway. No Pixel Token Compaction (removing the compress-expand pathway) results in 82,247 GFLOPs and out-of-memory at the training scale used — quantifying why naive per-pixel attention is infeasible. No Pixel-Pathway Attention (removing self-attention in PiT blocks while keeping pixel-wise AdaLN and FFN) reduces GFLOPs to 279 but degrades quality: at 80 epochs, gFID increases from 2.36 to 2.56 (IS drops from 282.3 to 256.9); at 160 epochs, gFID is 2.22 vs. 1.97 for the full model. This demonstrates that global attention in the pixel pathway provides meaningful global alignment beyond what local (FFN-only) processing can achieve, even though the benefit is moderate — the pixel-wise AdaLN and local processing carry most of the refinement burden, but attention provides necessary cross-patch coordination.

Pixel token compaction rate (Figure 11, Appendix C.3). Varying the compaction rate — keeping 1×, 2×, or 4× the number of compact tokens per patch relative to the base (one token per patch) — shows that the most aggressive compression (1×) achieves slightly better gFID: at 1M iterations, approximately 1.94 for 1× vs. ~1.96 for 2× and ~1.98 for 4×. The convergence gap is small but consistent throughout training (visible from 200K iterations onward). The paper hypothesizes that "a longer, redundant token sequence and a larger attention space can be more challenging to optimize and slower to converge," suggesting that the compact representation serves as a beneficial inductive bias. This is a non-obvious finding: one might expect that preserving more information (less compression) would improve quality, but the opposite is observed.

Patch size scaling across model sizes (Figure 5a). The patch size ablation across B, L, and XL models reveals an important interaction with model capacity. For PixelDiT-B, smaller patches dramatically accelerate convergence: at 200K iterations, p=32 achieves gFID ~48.5, p=16 achieves ~15.1, and p=4 achieves ~6.7. B/4 ultimately reaches ~3.4 gFID at 500K iterations. For PixelDiT-L, p=8 improves over p=16 more modestly (from ~2.72 to ~2.15 at 300K iterations). For PixelDiT-XL, the gap between p=8 and p=16 "essentially vanishes" — both converge to gFID near 2.0. This shows that the benefit of smaller patches diminishes as model capacity increases, suggesting that larger models can compensate for coarser patch tokenization through learned representations, but smaller models critically depend on finer spatial resolution. The practical implication: p=16 is near-optimal for XL models and offers the best compute-quality trade-off.

Model size scaling at fixed patch size (Figure 5b). At p=16, scaling from B to L to XL yields consistent improvements: at 200K iterations, gFID improves from 15.1 (B/16) to 4.95 (L/16) to 2.95 (XL/16). At 1M iterations, XL/16 reaches 1.94 gFID vs. approximately 2.1 for L/16. This demonstrates that PixelDiT benefits from standard model scaling, with the XL variant showing both better asymptotic quality and faster convergence to a given quality level.

Depth allocation between pathways (Figure 10, Appendix C.1). Under a fixed total budget of N+M ≈ 30 layers, several configurations are tested: (N=28, M=2), (N=26, M=4), (N=24, M=6), and (N=22, M=8). All dual-level configurations dramatically outperform the single-level baseline (not shown in this figure, but implied by the gFID range). The default (N=26, M=4) achieves the best overall trade-off: it reaches gFID ~2.34 by 300K iterations and ~1.94 at 1M iterations, outperforming both the shallower pixel pathway (N=28, M=2, which converges to ~2.1) and the deeper one (N=22, M=8, which converges similarly but slower in early epochs). This indicates that a moderate depth for the pixel pathway (4 out of 30 layers, or ~13% of total depth) is sufficient — deeper pixel pathways do not harm asymptotic quality but slow initial convergence.

Representation alignment (REPA) ablation (Table 11, Appendix C.2). Removing the REPA auxiliary loss causes a severe degradation: at 80 epochs, gFID increases from 2.36 to 6.58 and IS drops from 282.3 to 165.96. Even at 160 epochs, the no-REPA model reaches only gFID 4.33 and IS 242.4 — far behind the REPA-enabled model at 80 epochs (2.36/282.3). The appendix (Section G.3) further notes that training without REPA "became unstable and eventually diverged." This is perhaps the most dramatic ablation result and demonstrates that REPA is not an optional enhancement but a necessary component for PixelDiT's training stability. The paper's claim that pixel-space diffusion requires a "mature training recipe" is strongly supported by this result: REPA acts as a regularizer that anchors the semantic pathway to stable, semantically meaningful representations, preventing the pixel-space training signal from causing representational collapse.

CFG scale and interval sweep (Table 9, Appendix B.3). The optimal CFG settings shift with training duration. At 80 epochs, stronger guidance (scale 3.25, interval [0.10, 1.00]) is best (gFID 2.36). At 320 epochs, milder guidance (scale 2.75, interval [0.10, 0.90]) achieves gFID 1.61. Reducing the scale to 2.50 or expanding the interval to [0.10, 1.00] increases gFID to 1.69–1.71. This shift from strong to mild guidance with increased training is consistent with the understanding that better-trained models require less aggressive guidance to produce high-quality samples.

Inference solver and step count (Figures 8–9, Appendix B.1–B.2). FlowDPMSolver outperforms Euler and Heun samplers across most metrics at 100 steps (Figure 8), maintaining lower gFID and sFID with competitive IS, precision, and recall. The number of inference steps (Figure 9) shows diminishing returns: at the 1.6M-iteration checkpoint, 25 steps achieve gFID ~2.50, 50 steps achieve ~1.76, and 100 steps achieve ~1.61. The paper uses 100 steps for ImageNet to maximize quality and 25 steps for text-to-image to balance quality with throughput.

Extended training (Tables 17–18, Appendix F). PixelDiT continues to benefit from longer training. At 256², extending from 320 to 800 epochs improves gFID from 1.61 to 1.54 and IS from 292.7 to 297.0. At 512², extending fine-tuning from 40 to 530 epochs improves gFID from 2.21 to 1.81 and recall from 0.65 to 0.67. These results indicate that PixelDiT has not saturated at the reported training durations and that further quality improvements are possible with additional compute.

Failed attempts and negative results (Appendix G). The paper's appendix contains an unusually detailed section on experimental failures, which provides valuable context for understanding which design choices matter. Factorized spatial modulation (generating AdaLN parameters in a low-frequency DCT basis) did not match pixel-wise AdaLN performance, suggesting that per-pixel spatial resolution in modulation is important. Haar wavelet pre-transformation of patches did not improve over direct pixel-space modeling. Multiple alternative conditioning mechanisms (additive, cross-attention, self-attention concatenation, per-pixel additive) were tried; cross-attention and concatenation performed worse than AdaLN, while additive conditioning was only slightly worse but more parameter-efficient. In-context prefix tokens (learnable tokens prepended to the patch sequence) did not improve class-conditional generation. x-prediction (predicting clean images instead of velocity) did not outperform velocity prediction and required additional hyperparameter tuning. An encoder-decoder bottleneck for token compression was more sensitive to architectural choices and less robust than the PiT compaction mechanism. These negative results collectively support the paper's claim that the specific combination of pixel-wise AdaLN and token compaction is the result of systematic empirical search rather than arbitrary choice.

Critical Assessment

Does the paper demonstrate that "efficient pixel modeling" is the key factor enabling practical pixel-space diffusion?

The paper makes a strong circumstantial case but does not provide a direct, controlled experiment isolating "pixel modeling" as the causal factor. The ablation in Table 5 shows that the dual-level architecture with pixel token compaction (variant B) dramatically improves over the single-level DiT baseline (9.84 → 3.50 gFID). This is the closest the paper comes to proving its central claim, and it is convincing evidence that architectural factorization matters significantly. However, there is no experiment that keeps total compute fixed and varies only the "quality of pixel modeling" — for example, by comparing the dual-level design against a single-level design with equivalent FLOPs but different token granularity. The baseline vanilla DiT/16 uses the same patch size but no pixel-level pathway; adding a pixel-level pathway requires additional parameters and compute. The paper's GFLOPs analysis (Table 15) shows that PixelDiT-XL uses 311 GFLOPs vs. 238 for a latent DiT, but does not report the GFLOPs of the vanilla DiT/16 baseline in Table 5, making it unclear how much of the improvement comes from increased compute vs. better architecture.

The convergence speed argument (PixelDiT-XL achieving gFID 2.36 at 80 epochs vs. DiT-XL requiring 1400 epochs for gFID 2.27) is compelling but confounded by the use of modern training techniques (RoPE, RMSNorm, REPA, Rectified Flow, logit-normal sampling) that are known to accelerate DiT training independently of the dual-level design. A fairer comparison would apply these same techniques to a single-level pixel-space DiT and measure the residual benefit of the dual-level architecture.

Does the paper demonstrate that pixel-space models can match or approach latent diffusion quality?

Yes, with important qualifications at 256² and strong evidence at 512². At 256², PixelDiT-XL's 1.61 gFID is excellent for a pixel model but remains behind the best latent models (LightningDiT: 1.35; DDT-XL: 1.26; RAE-XL: 1.13). However, these latent models use 400–800 epochs of training (vs. PixelDiT's 320) and some include additional VAE improvements. PixelDiT's 800-epoch result of 1.54 gFID (Table 17) narrows the gap further. At 512², the case is stronger: PixelDiT's 1.81 gFID surpasses REPA (2.08), the primary latent baseline at this resolution. The paper's claim that the quality gap "can be largely closed" (Conclusion) is supported at 512²; at 256², the gap is narrowed but not fully closed, and the paper appropriately hedges: "These findings highlight effective and efficient pixel modeling as the key to practical pixel-space diffusion."

A missing comparison that would strengthen this claim: evaluating PixelDiT against latent models at the same training compute budget rather than at the same epoch count or same parameter count. PixelDiT-XL processes 256² images directly, while latent models process smaller latent representations (typically 32² or 64² spatial dimension). The per-iteration training cost differs, and a FLOPs-matched training comparison (similar to the FLOPs-matched inference comparison in Table 15) would clarify whether pixel-space training is genuinely more efficient per unit of compute or simply benefits from processing more information per iteration.

Does the text-to-image extension genuinely demonstrate that pixel-space diffusion scales to megapixel generation?

Yes, with clearly documented limitations. The GenEval score of 0.74 at 1024², surpassing FLUX-dev (0.67 at 12B parameters) and matching FLUX-schnell (0.71), is a strong result. The DPG-Bench score of 83.5 is competitive. The throughput of 0.33 samples/second at 1024² is practical for many applications. However, several caveats apply:

  • The training data is only ~26M image-text pairs, which is substantially less than the datasets used by FLUX and SD3 (typically hundreds of millions to billions). The paper acknowledges in the Limitations (Section I) that "limited model capacity and insufficient high-quality training data" cause struggles with complex objects. The strong GenEval results may partially reflect the training data quality rather than an inherent advantage of pixel-space over latent-space modeling — a comparison against a latent model trained on the same 26M pairs would be more informative.

  • The reported results use only 25 sampling steps. FLUX-dev typically requires more steps (the paper does not specify, but FLUX-dev with 25 steps would likely underperform its reported benchmarks). The FlowDPMSolver efficiency is an advantage of the Rectified Flow formulation, not specifically of pixel-space modeling — latent models using Rectified Flow would also benefit from fewer steps.

  • The model is only 1.3B parameters. The paper positions this as demonstrating efficiency, but it also means the model has limited capacity relative to state-of-the-art latent models (FLUX at 12B). The fact that a 1.3B pixel model can approach the quality of much larger latent models is impressive, but it is unclear whether pixel-space models would maintain this advantage as they scale up — or whether the training stability issues noted in Section I (loss spikes, sensitivity to architecture depth) would intensify.

Are the ablation results sufficient to attribute performance gains to specific mechanisms?

Mostly, with one notable gap in the pixel compaction analysis. The component-level ablation in Table 5 cleanly isolates the contributions of the dual-level design, compaction, and pixel-wise AdaLN. The REPA ablation in Table 11 demonstrates that this auxiliary loss is essential. The patch size and depth ablations (Figures 5, 10) provide nuanced guidance on architectural trade-offs. However, there is a significant missing ablation: no experiment directly measures the contribution of the residual connections in bypassing the compaction bottleneck. The paper's conceptual argument — that pixel token compaction differs from a VAE because residuals preserve high-frequency information — is central to its innovation but is never empirically isolated. One could test this by comparing the current design against a variant where the residual connections are removed (forcing the expansion layer to reconstruct all detail from the compact tokens), which would make the compaction behave more like a VAE bottleneck. If performance degrades significantly, it would validate the "lossless compression via residuals" claim. If it degrades only modestly, the compaction itself (even lossy) might be sufficient.

Additionally, the paper does not ablate the pixel hidden dimension D_pix. The value of 16 is used for all model sizes, but there is no sweep showing whether 8 or 32 would be better. The appendix (Section G.4) mentions that values from 4 to 128 were explored and that "a relatively compact (e.g., 16) pixel representation was sufficient when paired with adequate attention dimension," but no quantitative results are reported. Given that D_pix controls the memory footprint of the entire pixel-level pathway, this is a practically important hyperparameter.

Does the paper adequately support its claim about detail preservation in image editing?

Yes, but the evidence is from a single dataset and editing method. The FlowEdit experiment (Figure 1b, Table 14) is well-constructed: it quantifies background preservation using standard metrics (MSE, SSIM) on a public dataset of 281 editing pairs. The 6× lower MSE vs. FLUX is a genuinely large effect size. However, only one editing method (FlowEdit) is tested, and the comparison is against only two latent models (SD3 and FLUX). It would be stronger to show that the detail preservation advantage generalizes across multiple editing paradigms (e.g., SDEdit, prompt-to-prompt, ControlNet-based methods) and across a wider range of latent models. Additionally, the paper attributes the distortion entirely to VAE reconstruction error, but does not control for other differences between PixelDiT and the latent models (e.g., different training data, different text encoders, different diffusion formulations). A latent model with a higher-fidelity VAE (e.g., the improved VAEs from DC-AE) might show less distortion, which would test whether the advantage is specifically about "no VAE" vs. "better VAE."

Are there lurking issues with evaluation metric blind spots?

FID and GenEval measure different aspects of quality, and neither directly measures per-sample detail fidelity. FID measures distribution-level similarity — two models can have similar FIDs while one consistently blurs fine textures and the other preserves them, as long as the overall distribution of Inception features is similar. The paper argues that pixel-space models should preserve details better, but FID cannot validate this claim directly. The editing MSE/SSIM results partially address this, but only for the specific case of background preservation during editing. The paper does not report any metric that directly measures high-frequency detail preservation in unconditional or class-conditional generation (e.g., a frequency-domain analysis, a perceptual metric like LPIPS against reference images, or a user study). This is not a fatal weakness — FID remains the standard metric and PixelDiT excels at it — but it means the paper's motivating claim (that avoiding VAE compression improves detail) is empirically supported only through the editing application, not through the generation results that constitute the bulk of the evaluation.

What experiments would strengthen the paper?

  1. FLOPs-matched training comparison against latent DiTs: Train a latent DiT-XL and PixelDiT-XL with equal total training FLOPs and compare convergence. This would directly test whether pixel-space training is genuinely more efficient or simply benefits from processing more information per iteration.

  2. Same-data text-to-image comparison: Train a latent MM-DiT on the same 26M image-text pairs used for PixelDiT-T2I and compare GenEval/DPG. This would control for dataset quality differences.

  3. Compaction-without-residuals ablation: Remove residual connections around the pixel token compaction to test the "lossless compression" claim.

  4. Frequency-domain analysis of generated images: Compare power spectra or high-frequency content between PixelDiT and latent models to quantify whether pixel-space generation actually preserves more fine detail in unconditional generation.

  5. Scaling study for the pixel pathway: Systematically vary D_pix, M, and the compaction rate to establish scaling relationships for the pixel-level pathway analogous to the patch-size scaling study in Figure 5.

  6. Broader editing evaluation: Test across multiple editing methods (SDEdit, ControlNet-based, prompt-to-prompt) and multiple latent models (including those with improved VAEs) to establish the generality of the detail preservation advantage.

  7. Failure case analysis: The paper shows impressive successes in Figure 1b but does not show cases where PixelDiT fails to preserve details or where the editing result is worse than latent models. A balanced failure analysis would build credibility and help practitioners understand when the pixel-space advantage matters.

Overall assessment

The experimental results are substantial and well-documented. The paper reports results across two resolutions on ImageNet, text-to-image generation at two resolutions, and an editing application — a comprehensive evaluation suite. The ablation studies are thorough and include informative negative results. The paper's central claims are supported to different degrees: the claim that the dual-level architecture with pixel-wise AdaLN and compaction enables efficient pixel-space training is strongly supported by the ablation trajectory and FLOPs analysis; the claim that pixel-space models can approach latent diffusion quality is supported at 512² and partially supported at 256²; the claim that avoiding VAE compression benefits detail-critical applications is supported for the specific case of FlowEdit-based editing. The primary limitations are the lack of FLOPs-matched training comparisons against latent baselines, the absence of direct measurements of detail preservation in generation (as opposed to editing), and the use of a single editing method to validate the VAE-avoidance advantage. These limitations do not undermine the paper's contributions but represent opportunities for more rigorous future comparisons.

6. Limitations and Trade-offs

Training Stability: Pixel-Space Diffusion with Velocity Prediction Is Prone to Loss Spikes

PixelDiT trains directly on raw RGB pixel values using the Rectified Flow velocity prediction objective. The paper explicitly acknowledges that this combination is not benign from an optimization perspective:

"We observe that training pixel-space diffusion models with velocity prediction is prone to loss spikes, particularly for deeper architectures and during long training runs… fully eliminating loss spikes without sacrificing training efficiency remains an open challenge." (Appendix I, Limitations)

The consequence is not merely a nuisance — it is a fundamental training stability issue that constrains how large and how long PixelDiT models can be trained. The paper deploys multiple mitigation strategies: gradient clipping at aggressive values (1.0 initially, reduced to 0.5 later; 0.1 for high-resolution T2I training), the REPA auxiliary loss (which the appendix reveals is essential — training diverges without it), a two-stage learning rate schedule (1e-4 for 160 epochs, then 1e-5), and careful choices around prediction target (x-prediction was tried and did not outperform velocity prediction, per Appendix G.3). Despite all of these, loss spikes remain an unresolved threat. Table 8 shows that the T2I 1024² fine-tuning uses gradient clipping at 0.1 — an extremely aggressive value, 5–10× tighter than typical diffusion training — suggesting that stability at high resolution is especially precarious.

What evidence exists: The paper reports the existence of loss spikes qualitatively in Appendix I but provides no quantitative characterization — no loss curves showing spike frequency or magnitude, no ablation measuring how different clipping thresholds or stabilization techniques affect spike occurrence, and no analysis of whether spikes correlate with specific training phases, data examples, or architectural components. The appendix's "Failed Attempts" section (Section G) suggests that training instability was a persistent theme during development: alternative conditioning mechanisms (cross-attention, self-attention concatenation) "led to worse generation quality," and an encoder-decoder bottleneck was "highly sensitive" and "complex," both symptoms of an optimization landscape that is fragile to architectural choices.

Mitigation status: Partially mitigated, not solved. The paper's response is pragmatic — it identifies working configurations (REPA enabled, appropriate clipping, multi-stage LR) that make training feasible for the reported model scales, but it does not provide a principled understanding of why loss spikes occur or a guaranteed method for preventing them. The authors frame this as future work: "conducting more foundational research into the training dynamics of pixel-space diffusion to better understand and mitigate loss instabilities." For a practitioner, this means that training PixelDiT at larger scales (more parameters, higher resolution, longer schedules) may require extensive trial-and-error tuning of stabilization hyperparameters, and there is no guarantee that the current recipe will transfer. This limitation is especially consequential because one of the paper's central claims — that pixel-space diffusion can match latent diffusion — depends on the ability to scale up training, and instability is the primary obstacle to doing so.

The Headline FID Results Depend on Rapidly Diminishing Returns from Extended Training

PixelDiT-XL achieves its best reported gFID of 1.61 at 320 epochs on ImageNet 256², but the paper also reports a gFID of 1.54 at 800 epochs (Table 17). This improvement of only 0.07 gFID over an additional 480 epochs (2.5× the original training budget) represents severely diminishing returns. To put this in perspective: the first 80 epochs produce gFID 2.36 (improvement of 7.48 over the vanilla DiT baseline at 9.84), the next 240 epochs produce gFID 1.61 (improvement of 0.75), and the next 480 epochs produce gFID 1.54 (improvement of 0.07).

The consequence is that PixelDiT's training efficiency advantage over latent models is concentrated in early training and erodes with extended schedules. The paper emphasizes PixelDiT's fast convergence: 80 epochs to reach gFID 2.36, vs. DiT-XL requiring 1400 epochs to reach gFID 2.27. This is a genuine advantage for budget-constrained training. However, the diminishing returns at longer schedules imply that closing the remaining gap to the best latent models (e.g., LightningDiT at 1.35, DDT-XL at 1.26, RAE-XL at 1.13) would require substantially more training compute, and it is unclear from the available data whether the gap can be fully closed within any realistic budget. The reported 800-epoch result (1.54 gFID) is still 0.28–0.41 gFID behind the best latent models, and the trajectory suggests additional epochs would yield only incremental gains. The paper does not report 800-epoch results at 512², where PixelDiT's advantage over latent models is stronger (1.81 gFID at 530 epochs), so it is unknown whether training at that resolution also experiences similar diminishing returns.

What evidence exists: Table 17 provides the gFID trajectory across three checkpoints (80, 320, 800 epochs). The full convergence curves are not shown — only these three points. The appendix does not report longer training for other model sizes or resolutions, making the 800-epoch result an isolated data point rather than part of a systematic scaling study. The paper also does not compare against latent models trained for equivalent total epochs at 256² (most latent baselines in Table 1 report results at 400–1600 epochs, making direct comparison at matched training duration difficult since they process latent representations rather than pixels — per-iteration cost differs).

Mitigation status: Not addressed. The paper does not discuss diminishing returns, does not propose methods to extend the efficient training regime to later epochs, and does not estimate how much training would be needed to match the best latent models. This is a practical concern for any deployment where the highest possible quality matters: the "cheap to train well, expensive to train to the frontier" dynamic means that PixelDiT is an excellent choice for resource-efficient good-quality generation, but may not be the optimal path to state-of-the-art quality regardless of budget.

Text-to-Image Performance Cannot Be Disentangled from Data Quality and Model Scale

The paper reports strong text-to-image results — GenEval 0.74 at 1024², exceeding FLUX-dev (0.67) and FLUX-schnell (0.71) — but several confounds make it difficult to attribute this performance to pixel-space modeling specifically rather than to training data quality, model architecture choices, or evaluation methodology:

  • Training data is only ~26M image-text pairs, acknowledged as limited (Section I: "insufficient high-quality training data"). The paper does not describe the data source, curation process, or resolution distribution, making it impossible to assess whether the strong GenEval results reflect exceptional data quality (carefully filtered, high aesthetic score, strong text-image alignment) rather than an inherent advantage of pixel-space training.

  • The text encoder is Gemma-2, a 2B-parameter language model that may differ substantially from the T5-based or CLIP-based encoders used by competing models. The MM-DiT architecture also differs from competing architectures (e.g., FLUX uses a different multimodal attention design). Without an ablation that varies the data while holding the architecture constant, or varies the architecture while holding the data constant, the causal contribution of pixel-space modeling to the T2I results cannot be isolated.

  • DPG-Bench scores show PixelDiT-T2I trailing FLUX-dev/schnell (83.5 vs. 84.0/84.8) despite the GenEval advantage. This discrepancy (GenEval favors PixelDiT, DPG favors FLUX) suggests that the benchmarks measure different aspects of quality and that PixelDiT's advantage is not uniform. The category-wise breakdowns (Tables 10 and 12) reveal specific weaknesses: PixelDiT-T2I scores only 0.55 on GenEval "Colors" at 1024² vs. FLUX-dev's 0.79, suggesting that color attribute binding is a particular challenge for the pixel-space model at this scale.

The consequence is that the paper's T2I results establish feasibility (pixel-space models can generate reasonable text-aligned images at megapixel resolution) but do not establish competitiveness in a controlled comparison. A practitioner choosing between PixelDiT-T2I and a latent model cannot determine from the reported results whether PixelDiT-T2I's GenEval advantage would persist given the same training data, the same text encoder, and the same model scale.

What evidence exists: Tables 4, 10, and 12 provide the quantitative comparisons, but no ablation isolates data, encoder, or scale effects. The paper acknowledges the data and scale limitations in Section I, framing them as directions for future work ("scaling up the model capacity, curating larger and higher-quality training data"), which implicitly concedes that the current T2I results are not the final word on pixel-space text-to-image generation.

Mitigation status: Acknowledged as future work, not addressed experimentally. The paper's position is that the T2I results demonstrate "strong potential" and that "end-to-end pixel-space diffusion with dual-level design scales to text-to-image generation" — a feasibility claim rather than a superiority claim. This is appropriate given the confounds, but it means the T2I results should be interpreted as proof-of-concept rather than as evidence that pixel-space models are better than latent models at text-to-image generation.

The Detail Preservation Advantage Is Demonstrated Only for a Single Editing Method

The paper makes a compelling case that avoiding VAE compression benefits detail preservation during image editing, but the empirical evidence is narrow: a single editing method (FlowEdit), a single comparison metric (background MSE/SSIM on 281 samples), and comparisons against only two latent models (SD3 and FLUX). The paper states:

"operating directly in pixel space enables PixelDiT to bypass VAE reconstruction artifacts, leading to significantly better content preservation for image editing tasks" (Section 1)

The consequence is that the generalizability of this advantage is unknown. FlowEdit is an inversion-free editing method that manipulates the diffusion trajectory; it is plausible that other editing methods (SDEdit, which adds noise and denoises; prompt-to-prompt, which manipulates cross-attention maps; ControlNet-based methods, which add spatial conditioning) would interact differently with the VAE bottleneck. A method that injects strong structural guidance might override or mask the VAE distortions that FlowEdit exposes. Similarly, the comparison against SD3 and FLUX does not control for differences in the base model architecture, training data, or text conditioning — other latent models with higher-fidelity VAEs (e.g., DC-AE or the improved VAEs in recent work) might show less severe distortion.

What evidence exists: Figure 1b provides the qualitative evidence, and Table 14 quantifies it (MSE: 0.001522 for PixelDiT vs. 0.009105 for FLUX, a ~6× improvement). This is a genuine effect, but it is demonstrated on one task with two competitors. The paper does not test whether the detail preservation advantage appears in other editing paradigms, other latent models, or other types of fine-detail content (e.g., faces, text, repeating patterns, medical images). The appendix does not include additional editing experiments.

Mitigation status: Not addressed. The paper treats the FlowEdit result as a proof of principle — a demonstration that the VAE bottleneck has observable consequences that pixel-space models avoid — rather than as a comprehensive evaluation of the editing advantage. This is a reasonable scope for a paper primarily about generative modeling, but it means that practitioners interested specifically in image editing applications will need to run their own evaluations on their specific editing methods, content types, and latent-model baselines before adopting PixelDiT for this purpose.

The Representation Alignment Loss Is Essential but Binds the Model to a Frozen External Encoder

The REPA auxiliary loss — which encourages mid-level patch-pathway features to align with frozen DINOv2 encoder features — is empirically essential for PixelDiT's training stability and convergence. Removing REPA increases gFID from 2.36 to 6.58 at 80 epochs and causes training to eventually diverge (Table 11, Appendix G.3). However, this dependency creates several practical constraints:

  • DINOv2 is a specific pretrained model trained on a specific data distribution (LVD-142M, a curated dataset of 142M images). If a practitioner wants to train PixelDiT on a substantially different image domain — medical imaging, satellite imagery, scientific visualizations, non-photorealistic art — DINOv2 features may not provide meaningful semantic representations, and the REPA loss may either fail to stabilize training or bias the model toward DINOv2's visual prior. The paper does not explore whether other feature extractors (e.g., CLIP, MAE, domain-specific encoders) can substitute for DINOv2, or whether the REPA weight and alignment depth would need to be re-tuned for different domains.

  • The alignment depth is hand-tuned (8th block for XL, 6th block for T2I) and the REPA weight is fixed at 0.5. The paper does not ablate these choices, and there is no guarantee that the same settings would be optimal for different model sizes, resolutions, or data domains. The T2I training disables REPA entirely during 1024² fine-tuning (Table 8) without explanation — this suggests that REPA may become less beneficial or even harmful at higher resolutions, but the paper provides no analysis of this trade-off.

  • The frozen encoder represents a persistent architectural dependency, which partially undermines the paper's narrative of pixel-space models being "simpler" and "more elegant" than latent models. Latent models depend on a pretrained VAE; PixelDiT depends on a pretrained DINOv2. The dependencies serve different purposes (compression vs. representational regularization) but both introduce external models into the training pipeline. The paper acknowledges this only implicitly, by reporting the REPA ablation honestly but not discussing the architectural dependency as a limitation.

The consequence is that PixelDiT's training recipe is not self-contained — it requires access to a specific pretrained vision encoder, and the model's convergence behavior is coupled to that encoder's representational properties. For standard natural image generation, this is a manageable dependency (DINOv2 is publicly available and well-understood). For domain-specific applications, it may be a barrier.

What evidence exists: Table 11 quantifies the essential nature of REPA, and Appendix G.3 states that training diverges without it. The paper reports no experiments with alternative encoders, no ablation of the alignment depth or weight, and no analysis of how REPA interacts with different image domains. The T2I 1024² fine-tuning disables REPA (Table 8), but the reasons and consequences are not discussed.

Mitigation status: Partially mitigated by the paper's transparency, but not solved. The paper is honest about REPA's importance and reports the ablation, which allows practitioners to understand the dependency. However, no alternative stabilization techniques are explored, and the dependency on a specific frozen encoder (DINOv2) is not flagged as a limitation that might restrict domain transfer. Future work on self-contained training — perhaps using a jointly trained or momentum-updated target encoder, or replacing REPA with a different regularizer — would address this limitation.

Computational Trade-Off Between Pixel and Latent Models Is Not Fully Characterized

The paper argues that PixelDiT-XL achieves comparable inference cost to latent DiTs (311 GFLOPs for PixelDiT vs. 238 GFLOPs for DiT-XL/2 at 256², per Table 15) while avoiding the VAE bottleneck. However, this comparison does not account for how training cost and memory scale with resolution, which is where latent models derive their primary advantage. The VAE in latent diffusion models compresses a 256² image to a 32² latent representation — a 64× reduction in spatial tokens. This means that as resolution increases, the gap between pixel-space and latent-space training cost widens:

  • At 256², PixelDiT processes patch tokens at a 16 × 16 grid (L = 256 tokens) plus pixel-level processing. A latent DiT at 256² input processes a 32 × 32 latent grid (L = 1024 tokens, but in a compressed space). The compute advantage is modest (311 vs. 238 GFLOPs).

  • At 1024², PixelDiT processes 64 × 64 = 4096 patch tokens (plus pixel-level), while a latent DiT processes a 128 × 128 latent grid (16384 tokens). The spatial token count for PixelDiT grows linearly with pixel count (O((H/p)²) vs. O((H/8)²) for latent, where 8 is the VAE compression factor). The paper's GFLOPs breakdown (Table 16) shows 7147 GFLOPs for PixelDiT-XL at 1024² with p=16 — approximately 23× the 256² cost (311 GFLOPs). This is an unavoidable consequence of processing pixels directly, and the paper does not compare against the training cost of a latent DiT at 1024².

The consequence is that PixelDiT's efficiency advantage over latent models narrows at higher resolutions, and may reverse at the resolutions (2048²+) where the VAE compression provides the greatest benefit. The paper's throughput measurements (Table 4) show PixelDiT-T2I at 0.33 samples/second at 1024² — slower than FLUX-schnell (0.5) despite PixelDiT being a much smaller model (1.3B vs. 12B parameters). This suggests that the per-pixel processing cost is a significant drag on throughput that latent models avoid entirely.

What evidence exists: Table 16 provides GFLOPs across resolutions and patch sizes, showing the quadratic scaling with resolution. Table 15 compares PixelDiT against latent models at 256² only. The paper does not report the GFLOPs or training cost of a comparable latent model at 512² or 1024², so the efficiency comparison at higher resolutions is incomplete. The paper acknowledges that compute scales roughly quadratically with resolution (Table 16 caption), but does not discuss how this compares to latent model scaling.

Mitigation status: Partially addressed through throughput measurements, but training cost at scale is not analyzed. The inference throughput numbers (Table 4) provide a practical comparison at deployment time, showing that PixelDiT-T2I is within a factor of ~1.5× of FLUX-schnell at 1024². However, training cost — which is the dominant concern for practitioners training from scratch — is only characterized at 256² (Table 15) and through training epoch counts. The paper's claim that pixel-space diffusion is "practical" is supported for inference at the reported resolutions, but the training cost at megapixel scales and beyond is not fully characterized, and the comparison against latent model training cost is absent.

7. Implications and Future Directions

How This Work Changes the Landscape

PixelDiT does not introduce a new generative modeling paradigm — it operates within the established Rectified Flow + DiT framework — but it reframes what the field should consider a solved problem vs. an open architectural challenge. The conceptual shift is this: the quality gap between pixel-space and latent diffusion models has been widely attributed to the inherent difficulty of modeling raw pixels (higher variance, more complex noise distributions, harder optimization landscape). PixelDiT argues — and demonstrates empirically — that this attribution is wrong, or at least severely incomplete. The bottleneck was not pixel space itself, but the absence of an architecture that explicitly separates semantic reasoning from per-pixel detail refinement.

This reframing matters because it changes the default assumption researchers should make when designing future generative models. Before PixelDiT, the field's implicit flowchart was: "Want good image quality at scale → use a VAE." The VAE was treated as a de facto architectural requirement for transformer-based diffusion, not a design choice to be evaluated against alternatives. PixelDiT's results — 1.61 gFID at 256², 1.81 gFID at 512², GenEval 0.74 at 1024², all without any autoencoder — demonstrate that the VAE is architecturally optional, not computationally required, and that the efficiency argument for latent space can be matched by structuring computation differently (311 GFLOPs for PixelDiT-XL vs. 238 GFLOPs for DiT-XL/2, a ~1.3× factor rather than the orders-of-magnitude gap that motivates latent models in the first place).

The practical significance of this reframing is that the field can now debate the VAE as an engineering trade-off rather than an architectural necessity. For applications where detail preservation matters (image editing, inpainting, medical imaging, scientific visualization), the VAE is no longer the only way to achieve efficient attention — pixel token compaction provides an alternative compression mechanism that, crucially, is lossless with respect to the output because residuals bypass the compression. For applications where compression enables higher resolution or larger models within a fixed budget, latent models retain their advantage, but the trade-off is now quantitative ("how much quality do you lose from the VAE bottleneck vs. how much compute do you save?") rather than qualitative ("you must use a VAE to train at this scale").

The paper also resolves a tension that has existed since the original DiT paper (Peebles & Xie, 2023) and the subsequent shift to latent space: the observation that DiTs converge faster and scale better than U-Nets, but that the best DiTs use latent representations. PixelDiT demonstrates that the convergence speed and scaling behavior of DiTs can be realized directly in pixel space, provided the architecture is structured to handle the two qualitatively different tasks that pixel-space generation requires (semantic layout and texture refinement). This explains why naive pixel-space DiTs (the "Vanilla DiT/16" baseline in Table 5, which achieves only 9.84 gFID at 80 epochs) failed — not because pixel space is intrinsically hard, but because a single-pathway DiT forces semantic and textural information into the same token representation, creating an impossible optimization problem. The dual-level design solves this by factorization, which is a general principle (not specific to images) that could apply to other high-dimensional generation tasks where multi-scale structure exists.

Two research directions become more attractive as a result of this work, and two become less so:

More attractive:

  • Architectural alternatives to VAEs for efficient attention in high-dimensional generation. PixelDiT's pixel token compaction is one specific mechanism — compressing within-patch, attending across patches, expanding with residuals. But the general principle (compress only for the attention operation, preserve high-frequency information through residuals everywhere else) could be instantiated in many ways: learned clustering instead of fixed grid patches, dynamic routing based on content, progressive compression at multiple scales. The paper opens a design space that was previously foreclosed by the assumption that a VAE was necessary.

  • Pixel-space models for detail-critical applications. The FlowEdit experiment (Table 14: ~6× lower MSE than FLUX on background preservation) provides concrete evidence that the VAE bottleneck has observable consequences for downstream tasks. This should motivate systematic investigation of where VAE distortions matter: medical image reconstruction, document processing, fine-art reproduction, satellite imagery, and any domain where pixel-level fidelity in unedited regions is a requirement.

Less attractive:

  • Developing ever-more-sophisticated VAEs as a primary research direction for improving diffusion model quality. The paper demonstrates that the VAE-free path can achieve results competitive with the best VAE-equipped latent models. While improving VAEs remains valuable (for compression, for enabling larger models at fixed compute, for domains where a suitable autoencoder already exists), PixelDiT weakens the case that VAE research is the critical path to better image generation. The bottleneck is not the compression quality — it's the architectural handling of multi-scale information, which can be addressed within a single end-to-end model.

  • Cascaded and multi-stage pipelines that introduce separate models for different resolutions. The dual-level design integrates semantic and detail processing into a single model trained end-to-end, avoiding the error accumulation and engineering complexity of cascaded approaches. While cascaded models may still be useful for other reasons (e.g., enabling extremely high resolutions beyond what a single model's attention can handle), PixelDiT demonstrates that multi-resolution processing can be collapsed into a single architecture for at least up to 1024² resolution, reducing the motivation for multi-model pipelines.

Follow-Up Research This Work Enables

Scaling behavior of pixel-space vs. latent-space DiTs under FLOPs-matched training budgets. The paper compares inference GFLOPs (Table 15) but does not compare training cost. A critical open question: if you train a latent DiT and a PixelDiT with the same total training FLOPs (accounting for the fact that PixelDiT processes larger tensors per iteration but may converge in fewer iterations), which achieves better quality? The paper's convergence speed data (2.36 gFID at 80 epochs vs. DiT-XL's 2.27 at 1400 epochs) hints that PixelDiT is more sample-efficient, but the per-iteration cost is higher. A controlled study would train PixelDiT-XL and a latent DiT (e.g., DiT-XL/2 with a standard VAE) on ImageNet 256² with matched total FLOPs, reporting gFID as a function of cumulative petaFLOP-days. This would directly test the paper's implicit claim that pixel-space training is not just architecturally feasible but compute-efficient relative to latent training. The study should also vary the VAE compression factor to map out the trade-off curve: at what compression ratio does the latent model's efficiency advantage outweigh the VAE's quality degradation?

Does the dual-level design transfer to other high-dimensional generation domains? PixelDiT's core architectural insight — separate semantic and detail pathways with efficient per-pixel conditioning — is not specific to natural images. Direct candidates for transfer include: (a) video generation, where the semantic pathway could process a low-resolution spatiotemporal grid while pixel pathways refine per-frame details, potentially avoiding the need for a VAE that compresses both spatial and temporal dimensions; (b) 3D shape and radiance field generation, where the semantic pathway handles global structure (coarse geometry, part articulation) and a pixel-level-equivalent pathway refines local surface detail or view-dependent appearance; (c) audio spectrogram generation, where the semantic pathway captures long-range temporal and harmonic structure while the detail pathway refines fine-grained frequency content. A strong follow-up would implement the dual-level design for one of these domains, train on a standard benchmark (e.g., UCF-101 for video, ShapeNet for 3D), and compare against both latent and pixel-space baselines, reporting not just quality metrics but also the domain-specific detail preservation (e.g., motion sharpness for video, surface normal consistency for 3D).

Loss spike characterization and stabilization for pixel-space diffusion training. The paper acknowledges that training pixel-space models with velocity prediction is "prone to loss spikes" (Appendix I) but provides no quantitative characterization. A systematic study would: (a) log loss trajectories at per-iteration granularity for PixelDiT-XL across the full 320-epoch ImageNet training run, identifying spike frequency, magnitude distribution, and correlation with training phase (early vs. late), data examples (are certain ImageNet classes more spike-prone?), and architectural components (do spikes originate in the patch-level or pixel-level pathway?); (b) test stabilization interventions beyond gradient clipping, including spectral normalization, activation checkpointing strategies, learning rate warmup schedules, and alternative prediction targets (the paper mentions x-prediction was tried but did not outperform velocity prediction — a detailed comparison would be valuable); (c) measure whether loss spikes correlate with degradation in generated sample quality (do checkpoints saved immediately after a spike produce worse images?) or whether the model recovers. This would transform the qualitative observation of instability into actionable guidance for practitioners scaling PixelDiT to larger models and higher resolutions.

Can the REPA dependency be removed or replaced? REPA alignment to frozen DINOv2 features is empirically essential (Table 11: removing it increases gFID from 2.36 to 6.58 and causes eventual divergence), but it binds the model to a specific external encoder. A valuable follow-up would investigate whether REPA can be replaced with: (a) a self-supervised objective computed entirely within the model (e.g., a momentum encoder tracking the patch-level pathway's own features, similar to BYOL or MoCo); (b) a different frozen encoder (e.g., CLIP, MAE, or a domain-specific encoder for non-natural images) and whether the optimal alignment depth and weight transfer; (c) a simpler regularization technique (e.g., feature variance regularization, spectral contrastive loss) that prevents representational collapse without requiring an external encoder. The key metric would be training stability (does the alternative prevent divergence?) and final gFID (does it match or approach the 1.61 achieved with DINOv2 REPA?). The T2I result that REPA is disabled during 1024² fine-tuning (Table 8) is unexplained and suggests that REPA's importance may diminish at higher resolutions or with strong text conditioning — systematically varying resolution and conditioning strength while measuring REPA's marginal benefit would clarify when and why it matters.

Controlled comparison of editing detail preservation across methods and latent model VAEs. The FlowEdit experiment (Figure 1b, Table 14) demonstrates that PixelDiT preserves background details better than SD3 and FLUX, but the comparison is narrow. A comprehensive study would: (a) test PixelDiT against latent models with higher-fidelity VAEs (e.g., DC-AE, the improved VAEs from the reconstruction-generation dilemma literature cited in Section 2.1) to determine whether the detail preservation advantage is about "no VAE" vs. "any VAE" or about "no VAE" vs. "the specific VAEs used by SD3 and FLUX"; (b) test across multiple editing paradigms — SDEdit (add noise, denoise with new prompt), prompt-to-prompt (cross-attention manipulation), and inversion-based methods beyond FlowEdit — to determine whether the advantage is specific to FlowEdit's trajectory manipulation or general to VAE-free operation; (c) quantify detail preservation in generated (not edited) images by measuring high-frequency content via power spectrum analysis, comparing PixelDiT and latent models on texture-rich ImageNet classes (e.g., 970: alp, 291: lion, 263: Pembroke Welsh corgi). This would address the blind spot that FID cannot measure per-sample detail fidelity directly.

Scaling laws for the pixel-level pathway. The paper provides a scaling study for patch size (Figure 5) and depth allocation (Figure 10) but not for the pixel-level pathway's key hyperparameters: the per-pixel hidden dimension D_pix, the number of PiT blocks M, and the compaction rate. The appendix mentions that D_pix values from 4 to 128 were explored and that "a relatively compact (e.g., 16) pixel representation was sufficient," but no quantitative scaling curves are reported. A systematic study would train PixelDiT-B (to manage compute) on ImageNet 256² while independently varying D_pix (4, 8, 16, 32, 64), M (1, 2, 4, 8), and the compaction rate (1×, 2×, 4×, 8×), measuring gFID at fixed training iterations and total GFLOPs per forward pass. The goal would be to establish whether there are power-law relationships between these hyperparameters and quality (analogous to the model-size scaling in Figure 5b) and to identify the compute-optimal configuration for a given parameter budget. This would transform the pixel-level pathway design from an empirically-tuned choice to a principled scaling decision.

Practical Applications and Downstream Use Cases

Training-free image editing with pixel-perfect background preservation. The FlowEdit results (Figure 1b, Table 14) directly enable a deployment scenario where users perform localized edits (e.g., "change the bicycle to a motorcycle") and require that unedited regions — especially fine text, logos, textures, or faces — remain pixel-identical to the source image. PixelDiT's MSE of 0.001522 on background regions (~6× lower than FLUX's 0.009105) translates to visibly cleaner edits where small details survive the editing process intact. This matters for professional creative workflows (advertising, product photography, graphic design) where artifacts in supposedly-unchanged regions are unacceptable. The advantage is not incremental — the VAE reconstruction artifacts shown in Figure 1b (scene text smeared beyond legibility even before editing) are severe enough to make latent models unsuitable for detail-critical editing tasks, and PixelDiT presents a drop-in alternative that operates within the same FlowEdit framework.

Efficient high-resolution image generation without autoencoder overhead. The throughput numbers in Table 4 — 0.33 samples/second at 1024² on a single A100 with 25 sampling steps — make PixelDiT-T2I viable for batch generation pipelines where latent model throughput would require multiple GPUs or longer wait times. For a production system generating, say, 100K images at 1024², PixelDiT-T2I at 0.33 samples/second requires approximately 84 GPU-hours on a single A100. FLUX-dev at 0.04 samples/second would require approximately 694 GPU-hours — an 8.3× difference, though this comparison does not account for quality differences (FLUX-dev scores 0.67 on GenEval vs. PixelDiT-T2I's 0.74, so the cheaper model is actually better on text-alignment while being much faster). This matters for applications like synthetic data generation, content creation platforms, and on-demand image services where inference cost dominates the total cost of ownership.

Domain-specific pixel-space training where no suitable pretrained autoencoder exists. The paper's "no autoencoder needed" property is especially valuable for image domains where a high-quality VAE is not available off-the-shelf. Medical imaging (CT, MRI, histopathology), satellite and aerial imagery, scientific visualization (microscopy, spectroscopy), and specialized industrial imaging (semiconductor inspection, materials analysis) all have data distributions substantially different from the natural images that standard VAEs (SD, FLUX) are trained on. Using a mismatched VAE introduces domain-specific artifacts that are poorly understood. PixelDiT allows practitioners to train end-to-end in pixel space on their domain data, using the same architecture and training recipe, without needing to first train or adapt a VAE. The only external dependency is REPA with DINOv2 — which may itself need adaptation for non-natural images (a follow-up research direction identified above). The training cost (320 epochs on ImageNet 256² for 1.61 gFID) provides a reference point for estimating the budget needed for domain-specific training.