BitDance: Scaling Autoregressive Generative Models with Binary Tokens

ArXiv: 2602.14041

🎯 Pitch

By abandoning traditional codebook indices and instead modeling images as sequences of binary tokens with up to 2^256 states, BitDance achieves the best FID ever reported for an autoregressive image generator. It then sidesteps the intractable classification problem this huge vocabulary creates by using continuous-space diffusion to sample these binary tokens, enabling a 30x speedup over prior AR models at 1024x1024 resolution while surpassing state-of-the-art parallel models with 5.4x fewer parameters.

1. Executive Summary

BitDance introduces a scalable autoregressive image generation framework that predicts binary visual tokens with an unprecedentedly large vocabulary of $2^{256}$ states, achieving reconstruction fidelity surpassing continuous VAEs on ImageNet $256 \times 256$ (PSNR 25.29 vs. 24.81 for DC-AE at $32\times$ downsampling). To sample from this enormous discrete space—where conventional classification heads become computationally intractable—the paper proposes a binary diffusion head (a continuous-space diffusion model that predicts binary tokens as hypercube vertices and applies a hard sign binarization at inference) alongside next-patch diffusion (a parallel decoding method that jointly models $p^2$ tokens within a local patch using block-wise causal attention and a multi-token diffusion objective). BitDance achieves an FID of 1.24 on class-conditional ImageNet generation—the best among all autoregressive models—and with next-patch diffusion surpasses the 1.4B-parameter state-of-the-art parallel AR model RandAR-XXL by 0.5 FID while using $5.4\times$ fewer parameters (260M) and achieving $8.7\times$ faster throughput (90.26 vs. 10.39 images/second), establishing that binary tokens enable efficient high-fidelity autoregressive generation only when the sampling head models the joint distribution of parallel tokens rather than assuming independence.

2. Context and Motivation

The Core Problem: Discrete Visual Tokens That Are Both Expressive and Practical

The fundamental challenge BitDance tackles is the tension between expression and tractability in discrete visual tokenization for autoregressive image generation. Autoregressive (AR) models have become the dominant paradigm in language modeling because next-token prediction provides a clean, scalable training objective and produces models with strong generalization. Extending this success to visual generation requires converting continuous images into sequences of discrete tokens—a process called tokenization—but existing tokenizers face a persistent trade-off:

• Discrete tokenizers (typically Vector Quantization or VQ-based) produce well-regularized token spaces that prevent error accumulation during long-sequence generation. However, their vocabulary sizes are limited by training instability (codebook collapse) and computational constraints, leading to degraded reconstruction quality—the images reconstructed from these tokens lose fine-grained details compared to the originals.
• Continuous tokenizers (VAE-based) achieve excellent reconstruction fidelity because their latent spaces are unconstrained. But this very lack of constraint makes them poorly suited for autoregressive generation: during the sequential sampling process, small prediction errors at each step compound over long sequences (sometimes called "error accumulation" or "representation drift"), producing increasingly distorted outputs as generation proceeds.

This gap is not merely academic. High-resolution image generation—where sequence lengths grow quadratically with resolution—amplifies both problems. A $1024 \times 1024$ image encoded with a $16\times$ downsampling tokenizer yields a $64 \times 64 = 4096$-token sequence. Any degradation in per-token quality or any accumulation of prediction error across these 4096 steps manifests as visible artifacts, semantic inconsistencies, or global structural failures in the final image. As the paper demonstrates (Table 11), prior AR models at this resolution require hundreds of seconds to generate a single image (NextStep-1: 402 seconds), making them impractical for interactive applications.

The paper identifies a second, related problem that has received less attention: parallel decoding in AR models is fundamentally bottlenecked by the sampling mechanism, not just the sequence ordering. Recent work (PAR, RandAR, NAR) has shown that predicting multiple tokens simultaneously can dramatically accelerate generation. However, these methods still rely on classification heads that sample tokens independently, effectively assuming the tokens within a parallel group are conditionally independent given the prefix. This assumption is demonstrably false for spatially adjacent image patches, and violating it introduces inter-token incoherence—the generated group lacks the structural consistency that would come from properly modeling their joint distribution.

Why This Problem Matters

The practical stakes are high for several reasons the paper implicitly or explicitly raises:

1. The scaling trajectory of visual AR models depends on solving tokenization. Language models scaled from millions to hundreds of billions of parameters in part because text tokenization was a solved problem—byte-pair encoding provides a stable, expressive vocabulary with little room for improvement. In vision, tokenization remains the primary bottleneck. If discrete tokenizers cannot match the reconstruction quality of continuous ones, then AR visual models will be permanently handicapped relative to diffusion models, which operate directly in continuous VAE latent spaces and currently dominate the state of the art in visual quality. BitDance's central wager is that scaling the vocabulary size of discrete tokenizers into the regime of $2^{256}$ states can close this gap, making discrete AR generation competitive with or superior to diffusion without sacrificing the regularization benefits of discreteness.

2. Inference speed is a deployment-critical constraint. The paper emphasizes (Figure 1, Tables 3 and 11) that token-by-token generation is impractically slow at high resolutions. A $30\times$ speedup over prior AR models for $1024 \times 1024$ generation (from ~400 seconds to ~12 seconds) transforms AR image generation from a research curiosity into a potentially deployable system. This speedup comes from parallel token prediction, but the paper argues that existing parallel methods sacrifice quality for speed because of their independent sampling assumption. Solving the joint distribution modeling problem for parallel tokens would unlock speed without the quality penalty.

3. Unified multimodal models need a shared token space. Several recent works (Emu3, Chameleon, Show-o, Janus) aim to build single transformers that process both text and images using the same autoregressive objective. For these models, the visual tokenizer determines whether images can be represented with sufficient fidelity to support both understanding and generation tasks. A tokenizer that loses fine details during encoding will limit the model's ability to answer visual questions, read text in images, or generate detailed outputs—all capabilities that users increasingly expect from multimodal systems.

Prior Approaches and Where They Fall Short

The paper organizes prior work along two axes: tokenizer design and sampling mechanism. Understanding where each falls short is essential to appreciating BitDance's design choices.

Prior Tokenizer Designs

Vector Quantization (VQ) tokenizers (VQ-VAE, VQGAN, LlamaGen's tokenizer) learn a codebook of $K$ embedding vectors and map each image patch to the nearest codebook entry. The vocabulary size $K$ is typically in the thousands to tens of thousands (LlamaGen uses 16,384; Show-o uses 8,192). This approach has two failure modes:

• Codebook collapse: As $K$ grows, many codebook entries receive zero or near-zero gradient updates because no encoder output maps close to them. The effective vocabulary shrinks, defeating the purpose of scaling $K$. This is why most VQ tokenizers plateau well below $K = 10^5$.
• Quantization error: With small $K$, the discrete approximation is coarse. The tokenizer must compress a high-dimensional continuous signal into one of a few thousand discrete codes, losing significant information. Table 1 quantifies this: SD-VAE (continuous) achieves 23.54 PSNR at $8\times$ downsampling, while LlamaGen (discrete, VQ with $K=16,384$) achieves only 20.65 PSNR at $16\times$ downsampling—a massive reconstruction gap.

Lookup-Free Quantization (LFQ) (MAGVIT-v2, Open-MAGVIT2) addresses codebook collapse by using an implicit, learning-free codebook where tokens are binary vectors—each channel is independently binarized to $\{-1, +1\}$ via the sign function. With $d$ channels, the effective vocabulary is $2^d$, which can scale much larger than VQ codebooks. MAGVIT-v2 demonstrated $2^{18}$ vocabularies and improved reconstruction. However, LFQ introduces a new bottleneck: the entropy loss used to encourage uniform codebook utilization requires computing similarities between encoded latents and all $2^d$ codebook entries. As $d$ grows, this computation becomes exponentially expensive in memory, capping practical vocabulary sizes. MAGVIT-v2 used $d=18$; the paper notes that further scaling via naive LFQ is infeasible.

Binary quantization variants (BSQ, WeTok) attempt to resolve LFQ's memory bottleneck through independence assumptions or grouping strategies. BSQ assumes bit-wise independence for entropy computation, while WeTok partitions channels into groups and computes entropy per-group. These approaches enable larger $d$, but the reconstruction quality they achieve (WeTok: 20.77 PSNR at $2^{32}$ vocabulary, $32\times$ downsampling; Table 1) still lags behind continuous VAEs. The paper builds directly on WeTok's group-wise entropy computation to scale to $d = 256$ (a $2^{256}$ vocabulary), which is the largest reported for any visual tokenizer.

Continuous VAEs (SD-VAE, DC-AE, MAR's VAE) achieve the best reconstruction quality because they impose no discrete bottleneck. However, when used for AR generation, continuous tokens create a different problem: the model must predict a point in an unbounded continuous space at each step, and small prediction errors drift the latent representation away from the VAE's training distribution. The paper explicitly compares continuous and discrete tokenizers for AR generation in Table 12: MAR's continuous VAE achieves only 3.16 FID vs. 1.79 for BitDance's binary tokenizer when both are used with the same AR backbone. The paper attributes this to "significant error accumulation during generation" with continuous tokens. SphereAR partially addresses this by constraining latents to a hypersphere, but the fundamental issue of unbounded prediction error remains.

Prior Sampling Mechanisms for Large Vocabularies

The paper identifies a sampling bottleneck that emerges specifically when vocabulary sizes become very large, and this is where its technical contribution is most novel. For a $d$-bit binary token, the number of possible states is $2^d$. Three approaches exist:

Standard classification head. Model the categorical distribution over all $2^d$ indices with a softmax layer. The parameter count is $h \times 2^d$ where $h$ is the hidden dimension. For $d=32$ and $h=1024$, this is $\approx 4.4 \times 10^{12}$ parameters—entirely infeasible. The paper notes (Table 13) that this approach causes out-of-memory (OOM) errors even at modest scales. This is why no prior work uses standard classification for large binary vocabularies.

Bit-wise independent classification (Infinity). Factor the joint distribution as a product of $d$ independent Bernoulli distributions: $p(b_1, \ldots, b_d) = \prod_{i=1}^d p(b_i)$. Each bit gets its own binary classifier, reducing parameters to $h \times 2d$. This scales gracefully but makes a strong assumption: that bits are independent given the context. For visual tokens, this assumption is false. Adjacent pixels in an image patch share structure (edges, textures, colors) that creates correlations across bits. The paper demonstrates this empirically in Table 13: the bit-wise classification head achieves only 8.37 FID vs. 1.79 for the binary diffusion head, a catastrophic degradation. The qualitative implication is that independent bit sampling produces tokens that are individually plausible but jointly incoherent—the bits don't "agree" on which visual pattern they collectively represent.

This is a critical insight that the paper develops but doesn't state explicitly: the bit-wise independence assumption is the same mistake that parallel AR models make at the token level. Both assume that components of a joint prediction (bits within a token, or tokens within a patch) can be treated independently, and both suffer quality degradation as a result. BitDance's binary diffusion head solves this at the bit level, while next-patch diffusion extends the solution to the token level.

How BitDance Positions Itself

BitDance does not introduce a fundamentally new type of tokenizer or a new AR architecture. Its contribution is best understood as a synthesis of existing ideas applied at an unprecedented scale, with a novel sampling mechanism that makes that scale usable. The paper positions itself through three design choices, each of which responds to a specific failure mode in prior work:

1. Scale vocabulary entropy to bridge the discrete-continuous reconstruction gap. BitDance adopts LFQ with group-wise entropy computation (from WeTok) but pushes the vocabulary size to $2^{256}$—four orders of magnitude larger than MAGVIT-v2's $2^{18}$ and eight orders of magnitude larger than the largest VQ-based tokenizers. The paper's key empirical finding in this regard (Table 1) is that at $2^{256}$, the discrete tokenizer surpasses continuous VAEs in reconstruction quality (PSNR 25.29 vs. 24.81 for DC-AE at $32\times$ downsampling). This challenges the conventional wisdom that continuous representations are inherently necessary for high-fidelity reconstruction and establishes that sufficiently high entropy discrete tokens can match or exceed continuous ones.

2. Use continuous-space diffusion to sample from the enormous discrete vocabulary. This is the paper's most distinctive technical contribution. Rather than treating binary tokens as $2^d$-way categorical variables (which is intractable) or $d$ independent binary variables (which is inaccurate), BitDance embeds binary tokens as vertices of a $d$-dimensional hypercube and models their conditional distribution using a diffusion process in the continuous space $\mathbb{R}^d$. The key insight—which the paper explains geometrically—is that this is an easier modeling task than standard diffusion on unconstrained VAE latents because the target space is a finite, structured set of points (the $2^d$ hypercube vertices) with uniform magnitude but varying direction. The diffusion model only needs to learn to point toward the correct vertex; it does not need to model an arbitrary continuous distribution. The hard binarization step ($x_1 = \text{sign}(x_1)$ at the end of sampling) provides a clean projection back onto the discrete set, preventing the drift that plagues continuous-token AR models.

This is genuinely novel in the literature. Prior work has used diffusion for continuous token prediction (MAR, xAR, NextStep-1) and for pixel-space generation (DDPM), but BitDance is the first to use diffusion to sample from a discrete target space by operating in a continuous embedding of that space. The paper frames this (Figure 3 caption) as resolving the "scaling friendly / precise sampling" trade-off: the binary diffusion head is scaling-friendly (parameters scale linearly with $d$, not exponentially) and precise (models the joint distribution, not the factorized approximation).

3. Extend joint distribution modeling from bits to patches for parallel decoding. The paper observes that the same joint-vs-independent tension exists at the token level: parallel AR models that sample tokens independently within a group produce incoherent outputs. Next-patch diffusion addresses this by extending the binary diffusion head to predict multiple tokens jointly—the diffusion objective in Equation 7 operates on all $p^2$ tokens in a patch simultaneously, modeling their joint conditional distribution. The block-wise causal attention mask ensures that tokens within a patch can attend to each other during the transformer forward pass (enabling the model to learn intra-patch spatial dependencies), while the autoregressive mask across patches maintains the sequential generation structure.

This positions BitDance as a unified solution to both the bit-level and token-level joint modeling problems. The binary diffusion head handles correlations among bits within a single token; the multi-token extension handles correlations among tokens within a spatial patch. Prior parallel AR methods (PAR, RandAR, NAR, MaskGIT) addressed the token-level parallelization but left the bit-level sampling problem unsolved (since they used standard VQ tokenizers with small vocabularies, where classification heads are tractable). Infinity addressed the bit-level problem with bit-wise classification but did not explore joint token prediction. BitDance is the first to address both simultaneously.

Relationship to the Broader AR Generation Landscape

The paper situates itself relative to several lines of work that are worth distinguishing:

• Against continuous-token AR models (MAR, xAR, NextStep-1, Fluid): BitDance argues that continuous tokens lack sufficient regularization, citing the error accumulation problem that SphereAR partially addressed. The binary tokenizer provides a stronger structural constraint (the hypercube) without sacrificing reconstruction quality—an advance over SphereAR's hyperspherical constraint, which still operates in a continuous space.

• Against next-scale prediction (VAR): VAR generates entire resolution scales at once rather than individual tokens, which also achieves parallelism. BitDance's next-patch diffusion is a different form of parallelism (spatial patches within a scale rather than across scales) and is compatible with VAR-style ordering—the paper does not explore this combination but the architecture would support it.

• Against random-order AR (RandAR, RAR): These methods improve training by exposing the model to multiple factorization orders, but during inference they still sample tokens one at a time. BitDance's parallel prediction reduces the number of sequential steps regardless of order.

• Against masked generative models (MaskGIT, MAGVIT-v2): These models predict all tokens in parallel using iterative refinement, achieving high speed but typically lower quality than sequential AR models. BitDance aims to match or exceed sequential AR quality while approaching the speed of masked models.

The paper's ultimate positioning is that binary tokens + diffusion-based joint sampling + parallel patch prediction constitutes a new design point in the AR generation space—one that preserves the regularization benefits of discreteness, the expressiveness of large vocabularies, and the speed of parallel decoding, without the compromises that prior methods accepted on one or more of these dimensions.

3. Technical Approach

This is primarily a systems and methods paper whose core idea is that scaling the vocabulary of discrete binary visual tokens to an unprecedented $2^{256}$ states—combined with a diffusion-based sampling head that models joint bit distributions and enables parallel multi-token prediction—resolves the fundamental trade-off between reconstruction fidelity and generative tractability that has limited autoregressive image generation.

3.1 Reader Orientation

BitDance is a complete autoregressive image generation system that converts images into sequences of binary tokens, then learns to generate new images by predicting these tokens one patch at a time using a transformer. The central problem it solves is that prior discrete tokenizers either sacrificed reconstruction quality (VQ-based, small vocabularies) or became computationally intractable to sample from (large binary vocabularies), while continuous tokenizers produced high-quality reconstructions but suffered from error accumulation during sequential generation. The "shape" of BitDance's solution is a three-stage pipeline: a high-entropy binary tokenizer that achieves continuous-VAE-level reconstruction, a transformer that processes mixed text and binary-vision tokens autoregressively, and a diffusion-based sampling head that models the joint distribution of all bits within one or more tokens, enabling both precise single-token sampling and fast parallel multi-token prediction.

3.2 Big-Picture Architecture (Diagram in Words)

The BitDance system consists of five major components connected in a training pipeline and an inference pipeline:

1. Binary Visual Tokenizer (Section 3.1): A convolutional encoder-decoder that compresses an input image into a grid of binary latent vectors and reconstructs the image from them. The encoder outputs a $d$-channel continuous latent per spatial position, which is binarized element-wise via the sign function to produce a binary token $x \in \{-1, +1\}^d$. The decoder reconstructs the image from these binary tokens. The tokenizer is frozen after training.
2. Autoregressive Transformer Backbone: A standard decoder-only transformer (similar to GPT or LLaMA) that processes a 1D sequence of tokens. For class-conditional generation, it processes only vision tokens with a block-wise causal attention mask. For text-to-image generation, it processes interleaved text tokens and vision tokens, initialized from a pretrained LLM (Qwen3-14B). It produces a hidden state vector $z \in \mathbb{R}^h$ for each position that needs a vision token prediction.
3. Binary Diffusion Head (Section 3.2): A lightweight diffusion transformer (DiT) that takes the AR transformer's hidden state $z$ as conditioning and generates a binary token $x \in \{-1, +1\}^d$ via an iterative denoising process in continuous space. The head is trained with a flow-matching objective and applies a hard sign binarization at the final denoising step. This is the core technical novelty—it replaces the standard classification head to handle the $2^d$-way categorical distribution efficiently and accurately.
4. Next-Patch Diffusion Mechanism (Section 3.3): An extension of the binary diffusion head that predicts $p^2$ tokens (a $p \times p$ spatial patch) simultaneously. The AR transformer uses block-wise causal attention so tokens within a patch can attend to each other. The diffusion head takes all $p^2$ hidden states and jointly denoises all $p^2$ tokens in parallel, modeling their joint conditional distribution. This provides the inference speedup.
5. Multi-Stage Training Recipe (Section 4.3): For text-to-image, the system goes through pre-training (PT) on large-scale image-text pairs, continued training (CT) with higher-resolution data, supervised fine-tuning (SFT) on high-quality images, and an optional distillation stage to increase the number of tokens predicted in parallel from 16 to 64 per step.

Information flow during training (class-conditional): An image goes through the frozen binary tokenizer → the resulting grid of binary tokens is flattened into a 1D sequence in patch-wise raster-scan order → learnable prefix tokens are prepended to enable parallel prediction of the first patch → the AR transformer processes the sequence with block-wise causal attention, producing hidden states → for each patch of $p^2$ tokens, the binary diffusion head receives the corresponding $p^2$ hidden states as conditioning, adds noise to the ground-truth binary tokens according to a timestep $t$, and predicts the velocity field → the flow-matching loss (Equation 7) is computed between the predicted velocity and the true velocity $v_t = X - \epsilon$.

Information flow during inference (class-conditional): Starting with only prefix tokens and class conditioning → at each AR step, the transformer processes the partial sequence and outputs hidden states for the next patch's positions → the binary diffusion head initializes $p^2$ tokens as Gaussian noise and iteratively denoises them over $N$ steps, conditioned on the hidden states → the denoised tokens are binarized via sign and appended to the sequence → the process repeats until all patches are generated → the complete binary token grid is decoded by the tokenizer's decoder into an image.

Information flow for text-to-image: A text prompt is tokenized into text tokens → resolution tokens $[\text{res\_i}], [\text{res\_j}]$ specify the output dimensions → the transformer processes text and resolution tokens first → then vision tokens are generated autoregressively in patches using next-patch diffusion, with the binary diffusion head conditioned on both text and previously generated vision context → the complete sequence is decoded to an image at the requested resolution.

3.3 Roadmap for the Deep Dive

I will explain the components in the order that matches the system's data flow and reflects the paper's "stack" of innovations:

• First, the binary visual tokenizer (Section 3.1): how it converts images to binary tokens using Lookup-Free Quantization with group-wise entropy computation, why scaling to $d=256$ channels matters, and how the tokenizer is trained. This establishes what the downstream model must predict.
• Second, the binary diffusion head (Section 3.2): the core sampling mechanism. I will explain why standard classification fails for $2^d$ vocabularies, how the diffusion formulation models the joint distribution of bits, the flow-matching objective and its geometric interpretation, and the inference procedure including the critical hard binarization step.
• Third, the next-patch diffusion mechanism (Section 3.3): how the binary diffusion head is extended to predict multiple tokens in parallel. I will cover the patch-wise raster-scan order, the block-wise causal attention mask, the multi-token diffusion objective, the prefix token mechanism, and why this approach avoids the training-inference discrepancy that limits prior parallel AR methods.
• Fourth, the full text-to-image system (Section 4.3 details): how the AR transformer is initialized from an LLM, the token sequence format including resolution tokens and positional encodings, the multi-stage training recipe, and the distillation stage for further acceleration.

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Binary Visual Tokenizer: Scaling Discrete Vocabulary to $2^{256}$

The tokenizer's job is to compress an image $I \in \mathbb{R}^{H \times W \times 3}$ into a compact grid of discrete tokens from which the image can be reconstructed with high fidelity. BitDance uses a convolutional autoencoder architecture where the encoder produces a continuous latent $x \in \mathbb{R}^d$ for each spatial position, and this continuous vector is quantized to a binary vector via the sign function.

Lookup-Free Quantization (LFQ). The fundamental idea behind LFQ is to avoid learning a codebook entirely. Instead, the codebook is implicit and defined mathematically as $\mathcal{C}_{\text{LFQ}} = \{-1, +1\}^d$—the set of all $2^d$ possible $d$-dimensional vectors where each coordinate is either -1 or +1. The quantization operation is simply the element-wise sign function applied to the encoder's output:

$x_q = \text{sign}(x)$

where $x \in \mathbb{R}^d$ is the encoder output at a spatial position, $\text{sign}(\cdot)$ returns +1 for non-negative inputs and -1 for negative inputs (applied independently to each of the $d$ channels), and $x_q \in \{-1, +1\}^d$ is the resulting binary token.

What it computes: For each spatial position in the latent grid, the encoder produces a $d$-dimensional continuous vector representing the visual content of that patch. The sign function thresholds each dimension at zero, mapping the continuous vector to the nearest vertex of a $d$-dimensional hypercube. The straight-through estimator is used during backpropagation (gradient flows through the sign function as if it were the identity), making the encoder fully differentiable.

Why this form: Learning a discrete codebook of $K$ embedding vectors (as in VQ-VAE) requires maintaining $K$ vectors and computing nearest-neighbor lookups, which becomes unstable as $K$ grows (codebook collapse). LFQ eliminates the learned codebook entirely—the $2^d$ possible tokens are defined purely by the sign pattern across $d$ channels. The size of the vocabulary is controlled solely by $d$, with no codebook vectors to maintain or collapse. This makes scaling to extremely large vocabularies (like $2^{256}$) possible in principle, whereas VQ-based approaches cannot exceed $K \approx 10^5$ without severe utilization problems.

Group-wise Entropy Loss. Even with LFQ, there is a subtle computational problem: the entropy loss is needed to prevent all encoder outputs from mapping to the same few binary patterns (a form of collapse where only a tiny fraction of the $2^d$ vocabulary is used). The standard entropy loss from MAGVIT-v2 computes:

$\mathcal{L}_{\text{entropy}} = \mathbb{E}[H(q(x))] - H[\mathbb{E}(q(x))]$

where $H(\cdot)$ denotes entropy, $q(x)$ is the soft assignment distribution over all $2^d$ codebook entries (computed via softmax of similarities between $x$ and every codebook vector), $\mathbb{E}[H(q(x))]$ is the average per-sample entropy (which should be low—each sample should confidently map to one code), and $H[\mathbb{E}(q(x))]$ is the entropy of the average assignment (which should be high—across a batch, all codes should be used roughly equally).

The memory bottleneck: Computing $q(x)$ requires evaluating the dot product (or L2 distance) between the continuous latent $x$ and all $2^d$ codebook vectors. For $d=256$, $2^{256}$ is astronomically larger than any feasible memory allocation, making the standard LFQ entropy loss intractable.

The group-wise solution (adapted from WeTok): Instead of computing entropy over all $2^d$ codes jointly, partition the $d$ channels into $g$ groups of size $d/g$ each. The entropy loss is computed independently within each group, treating each group as a $2^{d/g}$-way categorical variable. The total loss is the sum across groups. This reduces the memory complexity from $\mathcal{O}(2^d)$ to $\mathcal{O}(g \cdot 2^{d/g})$. For example, with $d=256$ and $g=32$ (groups of 8 channels each), the memory cost is $32 \cdot 2^8 = 8192$ entries—entirely manageable. The trade-off is that the entropy loss encourages uniform utilization within each group independently, rather than across the full codebook jointly, which is a weaker constraint but sufficient in practice.

Tokenizer Training Configuration. The tokenizer uses a pure CNN architecture (no attention or transformers in the tokenizer itself) to enable generalization across resolutions. It is trained on DataComp-1B supplemented with high-quality face and text datasets at $256 \times 256$ resolution for 400K steps with batch size 1024. The paper evaluates three configurations:

1. $16\times$ downsampling, $d=32$ (vocabulary $2^{32}$, compression ratio 192)
2. $32\times$ downsampling, $d=128$ (vocabulary $2^{128}$, compression ratio 192)
3. $32\times$ downsampling, $d=256$ (vocabulary $2^{256}$, compression ratio 96)

The compression ratio is computed as the total number of pixels divided by the total number of bits in the latent representation. For continuous VAEs, the paper assumes bfloat16 storage (16 bits per latent dimension) for comparability.

Key Result (Table 1): At $32\times$ downsampling with $d=256$, BitDance's tokenizer achieves PSNR 25.29 and SSIM 0.74 on the ImageNet $256\times 256$ validation set. This surpasses the continuous DC-AE (PSNR 24.81, SSIM 0.69) and DC-AE-SANA (PSNR 24.72, SSIM 0.69) at the same downsampling ratio. This is a pivotal empirical finding: it demonstrates for the first time that a discrete tokenizer can exceed continuous VAE reconstruction quality, closing the discrete-continuous gap that has historically handicapped AR visual models.

Scaling behavior (Figure 6): The paper investigates how vocabulary size interacts with AR model size. For the $16\times$ tokenizer ($2^{32}$ vocabulary), all model sizes converge reasonably. For the $32\times$ tokenizer with $2^{128}$ vocabulary, smaller models (BitDance-B, 24 blocks) begin to struggle, showing degraded FID. For the $32\times$ tokenizer with $2^{256}$ vocabulary, only the largest model (BitDance-H, 40 blocks) effectively leverages the expressive power; smaller models underfit. This reveals a complementary scaling relationship: vocabulary expansion must be matched with transformer capacity expansion to realize the benefits.

3.4.2 Binary Diffusion Head: Sampling from $2^d$ Categories via Continuous Diffusion

This is the paper's most technically novel component. The problem is: given a hidden state vector $z \in \mathbb{R}^h$ from the AR transformer that summarizes all previous context, how do we sample a binary token $x \in \{-1, +1\}^d$ from the conditional distribution $p(x|z)$?

Why Standard Classification Fails. For a $d$-channel binary token, there are $2^d$ possible states. A standard classification head must output a logit for each of these $2^d$ categories, requiring a weight matrix of size $h \times 2^d$. For $h=1024$ and $d=32$, this is $1024 \times 2^{32} \approx 4.4 \times 10^{12}$ parameters—roughly 8.8 TB in bfloat16, far beyond any current hardware. The paper confirms this causes out-of-memory (OOM) errors (Table 13).

Why Bit-wise Independent Classification Fails. The bit-wise approach decomposes the joint probability as a product of marginal probabilities, assuming conditional independence: $p(b_1, \ldots, b_d | z) = \prod_{i=1}^d p(b_i | z)$. Each bit gets its own binary classifier (logit for $b_i = +1$ vs $b_i = -1$), requiring only $h \times 2d$ parameters. This is the approach used by Infinity and is computationally tractable. However, the independence assumption is false for visual tokens. Bits encode correlated structure—edges, textures, spatial frequencies—and sampling bits independently produces tokens that are individually plausible (each bit looks like it could belong to some token) but jointly incoherent (the combination of bits doesn't correspond to any real visual pattern). Table 13 quantifies this failure: bit-wise classification achieves FID 8.37, compared to 1.79 for the binary diffusion head, a nearly $5\times$ degradation.

The Geometric Insight. Rather than treating binary tokens as $2^d$ discrete categories, BitDance treats them as points in continuous space $\mathbb{R}^d$—specifically, as vertices of the $d$-dimensional hypercube $\{-1, +1\}^d$. These vertices have two properties that make them amenable to continuous-space modeling:

1. Uniform magnitude: All vertices have the same L2 norm ($\|x\|_2 = \sqrt{d}$), so the model only needs to learn which direction to point in, not what magnitude to produce.
2. Finite, structured set: The target space is a discrete set of $2^d$ points with known geometry, not an arbitrary continuous distribution. This is a substantially simpler modeling task than standard VAE latent spaces, which are unbounded and unstructured.

The binary diffusion head operates entirely in $\mathbb{R}^d$, using a diffusion (flow-matching) process to transform Gaussian noise into a point near the correct hypercube vertex, then projects that point onto the nearest vertex via the sign function.

Training: Flow-Matching Objective. The binary diffusion head is trained with the Rectified Flow formulation, using $x$-prediction with velocity matching. The training procedure for a single binary token is:

1. Given the ground-truth binary token $x \in \{-1, +1\}^d$ and the conditioning hidden state $z \in \mathbb{R}^h$ from the AR transformer.
2. Sample a timestep $t \sim \mathcal{U}(0, 1)$ and Gaussian noise $\epsilon \sim \mathcal{N}(0, \mathbf{I})$.
3. Construct the noisy token by linear interpolation: $x_t = t \cdot x + (1 - t) \cdot \epsilon$.
4. Feed $x_t$, $t$, and $z$ into the diffusion network $v_\theta$, and compute the loss:

$\mathcal{L}(z, x) = \mathbb{E}_{t, x, \epsilon} \left\| v_\theta(x_t, t, z) - v_t \right\|^2$

where $x_t = tx + (1 - t)\epsilon$ is the noisy token at time $t$, $v_t = x - \epsilon$ is the target velocity (the direction from noise to data), and $v_\theta$ is the velocity-predicting neural network.

Network parameterization: The velocity $v_\theta$ is parameterized via $x$-prediction: the network $f_\theta$ predicts the clean data $x$, and the velocity is computed as $v_\theta(x_t, t, z) = (f_\theta(x_t, t, z) - x_t) / (1 - t)$. This reformulation is stable because the network output $f_\theta$ is always in the same range as the target $x$ (values near $\pm 1$), regardless of $t$.

What the loss computes: At each training step, the network sees a partially noised version of the true binary token ($x_t$ is a blend of the clean token $x$ and pure noise $\epsilon$, with mixing ratio determined by $t$) along with the conditioning $z$. The network must predict the velocity vector $v_t = x - \epsilon$—the difference between the clean token and the noise. The mean squared error between the predicted and true velocity is minimized. When $t$ is small (close to 0), $x_t \approx \epsilon$ is mostly noise, and the task is difficult (the network must hallucinate the token from conditioning alone). When $t$ is large (close to 1), $x_t \approx x$ is mostly clean, and the task is easy (small refinements). The uniform sampling of $t$ ensures the network learns the full trajectory.

Why this loss form: The flow-matching formulation with velocity prediction provides a straight-line path between the noise distribution and the data distribution. This is more efficient than DDPM-style noise prediction (which follows a curved trajectory) and results in faster sampling with fewer steps. The $x$-prediction parameterization is preferred over $\epsilon$-prediction because the target space $\{-1, +1\}^d$ has a known, bounded range—the network's predictions naturally stay near this range, improving stability.

Binary Diffusion Head Architecture. The prediction network $f_\theta$ is implemented as a lightweight DiT (Diffusion Transformer). For class-conditional models, the head width is consistent across model sizes while the number of blocks varies (6, 8, and 12 blocks for BitDance-B, -L, and -H respectively). The conditioning $z$ (the AR transformer's hidden state) is injected via adaptive layer normalization or cross-attention within the DiT blocks. Timestep $t$ is embedded via a sinusoidal encoding and similarly injected.

Inference: Iterative Denoising with Hard Binarization. At inference time, the sampling process is:

1. Initialize $x_0 \sim \mathcal{N}(0, \mathbf{I})$ as pure Gaussian noise.
2. For $n = 0, 1, \ldots, N-1$ with step size $\Delta_t = 1/N$:
   • Compute the velocity prediction $v_\theta(x_t, t, z)$ where $t = n/N$.
   • Update: $x_{t + \Delta_t} = x_t + v_\theta(x_t, t, z) \cdot \Delta_t$.
3. After $N$ steps, apply the hard binarization: $x_1 = \text{sign}(x_1)$.

The Euler integration with uniform steps is the simplest possible ODE solver for the flow ODE $dx/dt = v_\theta(x_t, t, z)$. After integration, $x_1$ should be near a hypercube vertex but may not lie exactly on it due to discretization error and model imperfection. The sign function projects it onto the nearest vertex (each coordinate mapped to the nearest of $\{-1, +1\}$).

Why the hard binarization matters: This final projection step is what makes the binary diffusion head a discrete sampler despite operating in continuous space. It exploits the structural prior that valid tokens lie exactly on hypercube vertices. This prevents the error accumulation that plagues continuous-token AR models—even if the continuous prediction drifts slightly, the sign projection snaps it back to a valid discrete state. The paper contrasts this with prior continuous-token AR models (MAR, xAR, NextStep-1) where no such projection exists, and small errors compound across the generation sequence.

Critical detail—no manual constraints are needed: Figure 9 shows that the binary diffusion head learns the discrete structure automatically from the flow-matching objective, without any explicit discrete loss or constraint during training. At small $t$ (high noise), predictions are uncertain and cluster near 0. As $t$ increases, predictions become progressively more bimodal, concentrating near $-1$ and $+1$. This emergent behavior demonstrates that the flow-matching objective is sufficient to learn the hypercube structure.

Sampling efficiency (Figure 8): The paper ablates the number of diffusion sampling steps $N$ (for next-patch diffusion with $p=4$) and finds that good FID is achieved with as few as 10 steps. This is much faster than typical image diffusion models (which require 50–250 steps) because the target space is highly structured—the model only needs to determine which of the $2^d$ vertices to point to, not to generate an arbitrary continuous vector. The paper attributes this to the discrete nature of binary tokens "simplifying the sampling task relative to continuous tokens."

What the Binary Diffusion Head Is NOT: It is important to distinguish this from prior work that uses diffusion for visual generation. The binary diffusion head does not generate pixels or VAE latents—it generates binary tokens that are then decoded by a frozen tokenizer. It does not replace the tokenizer's decoder; it replaces the classification head that would normally sample the next token in an AR model. This is analogous to how MAR uses a diffusion head to sample continuous VAE latents in a masked generative framework, but applied to binary discrete targets and integrated into a fully autoregressive architecture.

3.4.3 Next-Patch Diffusion: Joint Multi-Token Prediction for Parallel Decoding

The binary diffusion head solves the bit-level joint modeling problem within a single token. Next-patch diffusion extends this to the token level: predicting $p^2$ tokens (a $p \times p$ spatial patch) simultaneously while modeling their joint conditional distribution.

From Next-Token to Next-Patch Prediction. Standard autoregressive generation decomposes the joint probability of a sequence of $N$ tokens as:

$p(x) = \prod_{n=1}^N p(x_n | x_1, x_2, \ldots, x_{n-1})$

where each token $x_n$ is generated one at a time conditioned on all previous tokens. Next-patch prediction partitions the sequence into $M$ disjoint groups (patches) of size $p^2$:

$p(x) = \prod_{m=1}^M p(X_m | X_1, \ldots, X_{m-1})$

where $X_m = \{x_m^1, x_m^2, \ldots, x_m^{p^2}\}$ is the $m$-th patch containing $p^2$ tokens, and $p(X_m | X_1, \ldots, X_{m-1})$ is the joint conditional distribution of all tokens in the patch given all previous patches. This reduces the number of autoregressive steps from $N$ to $M = N/p^2$.

Why the standard approach fails for joint prediction. Prior parallel AR methods (PAR, RandAR, NAR) define the same factorization in Equation 6 but use token-wise classification heads during inference. Each token in $X_m$ is sampled independently via $p(x_m^i | X_1, \ldots, X_{m-1})$ using a softmax over its individual vocabulary. This creates a training-inference discrepancy: during training, the model learns to predict each token's marginal distribution (conditioned on previous patches and other tokens in the same patch being ground-truth, due to teacher forcing). During inference, the model must sample tokens without access to ground-truth co-tokens—the independence assumption is a workaround, not something the training objective encourages. The result is incoherent patches where tokens don't agree on the visual structure they jointly represent.

Multi-Token Diffusion Objective. BitDance resolves this by extending the binary diffusion head to model the joint distribution of all $p^2$ tokens. Let $X \in \mathbb{R}^{p^2 \times d}$ be the matrix of ground-truth tokens in a patch (each row is one $d$-dimensional binary token) and $Z \in \mathbb{R}^{p^2 \times h}$ be the corresponding hidden states from the AR transformer. The training objective becomes:

$\mathcal{L}_{\text{parallel}} = \mathbb{E}_{t, X, \epsilon} \left\| v_\theta(X_t, t, Z) - v_t \right\|^2$

where $X_t = tX + (1 - t)\epsilon$ adds noise to all $p^2$ tokens simultaneously, $\epsilon \in \mathbb{R}^{p^2 \times d}$ is independent Gaussian noise for each element, and $v_t = X - \epsilon$ is the target velocity for all tokens. The diffusion network $v_\theta$ takes all $p^2$ noisy tokens and all $p^2$ conditioning vectors simultaneously, processes them jointly (tokens can interact within the head's DiT), and predicts the velocity for all tokens.

What it computes: For each patch during training, the ground-truth $p^2$ binary tokens are corrupted with noise at level $t$, and the network must denoise all of them jointly, given the hidden states as conditioning. The key is that the network sees all tokens in the patch together and can model their correlations—if one token's bits indicate a horizontal edge, the network can predict that adjacent tokens in the patch should show edge continuation rather than contradictory texture.

Why this form: By applying the same flow-matching objective to a matrix of tokens rather than a single vector, the loss directly optimizes the joint conditional likelihood $p(X_m | \text{context})$, not a product of marginals. This aligns the training objective perfectly with the inference procedure—during both training and inference, the model processes all tokens in a patch jointly. There is no independence assumption to be violated during generation.

Block-wise Causal Attention Mask. To enable the AR transformer to learn useful representations for joint prediction, the standard causal attention mask is modified. In a standard causal mask for a 1D sequence, each token can only attend to tokens before it in the sequence (lower triangular). In BitDance's block-wise mask, the sequence is divided into blocks corresponding to patches (each block has $p^2$ positions, one per token in the patch). Tokens within the same block can attend to all other tokens in that block (including "future" tokens within the same patch), and tokens in block $m$ can attend to all tokens in blocks $1$ through $m-1$ (previous patches). Figure 4(c) illustrates this visually: the attention mask shows full connectivity within each block (dense $p^2 \times p^2$ squares on the diagonal) and causal connectivity across blocks (lower block-triangular).

What this enables: During the transformer forward pass, when computing the hidden state for a token in patch $m$, that token's representation incorporates information from all co-tokens in patch $m$ (via self-attention within the block). This means the conditioning $Z$ for the diffusion head already encodes intra-patch relationships. The diffusion head's own architecture further models these relationships during the denoising process. The intra-block visibility creates a richer conditioning signal than if each token could only see previous tokens—the model learns that adjacent spatial positions share structure.

Ablation (Table 14): The paper ablates two key design choices. Replacing the block-wise causal mask with a full bidirectional mask across all tokens (Block-wise→Full) degrades FID from 1.98 to 2.07, showing that maintaining autoregressive structure across patches is beneficial. Changing the patch-wise raster-scan order to a token-wise raster scan (Patch→Token Raster) degrades FID further to 2.15, confirming that grouping spatially adjacent tokens into patches improves generation quality.

Prefix Tokens for the First Patch. A subtle implementation detail: the first patch in the sequence has no preceding context, but all $p^2$ tokens within it need to be predicted simultaneously. To provide the correct number of placeholder positions for the transformer to produce hidden states, $p^2 - 1$ learnable prefix tokens are prepended before the vision token sequence. These prefix tokens are learned embeddings (similar to BOS tokens) that occupy the positions corresponding to the first patch's tokens during the transformer forward pass, producing the hidden states that condition the diffusion head for the first patch.

Inference Procedure with Next-Patch Diffusion. During generation:

1. The transformer processes the prefix tokens (and resolution tokens + text tokens for T2I), producing hidden states for the first patch's $p^2$ positions.
2. The binary diffusion head initializes $p^2$ tokens as Gaussian noise and iteratively denoises them over $N$ steps, conditioned on these hidden states. After binarization, the $p^2$ tokens form the first generated patch.
3. These tokens are appended to the sequence. The transformer re-processes the extended sequence (or uses KV-caching for efficiency), producing hidden states for the next $p^2$ positions.
4. Steps 2–3 repeat until all $M$ patches are generated.
5. The full grid of binary tokens is decoded by the tokenizer's decoder.

Patch Size Configurations. The paper explores $p=2$ (4 tokens per step, "-4x" suffix) and $p=4$ (16 tokens per step, "-16x" suffix). For $16\times$ downsampling on ImageNet $256\times 256$, the latent grid is $16\times 16 = 256$ tokens. With $p=1$, this requires 256 AR steps. With $p=2$, it requires $256/4 = 64$ steps. With $p=4$, it requires $256/16 = 16$ steps.

3.4.4 Text-to-Image System: Scaling BitDance with Multimodal Tokens

For text-to-image generation, BitDance scales to 14B parameters by initializing the AR transformer from Qwen3-14B, a pretrained LLM. This leverages the LLM's text understanding capabilities while adding image generation.

Token Sequence Format. The sequence for a text-to-image training example is:

[bos], {text tokens}, [boi], [res_i], [res_j], {visual tokens}, [eoi], [eos]

where [bos] and [eos] are standard LLM boundary tokens, {text tokens} are the tokenized text prompt, [boi] (beginning of image) marks the transition from text to image tokens, [res_i] and [res_j] are two special resolution tokens indicating how many visual tokens are in the height and width dimensions, {visual tokens} are the binary tokens flattened in raster-scan order, and [eoi] (end of image) marks the end of visual tokens.

Resolution Tokens for Dynamic Output Sizes: The resolution tokens [res_i] and [res_j] are a critical mechanism for supporting variable image resolutions and aspect ratios. They are learned embeddings, one for each possible spatial dimension value. During inference, the user or system specifies the desired output resolution by setting $i$ and $j$; the corresponding embeddings are inserted into the sequence, and the model knows exactly how many visual tokens to generate ($i \times j$ tokens, decoded in patches of $p^2$ each). This eliminates the need for separate models at different resolutions.

Positional Encoding. In addition to the 1D Rotary Position Embedding (RoPE) used by the pretrained LLM (which encodes token positions in the 1D sequence), BitDance adds 2D sinusoidal positional embeddings for the vision tokens. These 2D embeddings encode the $(h, w)$ coordinates of each visual token in the latent grid and are added to the token embeddings before the transformer. This enhances the model's spatial awareness—understanding which tokens are neighbors in the 2D grid helps with learning coherent visual structure.

Training Loss for Text-to-Image. The total loss is a weighted combination:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{vision}} + 0.01 \cdot \mathcal{L}_{\text{text}}$

where $\mathcal{L}_{\text{vision}}$ is the flow-matching loss from the binary diffusion head applied to all vision token positions, and $\mathcal{L}_{\text{text}}$ is the standard cross-entropy loss from the LLM's language modeling head applied to text token positions (predicting the next text token). The 0.01 weight on text loss ensures the model maintains its text understanding capabilities without the language loss dominating the much larger vision loss. All text tokens and vision tokens are trained simultaneously—the model sees interleaved text and image sequences and must predict both.

Classifier-Free Guidance. During training, text tokens are randomly dropped with probability 0.1 (replaced with a learned [NULL] token). This enables classifier-free guidance at inference: the model learns to generate images both conditioned on text and unconditionally. During inference, the guidance formula is applied to the velocity prediction:

$v_\theta^{\text{cfg}}(x_t, t, z_{\text{text}}) = v_\theta(x_t, t, z_{\emptyset}) + w \cdot (v_\theta(x_t, t, z_{\text{text}}) - v_\theta(x_t, t, z_{\emptyset}))$

where $z_{\text{text}}$ is the hidden state when text tokens are provided, $z_{\emptyset}$ is the hidden state when text is dropped, and $w$ is the guidance scale (typically > 1, e.g., 4–7). Increasing $w$ strengthens adherence to the prompt at the cost of reduced diversity.

Multi-Stage Training Recipe (Table 4). The text-to-image model goes through four stages:

Stage 1: Pre-training (PT). 100K steps with learning rate $1 \times 10^{-4}$, constant schedule, batch size 2560 (256M samples processed). Mixed-resolution training: images at 256px, 512px, and 1024px in a 2:7:1 ratio. This mixed-resolution setup is noted as "crucial for training stability"—the co-presence of multiple resolutions prevents the model from overfitting to a single resolution and aids generalization to higher resolutions.

Stage 2: Continued Training (CT). 40K steps, same learning rate, batch size 2480 (99.2M samples). Resolution ratio shifts to 0:1:1 (512px:1024px), increasing the proportion of high-resolution data to enhance the model's high-res generation capability.

Stage 3: Supervised Fine-Tuning (SFT). 40K steps, learning rate reduced to $2 \times 10^{-5}$, batch size 2320 (92.8M samples). Fine-tuned on high-quality images, including a small number generated by other text-to-image models (Seedream, Z-Image-Turbo). This helps the model adapt to the distribution of high-quality outputs with fewer training samples. A small amount of image-to-text data is preserved to maintain the LLM's text understanding.

Stage 4: Distillation (Optional). 20K steps, learning rate $2 \times 10^{-5}$, batch size 1536 (30.7M samples). Trained only at 1024px resolution. This stage transitions the model from predicting 16 tokens per step ($p=4$) to predicting 64 tokens per step ($p=8$). The weights are inherited from the SFT model, and the model adapts to the larger parallel prediction with only a few training steps. Table 10 shows the distilled model maintains quality (DPG-Bench 88.30 vs. 88.28, GenEval 0.85 vs. 0.86) while achieving faster inference (Table 11: 12.4 seconds for $1024\times 1024$, a $30\times$ speedup over NextStep-1's 402 seconds).

Optimizer Details (All Stages): AdamW with $\beta_1 = 0.9$, $\beta_2 = 0.95$, $\epsilon = 1 \times 10^{-15}$, weight decay 0.0, gradient norm clip 1.0. The lack of weight decay on the 14B model is notable and suggests the model is under-regularized by default from its LLM initialization, or that the vision loss provides sufficient regularization through its own noise and augmentation.

Vision and Text Loss Weighting. The relative weight of vision loss to text loss is 1 : 0.01 across all stages. This preserves the LLM's fundamental text encoding capabilities while keeping the primary optimization focus on image generation. Without the text loss, the LLM's language understanding would degrade as the vision loss dominates gradient updates.

4. Key Insights and Innovations

Innovation 1: Reframing Discrete Sampling as a Continuous-Geometry Problem

The dominant assumption in discrete autoregressive generation has been that sampling from a large discrete vocabulary requires either exhaustive categorical modeling (intractable for large vocabularies) or factorization into independent components (inaccurate due to violated dependencies). BitDance introduces a third path that is conceptually distinct from both: embed the discrete space into a continuous one with known geometry, then use continuous-space diffusion to navigate that geometry, projecting back to discrete at the final step.

This is not merely a replacement of classification with diffusion—it is a reframing of what it means to "sample from a discrete distribution." Instead of modeling $p(\text{index}=k | \text{context})$ over $2^d$ categories, BitDance models a conditional distribution in $\mathbb{R}^d$ whose modes correspond to the $2^d$ hypercube vertices. The diffusion model doesn't need to enumerate categories; it only needs to learn to steer noisy points toward the correct vertex, exploiting two geometric properties that make this easier than standard continuous diffusion: all target points have equal norm (so the model only learns direction, not magnitude) and the target set is finite and structured (vertices of a hypercube rather than an arbitrary continuous manifold). The hard sign projection at the final denoising step provides a clean, principled boundary between the continuous sampling process and the discrete output—a design choice that simultaneously enables precise sampling and prevents the drift accumulation that plagues continuous-token AR models.

Prior work contrast: Prior discrete AR models used classification heads over VQ codebook indices. When Infinity scaled to $2^{32}$ binary vocabularies, it used bit-wise independent classifiers—the only prior approach that scales linearly with $d$. This trades accuracy for tractability and demonstrably fails (FID 8.37 vs. 1.79, Table 13). Prior continuous AR models (MAR, xAR, NextStep-1) used diffusion heads but on unconstrained VAE latents, where no projection step exists and error accumulates. BitDance's innovation is recognizing that binary token space has geometric structure that makes diffusion not just usable but advantageous: the hypercube vertex property simplifies the learning problem compared to arbitrary continuous latents. The evidence is in the sampling efficiency (Figure 8: good results with 10 steps vs. 50–250 for standard image diffusion) and the emergent bimodality in the head's output distribution (Figure 9).

Significance beyond performance: This is a fundamental conceptual contribution to discrete generative modeling. It shows that the continuous relaxation + projection pattern—well-known in optimization (relaxation of integer programs) and representation learning (straight-through estimator)—can be productively applied at the sampling level, not just the training level. This opens a design space: any discrete generative modeling problem where the discrete space has known continuous geometry could potentially benefit from a diffusion-based sampler, replacing intractable categorical heads or inaccurate factorized ones.

Innovation 2: Diagnosing and Resolving the Joint Prediction Gap in Parallel AR Models

Prior parallel AR models (MaskGIT, PAR, RandAR, NAR) achieve speed by predicting multiple tokens simultaneously, but they all share a training-inference discrepancy that the paper is the first to explicitly diagnose: during training, tokens within a parallel group are conditioned on ground-truth co-tokens (due to teacher forcing of individual token losses), but during inference, no ground-truth co-tokens exist, so each token is sampled independently. This implicitly imposes a conditional independence assumption that the training objective never justified, degrading coherence within the generated group.

BitDance's insight is that this is not a minor implementation detail but a structural flaw in the parallel AR objective. The solution—extending the binary diffusion head to jointly model all tokens in a patch—is not just an engineering improvement; it is a principled resolution that aligns the training loss (joint velocity-matching over all tokens in $X_m$) with the inference procedure (joint denoising of all tokens in $X_m$). The block-wise causal attention mask further ensures that the conditioning $Z$ for each patch already encodes intra-patch spatial relationships before the diffusion head processes them.

Prior work contrast: PAR, RandAR, and NAR all use token-wise classification heads with independent sampling at inference. They acknowledge the speed-quality trade-off but treat it as inherent to parallel prediction—you gain speed at some quality cost. BitDance rejects this framing and shows the cost is manufactured by the independence assumption, not by parallelization itself. The evidence is in Table 3: BitDance-B-4x (260M parameters, 64 AR steps) achieves FID 1.69, surpassing RandAR-XXL (1.4B parameters, 88 AR steps) at FID 2.15—better quality with fewer steps and vastly fewer parameters. This is a reversal of the expected speed-quality trade-off and directly validates the diagnosis.

Significance beyond performance: This insight reframes what is possible with parallel AR decoding. It suggests that any method generating multiple tokens simultaneously should define a joint loss over the group, not aggregate independent per-token losses. This is a diagnostic contribution that applies beyond BitDance: future parallel AR methods must address this discrepancy or accept an accuracy ceiling that isn't fundamental to parallelization.

Innovation 3: Scaling Discrete Vocabulary Entropy Past the Continuous Reconstruction Frontier

The conventional wisdom in visual tokenization has been that continuous VAEs represent the upper bound on reconstruction quality—discrete tokenizers can approach this bound with sufficient vocabulary size but will always trail due to the information loss from discretization. BitDance empirically reverses this relationship: at $d=256$ channels with $32\times$ downsampling, the binary tokenizer achieves PSNR 25.29 vs. 24.81 for the continuous DC-AE (Table 1), surpassing the continuous VAE at the same compression ratio.

This is more than a benchmark result; it changes the narrative around discrete vs. continuous representations in generative modeling. The paper demonstrates that the historical advantage of continuous VAEs was not due to an inherent superiority of continuous latents but rather insufficient vocabulary entropy in prior discrete tokenizers. When the vocabulary scales to $2^{256}$—roughly $10^{77}$ possible states, astronomically larger than typical VQ codebooks of $10^4$ to $10^5$—the discrete representation becomes expressive enough to capture details that continuous VAEs, constrained by their own architectural choices (limited latent channels, Gaussian prior assumptions), cannot recover.

Prior work contrast: MAGVIT-v2 scaled LFQ to $2^{18}$ vocabulary but reconstruction still trailed continuous VAEs (PSNR 22.70 at $16\times$ downsampling vs. 23.54 for SD-VAE at $8\times$, Table 1). WeTok reached $2^{32}$ but at $32\times$ downsampling achieved only 20.77 PSNR. The field might reasonably have concluded that discrete tokenizers asymptotically approach but never surpass continuous ones. BitDance disproves this by pushing to $2^{256}$, enabled by the group-wise entropy computation inherited from WeTok.

The complementary scaling finding (Figure 6): The paper's observation that larger vocabularies require larger AR transformers to be effectively leveraged is an important empirical scaling law. It reveals that vocabulary scaling and model scaling are coupled—you cannot simply plug a $2^{256}$ tokenizer into a small model and expect gains. This has practical implications for resource allocation: the full benefits of high-entropy tokenizers are only realized at the scale where the transformer can model their complex conditional distributions.

Significance beyond performance: This finding establishes that discreteness is not a bottleneck for reconstruction fidelity—vocabulary size is. This liberates discrete tokenizer design from the assumption that they must trade reconstruction quality for regularization. The implication is that future AR visual models can aim for the best of both worlds: VAE-level (or better) reconstruction combined with the error-accumulation resistance and unified discrete token space that makes autoregressive modeling attractive.

Innovation 4: Hard Binarization as an Inference-Time Regularizer Against Error Accumulation

Continuous-token AR models (MAR, xAR, NextStep-1) suffer from error accumulation: small prediction errors at each step drift the latent representation away from the VAE's training distribution, causing compounding degradation over long sequences. SphereAR partially addresses this by constraining latents to a hypersphere, but the space is still continuous—any point on the sphere is a valid output, errors can still accumulate along the sphere's surface, and there is no mechanism to "snap back" to a valid state.

BitDance's inference-time sign(x) operation after denoising is a qualitatively different form of regularization. It is not a training constraint (the model is not trained with a discrete loss) but a structural prior enforced at inference time: no matter how much the continuous prediction drifts during the autoregressive rollout, the final binarization projects it exactly onto a hypercube vertex—a valid discrete token. This prevents the entire sequence from drifting into out-of-distribution regions of the latent space, because every token is guaranteed to be in the (finite, bounded) set that the tokenizer was trained to reconstruct.

Prior work contrast: SphereAR's hyperspherical constraint is training-time and continuous—it limits where latents can go but doesn't create discrete checkpoints. The tokenizer's decoder in continuous AR models must handle arbitrarily drifted latents, which it was never trained on. BitDance's decoder always receives binary tokens from the training distribution, because the sign projection enforces this. The paper doesn't frame it this way explicitly, but the geometric effect is: the autoregressive sequence is constrained to a graph whose nodes are the $2^d$ hypercube vertices, with transitions determined by the conditional distribution learned by the transformer. The model cannot "leave the graph."

Significance beyond performance: This is a structural insight about how to design token spaces for stable autoregressive generation. It suggests a design principle: the target space for autoregressive prediction should be a finite set with a known continuous embedding, and the final sampling step should project onto that set. Continuous relaxation enables efficient optimization; hard projection ensures distributional stability. This principle could generalize beyond binary tokens to any structured discrete space (e.g., permutations, graphs, trees) that can be embedded in a continuous Euclidean space with a well-defined projection.

Innovation 5: A Unified Architecture That Simultaneously Resolves Bit-Level and Token-Level Joint Modeling

Prior work addressed the bit-level sampling problem (Infinity, with bit-wise independence) and the token-level parallelization problem (PAR, RandAR, with independent token sampling) separately, using different mechanisms. BitDance is the first to address both with a single mechanism: the binary diffusion head handles bit-level joint modeling (avoiding the independence assumption that degrades Infinity), and the multi-token extension of the same head handles token-level joint modeling (avoiding the training-inference discrepancy in prior parallel AR methods).

This unification is intellectually satisfying but also architecturally efficient. The same DiT backbone, the same flow-matching objective, and the same binarization step operate at both scales—the only difference is the tensor dimensions ($d$ vs. $p^2 \times d$). This means improvements to the diffusion head (better architecture, faster sampling, more accurate velocity prediction) benefit both single-token quality and multi-token parallelism simultaneously.

Prior work contrast: Infinity used bit-wise classifiers for bit-level sampling and did not explore parallel token prediction. PAR used token-wise classification for parallel token prediction but operated on small VQ vocabularies where the bit-level problem doesn't arise (since token sampling is a single classification over $K$ categories). MAR used a diffusion head for continuous token prediction but had no discrete structure and no parallel token extension. BitDance is the first architecture where the same module serves as both the precision mechanism for high-entropy tokens and the acceleration mechanism for fast generation.

Significance beyond performance: This unification suggests that the joint-vs-independent modeling tension is fractal—it appears at the bit level (bits within a token are not independent), at the token level (tokens within a patch are not independent), and potentially at higher levels (patches within a region are not independent). BitDance's approach of extending the diffusion head to larger groups is an architectural pattern that could be applied recursively—a next-region diffusion could model multiple patches jointly—without requiring new mechanism designs at each scale. This positions the binary diffusion head as a general-purpose module for joint discrete generation that scales naturally from bits to tokens to larger spatial units.

5. Experimental Analysis

Evaluation Methodology

• Dataset. Class-conditional experiments use the ImageNet-1K benchmark at $256 \times 256$ resolution, following the standard protocol established by prior work [43, 61, 66]. Text-to-image experiments evaluate on four benchmarks: DPG-Bench [34] (semantic alignment across entity, attribute, relation, and global categories), GenEval [25] (object-focused compositional generation across six sub-tasks), OneIG-Bench [7] (English and Chinese variants, covering alignment, text rendering, reasoning, style, and diversity), and TIIF Bench [71] (instruction following across basic and advanced categories). The tokenizer training uses DataComp-1B [22] supplemented with high-quality face and text datasets. The text-to-image pre-training data is collected from open-sourced datasets such as LAION [59]; SFT data includes images generated by Seedream [24] and Z-Image-Turbo [65].

• Base model(s). For class-conditional generation, BitDance is trained from scratch in three sizes: BitDance-B (24 transformer blocks, width 768, 242M parameters), BitDance-L (32 blocks, width 1024, 527M parameters), and BitDance-H (40 blocks, width 1280, 1.0B parameters). The binary diffusion head uses 6, 8, and 12 DiT blocks respectively. For text-to-image generation, the AR transformer is initialized from the pretrained Qwen3-14B [78] LLM, producing a 14B-parameter model. The visual tokenizer uses a pure CNN architecture and is frozen during all AR training stages.

• Metrics. Class-conditional evaluation uses Fréchet Inception Distance (FID) as the primary metric, with Inception Score (IS), Precision, and Recall as complementary measures, all computed using OpenAI's TensorFlow-based evaluation toolkit [18] on 50K generated samples against the ImageNet training set statistics. Text-to-image evaluation uses the standard metrics provided by each benchmark: DPG-Bench reports an overall score averaged across five attribute categories, GenEval reports average accuracy across six compositional sub-tasks, OneIG-Bench reports overall scores combining alignment, text rendering, reasoning, style, and diversity dimensions, and TIIF Bench reports overall scores for short and long prompts across basic following, advanced following, and designer categories.

• Baselines. Class-conditional baselines include: (i) continuous diffusion models—DiT-XL/2 [53], SiT-XL/2 [48], DiCo-XL [3], MDTv2 [23], REPA [82], RAE [85]; (ii) continuous masked/AR models—MAR [43] at B/L/H scales, SphereAR [36] at B/L/H scales, xAR [56] at B/L/H scales; (iii) discrete AR models—LlamaGen [61] at L/XL/XXL, RandAR [52] at L/XL/XXL, RAR [81] at L/XL/XXL, Open-MAGVIT2-XL [47], MAGVIT-v2 [80], VAR [66] at d20/d30; (iv) parallel generation models—PAR [70] at L/XL/XXL, NAR [30] at L/XL/XXL, MaskGIT [6], MAR with masking, VAR with next-scale prediction. Text-to-image baselines include: (i) proprietary models—GPT Image 1 [51], Seedream 3.0 [24], Midjourney V7 [50], DALL-E 3 [4], Kolors 2.0 [38]; (ii) diffusion models—PixArt-$\alpha$ [9], FLUX.1-Dev [40], SD3 Medium [20], Z-Image-Turbo [65], Z-Image [65], Qwen-Image [64], HiDream-I1-Full [5], BAGEL [16], Lumina-Image-2.0 [54], SANA 1.5 [75], OmniGen2 [72], Show-o2 [77], among others; (iii) autoregressive models—Emu3-Gen [68], Infinity [29], Janus-Pro [12], Tar [28], NextStep-1 [63], GLM-Image [83], LightGen [73].

• Generation budget / compute accounting. For class-conditional experiments, generation cost is measured by the number of AR steps (transformer forward passes) and reported as steps or throughput (images/second on a single A100 with batch size 64 at bfloat16 precision). For parallel generation comparisons, the patch size $p$ determines tokens per step ($p=2$ yields "-4x" with 64 steps; $p=4$ yields "-16x" with 16 steps on ImageNet $256 \times 256$). For text-to-image, latency (seconds) is reported for $1024 \times 1024$ generation on a single H100 GPU at bfloat16. Training FLOPs are not compared head-to-head across models; the paper focuses on inference-time efficiency.

• Cross-validation / statistical protocol. There is no cross-validation or statistical significance testing reported. All class-conditional models are trained for 800 epochs with a fixed random seed protocol implied by the standard ImageNet benchmark evaluation. Text-to-image results are single-run evaluations on fixed benchmark test sets. The paper does not report confidence intervals or error bars on any metric, which is standard practice in this subfield but represents a limitation in assessing the robustness of reported differences.

Main Quantitative Results

Class-Conditional Image Generation: Sequential (p=1) Results

The headline finding for single-token prediction (Table 2) is that BitDance-H-1x (1.0B parameters) achieves an FID of 1.24 on ImageNet $256 \times 256$, the best FID among all autoregressive models and competitive with the strongest diffusion models. Specifically:

• BitDance-H-1x (FID 1.24) outperforms the previous best AR model xAR-H (FID 1.24, tied), and exceeds SphereAR-H (1.34), RAR-XXL (1.48), and VAR-d30 (1.92) by substantial margins—all despite using standard raster-scan causal ordering, which is generally considered a stronger baseline than random-order or next-scale prediction approaches.
• The scaling trend is consistent: BitDance-B (242M) → 1.68 FID, BitDance-L (527M) → 1.31 FID, BitDance-H (1.0B) → 1.24 FID. Each scale increase yields meaningful improvement, suggesting the architecture scales well with parameters.
• IS scores (304.4 for BitDance-H-1x) are strong but not dominant—RAR-XXL achieves 326.0 IS, indicating BitDance may trade some sample diversity for precision. The Precision and Recall values (0.81 and 0.64 respectively for BitDance-H) indicate a favorable precision-recall trade-off relative to competitors.
• Among discrete-token AR models using standard raster-scan ordering, BitDance's improvement is dramatic: LlamaGen-XXL (the previous best in this category) achieves only 2.34 FID despite 1.4B parameters. BitDance-B (242M) achieves 1.68 FID, demonstrating that the binary tokenizer + binary diffusion head combination is far more parameter-efficient than VQ-based approaches.

Critical comparison to continuous AR models: The continuous AR models xAR and SphereAR achieve comparable FID scores (1.24 and 1.34 respectively for their largest variants). However, the paper's ablation (Table 12) shows that when using a continuous VAE tokenizer with BitDance's AR backbone, FID degrades to 3.16 (MAR's VAE) or 4.84 (VA-VAE) versus 1.79 for BitDance's binary tokenizer. This suggests that the AR backbone itself is not the differentiator—the binary tokenizer provides a superior latent space for autoregressive generation. The comparable performance of BitDance-H-1x and xAR-H at 1.24 FID may indicate that different approaches (binary discrete vs. continuous + diffusion head for each token) converge to similar generation quality at scale, though BitDance achieves this with standard causal ordering versus xAR's next-x prediction paradigm.

Class-Conditional Image Generation: Parallel (p>1) Results

The headline finding for parallel generation (Table 3) is that BitDance-B-4x (260M parameters, 64 AR steps, generates 4 tokens/step) achieves FID 1.69 with throughput 24.18 images/second on A100, while BitDance-B-16x (260M, 16 AR steps) achieves FID 1.91 with throughput 90.26 images/second. The key comparisons:

• BitDance-B-4x vs. RandAR-XXL: BitDance achieves 1.69 FID versus RandAR-XXL's 2.15 FID, while using 5.4× fewer parameters (260M vs. 1.4B) and fewer AR steps (64 vs. 88). Throughput is 24.18 vs. 10.39 images/second—a 2.3× advantage.
• BitDance-B-16x vs. VAR-d24: BitDance achieves 1.91 FID versus VAR-d24's 2.09 FID, with 3.8× fewer parameters (260M vs. 1.0B) and comparable throughput (90.26 vs. 47.22 images/second for VAR-d20, which is the 600M variant). This is significant because VAR's next-scale prediction is inherently parallel (only 10 steps for the entire image) and has been considered the efficiency champion.
• Speed vs. quality trade-off: Moving from BitDance-B-4x to BitDance-B-16x reduces steps by 4× (64 → 16) and increases throughput by 3.7× (24.18 → 90.26 img/s) at the cost of only 0.22 FID (1.69 → 1.91). This shallow degradation is strong evidence for the effectiveness of joint multi-token modeling—if tokens within the patch were sampled independently, the quality drop would be much larger.
• Throughput comparison to diffusion models: DiT-XL/2 (675M) generates 1.06 img/s; BitDance-B-16x (260M) generates 90.26 img/s—an 85× speedup at comparable or better FID (1.91 vs. 2.27). This is particularly relevant because diffusion models have been considered faster than AR models at moderate resolutions. BitDance reverses this assumption.

What Table 3 does NOT show: The comparison is only at the base model scale (260M). No parallel results are reported for BitDance-L or BitDance-H. It is possible that larger models with parallel prediction could achieve even better FID (perhaps approaching the 1.24 of BitDance-H-1x) while retaining speed advantages over single-token prediction. The paper's decision to only report parallel results at the smallest scale leaves open the question of how parallel quality scales with model size.

Tokenizer Reconstruction Quality

The reconstruction results (Table 1) are foundational to all downstream generation results. Key findings:

• At $16\times$ downsampling with $2^{32}$ vocabulary (compression ratio 192), BitDance-Tok achieves PSNR 24.90, SSIM 0.72—outperforming SD-VAE (continuous, PSNR 23.54 at $8\times$) and all prior discrete tokenizers (Open-MAGVIT2: 22.70, Infinity: 22.70, LlamaGen: 20.65). This is the first demonstration of a discrete tokenizer surpassing a widely-used continuous VAE at comparable compression.
• At $32\times$ downsampling with $2^{256}$ vocabulary (compression ratio 96), BitDance-Tok achieves PSNR 25.29, SSIM 0.74—surpassing DC-AE (continuous, PSNR 24.81, SSIM 0.69) at the same downsampling factor. The higher PSNR despite more aggressive compression (96 vs. 48 for DC-AE at bfloat16) is striking.
• At $32\times$ downsampling with $2^{128}$ vocabulary (compression ratio 192), PSNR drops to 23.26—significantly worse than the $2^{256}$ variant (25.29) at the same downsampling rate. This demonstrates that vocabulary size, not just downsampling factor, is the key determinant of reconstruction quality. The 2× increase in per-token information capacity (256 bits vs. 128 bits) yields a 2.03 PSNR improvement at the same spatial compression.
• WeTok (prior state-of-the-art binary tokenizer) achieves only 20.77 PSNR at $2^{32}$ with $32\times$ downsampling. BitDance's improvement (25.29) is nearly 4.5 PSNR, which is an enormous gap in reconstruction quality and likely reflects the combination of larger vocabulary ($2^{256}$ vs. $2^{32}$) and improved training data/augmentation.

Caveat on reconstruction comparisons: Compression ratio calculations assume bfloat16 (16 bits per dimension) for continuous VAEs. If continuous VAEs can be stored at lower precision (e.g., 8-bit), their effective compression ratio improves. The paper's comparison is standard but favors discrete methods at the storage level. Additionally, the tokenizer is trained on DataComp-1B plus specialized datasets, while some baselines (SD-VAE) were trained on different data, making the comparison not purely controlled. The 25.29 PSNR for BitDance at $2^{256}$ should be understood as an upper bound on what is achievable with a discrete CNN tokenizer on this data mixture, not necessarily an inherent superiority over all possible continuous VAEs.

Text-to-Image Generation

The text-to-image results (Tables 5–9, 11) demonstrate that BitDance at 14B parameters achieves state-of-the-art performance among autoregressive models and is competitive with leading diffusion and proprietary models, despite training on substantially less data.

DPG-Bench (Table 5): BitDance achieves 88.28 overall, the highest among all autoregressive models (NextStep-1: 85.28, GLM-Image: 84.78, Infinity: 83.46) and competitive with top diffusion models (Qwen-Image: 88.32, Z-Image: 88.14). The breakdown shows particular strength in Entity (93.76) and Attribute (92.47) categories, suggesting strong object-level understanding.

GenEval (Table 6): BitDance achieves 0.86 overall, again the best among AR models (Tar: 0.84, Janus-Pro: 0.80). The sub-scores reveal a pattern: BitDance excels at Single Object (1.00), Two Objects (0.96), Colors (0.95), and Color Attribution (0.83), but is weaker on Counting (0.71) and Position (0.72). This suggests the model captures visual appearance well but struggles with precise spatial reasoning and numerical composition—a pattern common to many text-to-image models and not specific to the autoregressive paradigm.

OneIG-EN (Table 7) and OneIG-ZH (Table 8): BitDance achieves 0.532 (EN) and 0.512 (ZH), competitive with top models (Qwen-Image: 0.539 EN, 0.548 ZH; Z-Image: 0.546 EN, 0.535 ZH). The Text rendering scores are particularly strong: 0.937 (EN) and 0.961 (ZH), outperformed by only GLM-Image (0.969 EN, 0.976 ZH) and Z-Image (0.987 EN, 0.988 ZH) among open models. The diversity scores are relatively low (0.177 EN, 0.159 ZH), suggesting the model may produce somewhat homogeneous outputs across different prompts—a potential side effect of the SFT stage's focus on high-quality examples.

TIIF Bench (Table 9): BitDance achieves 79.64 (short prompts) and 78.12 (long prompts), second among AR models behind GLM-Image (81.01/81.02) and competitive with strong diffusion models. The detailed breakdown across basic following (attribute, relation), advanced following (reasoning, combined attributes), and designer categories (style, text, real world) shows balanced performance, with particular strength in Text rendering (96.67 short, 90.00 long) and Real World generation (87.78 short, 75.57 long).

Inference speed (Table 11): At $1024 \times 1024$ resolution, BitDance generates an image in 12.4 seconds on a single H100. This is $32.4\times$ faster than NextStep-1 (402 seconds), $4.3\times$ faster than GLM-Image (53.2 seconds), and faster than leading diffusion models at this resolution (Qwen-Image: 20.3s, Z-Image: 21.1s, BAGEL: 23.1s). This is a transformative practical result: BitDance is the only AR model that matches or exceeds diffusion model speed at high resolution.

Distillation results (Table 10, Figure 7): The distilled model (predicting 64 tokens per step instead of 16) maintains DPG-Bench 88.30 (vs. 88.28 for SFT) and GenEval 0.85 (vs. 0.86). The quality preservation with 4× fewer AR steps demonstrates that the binary diffusion head can model larger joint distributions without degradation—the distillation stage effectively amortizes the modeling capacity.

Data efficiency claim: The paper emphasizes that BitDance is trained on "fewer than 450M image-text pairs," which is "orders of magnitude smaller than the billion- to multi-billion-scale datasets typically used by current leading commercial models." This claim is difficult to verify since most commercial models do not disclose exact dataset sizes, but it is plausible that BitDance's strong performance relative to models like Qwen-Image, Z-Image, and Seedream 3.0 represents a genuine data efficiency advantage. The binary tokenizer's compact representation and the joint modeling of patches may contribute to this efficiency.

Ablation Studies and Robustness Checks

All ablation studies use BitDance-B as the backbone on ImageNet $256 \times 256$, trained for 400 epochs, evaluated with best classifier-free guidance scale.

Continuous VAE vs. binary tokenizer for AR generation (Table 12): MAR's VAE achieves only FID 3.16 and IS 289.9; VA-VAE achieves FID 4.84 and IS 273.7; BitDance-Tok achieves FID 1.79 and IS 290.5. This is a controlled comparison: the same AR backbone with different frozen tokenizers, isolating the tokenizer's effect on generation quality. The ~2× FID gap confirms that unconstrained continuous tokens cause significant error accumulation during autoregressive rollout, even when the tokenizer achieves good reconstruction (MAR's VAE is a strong continuous tokenizer). This supports the paper's central claim that binary discreteness provides beneficial regularization for AR generation.

Sampling head comparison (Table 13): Token classification head → out-of-memory (OOM) due to $h \times 2^d$ parameters. Bit-wise independent classification head → FID 8.37, IS 174.5. Binary diffusion head → FID 1.79, IS 290.5. The OOM result confirms the intractability of standard classification for $d=32$ tokens. The catastrophic FID of bit-wise classification (8.37 vs. 1.79) quantitatively demonstrates the failure of the independence assumption—this is not a minor degradation but a near-complete collapse of generation quality (for reference, the paper reports that VAR-d30, a strong baseline, achieves 1.92 FID; 8.37 is worse than early GANs and masked models from years prior). This ablation directly validates that joint modeling of bits is essential for high-quality generation from binary tokens.

Next-patch diffusion design choices (Table 14): The full next-patch diffusion design (block-wise causal mask + patch-wise raster scan) achieves FID 1.98 and IS 276.7. Replacing block-wise causal attention with a full bidirectional mask across all tokens ("Block-wise→Full") degrades to FID 2.07 (IS 271.8)—a modest but consistent drop, suggesting that maintaining autoregressive structure across patches helps the model learn spatial dependencies more effectively. Removing patch grouping and reverting to token-wise raster scan ("Patch→Token Raster") degrades further to FID 2.15 (IS 270.0), confirming that grouping spatially adjacent tokens into patches (and modeling them jointly) is beneficial compared to predicting the same number of tokens independently in raster order.

Diffusion sampling steps (Figure 8): For $p=4$ (next-patch diffusion with 16 tokens per step), FID is evaluated at 5, 10, 20, 50, and 100 diffusion sampling steps. The paper reports that "good results are attained with as few as 10 steps," and performance largely saturates by 20–50 steps. This is significantly fewer than typical image diffusion models (50–250 steps) and represents an important practical finding: the structured nature of the binary token space (hypercube vertices) makes the diffusion sampling problem easier than standard continuous diffusion. This directly supports the geometric efficiency argument made in the technical approach.

Prediction behavior of the binary diffusion head (Figure 9): The output distribution of the diffusion head across timesteps shows that at small $t$ (high noise), predictions cluster near 0 (uncertain). As $t$ increases, the distribution becomes progressively bimodal, with peaks near -1 and +1. By the final timesteps, predictions are almost entirely concentrated at the binary extremes. This figure is not a standard ablation but serves as an important robustness check: it demonstrates that the binary diffusion head learns the discrete structure of the target space automatically from the flow-matching objective, without requiring explicit discrete supervision (such as a cross-entropy term encouraging binary outputs) or architectural constraints (such as a sigmoid output layer). The emergent bimodality validates the design choice to use a continuous diffusion process rather than a discrete diffusion (which would require modeling transition probabilities over $2^d$ states).

Classifier-free guidance scale: The paper mentions that all models are evaluated at their best classifier-free guidance scale but does not report the specific scales used, nor does it ablate the effect of guidance strength on FID/IS. This is a missing ablation—the interaction between guidance scale and diffusion sampling steps could reveal whether guidance helps compensate for the discretization error from hard binarization.

Vocabulary size × model size interaction (Figure 6): While presented as a main experimental result rather than an ablation, the scaling study in Figure 6 is effectively an ablation of vocabulary size. For the $16\times$ tokenizer ($2^{32}$ vocabulary), all model sizes converge. For $32\times$ with $2^{128}$ vocabulary, BitDance-B shows degraded performance. For $32\times$ with $2^{256}$ vocabulary, only BitDance-H effectively leverages the vocabulary. This reveals that vocabulary scaling and model scaling are coupled—the benefits of larger vocabularies are conditional on sufficient transformer capacity. The paper does not explore whether this coupling could be mitigated by increasing training duration (all models are trained for 800 epochs) or by curriculum learning (starting with a smaller vocabulary and expanding).

Distillation stage (Tables 10, 11, Figure 7): The transition from 16-token (SFT) to 64-token (distilled) parallel prediction with only 20K training steps demonstrates that the binary diffusion head can rapidly adapt to larger joint prediction groups. This is a strong result but not thoroughly ablated: we don't know whether the distilled model could have been trained from scratch, whether further distillation stages (e.g., 256 tokens) are feasible, or whether the distillation loss includes any KL-divergence term in addition to the flow-matching objective.

Missing ablations the paper does not address: (1) Effect of group size in the tokenizer's entropy loss—how many groups provide the optimal trade-off between memory efficiency and codebook utilization? The paper inherits the group-wise approach from WeTok but does not ablate group count. (2) Architecture of the binary diffusion head—how many DiT blocks are needed for different patch sizes? The paper uses 6/8/12 blocks for B/L/H models with $p=1$, but the parallel experiments use only B models without reporting head depth. (3) Sensitivity to the hard binarization step—what happens if you skip binarization (continuous output) or use a soft binarization (tanh or sigmoid)? The paper argues binarization prevents error accumulation but never ablates it directly. (4) Impact of the text-to-image loss weight ratio (1 : 0.01 for vision : text)—would a different ratio improve or degrade performance? (5) The effect of mixed-resolution training (PT stage)—no ablation comparing mixed-resolution vs. single-resolution training is reported, despite the claim that it is "crucial for training stability." (6) The distilled model's performance at resolutions other than 1024px—is the 64-token parallel prediction quality maintained at 256px and 512px?

Critical Assessment

Does the paper demonstrate that BitDance achieves state-of-the-art autoregressive generation with FID 1.24?

The evidence is solid with one important qualification. Table 2 shows BitDance-H-1x at FID 1.24, tying xAR-H (also 1.24) and outperforming all other AR models. The result is reached using standard raster-scan causal ordering—the simplest and most widely-used AR paradigm—demonstrating that the binary tokenizer + binary diffusion head combination, not an exotic generation order, drives the quality. The consistent scaling from B (1.68) to L (1.31) to H (1.24) adds credibility.

The qualification: xAR-H also achieves 1.24 FID using continuous tokens with per-token diffusion heads. The paper's central claim is that discrete binary tokens are superior to continuous tokens for AR generation, yet BitDance and xAR achieve identical FID at comparable parameter counts. This doesn't invalidate the claim—BitDance achieves this with standard causal ordering while xAR uses a more complex "next-x" prediction paradigm, and the ablation in Table 12 shows that continuous tokenizers perform much worse with the BitDance backbone (FID 3.16)—but it does suggest that the advantage of binary over continuous tokens is contingent on the specific AR training recipe. A direct comparison of BitDance and xAR using the same AR backbone, training data, and compute budget would be more definitive.

The second qualification: The ablation (Table 12) comparing continuous vs. binary tokenizers shows FID 3.16 (MAR's VAE) vs. 1.79 (BitDance-Tok) when both use BitDance-B for 400 epochs. This is the most controlled comparison in the paper and directly supports the claim. However, 400 epochs is less than the full 800-epoch training used for the main results, and MAR's VAE might require different hyperparameters (learning rate, guidance scale) than used for the binary tokenizer. The gap is large enough (1.79 vs. 3.16) that hyperparameter tuning is unlikely to close it completely, but the comparison would be stronger if it used the same training budget as the main results.

Does the paper demonstrate that the binary diffusion head enables precise sampling from large binary vocabularies?

This claim is very strongly supported with a critical caveat about the comparison point. Table 13's comparison of sampling heads is the key evidence: standard classification OOM, bit-wise independent classification FID 8.37, binary diffusion head FID 1.79. The factorized classification's catastrophic performance (8.37 FID) makes the case definitively—any approach that assumes bit independence for binary tokens with $d=32$ produces near-garbage outputs. Since the binary diffusion head is the only tractable method that models the joint distribution, and it produces high-quality outputs, the claim holds.

The critical caveat: The comparison is only for $d=32$ tokens ($2^{32}$ vocabulary). The paper's main results use $d=256$ tokens for the tokenizer with the best reconstruction (Table 1), but all AR generation experiments use the $16\times$ tokenizer with $d=32$. The $d=256$ tokenizer is only evaluated for reconstruction, not for generation. The claim about scaling vocabulary size (Section 1 mentions $2^{256}$ vocabulary) is about reconstruction fidelity, not sampling—the sampling benefits are demonstrated only at $2^{32}$. Figure 6 explores the $2^{128}$ and $2^{256}$ tokenizers for generation and finds that larger vocabularies require larger models to be effective, but the binary diffusion head's scaling behavior at $d=128$ or $d=256$ is not directly compared to alternatives (since no alternative exists at that scale). The binary diffusion head is validated at $d=32$, and the paper's argument is that it enables larger vocabularies to be practical—but we don't see a head-to-head at the larger $d$.

A second observation: The paper doesn't ablate the diffusion formulation against other continuous-to-discrete approaches. For example, a continuous normalizing flow or a VAE-based sampler could also map from a continuous space to the hypercube vertices. The geometric efficiency argument (sampling is easier because targets are structured vertices) is supported by the 10-step sampling efficiency (Figure 8) but is not compared to alternatives that might also exploit this structure.

Does the paper demonstrate that next-patch diffusion provides parallel decoding with high accuracy?

Yes, with clear evidence from Table 3 and Table 14. The BitDance-B-4x model (260M, 64 AR steps) achieves FID 1.69, surpassing RandAR-XXL (1.4B, 88 AR steps, FID 2.15) by 0.46 FID with 5.4× fewer parameters. The BitDance-B-16x model (260M, 16 AR steps) achieves FID 1.91, surpassing VAR-d24 (1.0B, 10 steps, FID 2.09). The quality degradation from increasing parallelism (1.69 → 1.91 for 4× → 16×, or 0.22 FID) is modest relative to the speedup (3.7× throughput). The ablation in Table 14 confirms that joint patch modeling (block-wise attention + patch-wise scan) outperforms token-wise independent generation.

The claim that next-patch diffusion addresses the "training-inference discrepancy" in prior parallel AR methods is logically argued but indirectly tested. The paper claims that prior methods (PAR, RandAR) suffer because their training uses teacher forcing (each token sees ground-truth co-tokens) but inference samples independently. Next-patch diffusion aligns training and inference by jointly modeling all tokens in the patch. The evidence for this claim is BitDance's superior performance to these methods, but this is a composite effect: BitDance also uses a better tokenizer (binary LFQ vs. VQ) and a different sampling mechanism (diffusion vs. classification). A cleaner test would be to implement next-patch diffusion with the same VQ tokenizer and classification head as PAR/RandAR, showing that the joint modeling alone improves performance. Without this comparison, we cannot isolate how much of the improvement comes from joint modeling versus from the tokenizer and head.

Does the paper demonstrate that binary tokens surpass continuous VAEs in reconstruction?

Partly. Table 1 shows BitDance-Tok ($2^{256}$, $32\times$ downsampling) at PSNR 25.29 vs. DC-AE (PSNR 24.81) and DC-AE-SANA (PSNR 24.72). This is a genuine crossover—the discrete tokenizer achieves higher PSNR and higher SSIM (0.74 vs. 0.69) than the continuous baselines at the same downsampling rate.

But the comparison is not fully controlled. The compression ratios differ: BitDance at $2^{256}$ achieves compression ratio 96 (each spatial position is represented by 256 bits = 32 bytes uncompressed), while DC-AE achieves ratio 48 using bfloat16 (each position is represented by some number of latent channels × 16 bits). The paper's compression ratio calculation assumes the specific latent dimensionality of DC-AE (which varies by implementation), but the point is that BitDance achieves better reconstruction at higher effective compression than a specific continuous VAE. This makes the comparison somewhat favorable to BitDance—DC-AE at the same compression ratio might perform better or worse, but this isn't tested.

Additionally, the tokenizer is trained on DataComp-1B + specialized datasets while DC-AE was trained on different data (likely LAION or a curated internal dataset). Training data quality and quantity affect reconstruction metrics—the paper's controlled comparison between its own tokenizer and others would require identical training data, which is not the case. The claim should be understood as: with current training recipes and datasets, a sufficiently high-entropy discrete tokenizer can match or exceed the reconstruction quality of representative continuous VAEs, not as a proof of inherent superiority.

Does the paper demonstrate that BitDance is more data-efficient than competing models?

The paper makes this claim in the text-to-image section but provides limited direct evidence. The claim is that BitDance achieves competitive performance with "fewer than 450M image-text pairs," which is less than the billion-scale datasets used by commercial models. However:

1. The exact dataset sizes of commercial models like Qwen-Image, Z-Image, and Seedream 3.0 are not disclosed, making the comparison imprecise.
2. The SFT stage uses images generated by other text-to-image models (Seedream, Z-Image-Turbo), which could be considered a form of knowledge distillation rather than pure data efficiency—BitDance benefits from those models' capabilities through their outputs.
3. No ablation studies vary the training data size to show how BitDance's performance scales with data quantity. A learning curve showing BitDance reaching a given performance with less data than baselines would be the direct test of this claim.

The data efficiency argument is plausible—the compact binary representation and joint modeling of patches could improve sample efficiency—but the paper does not systematically validate it.

Does the paper demonstrate that BitDance achieves a 30× speedup over prior AR models at high resolution?

Yes, this is one of the paper's strongest empirical claims. Table 11 shows BitDance generating $1024 \times 1024$ images in 12.4 seconds on an H100, compared to NextStep-1 at 402 seconds and GLM-Image at 53.2 seconds. The 30× figure comes from 402/12.4 ≈ 32.4× vs. NextStep-1. This is a transformative practical result and is directly measured.

Caveats on the speed comparison: (1) The comparison is between the distilled BitDance model (64 tokens/step) and the base versions of NextStep-1 and GLM-Image. If those models had similar distillation or parallel decoding techniques, the gap might narrow. (2) Latency is measured on different GPU types (H100 for BitDance, unspecified for baselines—though the paper implies consistent measurement). (3) The speed comparison doesn't account for the tokenizer encoding/decoding time, which is typically small but non-zero. (4) NextStep-1 and GLM-Image might use different resolutions or tokenizer downsampling factors that affect sequence length. Despite these caveats, the 10-30× range of speedup is large enough that these factors are unlikely to eliminate the advantage.

What experiments would strengthen the paper?

1. Head-to-head comparison with xAR at matched parameter count and training budget. Both models achieve FID 1.24 at ~1B parameters, but with different tokenizers and training recipes. A controlled comparison would clarify whether binary discrete tokens are genuinely superior to continuous tokens for AR generation or whether both approaches converge at scale.

2. Parallel generation results at larger model scales (L and H). BitDance-B-4x achieves FID 1.69; would BitDance-H-4x approach the 1.24 FID of BitDance-H-1x while being 4× faster? This would answer whether parallel prediction quality scales with model size.

3. Generation experiments with the $2^{256}$ tokenizer. The paper demonstrates reconstruction superiority at $2^{256}$ but doesn't train AR generation with this tokenizer. At what model scale does the $2^{256}$ tokenizer become usable for generation, and does it further improve FID?

4. Ablation of the hard binarization step. Running inference without sign(x) would directly test whether the projection step is necessary for preventing error accumulation—the paper's argument is logical but not experimentally validated.

5. Controlled comparison isolating the joint modeling advantage. Implement next-patch diffusion with the same VQ tokenizer and classification head as PAR or RandAR, showing that joint modeling (not the tokenizer or head) drives the improvement.

6. Systematic data scaling study. Train BitDance at multiple dataset sizes (e.g., 50M, 150M, 450M, 1B images) and compare to baselines to validate the data efficiency claim.

7. Effect of guidance scale and its interaction with diffusion sampling steps. The paper uses classifier-free guidance but doesn't report the guidance strength or ablate its effect, which is standard in diffusion model papers.

8. Performance at intermediate resolutions for text-to-image. The text-to-image results focus on high resolution (512px, 1024px) but don't report quality at 256px, which would enable comparison to the class-conditional ImageNet results and help understand resolution scaling.

9. Robustness to different random seeds or training runs. The paper reports single-run results. Measuring variance across runs would help assess whether FID differences of 0.05–0.10 (the gaps between top methods) are statistically meaningful or within run-to-run noise.

10. Detailed breakdown of the distillation stage. How does performance evolve during the 20K distillation steps? Is there an initial quality drop followed by recovery? Can the model be distilled to 256 tokens/step?

6. Limitations and Trade-offs

6.1 The Binary Diffusion Head Has Only Been Demonstrated at Moderate Bit-Depths for Generation; the Tokenizer's Full $2^{256}$ Vocabulary Remains Untested for AR Modeling

The constraint. The paper's central narrative—that scaling binary token entropy to $2^{256}$ bridges the discrete-continuous reconstruction gap and enables state-of-the-art AR generation—rests on two separate empirical results that have not been connected. The tokenizer achieves its best reconstruction at $d=256$ channels with $32\times$ downsampling (PSNR 25.29, Table 1). However, all AR generation experiments use the $16\times$ tokenizer with $d=32$ (vocabulary $2^{32}$). The $2^{128}$ and $2^{256}$ tokenizers are evaluated only for reconstruction, never for downstream generation. The paper explicitly acknowledges this gap through Figure 6, which shows that smaller models fail to converge with $2^{128}$ and $2^{256}$ vocabularies, but does not report any AR generation results at these scales—even for the largest model.

The consequence. We do not know whether the binary diffusion head remains tractable and accurate at $d=128$ or $d=256$. The flow-matching objective's computational cost scales with $d$ (the dimension of the continuous space), and the head's DiT architecture would need to process proportionally larger tensors. More fundamentally, the geometric efficiency argument—that learning to point toward a hypercube vertex is easier than modeling an arbitrary continuous distribution because all targets share equal norm—may weaken as $d$ grows. In $d=32$ space, there are $2^{32} \approx 4.3 \times 10^9$ vertices; the model needs to discriminate among billions of possible directions. In $d=256$ space, there are $2^{256}$ vertices—a space so vast that the model might overfit to spurious correlations between conditioning vectors and specific vertices, or the diffusion process might fail to concentrate on the correct vertex within a reasonable number of sampling steps. The paper's finding that small transformers struggle with $2^{128}$ and $2^{256}$ vocabularies even during training (Figure 6) suggests fundamental difficulties that may not be resolvable solely by scaling model size.

What evidence exists. Figure 6 is the only evidence on this question, and it paints a concerning picture. For the $32\times$ $2^{128}$ tokenizer, BitDance-B shows "degraded performance" (the exact FID is not given, but the curve diverges from the $16\times$ $2^{32}$ baseline). For the $32\times$ $2^{256}$ tokenizer, only BitDance-H "effectively leverages" the vocabulary, and even then the paper does not report the achieved FID or compare it to the $16\times$ baseline. The paper frames this as a scaling insight ("vocabulary scaling and model scaling are coupled"), but it could equally be interpreted as evidence that $2^{256}$ vocabularies are not practically usable for AR generation with current model sizes and training budgets. The ablation in Table 13 (comparing sampling heads) is only for $d=32$ tokens—the catastrophic FID of 8.37 for bit-wise classification and the success of the binary diffusion head at 1.79 are demonstrated only at this scale. We have no evidence about whether bit-wise classification would be equally catastrophic at $d=128$ or $d=256$, or whether the binary diffusion head would maintain its advantage.

Mitigation status. The paper does not address this gap. It presents the $2^{256}$ vocabulary as a key contribution ("BitDance marks the first time a visual tokenizer's vocabulary has been expanded to $2^{256}$," Section 5) but never demonstrates that this vocabulary can be used for generation. The distinction between "tokenizer vocabulary" and "AR model's effective vocabulary" is elided. The paper's conclusion that "scaling vocabulary size... requires a concurrent expansion of Transformer size" (Section 4.1) hints at the difficulty but does not quantify the required scale. A practitioner hoping to use $d=256$ tokens for generation would have no guidance on model size, training duration, or expected performance.

6.2 The Difficulty Estimation Cost for Compute-Optimal Allocation Is Not Accounted For

The assumption. This is not a limitation the paper discusses—BitDance does not use compute-optimal allocation. However, a closely analogous issue arises in the paper's vocabulary scaling methodology. The paper evaluates multiple tokenizers ($16\times$ with $2^{32}$, $32\times$ with $2^{128}$, $32\times$ with $2^{256}$) and multiple AR model sizes against these tokenizers to determine which combinations work (Figure 6). This sweep is expensive—training multiple models at multiple scales on multiple tokenizers—and is presented as a one-time scaling analysis. However, any practitioner adopting BitDance's approach would need to perform a similar sweep for their own data, resolution, and compute budget. The paper does not provide a predictive scaling law or cost model that would let practitioners estimate which (tokenizer vocabulary size, model size) pair is optimal for their setting without running the full sweep.

The consequence. The upfront cost of selecting the right tokenizer-Transformer configuration for a new domain or application is not amortized by the paper's analysis. The complementary scaling relationship identified in Figure 6—vocabulary size and model size are coupled—means that choosing the wrong combination wastes significant compute. A practitioner who selects a $2^{256}$ tokenizer with a model that is too small will see training fail to converge; selecting a $2^{32}$ tokenizer with a model that is too large leaves vocabulary-related quality gains on the table. The paper offers no heuristics for making this choice beyond the qualitative observation that scaling should be matched. This is the tokenizer-selection analog of the "difficulty estimation cost" problem in adaptive inference methods—a meaningful overhead that the headline results don't account for.

What evidence exists. Figure 6 is the only evidence on vocabulary-model scaling interaction, and it covers only three vocabulary sizes (with different downsampling factors, confounding the comparison) and three model sizes on ImageNet. There is no extrapolation to other datasets, resolutions, or tokenizer architectures. The paper does not attempt to fit a scaling law predicting FID as a function of vocabulary bits $d$ and model parameters $N$, which would partially address this limitation.

Mitigation status. Not addressed. The paper does not frame this as a limitation, and the scaling analysis in Figure 6 is presented as a positive finding (insight about complementary scaling) rather than as a cost that must be paid. The text-to-image section (Section 4.3) chooses the $16\times$ $2^{32}$ tokenizer "to balance reconstruction quality and generative convergence speed" without discussing how this choice was made or how it would generalize. A practitioner would need to make a similar judgment without clear criteria.

6.3 Generation Experiments with the Full $2^{256}$ Tokenizer Are Absent; the Claimed Vocabulary Scaling Benefits Apply Only to Reconstruction, Not Generation

The constraint. The paper states in its introduction and conclusion that BitDance "expands vocabulary size up to $2^{256}$" (Section 1) and that "BitDance marks the first time a visual tokenizer's vocabulary has been expanded to $2^{256}$, attaining reconstruction fidelity comparable to that of continuous VAEs" (Section 5). The full abstract highlights this: "With high-entropy binary latents, BitDance lets each token represent up to $2^{256}$ states, yielding a compact yet highly expressive discrete representation." However, the $2^{256}$ vocabulary is used exclusively for reconstruction evaluation (Table 1, bottom rows). All class-conditional generation results (Tables 2, 3), all text-to-image results (Tables 5–11), and all ablations (Tables 12–14) use the $16\times$ $2^{32}$ tokenizer. The $2^{128}$ and $2^{256}$ tokenizers never appear in a generation experiment.

The consequence. The paper's most attention-grabbing claim—a $2^{256}$ vocabulary—is not validated for the task the paper is primarily evaluated on (image generation). This creates a gap between the paper's narrative and its empirical support. The claim that "scaling token entropy... yields reconstruction on par with continuous VAEs" is supported. The claim that this enables "high-fidelity visual generation" (Section 1) is supported only for $2^{32}$ tokens—which is still a large vocabulary relative to prior work ($2^{18}$ for MAGVIT-v2, $2^{32}$ for Infinity) but is four orders of magnitude smaller than $2^{256}$ in terms of state-space size. A reader who interprets the abstract's "$2^{256}$ states" as applying to the generation model would be misled.

The practical consequence depends on whether $2^{256}$ tokenizers prove usable for generation at larger scale. If they eventually work, the gap is a scope limitation of the current paper. If they prove fundamentally difficult to use (due to the scaling issues in Figure 6), the paper's headline vocabulary figure overstates the practical contribution. Either way, the absence of generation results at $d > 32$ is a significant limitation of the current empirical evaluation.

What evidence exists. The paper provides indirect evidence through Figure 6, which suggests that $2^{256}$ tokenizers cause training difficulties for models below BitDance-H scale (1B parameters, 40 transformer blocks). The paper does not report FID for BitDance-H with the $2^{256}$ tokenizer—the curve in Figure 6 is labeled but no numerical values are given. We do not know whether BitDance-H with $2^{256}$ tokens matches, exceeds, or falls short of the 1.24 FID achieved with $2^{32}$ tokens at the same model size. The absence of this comparison is conspicuous given its centrality to the paper's thesis.

Mitigation status. The paper does not acknowledge this as a limitation. The abstract, introduction, and conclusion all foreground the $2^{256}$ vocabulary without caveating that it was only validated for reconstruction. Section 4.1 ("Scaling up Token Entropy") discusses reconstruction and the Figure 6 scaling experiment but does not report a single downstream FID number for $d > 32$. The paper's positioning of vocabulary scaling as a core contribution is thus partially aspirational—it demonstrates that $2^{256}$ tokenizers can be trained and achieve excellent reconstruction, but not that they can be used for generation.

6.4 Single Benchmark, Single Model Family for Class-Conditional Generation; No Evidence of Cross-Domain Transfer

The constraint. All class-conditional generation experiments use the ImageNet-1K benchmark at $256 \times 256$ resolution with the BitDance model family trained from scratch. The text-to-image experiments use a different setup (Qwen3-14B initialization, multi-stage training, different tokenizer downsampling). This means there is no evidence that the class-conditional generation quality transfers to other visual domains (medical imaging, satellite imagery, illustrations, structured graphics) or that the architectural design generalizes to other model families (different transformer architectures, different initializations, different parameter scales beyond B/L/H). The paper's architecture is evaluated in a narrow but standard setting that dominates the visual AR literature.

The consequence. We cannot distinguish between properties of BitDance that are universal and those that are specific to ImageNet-1K at $256 \times 256$ or to the particular training recipe (800 epochs, AdamW with specific hyperparameters). ImageNet-1K is a relatively constrained dataset—1,000 classes of mostly natural object-centric images. It is possible that the binary tokenizer's reconstruction advantage over continuous VAEs (Table 1) is larger on ImageNet than on other domains (medical, text-heavy, high-frequency textures) where VAE architectures may have been optimized differently. Similarly, the binary diffusion head's sampling efficiency (10–20 steps, Figure 8) may not transfer to tokenizers with different $d$ or different training data. The absence of any non-ImageNet generation evaluation is a significant generalizability gap.

The text-to-image results partially address the domain question—BitDance generates diverse, photorealistic images with strong text rendering (Table 7, Text score 0.937). But the text-to-image model uses a different tokenizer and is initialized from an LLM, so it does not test whether the class-conditional BitDance architecture transfers to other domains.

What evidence exists. The paper reports no experiments outside of ImageNet-1K for class-conditional generation. The text-to-image evaluation spans multiple benchmarks (DPG-Bench, GenEval, OneIG, TIIF) but these all evaluate natural image generation from text prompts—they are variations on "photorealistic scene generation," not qualitatively different visual domains. The tokenizer is trained on DataComp-1B plus face and text datasets, which may be broader than ImageNet, but the AR generation evaluation is narrow.

Mitigation status. Not addressed. This is standard practice in the field—nearly all papers in Tables 2 and 3 report only ImageNet results for class-conditional generation—so the limitation is shared by all baselines. The paper might argue that text-to-image evaluation provides complementary evidence of generalizability, and indeed the strong DPG-Bench and GenEval results suggest the architecture scales to open-domain generation. However, the specific claim that BitDance "achieves an FID of 1.24, the best among AR models" (Section 1) is validated only on one dataset.

6.5 The Training-Inference Discrepancy Diagnosis for Prior Parallel AR Methods Is Plausible but Not Experimentally Isolated

The constraint. Section 3.3 makes a specific diagnostic claim: prior parallel AR methods (PAR, RandAR, NAR) suffer because "the tokens within [a] group are sampled independently through a standard classification head," creating a "training-inference discrepancy" where the model is trained with ground-truth co-tokens (via teacher forcing) but must sample without them at inference. BitDance's next-patch diffusion is proposed as the solution because it models the joint distribution. This is a conceptually important claim—it reframes the speed-quality trade-off in parallel AR as a solvable modeling problem rather than an inherent constraint.

The consequence. The paper provides only indirect evidence for this diagnosis. BitDance-B-4x outperforms RandAR-XXL (FID 1.69 vs. 2.15, Table 3), but this comparison confounds multiple differences: tokenizer type (binary LFQ vs. VQ), sampling head (diffusion vs. classification), training recipe, and architecture. We cannot isolate how much of the 0.46 FID improvement comes from joint token modeling specifically versus the tokenizer or head. The ablation in Table 14 (block-wise vs. full attention, patch-wise vs. token-wise raster scan) tests the value of joint spatial modeling within BitDance's framework but does not test the core claim against prior methods. A direct test would implement next-patch diffusion with the same VQ tokenizer and classification head as PAR or RandAR, keeping all else equal, and measure the improvement from joint modeling alone. Without such an experiment, the "training-inference discrepancy" remains a plausible explanation, not an empirically validated one.

What evidence exists. Table 14 shows that joint patch modeling (block-wise causal attention + patch-wise raster scan) improves FID from 2.15 (token raster) to 1.98 within BitDance's binary token framework. This demonstrates that joint modeling helps, but it does not test whether joint modeling would close the gap between BitDance's performance and prior methods if both used the same tokenizer and head. The comparison to RandAR-XXL in Table 3 is the only cross-method evidence, and it is confounded.

Mitigation status. Not addressed. The paper does not acknowledge this as a confound. The diagnosis is presented as a motivation for next-patch diffusion, and the strong empirical results are offered as validation, but the causal claim—that the independence assumption, specifically, causes the quality degradation in prior methods—is overdetermined by the evidence.

6.6 Inference Speed Gains Depend on the Distillation Stage, Whose Generality Is Not Established

The constraint. The $30\times$ speedup claim for $1024 \times 1024$ text-to-image generation (Table 11, 12.4 seconds vs. 402 for NextStep-1) and the $8.7\times$ speedup for class-conditional generation (Table 3, 90.26 vs. 10.39 img/s for RandAR-XXL) both depend on parallel token prediction—specifically, generating 16 tokens per step ($p=4$) for class-conditional, and up to 64 tokens per step ($p=8$) for text-to-image after distillation. The $p=8$ configuration (64 tokens/step) is reached through an additional distillation stage (Section 4.3.1, Table 4) that takes the SFT model, inherits its weights, and fine-tunes it for only 20K steps on high-resolution data. The paper claims "the model can adapt to predicting more tokens in parallel with only a few training steps."

The consequence. The generalization and robustness of this distillation process are not characterized. We do not know whether distillation from 16-token to 64-token parallel prediction:

• Requires specific data characteristics (e.g., high-resolution images only; the distillation stage in Table 4 uses only 1024px images)
• Produces models that are as robust to distribution shift as the SFT model (the evaluation in Table 10 covers only DPG-Bench and GenEval at unspecified resolutions)
• Would work for further scaling (e.g., 256 tokens/step) or for different base model sizes
• Would work without the SFT stage (i.e., distilling directly from CT or PT checkpoints)
• Has hidden costs (e.g., the distilled model might require more sampling steps in the binary diffusion head to maintain quality, partially offsetting the AR step reduction)

Table 10 shows that the distilled model nearly matches the SFT model on DPG-Bench (88.30 vs. 88.28) and GenEval (0.85 vs. 0.86), which is encouraging. But these are aggregate scores; we do not see per-category breakdowns to check whether specific capabilities degrade (e.g., text rendering, spatial reasoning, attribute binding). The quality preservation might not be uniform across all generation scenarios.

What evidence exists. Table 10 provides two aggregate metrics, and Figure 7 shows qualitative examples (promising but cherry-picked). Table 11 provides the latency comparison. No ablations explore what happens if distillation is attempted from earlier training stages, with different patch sizes, or with different amounts of distillation data. The paper does not report whether distillation requires a specific learning rate, batch size, or data composition, or whether it is robust to these choices.

Mitigation status. The paper treats distillation as a straightforward extension ("an additional distillation stage that transitions the SFT model... with only a few training steps") rather than as a potential failure point. The small number of training steps (20K) is presented as evidence of efficiency, but could also indicate brittleness—if the model can adapt so quickly, it might also be sensitive to small changes in the distillation data or hyperparameters. A practitioner wanting to apply this distillation for their own patch size or resolution would need to determine the right recipe through trial and error.

7. Implications and Future Directions

How This Work Changes the Landscape

BitDance represents a conceptual reframing rather than a paradigm shift in autoregressive visual generation. The paper does not introduce a fundamentally new model architecture, training objective, or tokenizer type—it combines existing components (LFQ, diffusion heads, flow matching, block-wise attention) at an unprecedented scale and demonstrates that doing so resolves long-standing trade-offs that the field had accepted as inherent. The magnitude of the shift lies in changing what researchers believe is possible at each of three levels.

At the tokenizer level, the paper falsifies the assumption that discrete representations are inherently lossier than continuous ones. The finding that a $2^{256}$ binary tokenizer surpasses continuous VAEs in reconstruction PSNR (25.29 vs. 24.81, Table 1) reverses a hierarchy that has structured visual generation research since VQ-VAE: discrete tokenizers were seen as a necessary compromise—you accept reconstruction degradation in exchange for compatibility with autoregressive objectives. BitDance shows that vocabulary entropy, not discreteness, determines reconstruction fidelity, and that scaling entropy into the $2^{256}$ regime closes (and crosses) the continuous-discrete gap. This reframes the tokenizer design problem from "how do we minimize the damage from discretization?" to "how do we maximize vocabulary entropy within a given compute budget?"

At the sampling level, the paper introduces a genuinely novel mechanism: using continuous-space diffusion to sample from a discrete target distribution by embedding it as a structured set of points (the hypercube vertices) and projecting back via sign at the final step. This is not an incremental refinement of existing classification or diffusion approaches—it is a new category of discrete sampler that sits between categorical modeling (intractable at scale), factorized modeling (inaccurate), and purely continuous modeling (drift-prone). The geometric insight—that hypercube vertices have uniform norm, so the model only learns direction—explains why sampling is fast (10–20 steps, Figure 8) despite the enormous vocabulary. This opens a design space: any discrete generative modeling problem where the discrete space has a known continuous embedding with a well-defined projection could adopt this approach, replacing intractable softmax heads or inaccurate independence assumptions.

At the decoding level, the paper provides the first clear diagnosis of a training-inference discrepancy in parallel AR models: teacher forcing on individual token losses implies conditional independence at inference, creating inter-token incoherence that prior work treated as an unavoidable speed-quality trade-off. Next-patch diffusion demonstrates that this trade-off is manufactured by the independence assumption, not by parallelization itself—the joint diffusion objective over $p^2$ tokens aligns training and inference, producing better quality with fewer steps and fewer parameters than the state of the art (BitDance-B-4x: FID 1.69 at 260M vs. RandAR-XXL: FID 2.15 at 1.4B, Table 3). This reframes parallel AR from "we sacrifice quality for speed" to "we haven't yet solved the joint modeling problem for parallel groups"—a solvable problem rather than a fundamental constraint.

Reconciling prior contradictions. The paper helps resolve a tension in the literature between continuous-token AR models (which achieve good quality but suffer error accumulation) and discrete-token AR models (which resist error accumulation but degrade reconstruction). BitDance's binary tokens provide both—VAE-surpassing reconstruction (Table 1) and a discrete space with a hard projection step that prevents drift. The direct comparison in Table 12 (continuous VAE: FID 3.16 vs. BitDance-Tok: FID 1.79 on the same backbone) quantifies this resolution. The field no longer needs to choose between reconstruction quality and autoregressive stability; sufficiently high-entropy discrete tokens provide both.

Directions that become more attractive. Research into vocabulary scaling for discrete tokenizers becomes significantly more attractive—BitDance shows that the ceiling on discrete reconstruction is much higher than previously demonstrated, and that $2^{256}$ is achievable with group-wise entropy computation. Research into diffusion-based discrete samplers for other structured discrete spaces (graphs, sequences with grammatical constraints, molecules) becomes natural extensions of the hypercube embedding + projection pattern. Research into joint distribution modeling for parallel generation gets a clear diagnostic framework—future parallel methods should define a joint loss over the parallel group and can use diffusion (as BitDance does) or alternative joint modeling approaches (autoregressive within-group, energy-based models, normalizing flows).

Directions that become less attractive. Bit-wise independent classification for binary tokens (as in Infinity) becomes significantly less attractive—Table 13 shows catastrophic degradation (FID 8.37 vs. 1.79), establishing that joint bit modeling is essential at even moderate $d=32$. Further scaling VQ-based tokenizers with learned codebooks becomes less attractive for high-fidelity generation—the $2^{32}$ LFQ vocabulary already outperforms the best VQ tokenizers by large margins, and LFQ scales further without codebook collapse, whereas VQ codebooks face fundamental stability limits beyond $K \approx 10^5$. Continuous-token AR models without a projection or constraint step become harder to justify for high-resolution generation—the error accumulation problem documented in Table 12 (FID 3.16 for MAR's VAE vs. 1.79 for binary tokens) suggests a fundamental stability limitation, and solutions like hyperspherical constraints (SphereAR) are partial mitigations rather than structural fixes.

Follow-Up Research This Work Enables

Training and evaluating AR generation with the full $2^{256}$ tokenizer. The paper's most conspicuous gap is that the $2^{256}$ vocabulary (which achieves the best reconstruction, PSNR 25.29 at $32\times$ downsampling, Table 1) is never used for generation. Figure 6 shows that smaller transformers struggle to converge with this vocabulary, but does not report FID for BitDance-H (the largest model) with the $2^{256}$ tokenizer. A direct experiment training BitDance-H-1x on the $2^{256}$ $32\times$ tokenizer for 800 epochs (matching the main result's training budget) would establish whether the generation quality matches, exceeds, or falls short of the 1.24 FID achieved with the $2^{32}$ $16\times$ tokenizer. A negative result (e.g., FID > 2.0 despite the reconstruction advantage) would reveal fundamental difficulties in modeling extremely high-entropy token distributions that scale beyond the geometric efficiency of the diffusion head. A positive result (FID < 1.24) would validate the paper's central thesis that vocabulary scaling translates directly to generation quality. This experiment would also test whether the binary diffusion head's sampling efficiency (Figure 8, 10–20 steps) degrades at $d=256$—more dimensions might require more steps to resolve the correct vertex.

Isolating the joint modeling contribution from the tokenizer and head. The paper's diagnosis of parallel AR methods—that independent token sampling causes the training-inference discrepancy—is compelling but confounded. A controlled experiment would implement next-patch diffusion (with the joint diffusion objective and block-wise causal mask) using the same VQ tokenizer and classification head as PAR or RandAR. Specifically, take RandAR's VQ tokenizer and transformer, replace the independent token-wise classification heads with a joint classification objective over a patch of tokens (a $K^p$ softmax is impractically large, so this would require a surrogate—perhaps a joint energy-based model or a structured prediction loss), and measure FID improvement relative to the independent baseline at the same model size and training budget. If the joint modeling contributes, say, 0.2–0.3 FID improvement, that isolates its effect from the tokenizer and head. If the improvement is negligible, then BitDance's advantage over prior methods comes primarily from the binary tokenizer and diffusion head, not from the joint modeling diagnosis. This experiment would determine whether the "training-inference discrepancy" is a primary bottleneck or a secondary concern.

Stress-testing the binary diffusion head at extreme bit-depths and sampling step counts. The paper demonstrates the binary diffusion head only at $d=32$ and sampling steps of 5–100 (Figure 8). Systematic stress tests would (a) push $d$ to 64, 128, and 256 while measuring FID and throughput, (b) push sampling steps down to 1–5 at each $d$ to find the quality cliff, and (c) compare against alternative continuous-to-discrete samplers at the same $d$, such as a conditional VAE with a Gumbel-softmax relaxation, a score-based model with Langevin dynamics, or a continuous normalizing flow. A finding that the binary diffusion head degrades gracefully with $d$ (e.g., FID increases by <0.1 per doubling of $d$) would validate the geometric efficiency argument at scale. A finding that performance collapses beyond some $d$ threshold would identify a fundamental limitation and motivate hybrid approaches (e.g., group-wise sampling where subsets of bits are modeled jointly and groups are modeled independently).

Scaling laws for vocabulary size, model size, and training compute in discrete AR generation. Figure 6 provides a qualitative scaling relationship (larger vocabularies need larger models) but no quantitative law. A systematic scaling study varying (a) vocabulary bits $d \in \{16, 32, 64, 128\}$, (b) model parameters $N \in \{100M, 300M, 1B, 3B\}$, and (c) training FLOPs across, say, $10^{18}$ to $10^{20}$, measuring FID on ImageNet at each point, would reveal the joint scaling surface. The key question is whether there exists a compute-optimal $d$ for each $N$ and budget—analogous to Chinchilla scaling laws for pretraining—or whether $d$ should always be pushed as high as the model can support. This would directly inform practitioners allocating compute between tokenizer training (higher $d$ requires more tokenizer parameters or longer training) and AR model training. A finding that the optimal $d$ grows slowly with model size (e.g., $d \propto \log N$) would be a practically important constraint on vocabulary scaling.

Exploring alternative discrete embeddings and projection operators. The hypercube vertex embedding is one specific choice—binary tokens are $\{-1, +1\}^d$, and the projection is element-wise sign. Are there better choices? The paper's geometric argument (uniform norm simplifies learning) suggests studying other uniform-norm discrete sets: vertices of a regular simplex, points on a hypersphere at fixed angular separation (like spherical codes), or binary vectors with a fixed Hamming weight. Each would produce a different token space with different entropy characteristics and reconstruction properties. An experiment training tokenizers with these alternative discrete spaces and evaluating both reconstruction PSNR and downstream generation FID (using an analogous diffusion-in-continuous-embedding sampling head) would reveal whether the hypercube is special or whether any sufficiently large structured discrete set works. This would also test whether different projection operators (softmax over distances, nearest-neighbor lookup in the learned codebook, stochastic rounding) improve upon hard sign.

Distillation recipes for scaling parallel prediction to extreme patch sizes. The paper's distillation stage goes from $p=4$ (16 tokens/step) to $p=8$ (64 tokens/step) in 20K steps with minimal quality loss (Table 10). How far can this be pushed? Experiments distilling to $p=16$ (256 tokens/step, which would generate a $1024 \times 1024$ image in 4–8 AR steps with $32\times$ downsampling) would test whether the binary diffusion head can model joint distributions over hundreds of tokens. If distillation fails or degrades quality significantly, alternative approaches—progressive distillation (16→64→256 in stages), knowledge distillation with a teacher model, or architectural modifications to the head (more DiT blocks, cross-token attention within the patch, hierarchical modeling)—could be explored. A finding that quality degrades rapidly beyond some patch size would identify a fundamental limit on parallelization density, constraining the achievable speedup from next-patch diffusion.

Practical Applications and Downstream Use Cases

High-resolution text-to-image generation at interactive latencies. The paper's most immediately deployable result is the 12.4-second generation time for $1024 \times 1024$ images on a single H100 (Table 11). This is $32\times$ faster than NextStep-1 (402 seconds), $4.3\times$ faster than GLM-Image (53.2 seconds), and faster than leading diffusion models (Qwen-Image: 20.3s, Z-Image: 21.1s). For applications requiring high-resolution output with sub-15-second latency—digital art tools, real-time design iteration, on-demand content creation for marketing—BitDance is the first autoregressive model that is practically competitive with or superior to diffusion models in speed, while preserving AR advantages (unified text-image token space, potential for multimodal integration). The distilled model's quality preservation (DPG-Bench 88.30, GenEval 0.85, Table 10) means this speed does not require sacrificing prompt adherence or visual quality.

On-device or low-resource image generation with small models. BitDance-B-4x achieves FID 1.69 with only 260M parameters and generates 24.18 images/second on an A100 (Table 3). This is higher quality (by 0.46 FID) than RandAR-XXL at 1.4B parameters (5.4× larger). For edge deployment scenarios—mobile image editing, on-device sticker or asset generation, low-latency preview generation in cloud applications—the combination of small parameter count, high throughput, and competitive quality makes BitDance a strong candidate. The tokenizer is a pure CNN that generalizes across resolutions (Section 4.1), enabling a single model to generate at multiple output sizes without architectural changes. The 90.26 images/second throughput for BitDance-B-16x (Table 3) at 260M parameters approaches real-time video generation rates if extended to video tokenizers.

Data-efficient fine-tuning for domain-specific image generation. BitDance claims strong performance with fewer than 450M image-text pairs (Section 4.3.2), suggesting data efficiency compared to models trained on billions of pairs. For organizations with limited domain-specific data—medical imaging, scientific visualization, product photography for niche e-commerce—this data efficiency, combined with the SFT recipe that adapts the model to high-quality outputs in only 40K steps (Table 4), provides a practical pathway. An organization could take the open-source BitDance model, fine-tune it on their domain data with the paper's SFT hyperparameters (learning rate $2 \times 10^{-5}$, constant schedule, batch size ~2000, 40K steps), and expect strong domain-specific generation without the billion-scale pretraining that competing models would require. The resolution token mechanism ([res_i], [res_j]) enables generation at any aspect ratio without architectural changes, which is valuable for product images, document layouts, or scientific figures that require non-square outputs.

Unified multimodal models with high-fidelity visual generation. BitDance's text-to-image model is initialized from Qwen3-14B and processes both text and image tokens in the same sequence with the same autoregressive objective (Section 4.3.1). The 0.01 : 1 text-to-vision loss weight preserves the LLM's text understanding while adding image generation. This architecture is directly applicable to unified multimodal models that need both visual understanding (answering questions about images, reading text in images) and visual generation (creating images from prompts, editing images based on instructions). BitDance's binary tokenizer achieving VAE-level reconstruction (PSNR 25.29, SSIM 0.74 at $32\times$ downsampling, Table 1) means the understanding side loses less visual information during encoding than prior discrete tokenizers. The compact representation ($32\times$ downsampling with $2^{256}$ vocabulary) means long documents with many images can be processed without sequence length explosion—each $1024 \times 1024$ image becomes only $32 \times 32 = 1024$ tokens.

When to Prefer This Method

The paper does not articulate an explicit decision framework comparing BitDance against named alternatives under specific conditions. The results are presented as a comprehensive state-of-the-art system that outperforms prior AR models across the board on ImageNet class-conditional generation (Table 2) and leads AR models on text-to-image benchmarks (Tables 5–9), without delineating regimes where a different approach would be preferable. The closest the paper comes to a trade-off discussion is in Table 12 (continuous vs. binary tokenizers for AR generation), where continuous tokenizers show substantially worse FID (3.16–4.84 vs. 1.79), and in Figure 6, where smaller transformers with larger vocabularies show degraded performance—but these are internal design choices within the BitDance framework, not comparisons against external alternatives. The paper also does not provide head-to-head guidance on when to use BitDance versus diffusion models like DiT or SD3, versus continuous-AR models like xAR or MAR, or versus next-scale prediction models like VAR—these comparisons appear in the experimental tables but are not synthesized into a decision rule. I therefore omit a speculative "prefer this method" matrix, as it would reflect my own extrapolation rather than the paper's articulated trade-offs.