Heptapod: Language Modeling on Visual Signals

🎯 Pitch

By predicting an entire image’s missing 2D block distribution at each step—rather than a single next token—a clean causal Transformer achieves a CFG‑free FID of 2.70 on ImageNet, leapfrogging all prior autoregressive models without external semantic tokenizers or guidance. This “next‑2D‑distribution” objective forces the model to internalize holistic visual semantics that standard next‑token prediction misses.

1. Executive Summary

This paper introduces Heptapod, an image autoregressive model that returns to the first principles of language modeling—causal attention, reconstruction-focused tokenization, and no Classifier-Free Guidance—by reformulating next-token prediction as next 2D distribution prediction (predicting the distribution over all remaining spatial positions in parallel at each timestep, rather than a single designated next token). Evaluated on ImageNet-1K 256×256 class-conditional generation with a PaLM-style causal Transformer using only a standard VAE or VQ-VAE tokenizer, Heptapod-H achieves an FID of 2.70 and IS of 229.8 without CFG, outperforming prior causal autoregressive models by a wide margin (e.g., LlamaGen-3B scores 9.38 FID with over 3× the parameters). The results establish that holistic spatial prediction can compel a causal Transformer to learn global visual semantics implicitly, but only when the prediction window spans the full image—restricting the window to local chunks degrades performance even with equivalent computational supervision.

2. Context and Motivation

The Core Problem: Language Modeling Does Not Cleanly Transfer to Images

The fundamental question this paper tackles is why the recipe for large language models—a scalable Transformer with causal attention, an efficient BPE tokenizer for compression, and a simple next-token prediction objective—fails to produce comparably strong results when applied to visual generation. In the text domain, this formula works elegantly: the tokenizer (e.g., Byte-Pair Encoding) performs faithful compression without embedding semantic understanding, and semantic relationships between words and concepts emerge inside the Transformer as a byproduct of optimizing next-token prediction across billions of tokens. The tokenizer doesn't need to know that "king" and "queen" are semantically related; the Transformer learns that relationship by observing how these tokens co-occur in sequences.

The authors observe that this clean separation has not materialized in the visual domain. As illustrated in Figure 1 (Top Right), leading autoregressive visual models such as VAR and LlamaGen suffer steep performance degradation when Classifier-Free Guidance (CFG) is disabled at inference time. CFG is a separate inference-time correction mechanism that modifies the model's output distribution by interpolating between conditional and unconditional predictions—essentially acting as an external semantic corrector. Its necessity signals that the model itself has not adequately learned the target distribution from the next-token prediction objective alone. This is not a minor implementation detail; it is a diagnostic that the core learning paradigm is incomplete.

The paper characterizes this gap using the concept of external semantics: information or guidance that is not learned from the next-token prediction objective but is instead injected from outside the model. External semantics appear in two primary forms throughout prior work:

Inference-time correction (CFG): A Bayesian corrective that refines the generated distribution post-hoc, compensating for the model's inability to learn adequate semantics during training.
Tokenizer-embedded semantics: The practice of distilling knowledge from pre-trained self-supervised learning (SSL) models like DINO or CLIP into the visual tokenizer itself, creating "semantic tokens" that pre-package high-level relationships into the compression step.

Both strategies violate the language modeling principle that semantic understanding should be an emergent property of the Transformer under the next-token prediction objective, not something engineered into the tokenizer or patched in at inference time. The paper argues this departure is not merely stylistic—it introduces fundamental limitations that prevent visual generative models from achieving the same self-sufficiency and scalability as LLMs.

Why This Problem Matters

The significance of this gap operates on multiple levels, from theoretical principles to practical deployment:

Theoretical: A principled foundation for multimodal learning. The language modeling paradigm has demonstrated remarkable universality across textual tasks—translation, summarization, reasoning, question-answering—all emerge from the same next-token prediction objective applied to a simple BPE tokenizer. If vision requires fundamentally different machinery (CFG, semantic tokenizers, bidirectional attention), then the dream of a truly unified multimodal architecture—one where a single Transformer processes text, images, audio, and video under the same causal objective—becomes architecturally impossible. Each modality would need its own set of external correctives and tokenizer engineering, undermining the elegant universality that makes LLMs compelling as a paradigm.

Practical: CFG introduces well-documented drawbacks. Classifier-Free Guidance, while effective at improving sample fidelity, comes at a cost. The paper notes that CFG leads to "intensity oversaturation and reduced sample diversity"—the generated images become less varied because CFG pushes the model toward a narrower, high-confidence region of the distribution. For creative applications where diversity matters, this is a genuine limitation. Moreover, CFG requires a separate inference-time hyperparameter (the guidance scale) that must be tuned, adding complexity to deployment. A model that performs well without CFG is not just philosophically cleaner; it is more useful in practice.

Practical: Semantic tokenizers create a ceiling on capability. When the tokenizer distills from a pre-trained SSL model like DINO, the autoregressive model can only learn what that SSL model already knows. As the paper states, this approach is "inherently constrained by the capabilities of external SSL models and often suffers from information loss that impairs reconstruction quality." Any limitations in the SSL model's representation—biases, blind spots, resolution constraints—are inherited by the generative model. This creates a hard upper bound on performance that cannot be exceeded by scaling the autoregressive Transformer alone.

System design: The "impossibility triangle." The paper invokes a concept from prior work (Section 1): the dilemma of an "impossibility triangle" between reconstruction quality, generation quality, and semantic/SSL representation. Optimizing a tokenizer for semantic representation (via SSL distillation) tends to sacrifice reconstruction fidelity, because the SSL representation discards fine-grained perceptual details that are irrelevant for classification but critical for faithful image reconstruction. Conversely, optimizing purely for reconstruction (standard VQ-VAE) produces high-fidelity tokenizers that autoregressive models struggle to learn meaningful semantics from. This tradeoff means that practitioners face an uncomfortable choice: accept poor generation quality with a high-fidelity tokenizer, or accept poor reconstruction with a semantic tokenizer. Heptapod's central claim is that this triangle can be broken—not by finding a better compromise, but by changing the learning objective so that the reconstruction-focused tokenizer can be retained while semantics are learned inside the Transformer.

Deployment: Causal attention enables seamless LLM integration. The paper emphasizes that Heptapod "adheres to causal attention" and can therefore "be seamlessly integrated into a multimodal LLM, preserving the architectural coherence and deployment properties of language models" (Section 5.4). Non-causal or partially-causal approaches (like VAR with bidirectional attention within each scale, or MAR with fully bidirectional attention) would require divergent inference infrastructure and break the unified processing paradigm. In an era where frontier models increasingly aim to be natively multimodal, preserving causal attention is not an academic exercise—it directly impacts whether visual generation can be a native capability of the same model that handles text and code.

Where Prior Approaches Fall Short

The paper identifies specific limitations in the existing landscape along several axes:

Standard autoregressive approaches with reconstruction tokenizers (e.g., LlamaGen) underperform. When a causal Transformer is trained with standard next-token prediction on VQ-VAE tokens, the resulting generative quality is poor without CFG. Table 2 shows LlamaGen-XL at 15.55 FID and LlamaGen-3B at 9.38 FID—considerably worse than non-autoregressive methods and even small diffusion models. This is not a parameter scaling issue; LlamaGen-3B has 3 billion parameters yet still underperforms a 400M diffusion model (LDM-4 at 10.56 FID). The problem is fundamental to the learning objective, not the model capacity.

Semantic tokenizers introduce a non-autoregressive dependency. Approaches like DiGIT (Zhu et al., 2024), TiTok, and MAGVIT-v2 achieve strong results (DiGIT reaches 3.39 FID), but they do so by constructing semantic vocabularies from pre-trained SSL models—specifically by running K-Means clustering on DINO hidden states to create discrete tokens that encode high-level visual concepts. Figure 3 (Right) visualizes the consequence: attention maps from autoregressive Transformers trained on semantic tokens concentrate on "spatially distant yet semantically related regions," revealing that the long-range dependencies have been pre-computed by the SSL model. The autoregressive Transformer is not learning these dependencies from the generative objective; it is inheriting them from the tokenizer. This departs from the principle that made text LLMs successful—imagine if the BPE tokenizer for text were constructed by clustering GloVe or BERT embeddings, pre-packaging semantic relationships into the token identities themselves. The paper argues this would "run counter to the very principles that made LLMs successful" and would similarly limit the model to the semantic knowledge of the embedding model.

Masked autoencoding (MAE-style) approaches are non-causal. Models like MaskGIT and MAGVIT-v2 use bidirectional attention and masked prediction objectives (predicting randomly masked patches from visible context). While effective (MaskGIT achieves 6.18 FID), they abandon the causal attention mechanism entirely. This means they cannot be used for sequential generation in the standard autoregressive manner and cannot be naturally integrated into causal LLM architectures. The paper argues that "retaining the causal structure characteristic of LLMs" is worth pursuing because it preserves architectural compatibility with the dominant paradigm in language AI.

Diffusion models achieve strong results but are architecturally distinct. The top of Table 2 shows diffusion-based methods achieving excellent FID scores—LDM-4 at 10.56, DiT-XL at 9.62, and REPA at 5.90. However, diffusion models use iterative denoising rather than autoregressive sampling, making them fundamentally different from language models in their inference procedure, architecture, and training objective. They cannot be straightforwardly integrated into a unified next-token prediction system. The paper is not arguing that diffusion models are inferior, but rather that if the goal is a unified multimodal architecture, the autoregressive paradigm must be made to work for vision.

VAR and MAR achieve good results but compromise on causality. VAR (Visual Autoregressive Modeling) performs coarse-to-fine autoregression across scales but uses "bidirectional attention within each scale"—what the paper labels "partially non-causal" (Table 2). MAR (Masked Autoregressive) explicitly "abandons the causal attention mechanism" and uses a generalized autoregressive formulation with bidirectional attention. Both approaches demonstrate that relaxing causality improves performance on vision, but the paper sees this as a concession that shouldn't be necessary if the objective is properly designed. MAR achieves the best FID among autoregressive-style methods (2.35 for MAR-H), setting a practical ceiling that Heptapod aims to approach while maintaining full causality.

The root cause: the "curse of locality." The paper provides a specific mechanistic explanation for why standard next-token prediction fails on reconstruction tokenizers (Section 4.1). VQ-VAE tokens exhibit strong local correlations—neighboring patches in the 2D grid have highly similar token representations because they correspond to adjacent image regions with similar colors and textures. This is visualized in Figure 3 (Left), which shows "cosine similarity to all other tokens in the grid" for three reference tokens: the similarity concentrates heavily on spatial neighbors. Under a fixed raster scan order with teacher forcing, the autoregressive model discovers a shortcut: it can "substantially reduce loss by perfectly predicting adjacent, highly correlated tokens, with little incentive to capture the long-range dependencies that are crucial for global structure but provide only marginal additional loss reduction." The optimization landscape has a local minimum that excels at texture-level local interpolation—predicting the next pixel-like detail—while ignoring global composition, object identity, and semantic coherence. This is the "curse of locality": the objective signal is dominated by easy local predictions, drowning out the harder but more important global ones.

This analysis explains the empirical pattern: semantic tokenizers work because they "pre-package long-range semantic relationships into the tokens themselves" (Section 4.1), forcing the model to attend to semantically related but spatially distant regions. But this is a workaround, not a solution—it moves the semantic learning burden to the tokenizer rather than fixing the objective that prevented the Transformer from learning those semantics in the first place.

How Heptapod Positions Itself

Heptapod positions itself as a principled return to first principles that resolves the curse of locality not by engineering semantics into the tokenizer, but by changing what the autoregressive model is asked to predict. The key move is the analogy in Figure 4: in text, the notion of "next" is naturally defined by 1D temporal order—the model always predicts the token at position $t+1$ . In images, any patch could be "next" depending on spatial proximity or semantic relatedness. Previous work resolves this by fixing a scan order (raster, spiral) and shifting the 2D position sequence so the model knows which position to predict. Heptapod instead refuses to specify which position is next. At each timestep, the model must be prepared to predict the token distribution at every remaining spatial position in parallel.

This reframes the autoregressive task from "predict the token at the next designated position" to "predict the distribution over the entire 2D grid of the future." The vocabulary expands from a 1D token space to a 2D (position, token) space, but the underlying Transformer remains a standard 1D causal model. The complexity of modeling 2D space is offloaded to a specialized prediction head, described in Section 4.3 and Figure 5. This is the core innovation: by forcing the model to predict spatially nonadjacent patches—since it must be ready for any position—the local interpolation shortcut is eliminated. "In this new paradigm, understanding global semantics is no longer an option, but a necessity for optimizing the objective" (Section 4.1).

The paper also positions its objective as a unification of autoregressive language modeling with masked autoencoding (MAE). From a self-supervised learning perspective, the causal prefix (the tokens seen so far) serves as the unmasked context, while predicting the entire remaining 2D grid in parallel is analogous to reconstructing all masked patches. This bridges two previously separate training paradigms: the sequential, causal generation of autoregressive models and the holistic, bidirectional prediction of MAE. The unification means the model receives a learning signal analogous to MAE's global reconstruction loss while operating within a causal architecture that supports autoregressive sampling. The paper argues this is key to making semantics emerge: MAE forces models to learn global structure because predicting a masked central patch requires reasoning about the surrounding visible context. Heptapod applies the same pressure within a causal framework.

Relative to the broader landscape, Heptapod explicitly rejects:

CFG as an inference-time crutch that masks deficiencies in the learning objective
Semantic tokenizers as a violation of the separation between compression (tokenizer) and understanding (Transformer)
Non-causal or partially-causal architectures as unnecessary concessions that break compatibility with LLM architectures
Fixed scan orders as an arbitrary imposition of 1D structure on inherently 2D data

Instead, it argues for a single change—reformulating the prediction target from 1D to 2D—as sufficient to enable a standard causal Transformer with a standard reconstruction tokenizer to learn rich visual semantics from the generative objective alone. The paper's central claim is that this objective change addresses the root cause of the curse of locality, making the entire framework of external semantics unnecessary.

3. Technical Approach

3.1 Reader Orientation

Heptapod is a class-conditional image generation model built from a standard causal Transformer and a simple reconstruction-focused visual tokenizer (either VQ-VAE or VAE), trained with a novel next 2D distribution prediction objective instead of the usual single-next-token prediction. The system solves the problem that standard autoregressive models with reconstruction tokenizers fail to learn visual semantics (because they can cheat by focusing on easy local interpolation) by forcing the model, at each generation step, to predict the token distribution across all remaining spatial positions in the 2D image grid simultaneously—making global semantic understanding a mathematical requirement for reducing the loss, not an optional byproduct.

3.2 Big-Picture Architecture (Diagram in Words)

The Heptapod framework has three major components connected in a pipeline:

Visual Tokenizer (VQ-VAE or VAE) — compresses a 256×256 image into a 16×16 latent grid of 256 tokens. This component is purely reconstructive: its only job is faithful compression, with no semantic knowledge embedded. For discrete tokenization, the paper uses the VQGAN tokenizer from LlamaGen (rFID 2.19); for continuous tokenization, the VAE tokenizer from MAR (rFID 1.43). These are both off-the-shelf components—the paper does not train new tokenizers.
Causal Transformer Backbone — a standard decoder-only Transformer with 1D causal attention (the same architecture as a GPT-style language model). It ingests a sequence of already-generated (position, token) pairs as its causal prefix and outputs hidden states at the final position, which encode the model's understanding of the image so far. The backbone has no special vision-specific mechanisms—it processes 1D sequences exactly like a text LLM.
2D Distribution Prediction Head — a separate neural network (either a bidirectional Transformer for global prediction or a cross-attention + bidirectional module for local/chunk-based prediction) that takes the causal backbone's output hidden states and produces, in parallel, token distributions for every not-yet-generated spatial position in the 2D grid. A distribution sampler then selects one (position, token) pair to add to the causal prefix for the next step.

Information flows as follows: an image is tokenized into 256 latent vectors with 2D spatial coordinates → at each autoregressive step, the causal Transformer processes all previously selected (position, token) pairs → the prediction head maps the final hidden state to a full 2D distribution over all remaining positions → a sampler selects one (position, token) pair from this distribution → the pair is appended to the causal prefix → repeat until all 256 positions are filled → the completed latent grid is de-tokenized back to a 256×256 image.

3.3 Roadmap for the Deep Dive

First, the conceptual shift from 1D next-token prediction to 2D distribution prediction—what exactly changes in the autoregressive formulation and why this eliminates the local interpolation shortcut that cripples standard approaches.
Second, the mathematical framework that unifies discrete and continuous tokens as instances of distribution modeling, and why cross-entropy (discrete) versus diffusion MSE (continuous) have different training dynamics despite sharing the same probabilistic objective.
Third, the 2D prediction head architectures (global vs. local/chunk-based), their computational tradeoffs, and how they enable a 1D causal Transformer to output a full 2D distribution without abandoning causality.
Fourth, the inference procedure—how the model actually generates images autoregressively by sampling from the predicted 2D distribution at each step.
Fifth, the training objective and loss computation details, including how the loss is applied selectively to masked/unseen positions and how supervision density interacts with prediction window size.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper proposing a new learning objective for visual autoregressive models. The core idea is to reformulate next-token prediction—which is ill-defined for 2D data—as a holistic next 2D distribution prediction task where the model must predict the token distribution over the entire remaining spatial grid at each autoregressive step, thereby forcing the causal Transformer to learn global image semantics from a purely generative objective without CFG, semantic tokenizers, or non-causal attention.

The Conceptual Shift: From 1D Next-Token to 2D Distribution Prediction

The fundamental problem Heptapod addresses is that standard next-token prediction, when applied to images tokenized by a reconstruction-focused VQ-VAE, fails to incentivize learning of global semantics. The paper diagnoses this as what it calls the curse of locality (Section 4.1): VQ-VAE tokens exhibit strong local correlations—neighboring patches in the 2D grid have highly similar latent representations because they correspond to adjacent image regions with similar colors and textures. Under a fixed raster scan order with teacher forcing (where the ground-truth previous token is always provided during training), the autoregressive model discovers that it can substantially reduce its loss by simply interpolating from spatially adjacent tokens, predicting the next token based on its immediate neighbors. This local interpolation strategy provides most of the available loss reduction because adjacent tokens are so similar; the marginal loss reduction from learning long-range dependencies (e.g., that two eyes in a portrait are semantically related despite being spatially distant) is comparatively small. Optimization thus gravitates toward a local minimum that excels at texture-level detail but largely ignores holistic structure, object identity, and semantic coherence.

This explains the empirical pattern that has driven the field toward semantic tokenizers and CFG: the model can reduce loss effectively without learning semantics, so it does. Semantic tokenizers circumvent this by pre-packaging long-range dependencies into the tokens themselves (via SSL distillation), forcing the model to attend to semantically related but spatially distant regions. But this moves the semantic learning burden from the Transformer to the tokenizer, violating the language modeling principle where semantics emerge inside the model from the objective.

Heptapod's solution is to change what the model is asked to predict so that local interpolation is no longer a viable strategy. The key insight is about the specification of "next." In text, the notion of "next token" is naturally defined by the 1D temporal order—the model always predicts the token at position $t+1$ . The model doesn't need to decide which position to predict; it's always the subsequent one. The 1D position information (the index in the sequence) is sufficient to uniquely specify the prediction target.

For a 2D image, however, "next token" has no intrinsic meaning. Any of the remaining spatial positions could be considered "next"—by spatial proximity (adjacent patches), by semantic relatedness (the matching eye on the other side of the face), or by some other criterion. Previous autoregressive image models resolve this ambiguity by imposing an explicit generation order (raster scan, spiral, random) and left-shifting the 2D position sequence so that the model always knows which specific spatial position it should predict at the next step. Referring to Figure 4 (Middle), in vanilla image autoregressive models, the input consists of (token, 2D position) pairs where the 2D position sequence is shifted by one relative to the token sequence—the position embedding at time step $t$ tells the model "you will predict the token at spatial position $P_{t+1}$ ." The model's task is then: given context, predict the single token at the specified next position.

Heptapod eliminates this specification entirely. Figure 4 (Right) illustrates the difference: the 2D spatial positions in the input are not shifted. The model receives (token, its own 2D position) pairs as input, with no information about which spatial position it should predict next. The model's task becomes: given the causal prefix, predict the token distribution at every remaining spatial position in parallel. The model must be prepared to predict any of them.

To make this concrete, consider the mathematical reframing. In standard autoregressive image modeling with a fixed order, the model learns:

$P(x_{t+1} \mid x_{1:t}, p_{1:t+1})$

where $x_i$ are the token values and $p_i$ are the 2D spatial positions, with $p_{t+1}$ explicitly provided as part of the input (via the left-shifted position sequence). The position $p_{t+1}$ tells the model which token to predict.

In Heptapod, the model instead learns:

$P(x_{t+1}, p_{t+1} \mid x_{1:t}, p_{1:t})$

where both the token and its position are jointly predicted. The model outputs a distribution over the entire 2D vocabulary—all possible (position, token) pairs—rather than a distribution over the token vocabulary at a pre-specified position.

What this changes computationally: Instead of a softmax over $V$ token classes (where $V$ is the codebook size for VQ-VAE, typically 16,384), the model outputs a distribution over $V \times (N - t)$ possibilities, where $N$ is the total number of patches (256 for a 16×16 grid) and $t$ is the current step. This is a much larger output space, but it is handled by the specialized prediction head, not by expanding the Transformer's own output vocabulary.

Why this eliminates the local interpolation shortcut: Under the standard formulation, the model knows exactly which position it needs to predict, so it can focus its representational capacity on producing the correct value for that specific location—and since that location is typically spatially adjacent to the most recent token (under raster scan), local interpolation works. Under Heptapod's formulation, the model does not know which position it will need to predict next. To accurately predict the distribution over all positions, it must encode a representation that is informative about every unseen location. This means understanding global structure: what objects are in the image, where they are located, how they relate to each other spatially and semantically. Predicting a spatially distant patch (e.g., the other eye) requires reasoning about the global layout and object identity from the visible context. The model cannot rely on local smoothness because it doesn't know which position will be sampled—it must be prepared for any of them.

Connection to the vocabulary expansion perspective: The paper frames this as "expanding the vocabulary from a 1D token space to a 2D (position, token) space" (Section 4.2). Each element of this 2D vocabulary is a pair (position, token), and the model predicts a distribution over this expanded vocabulary. However—and this is a crucial architectural detail—the underlying causal Transformer still operates on 1D sequences. The 2D complexity is handled entirely by the prediction head, which maps the 1D causal hidden state to the 2D output distribution. The Transformer itself needs no modification to its causal attention mechanism; it simply processes the sequence of previously generated (position, token) pairs as a 1D sequence, exactly as it would process a text sequence. The prediction head serves as an adapter between the 1D causal processing and the 2D spatial output.

The Unified Probabilistic Framework: Discrete and Continuous Tokens as Distribution Modeling

Before detailing the prediction head architectures, the paper establishes a probabilistic foundation that unifies discrete (VQ-VAE) and continuous (VAE) tokens within the same framework (Section 3). This unifies the treatment because, from the perspective of the next-2D-distribution objective, both token types require the same thing: modeling the conditional distribution of the token at each position, given the causal context.

The core task of autoregressive language modeling is to model:

$P(z_t \mid h_{t-1})$

where $z_t$ is the token at time $t$ and $h_{t-1}$ is the hidden state from the causal Transformer encoding all previous tokens $z_{1:t-1}$ . The hidden state $h_{t-1}$ is the model's compressed representation of everything seen so far.

The choice between discrete and continuous tokens simply dictates how this conditional distribution is parameterized and optimized, but the underlying objective—accurate conditional density estimation—is identical.

For discrete tokens (VQ-VAE): The distribution $P(z_t \mid h_{t-1})$ is a categorical distribution over a finite vocabulary of size $V$ . This can be directly parameterized by a linear classification head (a fully-connected layer mapping from the hidden state dimension to vocabulary size) followed by softmax. The loss is cross-entropy:

$\mathcal{L}_{\text{discrete}} = -\log P(z_t \mid h_{t-1}) = -\log \left( \frac{\exp(f(h_{t-1})_{z_t})}{\sum_{v=1}^V \exp(f(h_{t-1})_v)} \right)$

where $f(h_{t-1})$ is the output of the linear head (logits) and $f(h_{t-1})_{z_t}$ is the logit corresponding to the ground-truth token $z_t$ .

What it computes: The negative log-likelihood of the true token under the predicted categorical distribution. The softmax normalizes the logits into a valid probability distribution over the $V$ possible tokens, and the cross-entropy loss drives the predicted probability of the correct token toward 1.

Why this form: Cross-entropy is the maximum-likelihood objective for categorical data. Minimizing it directly maximizes the log-likelihood of the observed token sequence under the model. It provides a "direct and sharply defined gradient signal" (Section 3) because it explicitly contrasts the ground-truth token against all other tokens—the gradient pushes up the logit for the correct token and pushes down logits for all distractors simultaneously. This yields unambiguous feedback at every step.

For continuous tokens (VAE): The distribution $P(z_t \mid h_{t-1})$ is a continuous distribution over $\mathbb{R}^d$ (where $d$ is the latent dimension). This distribution typically defies representation by a simple parametric form (e.g., a single Gaussian is insufficient to capture the complex multimodal distribution of latent image patches conditioned on partial context). The paper follows MAR (Li et al., 2024) in adopting a diffusion-style parameterization to model this complex continuous distribution.

The diffusion approach models the conditional distribution by learning to reverse a forward noising process. Specifically, the model learns a denoising function that predicts the clean token $z_t$ from a noised version, conditioned on the hidden state $h_{t-1}$ . The loss is mean squared error (MSE) between the predicted and true token:

$\mathcal{L}_{\text{continuous}} = \mathbb{E}_{\epsilon, \sigma} \left[ \| D_\theta(z_t + \sigma \epsilon, \sigma, h_{t-1}) - z_t \|^2 \right]$

where $z_t$ is the ground-truth continuous token, $\epsilon \sim \mathcal{N}(0, I)$ is random Gaussian noise, $\sigma$ is the noise level (sampled from a distribution over noise scales during training), $D_\theta$ is the denoising model (a small MLP head that takes the noised token, noise level, and conditioning hidden state as input), and $\| \cdot \|^2$ is the squared Euclidean norm.

What it computes: The expected squared error between the denoised prediction and the clean token, averaged over random noise samples and noise levels. At each training step, a noise level $\sigma$ is sampled, Gaussian noise scaled by $\sigma$ is added to the true token, and the denoising model must recover the original token from its noised version. The loss encourages the denoiser to accurately remove noise of all magnitudes, which implicitly learns the score function (gradient of the log-density) of the conditional distribution $P(z_t \mid h_{t-1})$ .

Why this form: This loss has been shown to be equivalent to maximizing a variational lower bound on the log-likelihood of the data under the continuous distribution (Ho et al., 2020; Kingma et al., 2021). While the MSE loss does not directly optimize log-likelihood like cross-entropy, it provides an implicit learning signal by training the model to map noised samples back to the data manifold. The diffusion approach can model arbitrarily complex continuous distributions because the denoising objective effectively learns the gradient field of the log-density.

Training dynamics comparison: The paper observes (Section 3, confirmed in Section 5.2 and Figure 6) that these two parameterizations have "significant implications for training dynamics" despite their shared probabilistic foundation:

Cross-entropy (discrete VQ) provides a sharper gradient signal. Because it explicitly contrasts the correct token against all distractors in a single step, the model receives unambiguous feedback about exactly which token was correct and which were not. The gradient is concentrated on the relevant dimensions of the output space. This leads to "markedly faster and smoother initial convergence" (Section 5.2)—the VQ-based model achieves lower FID and higher IS in early epochs (up to ~150 epochs in Figure 6).
Diffusion MSE (continuous VAE) provides a more diffuse but potentially richer signal. The loss is averaged over a continuum of noise levels, meaning each training step provides less specific feedback about the exact token value. The gradient is "diluted" across the noise distribution. However, this forces the model to learn a smooth density that captures the full distributional structure, not just point estimates. The VAE-based model improves more slowly initially but "improves dramatically after ~150 epochs, ultimately surpassing the VQ-based model" (Section 5.2).

Why the VAE eventually wins: The paper attributes the final performance gap between VQ and VAE models to the reconstruction fidelity of the tokenizer itself. The VAE tokenizer achieves rFID of 1.43, while the VQ tokenizer achieves rFID of 2.19. Since the autoregressive model's generative quality is fundamentally bounded by the information preserved in the tokenizer, the VAE's superior reconstruction sets a higher ceiling. The paper notes that "the reconstruction quality of the tokenizer sets a practical upper bound on the final generative performance" (Section 5.2). If a VQ tokenizer could match VAE reconstruction quality, the convergence speed advantage of cross-entropy might make it the strictly better choice.

The paper's treatment of this choice: "We step back from this ongoing debate and argue that from the perspective of language modeling, the distinction between discrete and continuous tokens is mathematically immaterial" (Section 3). The framework treats both as instances of the same core task—modeling conditional distributions—and the choice of parameterization is an implementation detail. The default for final results is the VAE tokenizer due to its superior final performance, but the paper explicitly notes that improving VQ reconstruction could flip this preference.

The 2D Distribution Prediction Head Architectures

The prediction head is the architectural component that enables the 1D causal Transformer to output a full 2D spatial distribution. It receives the causal Transformer's output hidden state $h_{1:t}$ (representing all tokens generated so far) and must produce, for every not-yet-generated position in the 2D grid, a distribution over possible token values. The paper explores two architectural variants that represent different points on the tradeoff between spatial modeling scope and computational cost, as illustrated in Figure 5.

Global Prediction Head (Bidirectional Transformer)

The global variant, shown in Figure 5 (Left), models dependencies across the entire image grid at each prediction step. The design directly parallels the decoder of a Masked Autoencoder (MAE), unifying autoregressive and masked modeling paradigms.

Architecture: The global prediction head is a bidirectional Transformer (i.e., with full self-attention across all positions, not causal masking). At each timestep $t$ , the head processes two types of input:

Visible tokens: The hidden states $h_{1:t}$ from the causal Transformer, corresponding to the $t$ tokens that have already been generated. These represent the unmasked, known portion of the image.
Mask tokens: A set of learnable embedding vectors, one for each remaining spatial position (positions $t+1$ through $N$ , where $N = 256$ for a 16×16 grid). Each mask token is augmented with a learnable spatial positional embedding corresponding to its specific 2D grid location. These positional embeddings encode the (row, column) coordinates of each mask position, allowing the bidirectional Transformer to distinguish which region of the image each mask token represents.

Information flow: The visible token hidden states and the learnable mask tokens (with position embeddings) are concatenated into a single sequence of length $N$ . This full sequence is processed by the bidirectional Transformer, where every position can attend to every other position—visible tokens attend to other visible tokens and to mask tokens, and mask tokens attend to visible tokens and to other mask tokens. This global attention allows the model to explicitly reason about spatial relationships: a mask token representing a position in the upper-right corner can attend to a visible token representing an already-generated lower-left patch, building a holistic understanding of how the partial image relates to the unseen regions.

Output and loss computation: The bidirectional Transformer produces output representations for all $N$ positions. However, the loss is computed only on the outputs associated with masked positions (positions $t+1$ through $N$ ). For each masked position, the corresponding output is fed through the distribution parameterization layer (a classification head for discrete tokens, or a diffusion MLP head for continuous tokens) to produce the predicted token distribution at that spatial location. The loss aggregates the prediction error across all $N - t$ unseen positions.

Why this architecture: The global head forces the model to perform "holistic prediction of the unseen future from the causal past" (Section 4.3). Because all mask tokens can attend to each other and to all visible tokens, the model must reason about the entire spatial layout simultaneously. This is exactly the pressure needed to break the curse of locality: predicting a token at a spatially distant position requires integrating information from visible tokens across the image, understanding the global structure that relates them.

Computational cost: The global head's cost scales with the grid size $N$ . The self-attention in the bidirectional Transformer is $O(N^2)$ in the sequence length (256 positions), and this cost is incurred at every autoregressive step. For the full 256-token grid, this is manageable but significant. The paper notes this scaling concern and offers the local variant as a more efficient alternative.

Local (Chunk-Based) Prediction Head (Cross-Attention + Bidirectional)

The local variant, shown in Figure 5 (Right), reduces computational cost by restricting the prediction scope to a local chunk of the image at each step.

Architecture: The local prediction head operates on non-overlapping spatial chunks of the image grid. At each timestep, rather than predicting distributions for all remaining positions, the head focuses only on the chunk that contains the next token to be generated. The head consists of two interleaved attention mechanisms:

Cross-attention layers: Learnable query vectors, corresponding to positions within the target chunk, attend to the historical context—the hidden states $h_{1:t}$ from the causal Transformer. This cross-attention allows each query to gather relevant information from all previously generated tokens, regardless of their spatial location. If a token in the upper-left was already generated and is relevant to predicting a token in the lower-right chunk, the cross-attention can route that information.
Bidirectional self-attention layers: Operating exclusively within the chunk, these layers allow queries corresponding to different positions in the same chunk to attend to each other. This models local spatial dependencies—how patches within a small region relate to each other—but unlike the global head, it does not model dependencies between different chunks directly through the head.

Why chunk-based: The paper partitions the image into non-overlapping chunks to reduce the computational complexity from $O(N^2)$ (full grid) to $O(C^2)$ where $C$ is the chunk size (much smaller than $N$ ). For example, with a 16×16 grid and chunk size of 4×4 = 16 positions, the self-attention cost per step drops by a factor of $(256/16)^2 = 256\times$ compared to the global head. However, this computational savings comes at the cost of restricting the spatial modeling scope—dependencies across chunk boundaries must be captured by the causal Transformer backbone (through its sequential processing across steps) rather than by the head directly.

The key tradeoff: The global head models all spatial dependencies at once but is more expensive per step. The local head is cheaper per step but can only directly model dependencies within a chunk; cross-chunk dependencies must be built up sequentially across autoregressive steps. The paper's ablation in Section 5.3 (Figure 7) evaluates this tradeoff empirically.

Configurations of the Prediction Head

The paper specifies several architectural details:

Matching size to backbone: Unless otherwise stated, the prediction head matches the causal Transformer in hidden width, number of layers, number of attention heads, and MLP expansion ratio. The local prediction head contains "one additional attention block per layer compared to the global variant" (Section 5.1) because it has both cross-attention and self-attention sub-layers, whereas the global variant only needs self-attention.
Depth allocation ablation (Appendix A, Table 4): The paper investigates how to split layers between the causal Transformer backbone and the prediction head. With a total depth of 32 layers for Heptapod-L:
- A balanced split (16 backbone / 16 head) achieves the best FID of 3.62 and IS of 190.8.
- Shifting layers to the backbone (24 backbone / 8 head) degrades FID to 4.88 and IS to 158.93.
- An extreme backbone-heavy split (31 backbone / 1 head) causes a sharp collapse: FID 14.74, IS 98.15.
This demonstrates that "the prediction head must retain sufficient depth to project the 1D causal context into 2D spatial distributions and to model position-wise interactions at scale" (Appendix A). A too-shallow head cannot adequately transform the 1D hidden state into a rich 2D distribution.
Diffusion MLP head configuration for continuous tokens: Following MAR, the diffusion-style denoising MLP head uses:
- Base model: 6 blocks, width 1024
- Large model: 8 blocks, width 1280
- Huge model: 12 blocks, width 1536
These are the standard diffusion head configurations from MAR, not novel to Heptapod.

The Training Objective and Loss Computation

The training objective for Heptapod is the natural extension of next-token prediction to the 2D distribution case. However, the mechanics of how the loss is computed—which positions contribute, how supervision density is controlled, and how the window size interacts with the learning signal—are critical to understanding the method's behavior.

The Core Loss Formulation

At each training step, the model receives a sequence of (token, position) pairs for a complete image (all $N = 256$ patches). For each prefix length $t$ (from 1 to $N-1$ ), the model should predict the token distribution at all positions $t+1$ through $N$ . The loss for a single training sample is:

$\mathcal{L} = \sum_{t=1}^{N-1} \sum_{j=t+1}^{N} \ell(z_j, \hat{P}_j(h_{1:t}))$

where $z_j$ is the ground-truth token at spatial position $j$ , $\hat{P}_j(h_{1:t})$ is the predicted distribution at position $j$ given the prefix $h_{1:t}$ , and $\ell$ is the token-level loss function (cross-entropy for discrete tokens, MSE for continuous diffusion tokens). The outer sum over $t$ iterates over all prefix lengths, and the inner sum over $j$ computes the prediction error for all remaining positions at each prefix length.

What this computes: For every possible prefix of the token sequence, the model must predict the tokens at all subsequent positions. This is the 2D analog of teacher-forced next-token prediction, but expanded to predict the entire future grid rather than just the immediate next position.

Why this form: By computing the loss over all prefix lengths and all future positions, the model receives a dense learning signal that penalizes any failure to understand global structure. Partial prefixes that reveal different subsets of the image (e.g., early prefixes that have only seen the upper-left corner) force the model to infer the rest of the image from limited context, exactly the skill needed for autoregressive generation.

Supervision Density and Window Size

The paper introduces two key hyperparameters that control the practical instantiation of this loss:

Window size ( $w$ ): The number of spatial positions for which the prediction head produces distributions. The global head uses $w = 256$ (the full 16×16 grid). The local chunk-based head uses smaller windows (e.g., $w = 16$ for a 4×4 chunk, $w = 64$ for an 8×8 region).

Supervision density ( $n$ ): The number of tokens (positions) per training sequence for which the loss is actually computed. Due to computational constraints, it is infeasible to compute the loss over all $N - t$ positions for every prefix length. Instead, $n$ positions are sampled (or the first $n$ positions are used) and the loss is computed only on those. For example, w256n1 means the global head with window size 256 produces distributions for all positions, but the loss is computed on only 1 randomly sampled position per sequence. w256n8 means the loss is computed on 8 positions per sequence.

The computational budget for supervision is approximately the product $w \times n$ . The paper keeps this product approximately constant when comparing different window sizes to ensure fair comparison: a model with a smaller window but higher density (e.g., w64n4) uses roughly the same supervision FLOPs as a model with a larger window but lower density (e.g., w256n1).

The Critical Interaction: Window Size vs. Supervision Density

The paper's ablation in Section 5.3 (Figure 7) reveals a finding that is central to understanding why Heptapod works:

When the window size is small (local chunk-based heads): Increasing supervision density provides negligible benefit. The configurations w16n16, w64n4, and w4n64 all perform substantially worse than w256n1, even when the $w \times n$ product is matched. Notably, doubling supervision density within a fixed small window (w16n16 → w16n32) accelerates early convergence slightly but converges to similar final performance. This indicates that when the model's prediction is restricted to a local chunk, it cannot learn global semantics effectively regardless of how many tokens it supervises on. More supervision on a local prediction task just reinforces local interpolation patterns.

When the window size is large (global head): Increasing supervision density consistently improves performance. Figure 8 shows that for the global window (w256), going from n1 to n8 improves both FID and IS across all epochs. This is because the global head already forces the model to reason about long-range dependencies; denser supervision provides richer gradients that improve both convergence speed and final quality. The model can actually use the additional supervision signal because it is already attending to the full spatial context.

The paper's conclusion from this ablation: "The spatial extent of the prediction target is more critical than the supervision density to learn long-term semantics" (Section 5.3). The window size determines whether the model can learn global structure at all; supervision density determines how efficiently it learns, but only once the window is large enough to make global learning necessary. This supports the central hypothesis: the holistic prediction task (large window) is what compels semantic learning; the local interpolation shortcut is only eliminated when the model must account for spatially distant positions.

Inference: Autoregressive Generation with 2D Distribution Sampling

During inference, Heptapod generates images autoregressively by repeatedly sampling from the predicted 2D distribution. The procedure is:

Initialization: The sequence begins with a learnable start token [S], analogous to the beginning-of-sequence token in language models. The causal Transformer processes this single token.
2D distribution prediction: For the current prefix (starting with just [S] at step 1), the causal Transformer produces hidden states. The prediction head takes these hidden states and outputs a distribution over all remaining spatial positions and their token values. For a discrete tokenizer, this is a categorical distribution over the 2D vocabulary; for a continuous tokenizer, the diffusion head can produce samples via the reverse diffusion process conditioned on the hidden states.
Sampling: A (position, token) pair is sampled from this joint distribution. The sampling can be done in various ways:
- Random sampling: Sample a position uniformly or according to the model's predicted position distribution, then sample a token from the conditional token distribution at that position.
- Argmax: Select the (position, token) pair with the highest predicted probability (or highest confidence score).
The paper uses random sampling to maintain diversity (the default autoregressive generation approach), consistent with the teacher-forcing training objective where positions are naturally randomized.
Appending: The sampled (position, token) pair is appended to the sequence. The token value and its 2D spatial position embedding are both added to the input for the next step.
Repeat: Steps 2–4 are repeated until all $N = 256$ positions have been filled. The model maintains an internal record of which positions have been generated and removes them from the prediction space to avoid sampling the same position twice.
De-tokenization: The completed grid of 256 tokens (with known 2D positions) is arranged into the 16×16 spatial layout and passed through the tokenizer's decoder (VQ-VAE decoder or VAE decoder) to reconstruct the 256×256 image.

The generation order is learned, not fixed: Unlike standard autoregressive models with a predetermined raster or spiral order, Heptapod's generation order is determined by the model's own predictions. At each step, the model decides (through its distribution over positions) which position to generate next. This means the model can adapt its generation order to the image content—generating structurally important regions first, then filling in details. This is a form of learned, content-dependent generation order, though the paper does not extensively analyze the specific orders that emerge.

Connection to training: During training, the positions are provided in random order within each sequence (the paper uses shuffled 2D positions, as shown in Figure 4). This randomization forces the model to be prepared for any position to be "next" at any point, which aligns with the inference procedure where the model itself chooses the position via sampling. The training distribution over generation orders matches the inference distribution, ensuring consistency.

Summary of Design Choices and Their Justifications

2D distribution prediction over single-next-token prediction: Eliminates the local interpolation shortcut by forcing the model to predict spatially distant tokens. Justified by the ablation in Table 1: 1D-random improves over 1D-raster, but 2D-random (Heptapod) provides dramatically larger gains (FID 3.97 vs. 13.07 vs. 19.23), showing that predicting the full 2D distribution matters beyond just randomizing the order.
Causal attention preserved without modification: The 2D complexity is handled by the prediction head, not by changing the Transformer's attention mechanism. This preserves architectural compatibility with language models and ensures the approach works with standard causal Transformer implementations.
VAE tokenizer as default over VQ: Chosen for superior reconstruction fidelity (rFID 1.43 vs. 2.19) which sets a higher performance ceiling. The paper acknowledges VQ's faster convergence and suggests improving VQ reconstruction as a path to combining the advantages of both.
Global prediction head as default: Chosen because the ablation in Figure 7 shows it consistently outperforms local chunk-based heads, even with matched computational budget. The window size (spatial extent of prediction) dominates supervision density in determining final quality.
Matched head depth to backbone depth: Table 4 shows that a balanced split (16/16 for Heptapod-L) outperforms backbone-heavy splits. The head needs sufficient capacity to translate 1D hidden states into rich 2D distributions.
No CFG, no semantic tokenizer, no external SSL model: These are not design "choices" per se but rather the paper's core constraint: the framework must work without them to demonstrate that semantics can emerge from the generative objective alone. The results in Table 2 validate that this constraint can be satisfied while achieving competitive performance.
Random position order during training: Ensures the model sees all possible generation orders during training, making it robust to any sampling order at inference. This is analogous to how randomly masked positions in MAE force the model to learn global reconstruction regardless of which specific patches are masked.

4. Key Insights and Innovations

Innovation 1: The "Curse of Locality" as a Diagnostic Concept That Explains Why Visual Autoregressive Models Fail Without External Semantics

The paper's most distinctive conceptual contribution is not the solution itself, but the diagnostic framework it introduces to explain a pattern the field had observed but never systematically analyzed: why standard autoregressive models with reconstruction-focused tokenizers fail to learn visual semantics. The paper coins the term "curse of locality" (Section 4.1) to name a specific optimization failure mode that had previously been treated as a mysterious domain gap between text and images—something practitioners circumvented with CFG and semantic tokenizers without articulating why the gap existed.

What the curse of locality is, at the idea level: VQ-VAE tokens of adjacent image patches are overwhelmingly similar because natural images are spatially smooth—neighboring pixels share colors, textures, and low-level statistics. Under teacher forcing with a fixed scan order, the autoregressive model's loss function is dominated by these local correlations. The model can achieve substantial loss reduction by learning to interpolate from spatially adjacent tokens, a strategy that requires only local texture-level reasoning. Learning long-range dependencies—the kind that encode object identity, global composition, and semantic relationships—provides only marginal additional loss reduction because predicting a distant patch correctly when you've already gotten 90% of the way there with local interpolation barely moves the cross-entropy. The optimization landscape has a deep, broad local minimum at "good local predictor, terrible global reasoner," and standard next-token prediction with a fixed scan order drives the model straight into it.

Why this is a genuine insight rather than an obvious observation: Prior work had documented the symptoms—that LlamaGen performs poorly without CFG, that semantic tokenizers boost performance, that attention maps from VQ-VAE-trained models concentrate on local neighbors (Figure 3, Right). But the field's response was to treat these as evidence that vision is fundamentally harder than text, requiring either inference-time correction (CFG) or pre-computed semantics (SSL-distilled tokenizers). The curse of locality reframes these symptoms as a single, coherent optimization pathology: the model can succeed at its training objective without learning semantics, so it does. This is not a failure of architecture or scale—it is a failure of the objective function to specify what kind of understanding is required.

Comparison to prior assumptions: The dominant assumption in prior autoregressive visual generation was that if you scale up the model and train it long enough on enough data, semantics would eventually emerge from next-token prediction, just as they do in text. This assumption motivated efforts like LlamaGen (Sun et al., 2024), which scaled autoregressive Transformers to 3B parameters on VQ-VAE tokens. The curse of locality explains why this assumption is wrong: the optimization signal from local interpolation is always easier to exploit than the signal from global reasoning, regardless of model capacity. A 3B-parameter model can interpolate locally just as well as a 300M-parameter model. Scaling capacity without changing the objective just scales the model's ability to settle into the local minimum more efficiently. This is a fundamental diagnosis, not an incremental observation—it identifies a structural failure of the objective that cannot be fixed by scaling.

Evidence anchoring the claim: The paper provides converging evidence for this diagnosis:

Figure 3 (Left): Cosine similarity maps of VQ-VAE tokens show strong local correlations—adjacent tokens are overwhelmingly similar. This is the data property that enables the shortcut.
Figure 3 (Right): Attention maps from a standard autoregressive Transformer trained on VQ-VAE tokens show attention concentrated on spatial neighbors (local interpolation), while the same architecture trained on semantic tokens shows attention on spatially distant but semantically related regions. This visualizes what the model learns under each tokenizer regime.
Table 1: Simply randomizing the generation order (1D-random) improves FID from 19.23 to 13.07 over a fixed raster scan. This is a partial mitigation—randomization forces the model to sometimes predict spatially distant tokens—but it doesn't eliminate the problem because the model still only predicts one token at a time and the average token is still spatially near previously generated ones. The dramatic further improvement from 2D-random (FID 3.97) confirms that eliminating the local shortcut requires predicting the full 2D distribution, not just randomizing the 1D order.

Significance beyond Heptapod: The curse of locality is a portable diagnostic concept that can be applied to any domain where autoregressive modeling is applied to data with strong local correlations—video, audio, 3D geometry. It explains when the standard language modeling recipe will fail (when the data has high local redundancy) and what needs to change (the objective must force prediction of spatially or temporally distant elements). This is a theoretical advance in understanding the limits of next-token prediction across modalities, not just an engineering fix for image generation.

Innovation 2: Reframing Autoregressive Modeling as Holistic 2D Distribution Prediction, Unifying It with Masked Autoencoding

The paper's core technical innovation is the reformulation of the autoregressive prediction target from a 1D token at a specified position to a joint distribution over the entire 2D spatial grid of remaining positions. This is more than an architectural change—it is a conceptual reframing of what autoregressive modeling means for non-sequential data.

The conceptual move: Standard autoregressive models for sequential data (text, audio) have an implicit assumption: the data has a natural order, and "next" is well-defined. For 2D images, this assumption breaks—there is no canonical scan order, and any imposed order is arbitrary. Prior work dealt with this by choosing an order (raster, spiral, random) and treating it as if it were natural, effectively converting the 2D problem into a 1D problem by fiat. Heptapod makes the opposite move: instead of imposing a 1D order on 2D data, it expands the prediction vocabulary to 2D, asking the model to predict the distribution over (position, token) pairs. The data remains 2D; the model's output space becomes 2D; only the processing architecture (the causal Transformer backbone) remains 1D, and that 1D processing is an implementation detail, not a conceptual constraint.

Why this is a principled departure from prior work: This reframing addresses the ambiguity of "next token" not by resolving it (choosing a better order) but by removing the need to resolve it. The model is never told which position is "next"; it must be prepared for any. This transforms the problem from "given a sequence order, predict the next element" to "given any partial observation, predict the unseen remainder"—a formulation that is identical to masked autoencoding (MAE) but executed autoregressively. The paper explicitly draws this connection (Section 4.2, Abstract): "This learning objective unifies the sequential modeling of autoregressive framework with the holistic self-supervised learning of masked autoencoding, enabling the model to capture comprehensive image semantics via generative training."

Comparison to prior attempts at unifying AR and MAE: The idea of combining autoregressive and masked modeling objectives is not entirely new—approaches like MaskGIT (Chang et al., 2022) and MAGVIT-v2 (Yu et al., 2024) use bidirectional attention with iterative masked prediction, which is neither fully autoregressive nor fully causal. What distinguishes Heptapod's unification is that it achieves MAE-style holistic prediction while preserving full causality in the backbone. The bidirectional processing is confined to the prediction head, which operates on the causal backbone's output. This means the model can be sampled autoregressively (one token at a time in a causal order) while being trained with a loss that demands holistic understanding. This is a non-trivial architectural insight: the separation of causal sequence processing from 2D spatial prediction allows the best of both paradigms without compromise.

Significance beyond images: This reframing suggests a general principle for adapting autoregressive modeling to non-sequential modalities: expand the prediction vocabulary to include the structural dimensions of the data, rather than reducing the data to fit a 1D sequence. For video, this would mean predicting spatiotemporal distributions. For 3D shapes, predicting distributions over spatial coordinates. For graphs, predicting distributions over node-and-edge spaces. The paper doesn't explore these, but the conceptual framework is general.

Evidence that the unification matters: The ablation in Table 1 is the clearest proof. 1D-random already removes the fixed-scan-order bias but still underperforms dramatically (FID 13.07 vs. 3.97 for 2D-random). This isolates the effect of the 2D distribution prediction specifically: even when the model must predict tokens at random positions (reducing local interpolation opportunities), predicting one token at a time is insufficient. The model needs the pressure of predicting all unseen tokens simultaneously to develop the holistic representation that the loss demands. This confirms that the unification with MAE-style prediction is the active ingredient, not just order randomization.

Innovation 3: Demonstrating That a Causal Model Without CFG or Semantic Tokenizers Can Approach the Performance of Non-Causal or Externally Augmented Models

The paper's primary empirical contribution is a existence proof: it demonstrates that a fully causal autoregressive model, trained with only a reconstruction-focused tokenizer and no inference-time CFG, can achieve competitive generative performance on a standard benchmark. At the time of writing, this was not obvious—the dominant trend in the field was toward increasingly sophisticated external semantics (semantic tokenizers, SSL distillation, CFG) and away from strict causality (VAR's bidirectional within-scale attention, MAR's fully bidirectional attention).

What this changes about the research landscape: Before Heptapod, a researcher wanting to build a visual generative model faced an uncomfortable set of tradeoffs:

Use a reconstruction tokenizer with standard AR → poor results without CFG (LlamaGen: 9.38 FID at 3B parameters)
Use a semantic tokenizer → better results but bounded by SSL model quality and reconstruction fidelity (DiGIT: 3.39 FID with DINO tokens)
Abandon causality → better results but lose LLM compatibility (MAR: 2.35 FID with bidirectional attention)
Use diffusion → strong results but architecturally divergent from language models

Heptapod demonstrates that a fourth option exists: keep the reconstruction tokenizer, keep full causality, eliminate CFG, and change the objective. The result (FID 2.70, IS 229.8 at 941M parameters) approaches MAR's best (FID 2.35 at the same parameter count) while maintaining full causality. The gap between "causal and clean" and "best overall" shrinks from enormous (LlamaGen's 9.38 vs. MAR's 2.35 = 4× worse) to modest (2.70 vs. 2.35 = ~15% worse). This is not a marginal improvement—it is a qualitative change in what is possible within the causal paradigm.

Why this is more than a benchmark result: The significance is not the absolute FID number (which is competitive but not state-of-the-art—LightningDiT achieves 2.17, MAETok achieves 2.31). The significance is that this level of performance is achieved without the mechanisms the field had assumed were necessary. It challenges the implicit consensus that causal autoregressive models are inherently disadvantaged for vision and that external semantics or non-causal architectures are required to close the gap. By showing that a single objective change can recover most of the performance difference, the paper suggests that the prior gap was not due to any fundamental limitation of causality but rather due to a misformulated training objective that failed to demand semantic learning.

The specific contrast with prior causal autoregressive models (e.g., LlamaGen): LlamaGen (Sun et al., 2024) represents the best effort at scaling standard next-token prediction on VQ-VAE tokens with a causal Transformer. Its 3B-parameter model (3× larger than Heptapod-H) achieves 9.38 FID. Heptapod-H achieves 2.70 FID—a 3.5× improvement in FID with one-third the parameters. This gap is so large that it cannot be attributed to training details, tokenizer quality, or hyperparameter tuning. It must be attributed to the objective. This provides compelling evidence that the next-2D-distribution objective is not just a theoretical curiosity but a practically necessary reformulation for causal visual autoregressive modeling.

The significance of "no CFG" specifically: The paper reports all results with CFG disabled, which is unusual in the visual generation literature. Most papers report both CFG-enabled and CFG-disabled results, with CFG-enabled being the primary numbers. Heptapod's choice to disable CFG entirely is a deliberate methodological statement: if the model truly learns the data distribution from the generative objective, CFG should be unnecessary. The fact that Heptapod achieves strong results without CFG suggests that the next-2D-distribution objective genuinely produces a better-learned unconditional/conditional distribution, reducing the need for inference-time correction. This is not a small point—it means the model's generative capabilities are intrinsic rather than patched in, aligning with the language modeling philosophy where next-token prediction alone produces coherent text without post-hoc correction.

Evidence and caveats: Table 2 provides the primary evidence. However, the reader should note that MAR (the closest comparator in terms of approach) still maintains an advantage (2.35 vs. 2.70 FID). The paper is transparent that MAR's bidirectional attention provides benefits that causality constrains. The contribution is not "causal beats bidirectional" but rather "causal can be competitive if the objective is right"—a more nuanced and defensible claim.

Innovation 4: The Empirical Discovery That Prediction Window Size, Not Supervision Density, Is the Critical Factor for Semantic Learning

The ablation study in Section 5.3 (Figure 7) produces a finding that is both counterintuitive and practically significant: the spatial extent of the prediction target (window size) dominates the number of supervised tokens in determining whether the model learns global semantics. This is not an obvious result—a reasonable prior would be that more supervision (more positions per sequence contributing to the loss) should always help, especially if the total computational budget is held constant.

The finding in detail: When the prediction window is small (e.g., a 4×4 chunk of 16 positions), increasing the number of supervised tokens within that window from 16 to 32 (doubling the loss signal) produces negligible improvement in final generative quality. When the prediction window is large (the full 16×16 grid of 256 positions), increasing the number of supervised tokens from 1 to 8 produces consistent, meaningful improvements in both FID and IS. The total computational budget for supervision (window size × supervision density) is approximately matched across these comparisons, isolating the effect of window size from total FLOPs.

Why this is non-obvious and important: Intuitively, one might think that predicting 32 tokens in a small window provides a richer learning signal than predicting 1 token in a large window—32 loss terms versus 1. But the experiment shows the opposite is true. The reason, which the paper articulates, is that the semantic content of the learning signal depends on what the model is asked to predict. Predicting tokens within a local chunk, even many of them, only provides gradients about local texture and spatial smoothness—the model can succeed by interpolating within the chunk. Predicting even a single token at an arbitrary position in the full grid forces the model to encode information about the global layout, because that token could be anywhere—spatially distant from the visible context, requiring long-range reasoning. One globally informative supervision signal is worth more than many locally uninformative ones.

What this reveals about the learning dynamics: This finding supports the curse of locality diagnosis in a subtle way. It shows that the problem isn't just that local interpolation provides an easier gradient signal—it's that local supervision provides a different kind of signal that does not contain the information needed for semantic learning, no matter how much of it you have. You cannot learn global structure by summing many local predictions; you need prediction targets that span the global structure. This is reminiscent of the difference between learning from reconstruction (which can be done patch-by-patch) and learning from contrastive objectives (which require global comparisons)—the window size in Heptapod plays an analogous role to the global negative sampling in contrastive learning.

Practical significance: This finding has direct implications for anyone designing autoregressive models for non-sequential data. It suggests that the spatial/temporal extent of the prediction target should be maximized, even at the cost of reduced supervision density per step. A model that predicts the entire future at each step but only supervises on a few positions will outperform a model that predicts a local neighborhood with dense supervision. This is a actionable design principle that generalizes beyond Heptapod to video, audio, and other modalities.

Evidence: Figure 7 shows the main comparison (w256n1 outperforming w64n4, w16n16, w16n32, and w4n64 across all epochs). Figure 8 shows the complementary finding that within the global window, increasing density helps (w256n8 outperforming w256n1). Together, these establish the interaction: window size determines the ceiling of what can be learned; supervision density determines how efficiently you reach that ceiling. But without sufficient window size, no amount of density can compensate.

Innovation 5: The Conceptual Separation of Reconstruction (Tokenizer) from Semantic Learning (Transformer) as a Principled Design Philosophy

While not a technical innovation in itself, the paper articulates and rigorously adheres to a design philosophy that cleanly separates concerns between the tokenizer and the generative model. This philosophy—stated explicitly in the abstract and Section 1—is that the tokenizer should focus solely on faithful reconstruction and the Transformer should learn semantics implicitly from the generative objective, with no external semantic injection at either stage.

Why this is a meaningful contribution rather than just a constraint: This separation was the original philosophy behind language modeling (BPE tokenizers are semantically agnostic), but the visual generation field had drifted away from it, treating semantic tokenizers and CFG as necessary evils. The paper's contribution is to rehabilitate this philosophy as achievable and practical for vision. It demonstrates that the drift toward external semantics was not inevitable—it was a response to a specific failure mode (the curse of locality) that can be addressed at the objective level rather than the tokenizer level.

Comparison to the status quo: The dominant approaches that violate this separation include:

Semantic tokenizers (DiGIT, TiTok, MAGVIT-v2, TokenFlow): Distill knowledge from DINO, CLIP, or other SSL models into the tokenizer, creating tokens that already encode semantic relationships. The Transformer then learns to sequence these pre-semanticized tokens, but the semantics themselves come from the external model, not from the generative objective.
CFG-dependent models (VAR, LlamaGen, most diffusion models): Use inference-time bayesian correction to refine generated distributions, compensating for semantic deficiencies in the learned model.
Representation alignment methods (REPA): Inject SSL representations directly into the generative model's intermediate layers during training, providing an auxiliary semantic signal.

Heptapod rejects all of these, demonstrating that a reconstruction-only tokenizer and a purely generative objective can produce competitive results if the objective is properly formulated. This is not a methodological purity argument—it is an empirical claim that the separation is viable, backed by the results in Table 2.

The significance for multimodal systems: If visual generation can work under the same principles as text generation (semantically agnostic tokenizer + next-distribution prediction objective), then the path to unified multimodal models becomes clearer. A single Transformer could process interleaved text and image tokens generated by semantically agnostic tokenizers for each modality, trained with a unified objective (next-token prediction for text, next-2D-distribution prediction for images). The clean separation of concerns means the tokenizer can be optimized independently for each modality's compression properties without worrying about semantic compatibility, while the Transformer learns cross-modal and within-modal semantics from the joint objective.

The "impossibility triangle" reframing: The paper invokes the concept of an impossibility triangle between reconstruction quality, generation quality, and semantic representation (Section 1, citing prior work). The standard interpretation is that you can optimize at most two of these three—a tokenizer good at reconstruction and semantics will be bad for generation, etc. Heptapod's philosophy effectively exits the triangle by removing "semantic representation" from the tokenizer's responsibilities entirely. The tokenizer only needs reconstruction quality; the Transformer handles semantics through the generative objective. Whether this fully resolves the tradeoffs or simply shifts them is an open question, but the paper's results suggest the shift is productive.

Evidence: Table 2 shows Heptapod using reconstruction-focused tokenizers (VQGAN and VAE) with no SSL distillation, no CFG, achieving competitive results. The comparison with DiGIT (3.39 FID with DINO-based semantic tokens) is particularly informative: Heptapod-H (2.70 FID) outperforms DiGIT (3.39 FID) while using fewer parameters (941M vs. 732M for DiGIT, though Heptapod-H has more) and no external SSL model. This directly validates the claim that the separation of concerns is not just philosophically appealing but practically competitive.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the ImageNet-1K benchmark at 256×256 resolution, comprising approximately 1.28 million training images and 50,000 validation images across 1,000 object classes. The paper uses the standard training split for model training and evaluates generation quality on the validation split using the standard protocol of generating 50,000 images (one per class, conditioned on class labels).
Base model(s). Heptapod uses a decoder-only causal Transformer (GPT-style) at three scales: Heptapod-B (Base: 12 layers, 768 hidden size, 3072 FFN, 12 attention heads, 208M parameters), Heptapod-L (Large: 16 layers, 1024 hidden size, 4096 FFN, 16 attention heads, 478M parameters), and Heptapod-H (Huge: 20 layers, 1280 hidden size, 5120 FFN, 16 attention heads, 941M parameters). The backbone is a standard causal Transformer—no vision-specific modifications. The tokenizer is either a discrete VQGAN (from LlamaGen, rFID 2.19) or a continuous VAE (from MAR, rFID 1.43), both off-the-shelf. The paper explicitly states it "believe[s] this model is representative of the capabilities of many contemporary LLMs" and its purpose is to establish that the objective change, not model scale, drives the performance gains.
Metrics. The paper reports Fréchet Inception Distance and Inception Score as the primary generation quality metrics. FID measures the distributional distance between generated and real images in the feature space of a pre-trained Inception network—lower is better, capturing both fidelity and diversity. IS measures the KL divergence between the conditional class distribution and the marginal class distribution of generated images—higher is better, capturing sample quality and diversity but not fidelity to real images. Both metrics are computed on 50,000 generated samples following the standard ImageNet evaluation protocol. The paper also reports rFID (reconstruction FID) for tokenizers, which measures how well the tokenizer's autoencoder reconstruction preserves image fidelity—this serves as a theoretical lower bound on the generative model's achievable FID.
Baselines. The paper compares against a broad spectrum of generative models on ImageNet-1K 256×256, all evaluated without CFG where the paper's methodology applies:
- Diffusion models: LDM-4 (Rombach et al., 2022), DiT-XL (Peebles & Xie, 2023), SiT-XL (Ma et al., 2024), REPA† (Yu et al., 2025—uses external SSL model), MAETok† (Chen et al., 2025—uses external SSL model), LightningDiT† (Yao et al., 2025—uses external SSL model)
- Non-autoregressive (NAR) masked prediction: MaskGIT (Chang et al., 2022), MAGVIT-v2 (Yu et al., 2024), TiTok (Yu et al., 2024)
- Partially causal (VAR): VAR-d20, VAR-d24, VAR-d30 (Tian et al., 2024—bidirectional attention within each scale)
- Non-causal autoregressive-style (MAR): MAR-B, MAR-L, MAR-H (Li et al., 2024—abandons causal attention)
- Causal autoregressive: RQ-Transformer (Lee et al., 2022), LlamaGen-XL, LlamaGen-XXL, LlamaGen-3B (Sun et al., 2024), DiGIT† (Zhu et al., 2024—uses external SSL model for semantic tokens)
The key comparison group is the causal autoregressive models (LlamaGen, RQ-Transformer, DiGIT), as these share Heptapod's architectural constraints. MAR serves as the upper bound for what is possible without full causality.
Generation budget / compute accounting. The paper uses number of generation steps as the primary measure of inference compute. For all autoregressive models, one step equals one forward pass through the causal Transformer plus prediction head, generating one (position, token) pair. Heptapod requires exactly 256 steps to generate a full 256-token image (one per spatial position), same as standard autoregressive models. For training, the paper controls computational budget through the supervision density ( $n$ , the number of tokens per sequence used for loss computation) and window size ( $w$ , the number of positions the head predicts distributions for). The product $w \times n$ approximates the supervision FLOPs per training step. All models are trained for 800 epochs with batch size 2048 and learning rate $8 \times 10^{-4}$ using AdamW ( $\beta_1 = 0.9, \beta_2 = 0.95$ ).
Cross-validation / statistical protocol. The paper does not employ cross-validation or multiple training runs with error bars. All results are from single training runs of 800 epochs. The primary methodological rigor comes from the controlled ablation studies where individual components are varied while holding others constant (e.g., window size vs. supervision density in Figure 7, head depth in Table 4, tokenizer type in Figure 6). The lack of error bars or multiple seeds means the reported FID/IS numbers should be interpreted as point estimates rather than confidence intervals, though the consistency of trends across multiple comparisons (different window sizes, densities, model scales) provides informal robustness.

Main Quantitative Results

Convergence Efficiency: Discrete vs. Continuous Tokens (Section 5.2, Figure 6)

The paper first establishes that both discrete (VQ) and continuous (VAE) tokenizers are viable under the next-2D-distribution objective, but with significantly different training dynamics.

Headline result: The VQ-based model converges faster initially but plateaus at a higher FID; the VAE-based model converges slower initially but eventually surpasses the VQ model and achieves superior final generative quality.

Detailed findings from Figure 6:

At epoch ~50, the VQ-based Heptapod-L achieves substantially lower FID (~30) and higher IS (~60) compared to the VAE-based model (FID ~80, IS ~35).
The VQ model's FID continues improving smoothly through epoch ~400, reaching approximately 6.5–7.0.
The VAE model shows a dramatic inflection point around epoch 150: its FID drops sharply from ~40 to ~15 between epochs 150–300, then continues improving more gradually, ultimately reaching approximately 4.0 at epoch 800.
By epoch 800, the VAE model achieves FID of ~4.0 vs. the VQ model's ~6.5, and IS of ~180 vs. ~140.
The dotted lines in Figure 6 show each tokenizer's reconstruction FID: ~1.43 for VAE, ~2.19 for VQ. The gap between the models' final generative FID (4.0 vs. 6.5) closely mirrors this reconstruction gap (difference of 0.76 for reconstruction, difference of ~2.5 for generation).

Interpretation: The paper attributes the VQ model's faster initial convergence to cross-entropy providing a "direct and sharply defined gradient signal" (Section 3, confirmed in Section 5.2). The VAE model's eventual superiority is attributed to the VAE tokenizer's higher reconstruction fidelity setting a higher performance ceiling: "the reconstruction quality of the tokenizer sets a practical upper bound on the final generative performance" (Section 5.2). The paper takes this as evidence that if a VQ tokenizer could match VAE reconstruction quality, it might offer "the best of both worlds"—fast convergence and high final performance.

Design consequence: Based on these results, the paper adopts the VAE tokenizer as default for all subsequent experiments (Sections 5.3–5.5, Table 2).

Ablation of Prediction Target: 1D vs. 2D Distribution Prediction (Section 5.3, Table 1)

This experiment isolates the effect of the next-2D-distribution objective by comparing three variants that differ only in what the model is asked to predict and how the generation order is determined.

Headline result: Moving from fixed-scan 1D prediction to randomized 1D prediction provides a substantial but partial improvement (FID 19.23 → 13.07). Moving from randomized 1D prediction to 2D distribution prediction provides a dramatically larger improvement (FID 13.07 → 3.97).

Detailed findings from Table 1 (Heptapod-L, 400 epochs, CFG disabled):

Sampler & Order	FID↓	IS↑
1D-raster	19.23	62.3
1D-random	13.07	91.4
2D-random (Heptapod)	3.97	185.3

1D-raster: Predicts a single next token at a fixed raster scan position. This represents the standard autoregressive visual generation approach (as in LlamaGen). The poor FID of 19.23 and IS of 62.3 confirm the baseline failure: standard next-token prediction with a reconstruction tokenizer does not learn adequate semantics, consistent with the curse of locality diagnosis.
1D-random: Predicts a single next token, but the position is sampled uniformly at random from the remaining grid rather than following a fixed scan. This forces the model to sometimes predict spatially distant tokens, partially disrupting the local interpolation shortcut. The improvement to FID 13.07 and IS 91.4 is substantial (approximately 32% FID reduction), showing that randomization alone helps.
2D-random (Heptapod): Predicts the distribution over all remaining positions in parallel, then samples one (position, token) pair. The leap to FID 3.97 and IS 185.3 is dramatic—a 70% further reduction in FID from 1D-random. This isolates the specific benefit of predicting the full 2D distribution: even when the generation order is already randomized (removing the fixed-scan shortcut), predicting multiple positions simultaneously forces the model to develop representations that are informative about the entire unseen grid, which single-position prediction (even with randomized positions) does not require.

What this proves: The 1D-random vs. 2D-random comparison is the paper's cleanest causal test. Both use randomized generation orders, so the local interpolation shortcut is already disrupted in both conditions. The only difference is whether the model predicts one token or the full distribution. The large performance gap demonstrates that the holistic prediction requirement—not just order randomization—is the active ingredient. This directly supports the paper's central claim that the 2D distribution objective compels semantic learning in a way that 1D prediction cannot.

Note on experimental design: All three variants use the same model architecture (Heptapod-L), same tokenizer (VAE), same training duration (400 epochs), and no CFG. The only difference is the prediction target formulation. This is a well-controlled ablation that cleanly isolates the variable of interest.

Effect of Window Size and Supervision Density (Section 5.3, Figures 7–8)

This pair of experiments investigates the interaction between the spatial extent of the prediction target (window size $w$ ) and the number of tokens per sequence used for loss computation (supervision density $n$ ).

Window Size Ablation with Matched Computational Budget (Figure 7)

Headline result: Under approximately matched total supervision FLOPs (product $w \times n$ held roughly constant), larger prediction windows consistently outperform smaller windows, and the global window ( $w = 256$ ) achieves the best performance despite supervising on only a single token per sequence.

Detailed findings from Figure 7 (Heptapod-B, VAE tokenizer):

The paper compares five configurations:

w256n1 (global head, 1 supervised token per sequence): Best performance across all epochs. At epoch 800, achieves FID ~6.0 and IS ~140.
w64n4 (local head, 4 supervised tokens): Worse than w256n1 at all epochs. At epoch 800, FID ~7.5, IS ~128.
w16n16 (local head, 16 supervised tokens): Worse than w256n1 and w64n4. At epoch 800, FID ~9.0, IS ~120.
w16n32 (local head, 32 supervised tokens—doubled density): Similar to w16n16 at convergence; slightly faster early convergence but converges to the same approximate FID. At epoch 800, FID ~8.8, IS ~118. The additional supervision within the same small window provides negligible final benefit.
w4n64 (local head, 64 supervised tokens): Worst performance. At epoch 800, FID ~11.0, IS ~110.

The non-obvious finding: Doubling supervision density within a small window (w16n16 → w16n32) provides almost no improvement in final quality, while using a larger window with dramatically fewer supervised tokens (w16n16 → w256n1, going from 16 to 1 supervised token) provides a substantial improvement. This demonstrates that the spatial scope of what the model is asked to predict matters more than how many loss terms it receives. One globally informative supervision signal (a random position in the full 256-token grid) is more valuable than 16 locally uninformative signals (all within a 4×4 chunk), even though the latter provides 16× more gradient information about local texture.

Supervision Density Scaling Within Global Window (Figure 8)

Headline result: When the prediction window is global ( $w = 256$ ), increasing supervision density from 1 to 8 tokens per sequence provides consistent, meaningful improvements in both FID and IS, demonstrating that dense supervision is beneficial only when the prediction target already demands global reasoning.

Detailed findings from Figure 8 (Heptapod-B, VAE tokenizer):

Comparing w256n1 vs. w256n8:

At epoch 200: w256n8 achieves FID ~7.5 vs. w256n1 at ~9.0. The higher-density model converges faster.
At epoch 400: w256n8 achieves FID ~5.5 vs. w256n1 at ~6.5. The gap persists and widens.
At epoch 800: w256n8 achieves FID ~5.0 vs. w256n1 at ~6.0. The final performance gap is approximately 1.0 FID point.
IS shows a parallel trend, with w256n8 achieving ~150 vs. w256n1 at ~140 at epoch 800.

Synthesis of Figures 7 and 8: Taken together, these two figures reveal a critical interaction: window size determines the ceiling of semantic learning; supervision density determines how efficiently you approach that ceiling, but the ceiling itself is set by the spatial scope of the prediction task. With a small window, no amount of supervision can raise the ceiling because the model never needs to reason about long-range dependencies—every supervised token is spatially near the context. With a large window, the model must reason globally to predict even a single distant token, so additional supervision provides richer gradients that accelerate and improve this learning process.

Practical implication: The paper concludes that "the spatial extent of the prediction target is more critical than the supervision density to learn long-term semantics" (Section 5.3). Based on this, the global prediction head (w256) is adopted as the default for all subsequent experiments, with w256n1 for main results (Table 2) and w256n8 explored as a scaling direction (Section 5.5).

Main Benchmark: ImageNet 256×256 Class-Conditional Generation Without CFG (Section 5.4, Table 2)

This is the paper's primary quantitative evaluation, comparing Heptapod against the broader landscape of generative models on the standard ImageNet benchmark.

Headline result: Heptapod-H (941M parameters, causal attention, VAE tokenizer, no CFG, no semantic tokenizer) achieves FID 2.70 and IS 229.8, dramatically outperforming all prior causal autoregressive models and approaching the performance of non-causal or externally-augmented approaches.

Detailed findings from Table 2:

Comparison with causal autoregressive models (the primary apples-to-apples comparison):

Model	#Params	FID↓	IS↑
RQ-Transformer	1.4B	8.71	119.0
RQ-Transformer	3.8B	7.55	134.0
LlamaGen-XL	775M	15.55	79.2
LlamaGen-XXL	1.4B	14.65	86.3
LlamaGen-3B	3.0B	9.38	112.9
DiGIT†	732M	3.39	205.96
Heptapod-B (Ours)	208M	5.85	149.6
Heptapod-L (Ours)	478M	3.62	190.8
Heptapod-H (Ours)	941M	2.70	229.8

Heptapod-H (2.70 FID) outperforms LlamaGen-3B (9.38 FID) by a factor of ~3.5× in FID while using less than one-third the parameters (941M vs. 3.0B). This gap is decisive—it cannot be explained by tokenizer quality differences (both use reconstruction-focused VQ-VAE variants), architecture details, or training hyperparameters. It directly measures the benefit of the next-2D-distribution objective over standard next-token prediction.
Heptapod-H (2.70 FID) also outperforms DiGIT (3.39 FID), which uses semantic tokens distilled from DINO (an external SSL model, marked with †). Heptapod achieves better performance with fewer external dependencies, validating the paper's claim that semantics can emerge from a well-designed generative objective without being engineered into the tokenizer.
The scaling trend across Heptapod sizes is healthy: Base (208M, 5.85 FID) → Large (478M, 3.62 FID) → Huge (941M, 2.70 FID). Doubling parameters from Large to Huge reduces FID by ~0.9 points. The trend does not appear to be saturating, suggesting further scaling could yield additional improvements.

Comparison with non-causal and partially-causal autoregressive-style models:

Model	#Params	FID↓	IS↑	Causal?
VAR-d30	2.0B	5.26	175.6	Partially (bidirectional within scale)
MAR-B	208M	3.48	192.4	No (bidirectional)
MAR-L	479M	2.60	221.4	No (bidirectional)
MAR-H	943M	2.35	227.8	No (bidirectional)
Heptapod-H (Ours)	941M	2.70	229.8	Yes (causal)

Heptapod-H (2.70 FID) approaches MAR-H (2.35 FID) at the same parameter count (941M vs. 943M). The gap is 0.35 FID points, or approximately 15% relative. This is notable because MAR explicitly abandons causal attention and uses bidirectional attention in the backbone—a strictly more powerful architectural choice for generation quality. The fact that Heptapod nearly matches MAR while maintaining full causality is the paper's strongest evidence that the next-2D-distribution objective recovers most of what is lost by enforcing causality.
Heptapod-L (3.62 FID) outperforms MAR-B (3.48 FID) despite MAR-B using bidirectional attention—but Heptapod-L has more than twice the parameters (478M vs. 208M), so the comparison is not parameter-matched.
Heptapod-H (2.70 FID) substantially outperforms VAR-d30 (5.26 FID), which uses 2.0B parameters (more than twice Heptapod-H) but only partial causality (bidirectional within each scale). This demonstrates that the objective matters more than model scale or relaxed causality for this task.

Comparison with diffusion models and externally-augmented approaches:

Model	#Params	FID↓	IS↑	External SSL?
LDM-4	400M	10.56	103.5	No
DiT-XL	675M	9.62	121.5	No
SiT-XL	675M	8.30	131.7	No
REPA†	675M	5.90	157.8	Yes
MAETok†	675M	2.31	216.5	Yes
LightningDiT†	675M	2.17	205.6	Yes
Heptapod-H (Ours)	941M	2.70	229.8	No

Heptapod-H (2.70 FID) outperforms all standard diffusion models without external SSL (LDM-4, DiT-XL, SiT-XL) by substantial margins. This is notable but expected—diffusion models at this parameter scale are known to be outperformed by autoregressive and masked approaches on ImageNet without CFG.
MAETok (2.31 FID) and LightningDiT (2.17 FID) achieve better FID than Heptapod-H, but both use external SSL models (marked with †) to inject semantic knowledge. Heptapod achieves competitive results without this dependency.
Heptapod-H achieves the highest IS (229.8) among all models in the table—higher than MAR-H (227.8), MAETok (216.5), and LightningDiT (205.6). The paper does not extensively discuss IS, but higher IS generally indicates better sample quality and diversity as measured by the Inception classifier's confidence. This is a point in Heptapod's favor, though IS is known to be less reliable than FID as a holistic quality metric.

Key observations about the results:

The causal-to-non-causal gap narrows dramatically. In the prior generation of causal models, the gap between causal (LlamaGen-3B: 9.38) and non-causal (MAR-H: 2.35) was enormous—a 4× difference in FID. Heptapod shrinks this gap to 15% (2.70 vs. 2.35), demonstrating that most of the prior gap was attributable to the objective, not the causality constraint.
Scaling behavior appears healthy. The three model sizes (208M, 478M, 941M) show monotonic improvement in both FID and IS. The curve does not appear to be flattening, suggesting further scaling (to >1B parameters) could close the remaining gap with MAR and externally-augmented approaches.
The tradeoff with external SSL models is favorable. Heptapod-H outperforms DiGIT (which uses DINO-based semantic tokens) by 0.69 FID points, and approaches MAETok and LightningDiT (which also use external SSL) without that dependency. This validates the paper's central philosophy: external semantics are not necessary if the generative objective is properly formulated.
All Heptapod results are without CFG. This is explicitly noted in the table and throughout the paper. The fact that competitive results are achieved without CFG is a significant methodological statement, as the paper argues CFG masks deficiencies in the learned distribution. The strong IS scores (Heptapod-H at 229.8 is the highest in the table) suggest the model achieves diversity without CFG's tendency to reduce it.

Scaling Supervision Density Under Global Window (Section 5.5, Figure 8)

This experiment, discussed above in the ablation context, is also a forward-looking scaling result that points toward future improvements.

Headline result: Increasing supervision density from 1 to 8 tokens per sequence under the global prediction window (w256) yields consistent improvements in both convergence speed and final quality, suggesting that the w256n1 results in Table 2 (the main benchmark) are not the ceiling for what the objective can achieve.

Quantitative detail: At epoch 800, w256n8 achieves approximately FID 5.0 and IS 150 compared to w256n1's FID 6.0 and IS 140 (Heptapod-B, Figure 8). If this relative improvement (~17% FID reduction from increased supervision) scales to larger models, Heptapod-H with w256n8 might achieve FID in the 2.2–2.4 range—competitive with or surpassing MAR-H (2.35). However, the paper notes that "scaling supervision density markedly accelerates convergence and improves final performance, [but] also increases the computational cost," and does not report larger-scale results with increased density. The computation-vs-performance tradeoff of higher density at the Huge scale remains an open question.

Why this matters: This result, combined with the window size ablation, establishes a roadmap for future work: maximize window size first (to establish the semantic learning ceiling), then increase supervision density (to approach that ceiling more efficiently). The paper's main results (Table 2) use w256n1, which is the minimum density for the global head. This means the reported FID of 2.70 for Heptapod-H is a lower bound on what the framework can achieve with optimized density.

Ablation Studies and Robustness Checks

Prediction head depth allocation (Appendix A, Table 4): Varying the split of layers between the causal Transformer backbone and the 2D prediction head (total depth fixed at 32 for Heptapod-L) reveals that a balanced split (16 backbone / 16 head) achieves the best performance (FID 3.62, IS 190.8). Shifting layers toward the backbone (24/8) degrades FID to 4.88 and IS to 158.93—a meaningful but not catastrophic drop. An extreme backbone-heavy split (31/1) causes a sharp collapse: FID 14.74, IS 98.15. This demonstrates that the prediction head requires sufficient depth to effectively translate 1D causal hidden states into 2D spatial distributions. A single-layer head cannot adequately perform this translation, regardless of backbone capacity. The balanced split suggests that the 2D prediction task is approximately as complex as the causal sequence modeling task, requiring comparable representational capacity.

Tokenizer type (discrete VQ vs. continuous VAE) (Section 5.2, Figure 6): (Discussed in detail above under Main Quantitative Results.) Key finding: both tokenizer types are viable, but VQ converges faster (cross-entropy gradient) while VAE achieves higher final quality (better reconstruction fidelity). The final performance gap mirrors the reconstruction quality gap. This ablation validates that the next-2D-distribution objective works with both discrete and continuous token representations, supporting the paper's claim that the framework is agnostic to tokenization type.

Generation order (1D-raster vs. 1D-random vs. 2D-random) (Section 5.3, Table 1): (Discussed in detail above.) This is the most critical ablation because it isolates the effect of the next-2D-distribution prediction from order randomization. The result that 2D-random dramatically outperforms 1D-random (FID 3.97 vs. 13.07) is the strongest evidence for the paper's central mechanism.

Window size vs. supervision density (Section 5.3, Figures 7–8): (Discussed in detail above.) The finding that window size dominates supervision density—and that higher density only helps when the window is already large—is the paper's most practically significant and non-obvious ablation result. It provides a design principle (maximize window size first) that is both actionable and likely to generalize beyond image generation.

Model scale (Section 5.4, Table 2): The three model sizes (Base: 208M, Large: 478M, Huge: 941M) demonstrate smooth scaling behavior. No ablation of other architectural hyperparameters (attention head dimension, MLP expansion ratio, number of diffusion head blocks) is reported, suggesting the configurations were adopted from MAR without extensive tuning. The paper does not explore whether the optimal depth split between backbone and head changes with model scale (the 16/16 split was only tested at the Large scale), though the consistent performance across scales suggests the balanced split is robust.

CFG ablation (Table 2, implicitly): All Heptapod results are reported without CFG, but no explicit CFG-on vs. CFG-off comparison is reported for Heptapod. The paper's Figure 1 (Top Right) shows CFG ablation results for prior models (VAR, LlamaGen), demonstrating their steep performance drops without CFG. The implicit claim is that Heptapod does not need CFG, but without a within-model CFG ablation, the reader cannot distinguish between "Heptapod genuinely doesn't need CFG" and "Heptapod would benefit from CFG but the paper chose not to use it." This is a methodological gap—even if CFG is philosophically undesirable, reporting the CFG-on performance would establish an upper bound and quantify how much the model depends on external correction.

Critical Assessment

Claim 1: The next-2D-distribution prediction objective enables causal autoregressive models with reconstruction tokenizers to learn visual semantics implicitly, eliminating the need for external semantics (CFG, semantic tokenizers).

What the experiments actually demonstrate: The ablation in Table 1 provides strong causal evidence that the 2D distribution objective (2D-random) dramatically outperforms 1D prediction (1D-random and 1D-raster) under otherwise identical conditions. This establishes that the objective change is responsible for the performance improvement—it is not an artifact of model scale, architecture, or tokenizer quality.

However, the claim that this improvement is specifically due to "learning visual semantics implicitly" is an interpretation rather than a directly measured quantity. The paper provides suggestive evidence (Figure 3: attention maps show local concentration with VQ-VAE tokens vs. long-range attention with semantic tokens; Heptapod's attention maps are not shown), but does not include a direct probe of semantic understanding—for example, linear probing of intermediate representations, object detection or segmentation performance on generated images, or analysis of whether the model's internal representations correlate with semantic categories. The FID and IS metrics measure sample quality and diversity, not semantic understanding per se. It is possible that Heptapod achieves better FID through improved texture and structure modeling without necessarily developing the kind of semantic representations that would transfer to downstream tasks. The paper's central narrative—that semantics "emerge" inside the Transformer—would be strengthened by experiments demonstrating that Heptapod's internal representations encode object identity, spatial relationships, or other semantic properties beyond what is needed for generation quality.

What would strengthen this claim: (1) Linear probing or fine-tuning on ImageNet classification using Heptapod's frozen features, comparing to representations from semantic tokenizer-trained models. (2) Analysis of attention maps from Heptapod's causal Transformer to verify that they attend to semantically related but spatially distant regions (analogous to Figure 3, Right). (3) Controlled experiments measuring whether Heptapod generates images with correct object counts, spatial relationships, and attribute bindings—classes of errors that indicate semantic understanding failures specifically. Without these, the claim of "semantic emergence" is supported by generative quality metrics but not by direct semantic probing.

Claim 2: Heptapod demonstrates that a causal model without CFG or semantic tokenizers can approach the performance of non-causal or externally augmented models.

What the experiments actually demonstrate: Table 2 provides strong evidence that this claim is true on ImageNet-1K 256×256 class-conditional generation measured by FID and IS. Heptapod-H (2.70 FID, causal) is within 15% of MAR-H (2.35 FID, non-causal) at the same parameter count, and outperforms many externally-augmented approaches (DiGIT, REPA, VAR). This is a genuine advance over prior causal models (LlamaGen-3B at 9.38 FID).

Conditions and caveats:

Single benchmark, single resolution. All results are on ImageNet-1K at 256×256. The paper does not report results on other generation benchmarks (e.g., ImageNet at higher resolutions, COCO, LAION subsets, or text-to-image generation). Whether the approach transfers to higher resolutions (where the 2D grid size grows, increasing the prediction head's cost quadratically) or to more diverse datasets (where the "curse of locality" might manifest differently due to greater visual diversity) is unknown.
The MAR gap persists. While the gap is much smaller than for LlamaGen, Heptapod-H still underperforms MAR-H by 0.35 FID points. For applications where FID is a primary metric, MAR's bidirectional attention provides a genuine advantage that the next-2D-distribution objective does not fully recover. The paper's claim is "approaches" rather than "matches," which is accurate, but the practical significance of the remaining gap depends on the application.
The externally-augmented ceiling may be moving. LightningDiT (2.17 FID) and MAETok (2.31 FID) still outperform Heptapod, and both use external SSL. The paper's philosophical argument is that external semantics should be unnecessary, but empirically, they still provide measurable benefits. Whether this gap can be closed by scaling Heptapod (higher supervision density, larger models, longer training) or represents a fundamental ceiling for SSL-free causal approaches is not resolved by the current experiments.
Model family specificity. All results use a GPT-style causal Transformer with VAE tokenizer. Whether the findings transfer to other architectures (e.g., different attention mechanisms, different backbone designs) or tokenizer families is untested.

Claim 3: The spatial extent of the prediction target (window size) is more critical than supervision density for learning visual semantics.

What the experiments actually demonstrate: Figures 7 and 8 provide clean evidence for this claim within the specific experimental setup (Heptapod-B, VAE tokenizer, 800 epochs, ImageNet-1K). The finding is robust across the tested configurations: larger window with lower density consistently outperforms smaller window with higher density at matched computational budget, and increasing density only helps within the large-window condition.

Caveats:

Single model scale (Base). The window-size-vs-density ablation is only reported for Heptapod-B (208M parameters). It is possible that at larger scales, the relative importance of window size vs. density changes—larger models might extract more signal from dense local supervision, or might saturate the benefits of global prediction earlier. The paper implicitly assumes the finding scales with model size (since the global head is used for all subsequent experiments), but this is an extrapolation rather than a demonstrated fact.
Dichotomous comparison. The paper compares a single global configuration (w256) against a few local configurations (w64, w16, w4). This establishes the direction (bigger window is better) but does not characterize the shape of the relationship—for example, is w128 nearly as good as w256, or is there a threshold effect? Does the benefit of window size saturate at full-image coverage, or would an even larger window (if the grid were larger) continue to help? These questions are relevant for scaling to higher resolutions.
The "matched computational budget" approximation. The paper uses the product $w \times n$ as a proxy for supervision FLOPs, but this ignores the cost of the prediction head's self-attention (which scales with $w$ and with the number of layers in the head). A w256n1 configuration with a deep bidirectional Transformer head may have different actual FLOPs than a w16n16 configuration with a shallow cross-attention head. The paper acknowledges this indirectly by noting the head depth ablation (Table 4) but does not compute exact FLOPs for each configuration in Figure 7.

Claim 4: The framework is tokenizer-agnostic—both discrete (VQ-VAE) and continuous (VAE) tokenizers are viable.

What the experiments actually demonstrate: Figure 6 shows that both tokenizer types converge and achieve non-trivial performance. The VQ model achieves FID ~6.5, which is substantially better than LlamaGen's results with similar tokenization, confirming that the objective (not just the tokenizer) is driving the improvement.

Caveats:

The final performance gap is significant. The VAE model (FID ~4.0) clearly outperforms the VQ model (FID ~6.5) at convergence—a 60% relative improvement. Calling them both "viable" is accurate but understates the practical difference. For practitioners choosing a tokenizer, the VAE's 60% better FID would be decisive.
The VQ model's plateau suggests a systematic limitation. The VQ model in Figure 6 appears to plateau around FID 6.5 while the VAE model continues improving. The paper attributes this to reconstruction fidelity, but alternative explanations—codebook collapse, limited codebook expressiveness, or the categorical bottleneck—are not ruled out. If VQ performance is fundamentally limited by the discrete bottleneck rather than reconstruction quality, then improving VQ reconstruction might not close the gap, and the "best of both worlds" scenario (VQ speed + VAE quality) might not be achievable.
Single VQ tokenizer tested. The paper uses only the LlamaGen VQGAN tokenizer. Other VQ-VAE variants (e.g., with larger codebooks, improved quantization techniques like finite scalar quantization) might produce different results. The claim of tokenizer-agnosticism is supported by the two tested tokenizers but would be strengthened by testing additional variants.

Missing Experiments That Would Strengthen the Paper

CFG ablation for Heptapod: What is Heptapod's performance with CFG? If CFG provides little or no improvement, that would strongly support the claim that the objective eliminates CFG dependence. If CFG provides substantial improvement, it would suggest the curse of locality is mitigated but not eliminated.
Attention map visualization for Heptapod: The paper shows attention maps for standard autoregressive models with VQ-VAE vs. semantic tokenizers (Figure 3, Right) but does not show Heptapod's attention maps. This is a notable omission—the central claim is that the next-2D-distribution objective forces the model to learn long-range dependencies, and attention maps would provide direct visual evidence.
Higher resolution experiments: All results are at 256×256. The 2D grid size (16×16) is relatively small. At 512×512 or 1024×1024, the grid grows to 32×32 or 64×64 tokens (1024 or 4096 tokens), making the global prediction head's bidirectional self-attention quadratically more expensive. Whether the approach scales computationally to higher resolutions—and whether the global prediction window remains critical at larger grid sizes—is unexplored.
Text-to-image generation: The paper focuses exclusively on class-conditional generation. Extending to text-to-image would test whether the next-2D-distribution objective works with more complex conditional information and whether it scales to larger, more diverse datasets (e.g., LAION, COCO). This is particularly relevant because the paper's philosophical motivation (unified multimodal LLMs) points toward text-to-image as the natural application.
Semantic probing experiments: Linear classification, object detection, or segmentation on generated images would test whether Heptapod's internal representations encode semantic information beyond what is needed for generation quality. This would directly support the "semantics emerge implicitly" narrative.
Training efficiency comparison: The paper discusses convergence speed differences between VQ and VAE tokenizers but does not report wall-clock training time, total FLOPs to reach a given FID, or GPU memory consumption. For practitioners deciding whether to adopt the method, these practical metrics matter as much as final FID.
Stochasticity analysis: All results are from single training runs. The significance of the 0.35 FID gap between Heptapod-H and MAR-H depends on run-to-run variance, which is not reported. In the generative modeling literature, FID standard deviations of 0.1–0.3 across seeds are common, so the gap might be partially within noise.
Generation order analysis: Heptapod learns to choose its own generation order (by sampling from the predicted position distribution). The paper does not analyze what kinds of generation orders emerge—do they correspond to coarse-to-fine, center-out, or some other pattern? This would provide insight into what the model learns and connect to prior work on generation order in autoregressive models.

Overall Assessment

The experimental section provides convincing evidence for the paper's core mechanistic claim: that the next-2D-distribution objective substantially improves causal autoregressive image generation compared to standard next-token prediction, and that this improvement is attributable to the holistic prediction requirement rather than to order randomization or other confounds. The ablation design is clean, the comparisons are well-controlled, and the results are quantitatively decisive (Table 1, Figures 7–8).

The benchmarking results (Table 2) establish that this objective change enables causal models to approach the performance of non-causal or externally-augmented approaches, representing a genuine advance in the state of the art for fully causal visual generation. However, the remaining gaps (to MAR's bidirectional attention, to LightningDiT's external SSL) suggest the objective is a significant step toward principled visual autoregressive modeling but not a complete solution to all its challenges.

The primary limitations of the experimental evaluation are: (1) the single-benchmark, single-resolution scope, (2) the absence of direct semantic probing to support the "semantics emerge" narrative, (3) the lack of CFG ablation for Heptapod itself, (4) unexplored scaling dimensions (higher resolution, larger supervision density at Huge scale, different tokenizers), and (5) single-run results without variance estimates. These limitations do not undermine the paper's claims, but they bound their generality and leave open questions that future work will need to address.

6. Limitations and Trade-offs

The Cost of Difficulty Analogy: Prediction Window Size Creates a Quadratic Scaling Bottleneck the Paper Does Not Resolve

The assumption or constraint. Heptapod's central finding—that the global prediction window (w=256, the full 16×16 grid) is necessary for semantic learning—carries a computational cost that scales quadratically with image resolution. The global prediction head uses a bidirectional Transformer with full self-attention across all 256 positions at every autoregressive step, costing O(N²) in the grid size N. The paper acknowledges this tension in Section 4.3 when introducing the local chunk-based head as an alternative designed "to improve efficiency" because the global head's "computational cost scales with the grid size." However, Section 5.3 then demonstrates that the local head—the efficiency-motivated alternative—performs substantially worse: w256n1 achieves FID ~6.0 while w16n16 reaches ~9.0, and even w64n4 only reaches ~7.5 (Figure 7, Heptapod-B). The paper's own ablation forces a difficult choice: use the global head for quality but accept the quadratic cost, or use the local head for efficiency but accept worse generation.

The consequence. At the 256×256 resolution used in the paper (16×16 latent grid, 256 tokens), the global head's O(256²) = 65,536 pairwise attention operations per step are manageable. But ImageNet generation at higher resolutions—512×512, 1024×1024—is a standard benchmark that the paper does not evaluate. At 512×512 resolution, a typical VAE tokenizer would produce a 32×32 latent grid of 1,024 tokens, making the global head's self-attention cost O(1024²) = 1,048,576 operations per step—a 16× increase from 256×256. At 1024×1024 (64×64 grid, 4,096 tokens), the cost would be O(4096²) ≈ 16.8M operations per step—a 256× increase. Since the head runs at every autoregressive step (256 steps for a 256-token image, 1,024 steps for a 1,024-token image), the total inference cost scales as O(N³) for the global head: N steps × O(N²) self-attention per step.

This scaling behavior means that Heptapod's headline approach (global prediction head) is effectively restricted to low-resolution generation unless the head architecture is modified. A practitioner wanting to generate 1024×1024 images faces three unattractive options: (1) use the global head and accept ~256× more inference compute than at 256×256, (2) switch to the local head and accept the quality degradation documented in Figure 7, or (3) develop a hierarchical or multi-scale variant not explored in the paper.

What evidence exists in the paper. The paper provides no higher-resolution experiments (Table 2: all results at 256×256). Figure 7 shows the quality degradation from using smaller windows, quantifying the penalty for the local head (the only efficiency-motivated alternative presented). Section 5.3 explicitly notes the scaling concern: "While this global head captures long-range dependencies effectively, its computational cost scales with the grid size." The local chunk-based head is presented as the mitigation, but its performance penalty is documented in the same section. The paper does not explore intermediate architectures (e.g., hierarchical prediction heads, factorized attention, or progressive generation across scales) that might preserve global context with sub-quadratic scaling.

Mitigation status. The paper does not resolve this tradeoff. Section 6.2 gestures at audio applications where "the chunk-based approach... naturally aligns with this characteristic [long or unbounded sequences]," but this is framed as a feature for audio, not a resolution for higher-resolution images. The local head exists as a fallback but is shown to underperform. No architectural innovations to reduce the global head's cost (e.g., linear attention, sparse attention, or learned sparsity patterns) are explored. The paper acknowledges the tradeoff transparently but leaves it as an open problem for future work.

The "Semantics Emerge Implicitly" Claim Lacks Direct Evidence Beyond Generation Quality Metrics

The assumption or constraint. The paper's central narrative—stated in the abstract, Section 1, and throughout—is that the next-2D-distribution objective enables the causal Transformer to "learn visual semantics implicitly" (Section 5.4) without needing external SSL models or CFG. The introductory question the paper poses is: "Can we devise a visual generative learning paradigm... where the tokenizer is dedicated solely to faithful reconstruction, and complex semantics emerge implicitly within the Transformer through the next-token prediction objective?" (Section 1). The paper claims Heptapod answers this affirmatively.

However, the evidence provided for "semantic emergence" is entirely through generation quality metrics—FID and IS—which measure the distributional similarity and sample quality of generated images. These metrics are sensitive to texture fidelity, mode coverage, and low-level realism, but they do not directly probe whether the model's internal representations encode semantic properties (object identities, spatial relationships, attribute bindings, part-whole hierarchies). A model could achieve good FID by producing images with realistic textures and plausible local structure while still making systematic semantic errors—generating dogs with too many legs, cars with inconsistent perspectives, or objects in physically impossible spatial configurations.

The consequence. The paper's philosophical contribution—that the language modeling principles of separating compression from semantic learning can be preserved for vision—rests on the claim that semantics are learned, not just that generation quality is adequate. If Heptapod achieves good FID primarily through improved texture and local structure modeling (the very "local interpolation" the paper criticizes in standard autoregressive models), while still failing on systematic semantic reasoning tasks, the narrative of "semantic emergence" would be overstated. The distinction matters because the paper positions Heptapod as a principled alternative to semantic tokenizers, arguing that SSL-based tokenizers "pre-package long-range semantic relationships" (Section 4.1) while Heptapod learns them from scratch. If Heptapod does not actually learn rich semantic representations—or learns them only to the extent required for generation quality on a constrained-class dataset like ImageNet—then the claimed advantage over semantic tokenizers (beyond avoiding a dependency on external models) is unsubstantiated.

Specific failure modes that generation metrics might mask:

Attribute binding errors: Generating a "white cat and black dog" as a "black cat and white dog" because the model captures texture statistics but not compositional semantics.
Counting errors: Generating images with incorrect numbers of objects (ImageNet classes have consistent object counts).
Viewpoint consistency: Generating objects from impossible or inconsistent viewpoints.
Part-whole relationships: Misplacing object parts (wheels on a car, eyes on a face) while maintaining local texture quality.

What evidence exists in the paper. The paper provides:

Figure 3 (Left): Cosine similarity maps of tokenizers, showing that VQ-VAE tokens have strong local correlations. This supports the curse of locality diagnosis but does not measure semantic learning in Heptapod.
Figure 3 (Right): Attention maps from autoregressive Transformers trained on VQ-VAE vs. semantic tokens. No attention maps are shown for Heptapod itself. This is a notable omission: if Heptapod's causal Transformer learns long-range semantic dependencies, its attention maps should resemble the semantic-token case (attending to spatially distant but semantically related regions). The absence of this visualization is a missing piece of evidence for the central claim.
FID and IS improvements (Table 2, Figures 6–8): These show generation quality improvement but do not isolate semantic understanding from other factors (texture quality, mode coverage, training stability).
Qualitative samples (Figure 9): The paper includes generated examples, but without systematic analysis or comparison to baselines on semantic correctness criteria.

No experiments directly probe semantic understanding:

No linear probing or fine-tuning of Heptapod's features on ImageNet classification to test whether internal representations encode class-discriminative semantic information.
No evaluation on semantic consistency benchmarks (e.g., attribute binding, spatial relationship understanding, counting accuracy).
No comparison of Heptapod's internal representations against those from models trained with semantic tokenizers (e.g., via representational similarity analysis).

Mitigation status. The paper does not acknowledge this gap as a limitation. The claim that semantics "emerge implicitly" is treated as supported by the FID/IS results and the conceptual argument (that the holistic prediction objective must require global understanding). The missing evidence is a genuine gap between what the paper claims and what the experiments demonstrate. Future work could close this gap with semantic probing experiments, attention map analysis for Heptapod, or systematic evaluation of semantic error patterns in generated images.

The Framework Has Only Been Validated on a Single Benchmark (ImageNet-1K at 256×256) With a Single Model Family

The assumption or constraint. All experiments in the paper—convergence analysis (Figure 6), ablations (Figures 7–8, Table 1), and benchmarking (Table 2)—are conducted exclusively on ImageNet-1K class-conditional generation at 256×256 resolution using GPT-style causal Transformers. The paper does not report results on any other dataset, any other resolution, any other generation task (e.g., text-to-image, unconditional generation, inpainting), or any other model architecture. The VAE and VQ-VAE tokenizers are both off-the-shelf components from prior work (MAR and LlamaGen, respectively), but the backbone is a single architectural family (standard decoder-only Transformer).

The consequence. This single-benchmark scope creates several important uncertainties:

Dataset diversity: ImageNet-1K is a curated dataset of 1,000 object classes with relatively clean, object-centric images. Natural images in the wild (e.g., from LAION, COCO, or web-scale datasets) contain significantly more compositional complexity, scene clutter, text, multiple objects with complex spatial relationships, and long-tail visual concepts. The curse of locality analysis (Section 4.1) argues that VQ-VAE tokens have strong local correlations because "neighboring patches... are overwhelmingly similar"—this property might be less pronounced in complex scenes with many objects, occlusions, and varied textures. In such settings, the local interpolation shortcut might be less available, potentially reducing the relative advantage of the 2D distribution objective over standard next-token prediction (since local prediction would already be harder). Conversely, the longer-range dependencies in complex scenes might be even harder to capture, potentially making the 2D objective even more important. The paper provides no evidence either way.

Task diversity: Class-conditional generation is the simplest form of conditional image generation. The paper's philosophical motivation—enabling unified multimodal LLMs—points toward text-to-image generation, in-context visual generation, and instruction-following visual tasks. Extending Heptapod to text conditioning would require integrating cross-attention or prefix-based conditioning into the causal backbone, which is architecturally straightforward but untested. The interaction between the next-2D-distribution objective and complex natural language conditions (which introduce their own long-range semantic dependencies) is unknown.

Resolution scaling: As discussed in the first limitation above, the global prediction head's quadratic cost creates a resolution barrier. But even apart from computational cost, the paper provides no evidence that the 2D distribution objective remains beneficial at larger grid sizes. At a 32×32 grid (1,024 tokens), predicting the distribution over all remaining positions might become substantially harder, potentially requiring larger models or different head architectures. The paper's window-size ablation (Figure 7) only studied up to w=256, which is the full grid at 16×16. The optimal window size at larger grids might not scale linearly with resolution.

Architecture specificity: The paper uses standard GPT-style Transformers without vision-specific modifications. Whether the benefits of the next-2D-distribution objective transfer to other causal architectures (e.g., state-space models, linear attention, or hybrid architectures) is untested. The ablation in Table 4 shows that the depth split between backbone and head matters significantly—the balanced 16/16 split outperforms the 31/1 split by a large margin (FID 3.62 vs. 14.74). This sensitivity suggests the objective might interact with architectural choices in non-obvious ways that would need to be re-tuned for different backbone designs.

What evidence exists in the paper. All evidence comes from a single experimental configuration. The paper does not discuss these generalization limitations, nor does it claim the results generalize (which is appropriate, but the absence of discussion is itself a limitation). The model scaling results (Base → Large → Huge in Table 2) provide within-configuration evidence that the approach scales with model size on ImageNet, but this does not constitute cross-domain or cross-task validation.

Mitigation status. The paper does not acknowledge this as a limitation. The abstract and conclusion present the method as a general framework for "language modeling on visual signals" without qualifying the scope of validation. Given ImageNet-1K's role as the standard benchmark for image generation methods, single-benchmark evaluation is common in the field and not inherently disqualifying. However, the paper's strong philosophical claims about visual semantics, multimodal unification, and principled language modeling would benefit from at least preliminary evidence on a more diverse generation task.

The Supervision Density That Maximizes Performance (w256n8) Is Not Used for the Main Results, and Its Scaling Behavior at Larger Model Sizes Is Unexplored

The assumption or constraint. The paper's main benchmarking results in Table 2 use the w256n1 configuration (global prediction window, loss computed on a single randomly sampled token per sequence). However, Section 5.5 and Figure 8 demonstrate that increasing supervision density to w256n8 (loss computed on 8 tokens per sequence) provides consistent improvements: at epoch 800, Heptapod-B achieves FID ~5.0 with w256n8 versus ~6.0 with w256n1—a ~17% relative reduction in FID. The paper does not report w256n8 results for the larger models (Heptapod-L and Heptapod-H) used in the main benchmark.

The consequence. The reported results in Table 2—Heptapod-H at 2.70 FID, Heptapod-L at 3.62 FID—represent a lower bound on what the framework can achieve at those model scales. If the ~17% relative improvement from n8 over n1 observed at the Base scale transfers to larger models, Heptapod-H with w256n8 might achieve FID in the 2.2-2.3 range. This would:

Close or eliminate the gap with MAR-H (2.35 FID), the best non-causal autoregressive model. The paper's central claim—that a causal model can approach non-causal performance—would be even stronger if Heptapod matched or exceeded MAR at equal parameter count.
Approach or surpass externally-augmented methods like MAETok (2.31 FID). This would substantially strengthen the paper's philosophical argument that external semantics are unnecessary.
Establish a clearer scaling trend by providing data points at higher supervision densities across model scales, enabling practitioners to make informed compute-vs-quality tradeoffs.

The absence of these results leaves a gap: the paper cannot distinguish between "Heptapod fundamentally cannot close the remaining gap with MAR" and "Heptapod can close the gap but the paper did not run the experiment." This is particularly relevant because Section 5.5 frames supervision density scaling as a direction for "future research," implying the paper's authors believe further improvements are possible, but they do not quantify the potential gain at scale.

Why this matters for the paper's claims. The paper argues that the objective change (1D → 2D distribution prediction) is the key innovation, and that external semantics are unnecessary. But the measurable gap between Heptapod-H (2.70) and MAR-H (2.35) could be used to argue that bidirectional attention still provides a meaningful advantage that the objective cannot fully recover. If w256n8 at Heptapod-H scale closes this gap, that counterargument weakens. Without the experiment, the relative contribution of objective vs. density vs. model scale to closing the gap remains ambiguous.

What evidence exists in the paper. Figure 8 provides the only supervision density scaling data, and only at the Base model size. The trend is clear and monotonic—n8 outperforms n1 across all epochs, with both faster convergence and better final quality. The paper acknowledges that "scaling supervision density markedly accelerates convergence and improves final performance, [but] also increases the computational cost" (Section 5.5), and defers efficiency improvements to "future research." The computational cost of n8 over n1 is approximately 8× more loss computation per training step (since 8 tokens are supervised instead of 1), though the forward pass through the Transformer backbone is shared. For the larger models, this additional cost may be non-trivial in absolute GPU-hours.

Mitigation status. The paper explicitly flags this as future work (Section 5.5: "developing methods to enhance the training efficiency of our framework remains an important direction for future research"), which is appropriate transparency. However, the lack of even a single w256n8 data point at the Large or Huge scale (which would require only a few additional training runs) weakens the empirical claims. The paper's stated reason for using n1 in the main results—computational cost—is understandable but leaves the strongest version of the method unevaluated.

The assumption or constraint. The paper's motivation is explicitly tied to the vision of unified multimodal language models. The abstract positions Heptapod as "language modeling on visual signals," Section 5.4 emphasizes that "Heptapod's adherence to causal attention ensures that it can be seamlessly integrated into a multimodal LLM," and the concluding section (7) states the framework "lays a foundation for unified models that integrate generative and understanding capabilities across modalities." Despite this framing, all experiments are on class-conditional generation—where the conditioning signal is a single discrete class label (one of 1,000 ImageNet categories), not natural language text.

The consequence. Class-conditional generation is a fundamentally simpler conditioning problem than text-to-image generation. ImageNet classes are atomic labels (e.g., "golden retriever," "volcano," "library") that map directly to visual categories the model has seen during training. The model only needs to associate a class index with a visual prototype. Text-to-image generation, by contrast, requires:

Compositional understanding: Parsing complex descriptions with multiple objects, attributes, spatial relationships, and interactions ("a red cube on top of a blue sphere, with a small green cone to the left").
Open-vocabulary generalization: Generating concepts described in ways not seen during training, requiring the model to compose visual elements from textual descriptions.
Cross-modal alignment: Learning correspondences between linguistic semantics and visual appearance, which may require different representational pressures than within-modality visual semantics.

The paper's claim that Heptapod's causal attention "ensures seamless integration into a multimodal LLM" is an architectural claim (the Transformer is compatible with text token processing), but the learnability claim—that the next-2D-distribution objective would work effectively when conditioned on complex text prefixes—is untested. Text conditioning introduces text tokens with their own sequential structure and long-range dependencies; the interaction between text-level autoregressive modeling and image-level 2D distribution prediction in a shared causal Transformer is an open empirical question.

Specific unknowns:

Does the next-2D-distribution objective remain effective when the causal prefix contains text tokens (with very different statistical structure from image patches)?
Does the model learn to attend appropriately between text tokens and image tokens—e.g., linking descriptive phrases to the spatial regions they describe?
Does the image prediction head need to be modified to incorporate text context explicitly (e.g., cross-attention to text tokens), or does the causal attention in the backbone suffice?
How does the training procedure change when the dataset includes text-image pairs with variable-length, unstructured natural language captions rather than fixed class labels?

What evidence exists in the paper. No text-to-image experiments are reported. The paper does not discuss text conditioning, cross-modal attention, or the challenges of extending to multimodal generation. The architecture described in Section 4 and Figure 5 assumes the causal prefix consists only of (position, token) pairs from the image grid; the start token [S] is the only non-image token. Extending this to text conditioning would require architectural changes (at minimum, an embedding layer for text tokens and mechanisms to interleave text and image tokens in the causal sequence) that are not described or evaluated.

Mitigation status. The paper does not acknowledge this limitation. The "seamless integration" claim is presented as a consequence of causal attention compatibility, but integration requires more than architectural compatibility—it requires that the learning objective and training procedure work in the multimodal setting. The paper's philosophical narrative (returning to language modeling first principles, enabling unified multimodal models) creates an expectation of multimodal evidence that the experiments do not fulfill. Acknowledging this as future work—and potentially demonstrating a simple text-to-image variant on a small dataset—would strengthen the paper's positioning.

The Framework Provides No Evidence of Learning Transferable Visual Representations, Limiting the "Unified Understanding and Generation" Narrative

The assumption or constraint. The paper claims Heptapod "lays a foundation for unified models that integrate generative and understanding capabilities across modalities" (Section 7) and positions the next-2D-distribution objective as a unification of autoregressive modeling with MAE-style self-supervised learning (Section 4.2: "from a self-supervised learning perspective, our objective unifies autoregressive modeling with Masked Autoencoding"). The implicit promise is that training a model with this objective produces visual representations that are useful for both generation (synthesizing images) and understanding (classifying, detecting, segmenting).

The consequence. This claim remains entirely unsubstantiated. The paper evaluates Heptapod exclusively on generation quality (FID, IS). No experiments test whether the causal Transformer's internal representations—the hidden states that encode the "holistic understanding" the objective is designed to produce—transfer to recognition tasks. This is a significant gap because:

The MAE connection is structural (both predict masked content from visible context) but the paper provides no evidence that this structural similarity yields MAE-like representational quality. MAE representations are known to be strong for downstream vision tasks (He et al., 2022)—if Heptapod's representations were comparably strong, this would be a major result supporting the unification claim.
The paper's critique of semantic tokenizers is that they "pre-package" semantic knowledge from SSL models like DINO. If Heptapod's generative training does not produce representations that rival DINO for recognition tasks, then semantic tokenizers still provide a capability (recognition-quality representations in the tokenizer) that Heptapod does not replicate.
The unified model vision requires that the same Transformer can both generate images and perform visual understanding tasks (answering questions about images, detecting objects). Without evidence of understanding capabilities, Heptapod is a generative model with a philosophically clean design, not a step toward unified understanding-and-generation.

What evidence exists in the paper. Zero. The paper does not report any of the following:

Linear probe accuracy on ImageNet classification using frozen Heptapod features (the standard evaluation for self-supervised representation quality).
Fine-tuning performance on downstream vision tasks (detection, segmentation).
Comparison of Heptapod's representational quality against MAE, DINO, or other SSL methods.
Nearest-neighbor retrieval or other representational similarity analyses.

The only indirect evidence is the FID/IS improvements, which indicate the model learns something useful beyond local texture, but do not isolate representational quality from generation-specific capabilities (e.g., the decoder's ability to render realistic textures from latent codes).

Mitigation status. The paper does not acknowledge this gap. The unification claim is presented as a conceptual contribution rather than an empirically demonstrated one, which is appropriate for a method paper but creates a disconnect with the strong language about "laying a foundation for unified models." The paper would benefit from either (a) preliminary representation-quality experiments (e.g., linear probe on ImageNet), (b) a clear statement that representation learning is future work, or (c) more carefully scoped claims about what the current results do and do not demonstrate about understanding capabilities.

7. Implications and Future Directions

How This Work Changes the Landscape

Heptapod does not merely improve the state of the art for causal autoregressive image generation—it reframes why prior causal models underperformed, and in doing so changes what the field should consider the primary bottleneck. Before this work, the dominant narrative was that vision is fundamentally harder for autoregressive modeling than text: images lack natural sequential order, have high local redundancy, and require semantic understanding that simple next-token prediction cannot deliver. The field's response was to compensate with external crutches—CFG to correct learned distributions at inference time, SSL-distilled semantic tokenizers to pre-package long-range relationships into the compression step, or non-causal architectures (MAR, VAR) that abandon the causal constraint entirely. Each of these responses implicitly accepted that standard next-token prediction was insufficient for vision, treating the failure as a modality gap rather than an objective design failure.

Heptapod's central contribution is to re-diagnose the failure: the problem is not that causality constrains vision, nor that reconstruction tokenizers lack semantic information, nor that images are intrinsically harder than text. The problem is that standard next-token prediction with a fixed scan order creates an optimization landscape where the easiest way to reduce loss is to learn local interpolation, a strategy that saturates most of the available loss reduction without requiring global semantic understanding. The model does not fail to learn semantics because it cannot—it fails because the objective does not demand it.

This diagnosis is important because it is actionable and specific. Prior failure explanations ("vision is harder," "reconstruction tokenizers lose too much information," "causality is too constraining") point toward solutions that add complexity: better tokenizers, stronger guidance, relaxed architectural constraints. Heptapod's diagnosis points toward a simpler intervention: change what the objective asks the model to predict, from a single designated-next token to the full distribution over all remaining spatial positions. The dramatic improvement from this single change—FID dropping from ~19 (1D-raster) to ~4 (2D-random) in the controlled ablation of Table 1—validates that the objective, not the modality, was the bottleneck.

Several specific shifts follow from this reframing:

1. External semantics are reframed as compensations for an objective failure, not as necessary components of visual generative models. The paper demonstrates that a reconstruction-only tokenizer (VAE, rFID 1.43) combined with the right objective achieves 2.70 FID without CFG, outperforming DiGIT (3.39 FID) which uses DINO-based semantic tokens. This does not prove that semantic tokenizers are never useful—LightningDiT at 2.17 FID and MAETok at 2.31 FID still outperform Heptapod, and both use external SSL. But it establishes that the prior gap between semantic-tokenizer and reconstruction-tokenizer autoregressive models was largely attributable to the objective, not to the tokenizer's semantic content. For researchers who adopted semantic tokenizers because reconstruction tokenizers "didn't work" for autoregressive generation, Heptapod shows that the tokenizer was not the problem—the training objective was.

2. The search for better generation orders is reframed as solving the wrong problem. Prior work invested significant effort in designing generation orders for autoregressive image models—raster scan, spiral, coarse-to-fine (VAR), random (Randomized AR), and learned orders. Each order design was an attempt to impose structure on the inherently unordered 2D space so that next-token prediction would be better-behaved. Heptapod demonstrates that the entire framing—"which order should the model generate in?"—is an artifact of the 1D prediction paradigm. When the model predicts the full 2D distribution, the generation order becomes a sampling decision rather than a modeling constraint. The model can adapt its order to the image content, generating structurally important regions first regardless of their spatial position. The paper does not deeply analyze the emergence of generation orders, but the principle is clear: do not specify an order; make the model prepared for any order. This shifts research attention from order design (an inductive bias problem) to distribution modeling (a representation learning problem).

3. The gap between causal and non-causal architectures is substantially narrowed, making causality a more practical design choice. Before Heptapod, the performance penalty for full causality was severe—LlamaGen-3B (causal) at 9.38 FID vs. MAR-H (bidirectional) at 2.35 FID represents a 4× difference. For a practitioner building a multimodal LLM, this gap made causality a hard constraint that severely limited image generation quality. Heptapod shrinks this gap to ~15% (2.70 vs. 2.35 at matched parameter count), making causal autoregressive generation practically competitive for the first time. This does not eliminate the advantage of bidirectional attention—MAR still wins on FID—but it changes the tradeoff calculus. A system designer who values architectural coherence (unified causal processing for all modalities) can now achieve generation quality that was previously only available by abandoning causality. The remaining 15% gap is a target for future work, not a proof that causality is fundamentally limited.

4. The "impossibility triangle" between reconstruction, generation, and semantic representation is reframed as a problem of objective misalignment rather than an inherent tension. Prior work characterized a three-way tradeoff where optimizing a tokenizer for reconstruction quality compromised generation quality, and optimizing for semantics compromised reconstruction. Heptapod's solution is to exit the triangle entirely by removing "semantic representation" from the tokenizer's responsibilities. The tokenizer handles only reconstruction; the Transformer handles semantics through the generative objective. Whether this fully resolves the tradeoffs or simply shifts them (e.g., to a tradeoff between window size and computational cost) is an open question, but the paper's results demonstrate that the separation-of-concerns philosophy is practically viable at competitive quality levels. This provides a template for other modalities—audio, video, 3D—where similar impossibility triangles have been observed.

5. The unification of autoregressive and masked autoencoding objectives within a causal framework establishes a new point in the design space for self-supervised visual learning. The paper's connection between next-2D-distribution prediction and MAE (Section 4.2: the causal prefix as unmasked context, predicting the remaining grid as masked reconstruction) is more than an analogy—it is a specific architectural proposal for how to combine the sequential sampling capability of autoregressive models with the holistic representation learning of MAE, all without abandoning causality. This opens a design space that was previously unexplored: objectives that demand both sequential coherence (through causal processing) and global understanding (through holistic prediction). Future work on visual representation learning, video prediction, and multimodal models can operate in this space rather than choosing between autoregressive and masked paradigms.

Reconciling prior contradictions. The paper resolves an apparent contradiction in the literature: why some works found that autoregressive models with reconstruction tokenizers fail to learn semantics (e.g., LlamaGen requiring CFG for competitive results), while others found that autoregressive models can succeed when given semantic tokenizers (e.g., DiGIT achieving strong results with DINO tokens). The resolution is that reconstruction tokenizers do not inherently prevent semantic learning—they simply make it unnecessary under a 1D next-token objective, because local interpolation provides an easier path to loss reduction. Changing the objective to demand holistic prediction removes the local shortcut and reveals that reconstruction tokenizers are fully capable of supporting semantic learning. The prior negative results were measuring the objective's failure to demand semantics, not the tokenizer's failure to support them.

Research directions that become more attractive:

Improving reconstruction tokenizers for autoregressive generation. Before Heptapod, improving VQ-VAE reconstruction quality had limited value because autoregressive models could not exploit the additional fidelity. Heptapod's results (where the VAE model's quality mirrors the VAE tokenizer's rFID advantage) suggest that tokenizer quality directly sets the ceiling. Better tokenizers—larger codebooks, improved quantization, higher compression ratios—are now directly impactful for generative quality under the 2D objective.
Developing training-efficient global prediction architectures. The paper's finding that window size dominates supervision density (Figure 7) creates a clear target: make global prediction computationally feasible. Sub-quadratic attention mechanisms, factorized spatial attention, hierarchical prediction heads, or learned sparsity patterns could reduce the global head's cost without sacrificing the window size benefit.
Scaling supervision density at large model sizes. The consistent improvement from w256n1 to w256n8 at Base scale (Figure 8) suggests a straightforward path to better results: train larger models with higher density. The unexplored question is whether this improvement saturates, scales linearly with model size, or interacts with other hyperparameters.

Research directions that become less attractive:

Designing hand-crafted generation orders for autoregressive image models. If the model can learn to sample in any order (and Heptapod demonstrates strong results with random order during training), the marginal benefit of sophisticated order design is likely small compared to improving the prediction objective.
Semantic tokenizer design as a primary research focus for autoregressive generation. While semantic tokenizers may remain relevant for specialized applications (e.g., when extreme compression is needed and reconstruction fidelity is secondary), Heptapod demonstrates that they are not necessary for competitive autoregressive generation. Research effort may be better spent on reconstruction tokenizer quality and objective design.
Inference-time correction techniques (CFG) as a default component of visual autoregressive pipelines. If the objective can produce well-calibrated distributions without CFG (as Heptapod's results suggest), CFG becomes a fallback for cases where the objective is insufficient, rather than a universal requirement. This shifts research focus from tuning CFG schedules to improving the training objective.

Follow-Up Research This Work Enables

Scaling supervision density at the Large and Huge scales, and establishing the density-vs-quality Pareto frontier. The paper demonstrates that increasing supervision density from n1 to n8 under the global window (w256) improves Heptapod-B's FID from ~6.0 to ~5.0 at epoch 800 (Figure 8). However, all main benchmarking results in Table 2 use w256n1. The most immediate follow-up is simply to train Heptapod-L and Heptapod-H with w256n8 (and potentially n16, n32) to establish whether the ~17% relative FID reduction observed at Base scale transfers to larger models. A strong result—Heptapod-H achieving FID in the 2.2–2.4 range at n8, matching or surpassing MAR-H at 2.35—would substantially strengthen the paper's claim that causal models can approach non-causal performance. A negative result—diminishing returns from density at larger scales, or a plateau at FID ~2.5—would suggest that the remaining causal-to-bidirectional gap is due to architectural constraints rather than supervision efficiency. Either outcome is informative. The experiment requires no methodological innovation, only additional compute. The key numbers to report would be FID/IS vs. density at each model scale (B, L, H) at matched training duration, plus FID-vs-training-FLOPs curves to characterize the efficiency tradeoff.

Measuring semantic understanding directly through linear probing and attention analysis of Heptapod's internal representations. The paper's central narrative—that the next-2D-distribution objective causes semantic understanding to "emerge implicitly within the Transformer" (Section 1, Section 5.4)—is supported indirectly by FID/IS improvements but lacks direct evidence. A critical follow-up would evaluate Heptapod's frozen intermediate representations on standard self-supervised learning benchmarks: linear probe accuracy on ImageNet classification (following the MAE protocol of He et al., 2022), nearest-neighbor retrieval precision, and possibly transfer to detection or segmentation. Comparing these numbers against representations from (a) standard autoregressive models with reconstruction tokenizers (e.g., LlamaGen features), (b) models trained with semantic tokenizers (e.g., DiGIT features), and (c) dedicated SSL methods like DINO and MAE would directly test the claim of semantic emergence. A strong result would show Heptapod features achieving classification accuracy competitive with or exceeding DiGIT's features (despite Heptapod not using DINO at any stage), suggesting that the generative objective genuinely induces semantic representations. A weak result—Heptapod features performing near random or far below SSL baselines despite good FID—would indicate that the improved generation quality comes from better texture/structure modeling rather than semantic understanding, substantially weakening the paper's philosophical narrative. Additionally, visualizing Heptapod's attention maps (analogous to Figure 3, Right) would provide qualitative evidence: do they show the long-range, semantically-structured patterns of the semantic-tokenizer models, or the local-concentration patterns of the standard VQ-VAE models? The paper's omission of these visualizations is conspicuous given the central role attention maps play in motivating the method.

Extending the next-2D-distribution framework to text-to-image generation to test cross-modal semantic learning. The paper's motivation is explicitly tied to unified multimodal LLMs (Section 5.4: "Heptapod's adherence to causal attention ensures that it can be seamlessly integrated into a multimodal LLM"), but all experiments are class-conditional. The natural extension is to train Heptapod on text-image pairs (e.g., LAION-400M or COCO) with the same next-2D-distribution objective, where the causal prefix includes text tokens followed by image tokens. The key experimental questions: (1) Does the 2D distribution objective remain effective when the prefix contains natural language rather than fixed class labels? The text tokens have very different statistical structure from image tokens, and the model must learn cross-modal attention patterns. (2) Does Heptapod's learned generation order adapt to text semantics—e.g., generating named objects before background regions? (3) Can the same architecture handle both text-to-image generation and image understanding (via text tokens appended after image tokens)? A strong result would show that the text-conditioned Heptapod achieves competitive FID on MS-COCO (e.g., <10 FID at 256×256) without CFG, while also achieving non-trivial image captioning or VQA performance when run in reverse (image → text tokens). This would demonstrate that the objective induces joint multimodal representations, not just visual semantics. A more modest but still valuable result would be text-to-image generation alone, with analysis of how the text conditioning interacts with the learned generation order. The architectural changes needed are modest (adding a text tokenizer and embedding, interleaving text and image tokens in the causal prefix) but the training dynamics and optimal window-size-vs-density choices may differ substantially from the class-conditional setting.

Evaluating Heptapod at higher resolutions (512×512 and beyond) to characterize the global prediction head's scaling behavior and test hierarchical or factorized alternatives. The paper identifies the global head's quadratic cost as the primary bottleneck for resolution scaling (Section 4.3: "[the global head's] computational cost scales with the grid size"), but provides no empirical characterization of how FID, wall-clock time, and GPU memory scale with grid size. A systematic scaling study would: (1) Train Heptapod at 512×512 (32×32 latent grid, 1024 tokens) with the global head and report FID and per-step inference time relative to 256×256, establishing the practical cost-vs-quality curve. (2) Evaluate whether the global head remains necessary at larger grids—perhaps at 1024 tokens, a window of w=512 (half the grid) is sufficient to capture long-range dependencies, reducing the quadratic cost by 4×. (3) Develop and test hierarchical prediction heads that decompose the 2D distribution into coarse-to-fine levels: a lightweight global head predicts a coarse distribution (e.g., over 8×8 super-patches), and a local head refines within each super-patch. This would test whether the paper's finding that "window size matters more than density" (Section 5.3) generalizes to hierarchical architectures where the effective window size can be maintained while controlling cost. The experiment would measure the pareto frontier of FID vs. FLOPs per image at multiple resolutions, providing the first systematic characterization of how Heptapod-style objectives scale with spatial dimensions. The negative result—that the global head's cost becomes prohibitive before FID improvements saturate—would motivate a focused search for efficient approximations to global spatial attention in the prediction head.

Combining Heptapod's 2D distribution objective with discrete VQ-VAE tokenizers that approach VAE reconstruction quality, to test whether fast convergence and high final quality can be unified. Figure 6 reveals a clear tradeoff: the VQ-based model converges faster and more smoothly (due to cross-entropy's sharper gradient signal) but plateaus at higher FID (~6.5), while the VAE-based model converges slowly but ultimately achieves better FID (~4.0). The paper attributes the final performance gap primarily to reconstruction fidelity (rFID 2.19 for VQ vs. 1.43 for VAE) and suggests that "a discrete VQ tokenizer, if its reconstruction quality were improved to match or exceed that of the VAE, could potentially offer the best of both worlds" (Section 5.2). A direct test of this hypothesis would use a modern high-performance VQ-VAE—such as one employing finite scalar quantization (Mentzer et al., 2024), large codebooks with improved utilization (Zhu et al., 2024), or look-up free quantization (LFQ)—that achieves rFID comparable to or better than the MAR VAE's 1.43. Training Heptapod with this tokenizer under identical settings (model size, window, density, epochs) and comparing FID convergence curves against the VAE baseline would test whether the reconstruction quality ceiling is the only factor, or whether the discrete/continuous distinction introduces inherent tradeoffs (e.g., codebook collapse, categorical bottlenecks) that persist regardless of reconstruction quality. A positive result—VQ matching or exceeding VAE final FID while retaining faster convergence—would be practically significant because it would provide a faster-training alternative for large-scale models. A negative result—VQ plateauing below VAE despite matched rFID—would suggest that the diffusion loss on continuous tokens provides benefits beyond reconstruction quality (e.g., smoother gradient landscape, better distribution coverage) that are fundamental to the continuous parameterization.

Probing the negative result: does the ReST-style revision model degradation (Appendix K, Figure 16) reveal a fundamental instability in on-policy training of autoregressive vision models, or is it specific to the revision task? Although this direction is mentioned in the paper's Appendix K and reflects a negative result from their revision model experiments, it connects to a deeper question about whether the next-2D-distribution objective is compatible with on-policy or RL-based training methods that iterate on the model's own outputs. The paper found that applying ReST^{EM}-style optimization to the revision model caused "substantial degradation," hypothesizing that "on-policy data collection... exacerbates spurious correlations." A systematic follow-up would test whether Heptapod's generative model (not the revision model) is stable under on-policy training—e.g., by using Heptapod to generate images, filtering by FID or a learned quality estimator, and retraining on the filtered outputs for multiple iterations. Does the next-2D-distribution objective provide a sufficiently robust learning signal to prevent mode collapse or quality degradation under distribution shift, or does it also suffer from the same spurious correlation problems observed in the revision case? The experiment would characterize whether Heptapod's outputs—without CFG—have the right properties (coverage, calibration) to serve as training data for self-improvement loops. A positive result would open the door to data-generation pipelines where Heptapod generates synthetic training data for itself or other models. A negative result would suggest that even with the improved objective, autoregressive image generation requires careful data curation and is sensitive to distribution shift in training data, an important caveat for deployment in self-improving systems.

Practical Applications and Downstream Use Cases

1. Unified multimodal language models with native image generation. The most direct application of Heptapod is as the image generation component in a multimodal LLM that processes text, code, and images under a single causal Transformer with a unified architecture. Heptapod's 2.70 FID without CFG means that image generation quality is no longer the bottleneck preventing such unified architectures from being practically useful. A multimodal LLM built with Heptapod-style image generation could: (a) interleave text and image tokens in a single autoregressive sequence, (b) generate images from text descriptions using the same next-distribution-prediction mechanism for both modalities, (c) condition image generation on multi-turn conversation context, and (d) potentially run in reverse (image → text) for visual understanding using the same weights. The key practical benefit—beyond architectural elegance—is deployment simplicity: one model, one inference pipeline, one set of hyperparameters. The 4× reduction in parameters compared to LlamaGen-3B at much better quality (2.70 vs. 9.38 FID) makes on-device or lower-resource deployment of such models more feasible. The primary deployment risk is the global prediction head's computational cost at higher resolutions, which may require the local/chunk-based head or hierarchical variants for latency-sensitive applications.

2. High-quality data augmentation and synthetic data generation for vision tasks. Heptapod's class-conditional generation at 2.70 FID with IS 229.8 (the highest IS in Table 2) indicates strong sample quality and diversity within ImageNet classes. This makes it suitable for generating synthetic training data where class labels are available but images are scarce—for example, augmenting long-tail classes in object recognition, generating diverse training examples for fine-grained classification, or creating counterfactual examples (e.g., "what would this bird look like from a different angle?"). The practical advantage over prior autoregressive approaches is substantial: Heptapod-H at 941M parameters generates images more than 3× better (by FID) than LlamaGen-3B, meaning fewer computational resources for a given augmentation quality. The absence of CFG simplifies deployment—no guidance scale to tune, no inference-time hyperparameter that could affect sample diversity. However, the single-benchmark limitation applies: these benefits are demonstrated only on ImageNet-1K classes. Extending to custom class sets or open-vocabulary generation would require text-conditioned variants not yet demonstrated.

3. Foundation for video generation models using spatiotemporal distribution prediction. Heptapod's core idea—predict the distribution over the entire remaining spatiotemporal grid at each step—generalizes naturally from 2D (space) to 3D (space + time) for video generation. A video Heptapod would tokenize a video clip into a 3D grid (height × width × frames), then at each autoregressive step predict the distribution over all remaining spatiotemporal positions. The unification with MAE-style training would mean the model learns spatiotemporal representations that capture both spatial structure (within-frame) and temporal dynamics (across-frame), potentially enabling coherent long-range video generation. The paper's finding that window size matters more than supervision density (Figure 7) provides direct guidance for design: prioritize spatiotemporal coverage even at low density. The local chunk-based head (Section 4.3) is particularly well-suited here—predicting a spatiotemporal "tube" of nearby frames and positions could capture local motion while keeping cost manageable. The primary open question is whether the holistic prediction objective scales to the much larger token counts of video (e.g., 16 frames × 16×16 spatial grid = 4096 tokens), where the global head would be infeasible but chunk-based heads might still provide sufficient spatiotemporal context.

4. A principled baseline for research on causal visual autoregressive models, enabling clearer attribution of progress. Before Heptapod, evaluating a new causal autoregressive image model was ambiguous: was poor performance due to the objective, the tokenizer, the architecture, or CFG settings? Heptapod provides a clean, well-characterized baseline that isolates the objective's contribution: with an off-the-shelf VAE tokenizer, a standard causal Transformer, no CFG, and the next-2D-distribution objective, you can expect ~2.70 FID at 941M parameters on ImageNet-1K 256×256. Future work proposing new tokenizers, architectures, training procedures, or conditioning mechanisms for causal autoregressive vision can compare against this baseline to attribute improvements to their specific innovations rather than to objective quality. The ablation framework—window size, supervision density, head depth, tokenizer type—provides a template for characterizing new methods. A researcher proposing a new tokenizer can test whether it raises Heptapod's performance ceiling (as the VAE's rFID advantage did over VQ) or improves convergence (as VQ's cross-entropy did). A researcher proposing a new prediction head architecture can test whether it achieves w256n1-level quality at lower cost than the global bidirectional head. This standardization of the causal autoregressive baseline is a practical contribution that may accelerate progress more than any single architectural innovation.

When to Prefer This Method

The paper positions Heptapod against two specific classes of alternatives—standard causal autoregressive models with reconstruction tokenizers (e.g., LlamaGen) and non-causal or externally-augmented approaches (e.g., MAR, DiGIT, LightningDiT)—and articulates a clear tradeoff based on three desiderata: architectural compatibility with language models (causal attention), independence from external models (no SSL tokenizer, no CFG), and generation quality. The decision of when to prefer Heptapod follows directly from these comparisons:

Prefer Heptapod over standard causal autoregressive models (e.g., LlamaGen) in essentially all cases. They share the same architectural constraints (causal attention, reconstruction tokenizer, no CFG) and Heptapod dominates on generation quality by a large margin (2.70 vs. 9.38 FID at comparable or smaller parameter counts, Table 2). There is no documented tradeoff where LlamaGen-style approaches have an advantage—Heptapod's only additional cost is the prediction head, whose depth is matched to the backbone and whose overhead does not prevent dramatic quality gains. For any application where a causal autoregressive image model is desired, Heptapod's objective should replace standard next-token prediction as the default.
Prefer Heptapod over externally-augmented causal models (e.g., DiGIT) when independence from SSL models is valued. DiGIT achieves 3.39 FID using DINO-distilled semantic tokens, meaning its tokenizer depends on an external model that may have its own biases, limitations, and licensing constraints. Heptapod's 2.70 FID with a reconstruction-only tokenizer demonstrates that external semantics are not necessary for competitive quality, and avoiding an external dependency simplifies deployment (no separate SSL model to load, no SSL model biases to inherit). If the SSL model is already available in the deployment environment (e.g., DINO is used for other vision tasks), this advantage is reduced but not eliminated—the principle that semantics should emerge from the generative objective rather than being pre-packaged may still matter for representation quality and scalability.
Prefer Heptapod over non-causal approaches (e.g., MAR) when architectural coherence with language models is required. MAR achieves 2.35 FID (vs. Heptapod's 2.70) using bidirectional attention, representing a ~15% quality advantage at matched parameter count. However, bidirectional attention is incompatible with standard causal LLM architectures and requires separate training and inference pipelines. For applications where images are generated in isolation (standalone image generation, data augmentation, creative tools) and quality is the sole metric, MAR's bidirectional advantage may be worth the architectural divergence. For applications where image generation must be integrated into a multimodal LLM—interleaved text-and-image conversations, in-context visual generation, unified processing pipelines—Heptapod's full causality is a hard requirement, and the 15% quality gap is the cost of architectural compatibility. The paper's results suggest this gap can be narrowed through increased supervision density (Figure 8: w256n8 improved Heptapod-B by ~17%), potentially closing it entirely with optimized training recipes.
Prefer Heptapod over CFG-dependent models when sample diversity and calibration matter. CFG is known to reduce sample diversity (the paper notes "intensity oversaturation and reduced sample diversity" in Section 1) and requires tuning a guidance scale that controls the fidelity-diversity tradeoff. Heptapod without CFG achieves the highest IS in Table 2 (229.8), suggesting strong sample diversity without the guidance scale tuning burden. For applications requiring diverse outputs (creative tools, data augmentation, sampling-based search) or calibration (where the model's learned distribution should match the target distribution without post-hoc correction), Heptapod's CFG-free operation is a practical advantage. The absence of a CFG ablation for Heptapod itself means the paper cannot quantify how much quality is left on the table by not using CFG—it is possible that Heptapod would benefit from CFG, just less than prior models. This should be verified before deploying in quality-critical applications.

The key uncertainty in all these recommendations is the single-benchmark scope: they are validated on ImageNet-1K 256×256 class-conditional generation. The tradeoffs may shift on text-to-image tasks (where non-causal or diffusion models have a stronger track record), higher resolutions (where Heptapod's global head cost becomes prohibitive), or more diverse datasets (where the curse of locality may manifest differently). For deployments matching the paper's experimental conditions (class-conditional generation, moderate resolution, clean object-centric images), the recommendations are well-supported. For deployments outside this scope, Heptapod represents a promising but unvalidated approach that should be evaluated on the target domain before adoption.