Vision2024~103 minarXiv:2412.15119

Parallelized Autoregressive Visual Generation

score 9Yuqing Wang, Shuhuai Ren, Zhijie Lin +6↑ 53★ 184Dec 19, 2024cs.CVhttps://github.com/Epiphqny/PAR

🎯 Pitch

Autoregressive image generation can be sped up 9.5× with almost no quality loss by generating distant, weakly dependent tokens in parallel—without changing the model architecture at all. The key insight is that strong local token dependencies cause visible artifacts when parallelized naively, but splitting the image into independently generated regions with sequential initial anchors preserves coherence. This simple strategy achieves a 3.6× wall-clock speedup (e.g., 3.46s vs. 12.41s per image) while matching standard autoregressive FID scores.

1. Executive Summary

This paper proposes Parallelized Autoregressive Visual Generation (PAR), a method for accelerating autoregressive visual models without modifying the underlying architecture or tokenizer. Evaluated on ImageNet-256 and UCF-101 using PAR models scaled from 343M to 3.1B parameters, the approach introduces non-local parallel generation — generating weakly dependent tokens from spatially distant regions simultaneously while maintaining sequential generation for strongly dependent local tokens — guided by an explicit three-stage design principle (sequential initialization of region anchors to establish global structure, sequential within-region generation for local coherence, and parallel cross-region generation for efficiency). PAR achieves a 3.6× wall-clock speedup with comparable quality (FID 2.29 vs. 2.18 on ImageNet) and up to 9.5× speedup with minimal degradation (within 0.7 FID for images and 10 FVD for video), establishing that token dependencies — not architectural constraints — govern the feasibility of parallel visual generation, and that acceleration is achievable only when parallelized tokens are selected based on dependency strength rather than spatial proximity.

2. Context and Motivation

The Core Problem: Autoregressive Visual Generation Is Unacceptably Slow

The fundamental problem this paper addresses is straightforward but practically crippling: autoregressive visual generation is too slow for real-world deployment. Modern autoregressive image generators like LlamaGen [51] require 576 sequential token-by-token generation steps to produce a single image, taking 12.41 seconds on an A100 GPU (Table 2). For video generation, the situation is worse — a standard autoregressive video model requires 1,280 steps per clip (Table 3), with each step depending on the completion of all previous steps. This sequential bottleneck means that even as GPU hardware improves, the latency fundamentally scales with sequence length, not with available parallel compute.

This matters for two distinct reasons. First, there is a practical deployment barrier: applications requiring interactive or real-time visual generation (creative tools, video editing, gaming, live content production) simply cannot tolerate multi-second latencies per image, let alone per frame. A system that takes 12 seconds to generate one image is not viable for any application where a user is waiting in a feedback loop, such as iteratively refining a design or generating content in real time.

Second, there is a scaling economics problem: as autoregressive visual models grow larger and sequence lengths increase (higher resolutions, longer videos, more sophisticated tokenizations), the inference cost scales linearly with the number of generation steps. If each generation step costs $C$ FLOPs, the total inference cost is $C \times N_{\text{steps}}$ . Reducing $N_{\text{steps}}$ by a factor of $k$ directly reduces inference cost by the same factor — a crucial consideration for deploying these models at scale.

The Sequential Assumption: Where It Comes From

To understand why autoregressive visual generation is inherently sequential, it is helpful to recall the standard formulation. Given a sequence of visual tokens $\{v_1, v_2, \ldots, v_n\}$ produced by a discrete tokenizer (e.g., VQGAN [12]), an autoregressive transformer models the joint distribution as a product of conditional distributions:

$P(v_1, v_2, \ldots, v_n) = \prod_{i=1}^{n} P(v_i | v_1, v_2, \ldots, v_{i-1})$

At generation time, the model samples $v_1 \sim P(v_1)$ , then $v_2 \sim P(v_2 | v_1)$ , then $v_3 \sim P(v_3 | v_1, v_2)$ , and so on. Each token must be generated before the next can be predicted, because the model conditions on the full history of generated tokens. This is not an optional design choice — it is a mathematical consequence of the chain rule factorization of the joint distribution. Any deviation from strict sequential generation risks violating the autoregressive property and producing samples from an incorrect distribution.

The sequential bottleneck is not a flaw in any particular implementation; it is a structural property of the autoregressive modeling paradigm itself. This is why, despite radical improvements in transformer efficiency (FlashAttention, KV-cache optimizations, model quantization), the number of sequential steps — the generation depth — remains fundamentally unchanged. Engineering optimizations can reduce the cost per step but cannot reduce the number of steps.

Prior Approaches and Why They Fall Short

The literature contains three broad categories of attempts to accelerate sequential generation, each with significant limitations when applied to visual autoregressive models.

Approach 1: Speculative and Jacobi Decoding (Language Domain)

In language modeling, speculative decoding [6, 26, 28] uses a smaller, faster "draft" model to propose multiple tokens in parallel, which are then verified (and potentially rejected) by the main model. Jacobi decoding [22, 47] takes a different approach: it starts with a guess for all future tokens and iteratively refines them in parallel, converging to the autoregressive distribution.

Where these fall short for visual generation: Both approaches require either (a) training a separate draft model, which doubles the engineering and deployment complexity, or (b) multiple rounds of iterative refinement, which partially defeats the purpose of parallelization since each refinement round is itself a forward pass through the model. More fundamentally, these methods were developed for language, where the token vocabulary is small and token dependencies are largely local. Visual tokens exhibit fundamentally different dependency structures — adjacent pixels or patches have extremely strong correlations that make independent parallel prediction of nearby tokens much riskier than in text. The paper does not explicitly compare against speculative decoding variants for visual generation, but the core failure mode — independent sampling of strongly dependent visual tokens producing inconsistent local patterns (Figure 1b, Figure 5 bottom) — would affect any parallel proposal mechanism that does not account for visual token dependency structure.

Approach 2: Non-Autoregressive and Masked Modeling (Visual Domain)

MaskGIT [5] abandons autoregressive generation entirely, instead using a BERT-like masked modeling approach: the model starts with all tokens masked and iteratively unmasks tokens in parallel over multiple steps (typically 8 steps for ImageNet generation). This achieves dramatic speedups — 8 steps vs. 256+ for autoregressive models — but at the cost of generation quality. As shown in Table 2, MaskGIT achieves FID 6.18 vs. autoregressive models achieving 2.18-3.07, a substantial quality gap.

Where this falls short: The non-autoregressive paradigm represents a different modeling choice that gives up the autoregressive property entirely. This matters because the autoregressive formulation has proven to be uniquely scalable and flexible — it is the foundation of unified multimodal models that handle text, images, and video within a single framework [52, 65]. Switching to a masked modeling approach for visual generation breaks this unity, requiring separate architectures and training procedures for different modalities. The paper explicitly frames its goal as achieving speedup "while maintaining the simplicity and flexibility of standard autoregressive models" (Section 1), positioning itself against approaches that require architectural departures.

Approach 3: Next-Scale Prediction (VAR)

VAR [53] proposes a fundamentally different autoregressive paradigm: rather than predicting tokens in raster scan order, it predicts entire token maps at progressively higher resolutions ("next-scale prediction"). This reduces generation steps dramatically (10 steps for VAR-d30 in Table 2) and achieves excellent quality (FID 1.97).

Where this falls short: VAR requires a specially designed multi-scale tokenizer that produces token hierarchies at different resolutions. This is a non-trivial architectural constraint — standard VQGAN tokenizers produce flat token grids, and adapting VAR to work with arbitrary tokenizers requires retraining the tokenization pipeline. Additionally, VAR's multi-scale token sequences are longer than standard flat sequences because they include tokens at multiple resolution levels (coarse-to-fine). The paper notes this explicitly (Section 2): VAR "requir[es] specialized multi-level tokenizers and longer token sequences." This increases the total number of tokens the model must process, partially offsetting the step-count reduction in terms of total FLOPs. And like the masked modeling approaches, VAR departs from the standard autoregressive paradigm, making it less straightforward to integrate into unified multimodal models built around next-token prediction.

The Missing Piece: Structure-Aware Parallel Generation Within Standard AR Models

What the paper identifies as absent from the literature is a method that achieves all of the following simultaneously:

No architectural modifications to the autoregressive transformer
No additional models (draft models, refinement networks)
No specialized tokenizers (works with standard VQGAN, MAGVIT-v2)
Preservation of the autoregressive property (each prediction conditioned on all previously generated tokens)
Substantial speedup (3-10× reduction in generation steps with minimal quality loss)

This gap is the specific motivation for PAR. The paper's key insight is that the feasibility of parallel generation is not determined by the autoregressive framework itself but by which specific tokens are selected for parallel prediction. By identifying that visual token dependencies are spatially structured — strong locally, weak at distance — the paper can group tokens for parallel generation in a way that respects the underlying dependency structure while dramatically reducing the number of sequential steps.

How the Paper Positions Itself

The paper positions PAR as occupying a precise niche in the design space of efficient visual generation (Section 2, implicitly through Table 2 comparisons):

Approach	Steps	Quality	AR Property	Architecture Change
Standard AR (LlamaGen)	576	Excellent ✓	✓	None
MaskGIT	8	Degraded ✗	✗	New paradigm
VAR	10	Excellent ✓	Modified	New tokenizer
PAR (this paper)	51-147	Comparable ✓	✓	None

The core philosophical stance is that the autoregressive property is worth preserving — not because it is inherently superior for visual generation quality (indeed, VAR achieves slightly better FID), but because it enables unified multimodal modeling. The paper repeatedly emphasizes this point: "preserving the advantages of autoregressive modeling" (Section 1), "readily applicable to various visual tasks without specialized architectures or additional models" (Section 2), "compatible with standard autoregressive models without architectural modifications" (Section 1). This is not just marketing — it reflects a bet that the future of visual generation lies in large unified models that handle text, images, and video within a single autoregressive framework, and that acceleration methods must operate within that constraint to be practically relevant.

The Underlying Insight: Token Dependencies Are Spatial

The paper's most important conceptual move — and the insight that distinguishes it from prior work — is reframing the parallel generation problem from one of architecture design to one of token dependency analysis. Previous approaches asked: "How can we modify the model to generate multiple tokens at once?" The paper instead asks: "Which tokens have weak enough dependencies that they can be generated together without the model needing to be modified?"

This reframing is what enables PAR's simplicity. If the key challenge is token dependency rather than architectural limitation, then the solution involves reordering tokens before generation rather than redesigning the generation mechanism itself. The paper supports this insight through both empirical pilot studies (Figure 1b, Figure 5) and a theoretical analysis of conditional entropy (Appendix D), showing that:

Adjacent visual tokens have strong dependencies: generating them in parallel via independent sampling produces inconsistent local patterns (distorted faces, fragmented textures)
Distant visual tokens have weak dependencies: they can be generated simultaneously with minimal quality impact
Initial tokens in each region are special: even though spatially distant from each other, these tokens jointly determine global image structure and must be generated sequentially to maintain coherence (middle row of Figure 5)

This three-part observation directly motivates the paper's three design principles (Section 3.1) and provides a principled basis for the token grouping strategy that follows. It also explains why naive parallelization of adjacent tokens fails — a result that was perhaps intuitively expected but is demonstrated and analyzed with clarity in Figure 5 and Table 4(d).

Why This Problem Matters Now

The timing of this work reflects a specific inflection point in visual generation research. Several trends converge to make efficient autoregressive visual generation urgently important:

Model unification is accelerating: Models like Chameleon [52] and Emu3 [65] demonstrate that next-token prediction can unify text and visual generation within a single architecture. If this unification is the future, then the inference efficiency of the autoregressive backbone becomes the bottleneck for all modalities.
Video generation demands long sequences: A 17-frame video at modest resolution (128×128) produces 1,280 tokens with current tokenizers (Table 3). Longer videos, higher resolutions, and higher frame rates will push sequence lengths into the tens of thousands — making the sequential generation bottleneck exponentially more painful.
The scaling-efficiency tradeoff is tightening: As autoregressive visual models scale toward billions of parameters (Table 1 shows PAR up to 3.1B), the per-step computational cost grows. Reducing the number of steps becomes proportionally more valuable — a 4× step reduction on a 3B-parameter model saves far more total FLOPs than the same reduction on a 300M-parameter model.

The paper thus addresses a problem that is not merely academic but is actively blocking the practical deployment of autoregressive visual models at scale. Its contribution is less about proposing a radically new paradigm (like VAR or MaskGIT) and more about showing that the existing paradigm can be made dramatically more efficient through principled token ordering — a finding with immediate practical implications for any team building autoregressive visual generation systems.

3. Technical Approach

3.1 Reader Orientation

This paper presents PAR, a system that rearranges the order in which an autoregressive image generator produces visual tokens so that multiple tokens can be predicted simultaneously per forward pass, without changing the model architecture, tokenizer, or training objective. The core problem it solves is the sequential bottleneck of standard autoregressive visual generation — a 576×576 image encoded as 24×24 tokens requires 576 sequential prediction steps — and the solution takes the form of a dependency-aware token grouping strategy: identify which tokens have weak enough mutual dependencies to be safely generated in parallel, group them together, and maintain sequential generation only where strong dependencies demand it.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four conceptual components, all implemented within a standard autoregressive transformer:

Token Grid Partitioning — The 2D grid of visual tokens (e.g., 24×24) is divided into M×M spatial regions (e.g., 2×2 or 4×4). Each region is a contiguous block of tokens that shares strong internal dependencies.
Initial Anchor Generation (Stage 1) — One anchor token per region is generated sequentially (region by region, M² steps total), establishing a globally coherent structural scaffold before any parallel generation begins. This creates M² tokens that provide cross-region context for all subsequent predictions.
Cross-Region Parallel Generation (Stage 2) — After anchors are placed, tokens at the same relative position within each region are predicted simultaneously across all M² regions in each forward pass. For example, all "upper-left-corner" tokens across regions are predicted together, then all "one-step-right" tokens, and so on. This reduces the remaining generation steps from (total_tokens - M²) to (total_tokens - M²) / M².
Group-wise Attention with Global Autoregression — Within each parallel group, bidirectional attention allows tokens to see each other (enriching local context), while causal attention between groups preserves the autoregressive property. A short transition sequence of learnable embedding tokens signals the shift from sequential to parallel generation mode.

Information flows as follows: class token enters → Stage 1 generates one anchor token per region sequentially (1,2,...,M²) → learnable transition tokens are inserted → Stage 2 predicts groups of M² tokens in parallel, each group conditioned on all previous anchors and parallel groups → final token sequence is rearranged back to the original spatial order for decoding.

3.3 Roadmap for the Deep Dive

First, the formal analysis of token dependencies and parallel generation feasibility — why some tokens can be parallelized and others cannot, grounded in the joint distribution factorization and the conditional entropy analysis from Appendix D.
Second, the three design principles derived from this analysis, which serve as constraints governing all subsequent design choices.
Third, the token grouping mechanism and the two-stage generation procedure — how the token grid is partitioned, how anchors are sequenced, and how cross-region groups are formed.
Fourth, the attention pattern design — why standard causal masking fails for parallel groups, how group-wise bidirectional attention works, and how global autoregression is preserved.
Fifth, the model implementation details — the learnable transition tokens, 2D Rotary Position Embeddings, and the sequence structure that makes the whole scheme work within a standard transformer architecture.
Sixth, the extension to video generation — how the framework applies to spatiotemporal token grids and why temporal parallelization is deferred.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that the feasibility of parallel token prediction in autoregressive visual generation is governed by token dependency structure, not by any architectural limitation, and that by organizing tokens based on dependency strength rather than spatial proximity, one can achieve dramatic speedups without modifying the model.

Token Dependencies and the Parallel Generation Problem

The paper begins with a careful analysis of why naive parallel token prediction fails. Understanding this analysis is essential because every design choice in PAR follows from it — the method is not an arbitrary heuristic but a direct response to the structure of visual token dependencies.

The joint distribution factorization. Standard autoregressive generation samples from the joint distribution over visual tokens by factorizing it via the chain rule:

$P(v_1, v_2, \ldots, v_n) = \prod_{i=1}^{n} P(v_i \mid v_1, v_2, \ldots, v_{i-1})$

where $v_i$ is the i-th visual token in the generation sequence and $P(v_i \mid v_1, \ldots, v_{i-1})$ is the conditional distribution predicted by the model at step i.

What it computes: this equation says that the probability of an entire token sequence can be expressed as the product of individual token probabilities, where each token's probability is conditioned on all tokens that came before it. At generation time, we sample $v_1$ from its distribution, then sample $v_2$ conditioned on the realized $v_1$ , then $v_3$ conditioned on $(v_1, v_2)$ , and so on — exactly 576 sequential sampling operations for a 24×24 grid.

Why this form: the chain rule is exact — it involves no approximation. The factorization is always valid for any joint distribution. The cost is that sampling requires one step per token, because each token depends on all previous tokens through the conditioning.

What happens with parallel sampling. When we attempt to generate $k$ tokens simultaneously rather than one at a time, we are implicitly assuming that their joint conditional distribution factorizes into a product of independent distributions:

$P(v_{i+1}, \ldots, v_{i+k} \mid v_1, \ldots, v_i) \stackrel{?}{=} \prod_{j=1}^{k} P(v_{i+j} \mid v_1, \ldots, v_i)$

where the left-hand side is the true joint distribution of the next k tokens given the history, and the right-hand side is the product of marginals that results from independently sampling each token from its conditional distribution (without conditioning each token on the other parallel tokens).

What this means in practice: for the equality to hold (i.e., for parallel sampling to produce correct samples), the k tokens being generated together must be conditionally independent given the history. If they are not — if token $v_{i+2}$ depends strongly on token $v_{i+1}$ in ways not captured by conditioning only on $v_1, \ldots, v_i$ — then independent sampling produces tokens from an incorrect distribution, manifesting as inconsistent local patterns.

The spatial structure of visual dependencies. The paper's central empirical claim (supported by the pilot study in Section 3.1 and the conditional entropy analysis in Appendix D) is that for visual tokens:

Adjacent tokens are strongly dependent. Two neighboring patches in an image share edges, textures, and object boundaries. If you sample the left-half of a tiger's eye independently from the right-half (conditioned only on distant context but not on each other), they may not align — producing the distorted tiger faces in Figure 1b.
Distant tokens are weakly dependent. A patch in the top-left corner and a patch in the bottom-right corner of an image typically belong to different objects or background regions. Knowing the value of one provides relatively little information about the other beyond what is already captured in the shared global context.
Initial tokens are a special case. The first token generated in each spatial region plays an outsized role — it effectively "commits" that region to a particular structural interpretation (e.g., "this region contains the left edge of a face," "this region contains sky texture"). If all regions' first tokens are generated in parallel without awareness of each other's decisions, the regions may make incompatible structural commitments, producing images with duplicated objects, misaligned boundaries, or incoherent global layouts (middle row of Figure 5).

Conditional entropy quantification (Appendix D). The paper formalizes this intuition using conditional entropy. For a token $v_k$ and a conditioning set $\{v_j\}_{j<k}$ , the conditional entropy $H(v_k \mid \{v_j\}_{j<k})$ measures how much uncertainty remains about $v_k$ after observing $\{v_j\}_{j<k}$ . Lower entropy means the conditioning set is more informative (stronger dependency); higher entropy means the conditioning set provides less information (weaker dependency).

The paper estimates this using a residual covariance approach. The generative relationship is modeled as:

$v_k = f(\{v_j\}_{j<k}) + \epsilon_k$

where $f(\cdot)$ is a deterministic function (learned via a parameterized model $f_{\theta}$ ) and $\epsilon_k$ is the residual error term.

What it computes: for any target token $v_k$ and conditioning set, a neural network is trained to predict $v_k$ from $\{v_j\}_{j<k}$ . The residuals (prediction errors) are collected across 10,000 ImageNet images, and their covariance matrix $\Sigma$ is computed. The conditional entropy is then upper-bounded by the determinant of this covariance:

$H(v_k \mid \{v_j\}_{j<k}) \leq \frac{1}{2}\log((2\pi e)^d |\Sigma|)$

where $d$ is the token feature dimension and $|\Sigma|$ is the determinant of the residual covariance matrix.

Why this form: the inequality follows from maximum entropy theory — for a given covariance structure, the Gaussian distribution maximizes entropy, so the Gaussian entropy with covariance $\Sigma$ provides an upper bound on the true conditional entropy. The determinant $|\Sigma|$ captures the volume of the residual error distribution: larger determinant means more residual uncertainty (higher entropy, weaker dependency), smaller determinant means less residual uncertainty (lower entropy, stronger dependency).

Key findings from the entropy analysis. The entropy maps in Figure 11 show that when conditioned on a reference token (blue square), nearby tokens exhibit lower conditional entropy (darker red — stronger dependency) while distant tokens exhibit higher conditional entropy (lighter — weaker dependency). Crucially, Figure 12 compares the entropy increase when switching from sequential to parallel generation under two ordering strategies:

Raster scan parallelization (predict consecutive tokens simultaneously): large entropy increases (Figure 12f), indicating that adjacent tokens are highly dependent and parallel prediction introduces substantial uncertainty.
PAR's cross-region parallelization (predict tokens from different spatial blocks simultaneously): small entropy increases (Figure 12c), indicating that non-local tokens have weak dependencies and parallel prediction introduces minimal additional uncertainty.

This quantitative analysis provides the theoretical foundation for the paper's design principles.

The Three Design Principles

From the dependency analysis, the paper distills three design principles that constrain the parallel generation strategy (stated explicitly in Section 3.1):

Principle 1: Generate initial tokens for each region sequentially to establish proper global structure. The first token in each spatial region serves as an anchor that determines the region's structural role in the image. These anchors must be generated with full awareness of each other — if region A's anchor commits to "this is sky" and region B's anchor commits to "this is ground," they need to be mutually consistent about where the horizon falls. Sequential generation ensures that later anchors can condition on earlier anchors' decisions.

Principle 2: Maintain sequential generation within local regions where dependencies are strong. Within a contiguous spatial block (e.g., a 12×12 patch), adjacent tokens share edges, textures, and object boundaries. The raster scan order within each region is preserved, meaning that if we think of tokens position-by-position across regions, the tokens at each position form a group, but within each region the tokens advance one position at a time in coordination with all other regions.

Principle 3: Enable parallel generation across regions where dependencies are weak through proper token organization. Tokens at the same relative position in different regions (e.g., the (3,7) position in the top-left block and the (3,7) position in the bottom-right block) are spatially distant and belong to different local contexts. Their dependencies are weak enough that they can be generated simultaneously via independent sampling from their respective conditional distributions.

These principles act as constraints, not as an algorithm — they define what a valid parallel generation strategy must satisfy. The remainder of Section 3.2 and 3.3 is about how to operationalize these principles within a standard autoregressive transformer.

Token Grid Partitioning and Cross-Region Grouping

The first mechanical step in PAR is to reorganize the token sequence so that tokens destined for parallel generation are adjacent in the sequence order.

Grid partitioning. Given a visual token grid of size $H \times W$ (e.g., 24×24 for ImageNet at 384×384 with 16× downsampling), the grid is partitioned into $M \times M$ equally-sized spatial regions. Each region contains $k = (H/M) \times (W/M)$ tokens. For the default PAR-4× setting, $M=2$ , producing 4 regions each containing $12 \times 12 = 144$ tokens. For PAR-16×, $M=4$ , producing 16 regions each containing $6 \times 6 = 36$ tokens.

Why square partitions? Because they preserve the aspect ratio of local regions, ensuring that each region's internal raster scan follows natural 2D spatial progression. Non-square partitions (e.g., 1×4 strips) would create regions with unnatural aspect ratios and potentially different dependency structures along the long vs. short dimension.

Token indexing convention. Let $v^{(r)}_j$ denote the token at position $j$ (where $j \in \{1, \ldots, k\}$ indexes the raster-scan position within a region) in region $r$ (where $r \in \{1, \ldots, M^2\}$ indexes the regions, typically in raster-scan order of the regions themselves).

Cross-region grouping. The key reorganization is to group tokens by their within-region position $j$ across all regions:

$\Big\{ [v^{(1)}_1, \dots, v^{(M^2)}_1], [v^{(1)}_2, \dots, v^{(M^2)}_2], \dots, [v^{(1)}_k, \dots, v^{(M^2)}_k] \Big\}$

where each bracket $[v^{(1)}_j, \ldots, v^{(M^2)}_j]$ is a group of $M^2$ tokens — all at the same relative position $j$ in their respective regions — that will be generated in parallel.

What this accomplishes: instead of the original raster scan sequence that visits every token in region 1, then every token in region 2, etc. (producing strong local dependencies between consecutive tokens), the reorganized sequence visits all regions' position-1 tokens, then all regions' position-2 tokens, and so on. This means tokens that are adjacent in the generation sequence are spatially distant in the image (different regions), and therefore weakly dependent — exactly what we need for safe parallel prediction.

Why this grouping and not alternatives: one might consider grouping tokens diagonally (all tokens on the same diagonal of the image) or randomly. The region-based grouping has two advantages: (1) it respects the 2D spatial structure, making it natural to encode with 2D positional embeddings, and (2) it ensures that within each parallel group, tokens are approximately equidistant, producing uniform dependency weakness rather than a mix of weak and moderate dependencies.

Stage 1: Sequential Generation of Region Anchors

Before any parallel generation begins, the model generates one initial token for each region sequentially. This stage addresses Principle 1 — establishing global structural coherence before regions start generating independently.

Generation procedure. Starting with the first region (typically top-left in the region raster order), the model generates anchor tokens one region at a time:

$v^{(i)}_1 \sim P(v^{(i)}_1 \mid v^{(<i)}_1), \quad i \in \{1, \ldots, M^2\}$

where $v^{(i)}_1$ is the first token of region $i$ (the anchor), $v^{(<i)}_1$ denotes the anchors of all regions preceding region $i$ in the generation order, and $P(\cdot \mid \cdot)$ is the model's predicted conditional distribution.

What this computes: for $M=2$ (4 regions), this is 4 sequential generation steps. For $M=4$ (16 regions), this is 16 sequential steps. Each anchor is generated with full knowledge of all previously generated anchors, ensuring that region $i$ 's structural commitment is consistent with regions $1$ through $i-1$ .

Why sequential: as demonstrated in Figure 5 (middle row), generating all anchors in parallel leads to incoherent global structure — regions make conflicting decisions about what objects appear where, producing images with duplicated animal bodies or misaligned scene elements. The sequential procedure imposes a causal ordering on structural decisions: earlier regions "set the stage," and later regions adapt to fit.

Why only one anchor per region: a single token is the minimal commitment needed to establish a region's structural role. Generating more tokens sequentially per region would increase the sequential step count without proportionate benefit, since the first token already constrains the region's high-level interpretation. The paper's ablation (Table 4a) confirms that these initial sequential tokens are crucial: removing them degrades FID from 2.61 to 3.67.

Region generation order. The paper uses raster-scan order over regions (top-left, top-right for M=2; then bottom-left, bottom-right). This choice is natural but not deeply justified — the paper does not ablate alternative region orderings (e.g., center-out, random permutation). The implicit assumption is that a consistent spatial order provides a regular structure that the model can learn to exploit.

Transition tokens. Between Stage 1 (sequential anchors) and Stage 2 (parallel groups), the paper inserts $n-1$ learnable embedding tokens (where $n = M^2$ is the parallel group size). These are labeled $[M1, M2, M3]$ in Figure 4a for the 4× case. The paper states these tokens "help the model enter parallel prediction mode." Conceptually, they serve as a learned signal that the generation regime is changing — from "you are building global structure one region at a time" to "you are now filling in details across all regions simultaneously." These tokens are optimized during training and have the same dimension as regular visual tokens. They participate in attention but do not correspond to any output token (they appear only in the input, not the target).

Stage 2: Parallel Cross-Region Generation

After the anchors are established, the remaining $k-1$ positions within each region (where $k$ is the number of tokens per region) are generated in parallel across regions.

Generation procedure. For each position $j \in \{2, \ldots, k\}$ (following raster order within each region), the model generates all $M^2$ tokens at that position simultaneously:

$\{v^{(r)}_j\}_{r=1}^{M^2} \sim P(\{v^{(r)}_j\}_{r=1}^{M^2} \mid v_{<j})$

where $\{v^{(r)}_j\}_{r=1}^{M^2}$ is the set of all tokens at position $j$ across all $M^2$ regions, and $v_{<j}$ denotes the conditioning context: all anchor tokens ( $v^{(r)}_1$ for all $r$ ) plus all tokens from positions $1$ through $j-1$ across all regions.

What this means operationally: at the first parallel step (position $j=2$ ), the model takes as input all anchors (generated sequentially) and the transition tokens, then produces $M^2$ output tokens — one per region — in a single forward pass. At the second parallel step (position $j=3$ ), it takes all anchors plus all position-2 tokens as input, and produces all position-3 tokens. This continues until all $k$ positions are generated.

Step count calculation. The total generation steps are:

$\text{Steps}_{\text{PAR}} = M^2 + \frac{H \times W - M^2}{M^2}$

where $M^2$ steps come from Stage 1 (sequential anchors), and the remaining $H \times W - M^2$ tokens are generated in groups of $M^2$ per step during Stage 2. For ImageNet (24×24 = 576 tokens):

PAR-4× ( $M=2$ ): $4 + (576-4)/4 = 4 + 143 = 147$ steps (3.9× reduction from 576)
PAR-16× ( $M=4$ ): $16 + (576-16)/16 = 16 + 35 = 51$ steps (11.3× reduction from 576)

Independent sampling of parallel tokens. The critical implementation detail is that within each parallel group, tokens are sampled independently from their respective conditional distributions. The model outputs $M^2$ separate probability distributions (one per region-position), and each token is sampled via the standard sampling strategy (top-k, temperature) without conditioning on the other tokens being generated in the same step. This is why the tokens must be weakly dependent: independent sampling is only valid when the joint distribution approximately factorizes.

What about the autoregressive property? Each token's prediction is still conditioned on all previously generated tokens — including tokens from earlier parallel groups and, crucially, the anchor tokens. The only "missing" conditioning relative to full sequential generation is the tokens within the same parallel group: token $v^{(2)}_j$ does not condition on token $v^{(1)}_j$ (same position, different region) because they are generated simultaneously. The paper's claim — validated by the empirical results — is that this conditioning gap is harmless because these tokens are spatially distant and weakly dependent.

Internal raster order within regions. Within each region, tokens progress in standard raster scan order from position 1 to position $k$ . This means that when the model generates the position- $j$ group, each region's position- $j$ token can condition on that same region's position- $(j-1)$ token (generated in the previous parallel step), which is spatially adjacent within the region. This preserves the strong local dependencies that Principle 2 demands — the sequential progression within regions happens across parallel steps, not within them.

Attention Pattern Design: Group-wise Bidirectional Attention

The standard causal attention mask in autoregressive transformers prevents each token from attending to future tokens. Naively applied to PAR's parallel groups, this would severely restrict the context available to each token.

The naive causal masking problem. Figure 4b (right) illustrates the issue. In the parallel generation setting, if we simply apply causal masking to the reordered sequence, token $6b$ (position-6 token in region b) can only attend to tokens up to $5b$ in the sequence — it cannot see tokens $5a$ , $5c$ , or $5d$ from the previous parallel group, even though those tokens were generated in the same step as $5b$ and are fully available. This is because causal masking respects sequence position, and $5c$ and $5d$ appear after $5b$ in the reordered sequence.

The consequence: with naive causal masking, each token in a parallel group sees only a subset of the previous group — specifically, only the tokens at its position or earlier in the sequence. This drastically reduces the effective context for each prediction, making it harder for the model to maintain global coherence.

The solution: group-wise bidirectional attention. The paper enables bidirectional (full) attention within each parallel group while maintaining causal attention between groups. Specifically:

Within a group (e.g., all position- $j$ tokens $[v^{(1)}_j, \ldots, v^{(M^2)}_j]$ ): these tokens can attend to each other. This is implemented by not masking attention between tokens in the same group.
Between groups (e.g., tokens in group $j$ attending to tokens in groups $<j$ , and to the anchors, class token, and transition tokens): standard causal masking applies — group $j$ can attend to all previous groups but not to future groups.

Operationally: when predicting the position- $j$ group, every token in that group can attend to every token in all previous groups (position $1$ through $j-1$ ) AND to every other token in the position- $j$ group itself. It cannot attend to tokens in future groups (position $>j$ ). This is illustrated in Figure 4b (left), where the colored cells show the visible context: all tokens in group $[5a-5d]$ are visible when predicting any token in group $[6a-6d]$ .

What this accomplishes: the bidirectional attention within groups means that even though tokens are sampled independently (no conditioning on same-group tokens), the model's internal representations for each token are computed with awareness of all other tokens in the group. This is not equivalent to conditioning the sampling distribution on same-group tokens (that would require autoregressive ordering within the group), but it enriches the feature representations used to compute each token's output distribution. The ablation in Table 4(c) confirms that this matters: enabling full attention within groups reduces FID from 3.64 (causal only) to 2.61.

Why this preserves global autoregression: the crucial constraint is that no token can attend to tokens that haven't been generated yet in the autoregressive order. Since all tokens in group $j$ are generated simultaneously (no internal ordering), there is no "future" within the group — any within-group attention pattern is valid. Causal masking between groups ensures that group $j$ cannot see group $j+1$ , preserving the autoregressive property that predictions depend only on the past.

Compatibility with KV-cache: because the attention pattern is still causal at the group level (groups are processed sequentially), the standard KV-cache optimization remains applicable. The model can cache keys and values for all tokens up to the current group and reuse them for subsequent groups, avoiding recomputation. Within each group, bidirectional attention requires computing attention between all pairs of tokens in the group, but this is a small $O(M^4)$ overhead relative to the $O(M^2 \times \text{total\_tokens})$ cost of attending to the full history.

Model Implementation Details

The paper implements PAR within a standard autoregressive transformer architecture (Llama-style), with several specific design choices that make the parallel generation scheme work.

Input sequence structure. As shown in Figure 4a, the input sequence has three distinct parts concatenated together:

Class token [C]: a learned embedding representing the class condition (for class-conditional generation). This appears at the start of every sequence.
Stage 1 tokens [1, 2, \ldots, M^2]: the sequentially generated anchor tokens. During training, these are the ground-truth token embeddings from the dataset. During inference, they are the tokens sampled by the model one at a time.
Transition tokens [M1, M2, \ldots, M_{n-1}]: $n-1$ learnable embedding vectors (where $n = M^2$ ) that have no corresponding visual token but serve to signal the transition to parallel generation mode. These are model parameters optimized during training.
Stage 2 group tokens [[5a, 5b, 5c, 5d], [6a, 6b, 6c, 6d], \ldots]: the parallel groups, each containing $M^2$ tokens. During training, these are the ground-truth tokens. During inference, they are the tokens sampled by the model in groups of $M^2$ .

Target sequence. The target sequence is offset from the input sequence by $n = M^2$ positions. Specifically:

During Stage 1, the target at input position $i$ (which contains the class token and previous anchors) is the next anchor token $v^{(i+1)}_1$ .
During the transition, the target at the position containing the last anchor plus the beginning of the transition tokens is the first parallel group $[5a, 5b, 5c, 5d]$ .
During Stage 2, the target at the position containing the previous group is the next group $[6a, 6b, 6c, 6d]$ , etc.

This offset of $n$ positions between input and target is what enables predicting $n$ tokens simultaneously: the model's output at $n$ consecutive positions together form the $n$ tokens of the next group.

2D Rotary Position Embeddings (RoPE). Standard 1D RoPE encodes the position of each token in the sequence (position 0, 1, 2, ...). However, because PAR reorders tokens — the sequence position no longer corresponds to the spatial position — 1D RoPE would lose spatial information. The paper instead uses 2D RoPE, which encodes each token's $(x, y)$ coordinates in the original 2D token grid, regardless of where it appears in the reordered sequence.

Why 2D RoPE: this preserves the spatial relationships that the model learned during pretraining (or during standard autoregressive training, where sequence position equals raster scan position). If token $v^{(2)}_5$ originally came from position $(3, 7)$ in the image grid, it receives the positional encoding for $(3, 7)$ even though it appears at sequence position 42 in the reordered sequence. This means the attention mechanism can still learn position-dependent patterns ("tokens in the top-left attend to other tokens in the top-left") based on spatial proximity rather than sequence proximity.

Why learnable transition tokens rather than just starting parallel groups immediately: the model needs a signal that the generation mode is shifting. In standard autoregressive training, the model always predicts one next token given a prefix of tokens. In PAR's parallel mode, it predicts $n$ next tokens. The transition tokens provide a learned "mode switch" in the embedding space — they are optimized to produce hidden states that prepare the model to output $n$ distributions rather than one. Without them, the model would need to infer the mode switch purely from sequence structure, which might be learnable but would likely require more training data or capacity.

Training procedure. The model is trained with standard next-token prediction loss (cross-entropy) on the target tokens. During training, teacher forcing is used — the ground-truth tokens are provided as input regardless of what the model would have predicted. The loss is computed over all $H \times W$ visual token positions (the class token and transition tokens are not part of the loss). The training data consists of pre-tokenized images from ImageNet-1K, with the token sequence reorganized into the PAR order before being fed to the model. All other training hyperparameters (optimizer, learning rate schedule, etc.) follow the baseline LlamaGen configuration and are listed in Table 5 of the paper.

Inference procedure. During inference:

The class token [C] is provided.
Stage 1: the model generates one anchor token per region sequentially. At each step, the previously generated anchor(s) are fed back as input, and the next anchor is sampled.
The transition tokens are appended to the input (they are fixed learned embeddings, not generated).
Stage 2: for each position $j = 2, \ldots, k$ , the model takes the current prefix (all previous tokens) as input and generates $M^2$ tokens in parallel. These tokens are sampled independently using the standard sampling strategy (top-k with temperature, optionally with classifier-free guidance). The sampled tokens are appended to the input for the next step.
After all tokens are generated, they are rearranged from the PAR order back to the original spatial order (by inverting the cross-region grouping and removing transition tokens) and decoded by the VQGAN decoder to produce the final image.

Classifier-free guidance (CFG). Like LlamaGen, PAR uses CFG at inference time. During training, the class token is randomly dropped (replaced with a null-class embedding) with probability 0.1. At inference, the model is run twice — once with the true class token and once with the null-class token — and the logits are combined as:

$\text{logits}_{\text{CFG}} = \text{logits}_{\text{unconditional}} + s \cdot (\text{logits}_{\text{conditional}} - \text{logits}_{\text{unconditional}})$

where $s$ is the guidance scale (values in Table 5, e.g., 1.435 for PAR-XXL). This requires two forward passes per generation step but substantially improves sample quality.

Extension to Video Generation

The paper extends PAR to video generation with minimal modifications, treating the spatiotemporal token grid similarly to the spatial grid in the image case.

Video tokenization. Videos from UCF-101 (17 frames at 128×128 resolution) are tokenized using a reproduced MAGVIT-v2 tokenizer that applies 8× spatial compression and 4× temporal compression. The result is a 3D token grid of size $T \times H \times W = 5 \times 16 \times 16 = 1280$ tokens per video clip. Each "token" in this grid already aggregates information from multiple input frames (due to temporal compression), so it represents a spatiotemporal patch rather than a purely spatial one.

Applying PAR to the spatial dimensions. The paper applies the same parallel generation strategy along the spatial dimensions (H and W) while keeping the temporal dimension sequential. Specifically, within each temporal frame $t$ , the 16×16 spatial token grid is partitioned into $M \times M$ regions, anchors are generated sequentially, and then tokens at corresponding positions across regions are generated in parallel. The temporal frames themselves are processed in order (frame 0, then frame 1, ..., then frame 4).

Step count for video. With $M=2$ (PAR-4×): each spatial frame requires $4 + (256-4)/4 = 4 + 63 = 67$ steps. Across 5 temporal frames: $5 \times 67 = 335$ steps. But wait — the paper reports 323 steps for PAR-4× (Table 3), not 335. The discrepancy likely arises because the anchor generation spans only the first temporal frame (or some frames share anchors) — the paper does not specify the exact video generation procedure in sufficient detail to resolve this. With $M=4$ (PAR-16×), the reported step count is 95 (Table 3), a 13.5× reduction from the 1280-step sequential baseline.

Why not parallelize temporally? The paper explicitly states (Section 3.3): "we also explored parallel generation along the temporal dimension, we found it less effective than spatial parallelization. This is because temporal dependencies exhibit stronger sequential characteristics that are fundamental to video coherence." In other words, the frame-to-frame progression in video carries strong causal structure — frame $t+1$ depends heavily on frame $t$ in ways that cannot be captured by conditioning only on distant context. Generating tokens from multiple frames in parallel would require independent sampling of strongly dependent temporal neighbors, producing temporal inconsistencies (flickering, motion artifacts). The paper leaves temporal parallelization as future work.

3D positional embeddings. For video, 2D RoPE is extended to 3D, encoding each token's $(t, x, y)$ coordinates in the spatiotemporal grid. This preserves both spatial and temporal position information under the reordered sequence.

Video-specific hyperparameters. The video model uses a smaller architecture (792M parameters, matching the PAR-XL image model scale approximately) and is trained for 3000 epochs on UCF-101 (Table 6). Classifier-free guidance uses a guidance scale of 1.15 with top-k sampling (k=8000), reflecting the different distribution characteristics of video tokens vs. image tokens.

Summary of Design Choices and Their Justifications

Why region-based grouping over diagonal or random grouping: region-based grouping creates tokens that are approximately equidistant within each parallel group, ensuring uniform (weak) dependencies. Diagonal grouping would mix near and far tokens within the same group. Random grouping would make the model's job harder by requiring it to learn arbitrary long-range dependencies.

Why exactly one anchor per region: more anchors would increase sequential steps without proportionate benefit; fewer (zero) would violate Principle 1 and produce incoherent global structure, as demonstrated in the ablation (Table 4a).

Why group-wise bidirectional attention rather than full causal or full bidirectional: full causal within groups restricts context too much (FID degradation in Table 4c); full bidirectional across all tokens would violate the autoregressive property and prevent the use of standard autoregressive training objectives and KV-cache optimizations.

Why 2D/3D RoPE rather than learned position embeddings: RoPE naturally generalizes to unseen positions (enabling zero-shot higher-resolution generation, Appendix C), and 2D RoPE preserves the spatial relationships that the model learned during pretraining on raster-scan sequences. Learned absolute position embeddings would need to be retrained for every different parallelization scheme.

Why learnable transition tokens rather than structural cues: the model needs an explicit signal that the prediction mode has changed. Structural cues (e.g., observing that the sequence pattern has shifted) would require the model to infer the mode switch from data, which is learnable but less reliable and would likely require more training.

Why no additional models or architectural changes: this is the paper's defining constraint. Adding a draft model, a refinement network, or a specialized attention pattern would increase system complexity, break compatibility with standard transformer implementations, and reduce the approach's applicability to unified multimodal models. Every component of PAR — reordering, 2D RoPE, learnable transition tokens, group-wise attention — operates within the standard autoregressive transformer framework and can be implemented by modifying the input pipeline and attention mask, without touching the model architecture itself.

4. Key Insights and Innovations

Innovation 1: Reframing Parallel Generation as a Token Dependency Problem Rather Than an Architectural One

The paper's most fundamental conceptual move is what it rejects: the assumption that accelerating autoregressive generation requires modifying the model architecture, training a separate draft model, or switching to a non-autoregressive paradigm. Before this work, the dominant approaches to speeding up sequential generation fell into three camps — speculative decoding (add a draft model; Chen et al., 2023; Leviathan et al., 2023), Jacobi-style iterative refinement (add a refinement process; Song et al., 2021; Kou et al., 2024), or non-autoregressive modeling (abandon autoregression entirely; Chang et al., 2022). All three camps implicitly accept that the autoregressive transformer as is cannot generate multiple tokens per step without quality collapse.

PAR's reframing is to ask a different question entirely: not "how do we modify the generation mechanism?" but "which tokens have weak enough mutual dependencies that they can be independently sampled without the model needing to be modified?" This shifts the problem from architecture design to input organization — from how the model generates to what it generates in what order. The insight is that the autoregressive property (each prediction conditions on all prior tokens) is not the bottleneck; the bottleneck is which specific tokens are chosen to be predicted next. If you can identify tokens whose conditional independence approximately holds given the history, you can predict them simultaneously without architectural changes.

This is a fundamental reframing, not an incremental relaxation. It is analogous to the difference between "how do we make matrix multiplication faster?" (engineering) and "can we reorder the computation so that some multiplications become unnecessary?" (algorithmic insight). The paper's contribution is the latter: it identifies that the feasibility of parallel generation is a property of the data's dependency structure (which is spatially organized for visual tokens), not a property of the model's architecture. This is what enables the extreme simplicity of the solution — a sequence reordering plus an attention mask — because the hard problem (handling strong dependencies) is avoided by never asking the model to handle them in parallel.

The evidence for this reframing's validity is the sharp contrast between naive parallelization (raster scan order, adjacent tokens predicted together) and PAR's cross-region grouping (Table 4d). Both use the same architecture, the same training objective, and the same number of parallel tokens per step. But raster scan parallelization degrades FID from 2.62 to 5.64 (a catastrophic collapse), while cross-region parallelization maintains quality at 2.61. The only difference is which tokens are grouped — proving that token selection, not architecture, is the governing factor. The conditional entropy analysis in Appendix D provides the theoretical lens: Figure 12 shows that predicting adjacent tokens simultaneously produces large entropy increases while predicting distant-region tokens produces small ones.

Innovation 2: The Three-Stage Design Principle as a Diagnostic Framework for Where Parallel Generation Fails

The paper's second conceptual contribution is not the specific PAR algorithm but the diagnostic principles it extracts from the failure modes of naive parallelization. Section 3.1 identifies three distinct ways that parallel token prediction can go wrong, each corresponding to a different type of token dependency violation:

Global structure collapse: when the first tokens of different spatial regions are generated independently without awareness of each other's structural commitments, the image fragments into incompatible local interpretations (Figure 5, middle row).
Local detail corruption: when adjacent tokens within a region are generated in parallel, their strong mutual dependencies (edges, textures, boundaries) are violated, producing distorted local patterns (Figure 5, bottom row; Figure 1b).
Context impoverishment: when naive causal masking restricts parallel tokens to seeing only a subset of the available history, the model loses information that was available in sequential generation (Table 4c, causal vs. full attention within groups).

What makes this a conceptual contribution rather than just a list of observations is that these failure modes are diagnostically separable — the paper demonstrates each one independently, with its own ablation and its own visual signature, and ties each to a specific remediation (Principle 1 → sequential anchors; Principle 2 → within-region sequential progression; Principle 3 plus group-wise attention → cross-region parallelization with bidirectional context). This provides a failure analysis framework that future work can use to debug parallel generation schemes: if your method produces duplicated body parts, you violated the global structure principle; if it produces fragmented textures, you violated the local dependency principle; if it produces globally blurry or inconsistent output, you may have a context impoverishment issue.

Prior work on parallel decoding (speculative decoding, Jacobi decoding, non-autoregressive generation) did not provide this kind of diagnostic decomposition. Those methods either work or don't work as a whole, but the failure modes are opaque — is the draft model too weak? Is the refinement not converging? Is the mask schedule suboptimal? PAR's framework decomposes the problem along the natural structure of visual data (global vs. local dependencies, spatial distance as a proxy for dependency strength), making the failure modes interpretable and directly actionable.

The three principles also serve as transferable design constraints that go beyond the specific PAR implementation. A different grouping strategy (e.g., based on semantic segmentation rather than spatial partitioning, or dynamic rather than fixed regions) would still need to satisfy these principles to succeed. The principles thus function as a kind of API contract between the token grouping mechanism and the generation quality: violate any one, and quality degrades in a predictable way.

Innovation 3: Conditional Entropy as a Quantitative Justification for Spatial Dependency Assumptions

The paper provides a theoretical analysis (Appendix D) that elevates the spatial dependency observations beyond intuition or anecdotal pilot studies. Using a residual covariance-based upper bound on conditional entropy, the paper quantitatively demonstrates that:

Conditioned on a reference token, nearby tokens have substantially lower conditional entropy (stronger dependency) than distant tokens (Figure 11). This is not surprising — it confirms the spatial locality prior that underlies virtually all visual processing — but the paper quantifies it rather than assuming it.
More importantly, Figure 12 compares the entropy increase when switching from sequential to parallel generation under two orderings. The finding is that PAR's cross-region ordering produces significantly smaller entropy increases than raster scan ordering — meaning that parallel prediction under PAR's grouping introduces less additional uncertainty than naive parallelization of consecutive tokens. The gap is visually striking in the entropy maps: Figure 12c shows mostly blue (low entropy increase) while Figure 12f shows large red regions (high entropy increase).

What makes this an innovation rather than just a validation: this analysis provides a principled metric for comparing parallelization strategies without needing to train full generative models on each candidate. One could, in principle, use this entropy-based analysis to screen candidate token groupings — compute the entropy increase for a proposed parallelization scheme on a dataset of encoded images, and use that as a predictor of generation quality degradation — before committing to expensive model training. The paper does not fully develop this idea (all PAR variants are validated through full training), but the framework is there.

This is an incremental advance (the information-theoretic tools — conditional entropy, maximum entropy bounds, residual covariance — are standard) but applied to a problem domain (ordering tokens for parallel visual generation) where such analysis was previously absent. Prior work on parallel decoding in language or vision did not provide this kind of dependency-aware quantitative justification for why their particular grouping choices worked; they relied on empirical trial-and-error. The paper's entropy analysis gives the design principles a formal floor, making PAR's design choices derivable from data rather than merely validated by data.

Innovation 4: Evidence That Initial Tokens Play an Outsize Structural Role — and That This Generalizes Beyond Standard Autoregressive Generation

The finding that generating the first token of each region sequentially is crucial (Table 4a: removing initial sequential generation degrades FID from 2.61 to 3.67, with only 3 additional steps saved) is more than an ablation result — it reveals something fundamental about how autoregressive visual generation works. The initial tokens are not just "the first tokens"; they are structural anchors that commit each spatial region to a high-level interpretation. When generated independently (in parallel), these anchors make incompatible commitments, producing duplicated objects, misaligned scene elements, and incoherent global layouts (Figure 5, middle row).

This insight has implications beyond PAR. It suggests that in any autoregressive visual model, the earliest-generated tokens carry disproportionate responsibility for global coherence — they are the "foundation" on which local detail generation builds. This connects to broader questions in autoregressive generation about error propagation (early mistakes cascade) and about the relationship between generation order and output structure. VAR [53] implicitly exploits a similar idea by generating coarse-scale structure before fine-scale details, but through a different mechanism (multi-scale token hierarchies rather than spatial anchor tokens). PAR shows that the same principle — structure before detail — can be realized within a flat token grid through careful ordering, without multi-scale tokenizers.

The paper also demonstrates that this anchor effect is specific to cross-region initial tokens, not to initial tokens in general. In standard raster-scan autoregressive generation, the first few tokens (top-left corner) do influence global structure simply because they're generated first and all subsequent tokens condition on them. But PAR's Stage 1 explicitly creates M² anchor tokens — one per region — spreading the structural commitment across the entire image rather than concentrating it in one corner. This is a novel design choice: rather than letting the autoregressive order naturally privilege early positions, PAR actively engineers which tokens serve as structural anchors by designating one per region and generating them sequentially. This gives the model M² "seeds" distributed across the image rather than one, potentially providing more robust initialization of global structure.

The significance of this finding is that it identifies initial tokens as a scarce resource in autoregressive generation — they are the only tokens generated with full sequential context before any parallelization begins — and shows that allocating this resource across spatial regions (one anchor per region) is dramatically more effective than concentrating it (generating all of the first region before moving to the next). This resource-allocation perspective on autoregressive ordering is novel and suggests future work on optimal anchor placement (e.g., non-uniform anchors based on image content, or learned anchor positions rather than fixed spatial partitioning).

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses two primary datasets: ImageNet-1K [10] at 256×256 resolution for class-conditional image generation (1,281,167 training images across 1,000 classes, evaluated on the 50,000-image validation set), and UCF-101 [48] for class-conditional video generation (approximately 9,500 training videos across 101 action categories, each processed as 17-frame clips at 128×128 resolution). The choice of these specific datasets follows established benchmarks in prior autoregressive visual generation work — LlamaGen [51] for ImageNet and MAGVIT-v2 [71] for UCF-101 — enabling direct comparison against published baselines without introducing dataset-shift confounds.
Base model(s). All experiments use the PAR model family, which are Llama-style autoregressive transformers scaled from 343M to 3.1B parameters (Table 1: PAR-L at 343M with 24 layers, PAR-XL at 775M with 36 layers, PAR-XXL at 1.4B with 48 layers, PAR-3B at 3.1B with 24 layers and wider hidden dimensions). These architectures deliberately mirror the LlamaGen [51] configurations to ensure fair comparison — any quality differences can be attributed to the parallel generation strategy rather than differences in model capacity or architecture. For video generation, a single 792M-parameter model is used (PAR-4× and PAR-16× variants). The image generation models use a VQGAN tokenizer [12] with 16,384 codebook size and 16× downsampling ratio (384×384 images → 24×24 token grids of 576 tokens). The video models use a reproduced MAGVIT-v2 tokenizer [71] with 64K vocabulary, 8× spatial compression, and 4× temporal compression (17 frames at 128×128 → 5×16×16 token grids of 1,280 tokens). This tokenizer choice is deliberate: VQGAN connects PAR to the LlamaGen baseline, while MAGVIT-v2 demonstrates compatibility with a different tokenization paradigm and vocabulary size.
Metrics. Image generation quality is evaluated using Fréchet Inception Distance (FID) [15] as the primary metric, with Inception Score (IS) and Precision/Recall [24] as complementary measures. FID measures the distributional distance between generated and real images in Inception-v3 feature space — lower is better — and is the standard metric in the generative modeling literature for ImageNet. For video generation, the primary metric is Fréchet Video Distance (FVD) [57], which extends the FID concept to video by using a 3D convolutional network to extract spatiotemporal features. All metrics are computed on the full validation/test sets (50,000 images for ImageNet; all test videos for UCF-101). Generation steps (the number of sequential forward passes) and wall-clock time (measured with batch size 1 on a single A100 GPU) are reported to quantify the speed-quality tradeoff. Wall-clock time is the metric that matters for practical deployment, while step count isolates the algorithmic acceleration from engineering optimizations (KV-cache, CUDA graphs) that could benefit all methods.
Baselines. The paper compares against a comprehensive set of methods across five paradigms:
- GAN-based: BigGAN [3] (112M params, FID 6.95), GigaGAN [20] (569M, FID 3.45), StyleGAN-XL [44] (166M, FID 2.30) — one-shot generation, representing the quality ceiling on efficiency but typically worse FID than modern methods at scale.
- Diffusion-based: ADM [11] (554M, FID 10.94, 250 steps), CDM [17] (FID 4.88, 8,100 steps), LDM-4 [42] (400M, FID 3.60, 250 steps), DiT-XL/2 [37] (675M, FID 2.27, 250 steps) — iterative denoising at various step budgets.
- Mask-based: MaskGIT [5] (227M, FID 6.18, 8 steps) — the primary non-autoregressive visual baseline, representing extreme speed at the cost of quality.
- Next-scale prediction: VAR [53] (2B, FID 1.97, 10 steps) — the strongest quality competitor, but at the cost of specialized multi-scale tokenizers and longer token sequences.
- Autoregressive: VQGAN [12] (227M-1.4B, FID 5.20-18.65, 256-1024 steps), ViT-VQGAN [68] (1.7B, FID 3.04-4.17, 1024 steps), RQTran [25] (3.8B, FID 3.80-7.55, 256 steps), and most critically LlamaGen [51] (343M-3.1B, FID 2.18-3.07, 576 steps) — the direct autoregressive baseline that PAR's architecture and training configuration mirror. For video, baselines include diffusion methods (VideoFusion [31], Make-A-Video [45], HPDM-L [46]), masked methods (MAGVIT [70], MAGVIT-v2 [71]), and autoregressive methods (CogVideo [18], TATS [13], OmniTokenizer [64], MAGVIT-v2-AR [71]).
The critical baseline is LlamaGen at matched model sizes: PAR-L vs. LlamaGen-L (both 343M), PAR-XL vs. LlamaGen-XL (both 775M), PAR-XXL vs. LlamaGen-XXL (both 1.4B), PAR-3B vs. LlamaGen-3B (both 3.1B). This matched-parameter comparison isolates the effect of the parallel generation strategy — any quality difference is due to the token ordering and parallelization, not to differences in model capacity.
Generation budget / compute accounting. The paper measures test-time compute in two complementary units. Generation steps are the algorithmic unit: one step = one forward pass through the transformer. For LlamaGen, this equals 576 steps (one per token). For PAR-4×, this equals 147 steps (4 anchors + 143 parallel groups of 4). For PAR-16×, this equals 51 steps (16 anchors + 35 parallel groups of 16). The step reduction ratio is the primary efficiency metric — 3.9× for PAR-4×, 11.3× for PAR-16×. Wall-clock time (seconds per image/video) accounts for the actual computational cost of each forward pass — parallel groups require the model to process multiple token positions simultaneously, so a PAR step is computationally heavier than a LlamaGen step. The paper reports both numbers transparently: PAR-3B-4× achieves 3.46s vs. LlamaGen-3B's 12.41s (3.6× actual speedup), and PAR-3B-16× achieves 1.31s (9.5× speedup). The gap between algorithmic step reduction (3.9×) and actual speedup (3.6×) reflects the overhead of processing larger groups per step (wider attention patterns, more output tokens). Importantly, all time measurements use batch size 1 on a single A100 GPU, and no engineering optimizations (vLLM, CUDA graphs, quantization) are applied to either PAR or LlamaGen in the main comparisons — the speedup comes purely from algorithmic changes. Appendix B verifies that PAR's algorithmic speedup is orthogonal to engineering optimizations: adding simple PyTorch compilation to PAR-3B-4× reduces latency from 3.46s to 1.15s (a further 3× improvement), and PAR-3B-16× with compilation reaches 0.43s.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. The standard practice in generative modeling benchmarks is to report metrics computed once on the full validation/test set, and the paper follows this convention. The 50,000-image ImageNet validation set is large enough that sampling variance is low for FID — though the paper does not report confidence intervals. For the entropy analysis in Appendix D, 10,000 images are randomly sampled from ImageNet to estimate conditional entropy trends; this sample size is adequate for the qualitative comparisons being made (relative entropy differences between orderings), but the paper does not report error bars on these estimates. The lack of statistical rigor is a minor limitation — typical for this subfield — but does not undermine the central claims given the large effect sizes (FID differences of 3+ points in failure cases are clearly meaningful).

Main Quantitative Results

Image Generation: Competitive Quality with Dramatic Speedup

The headline result for image generation appears in Table 2. PAR-3B-4× achieves FID 2.29 with 147 steps and 3.46s wall-clock time, compared to the baseline LlamaGen-3B at FID 2.18 with 576 steps and 12.41s. This represents a 3.9× step reduction and 3.6× actual speedup with only 0.11 FID degradation — a negligible quality cost for more than tripling throughput. At more aggressive parallelization, PAR-3B-16× achieves FID 2.88 with 51 steps and 1.31s (9.5× speedup), trading 0.7 FID for near-real-time generation.

The matched-parameter comparison across model scales reveals an important scaling trend (Table 2, comparing PAR vs. LlamaGen at each size):

PAR-L-4× (343M): FID 3.76, vs. LlamaGen-L FID 3.07 — a 0.69 gap.
PAR-XL-4× (775M): FID 2.61, vs. LlamaGen-XL FID 2.62 — essentially identical (gap of -0.01, meaning PAR is marginally better).
PAR-XXL-4× (1.4B): FID 2.35, vs. LlamaGen-XXL FID 2.34 — again essentially identical (gap of 0.01).
PAR-3B-4× (3.1B): FID 2.29, vs. LlamaGen-3B FID 2.18 — gap of 0.11.

At small scales (343M), PAR-4× degrades quality noticeably relative to the sequential baseline. But as model capacity increases, the gap shrinks dramatically — at 775M and 1.4B, PAR matches the sequential baseline within measurement noise. At 3.1B, a small gap re-emerges (0.11 FID). The paper interprets this as evidence that "increased model capacity helps mitigate the quality trade-off from parallel prediction, suggesting stronger capability in modeling joint distribution of parallel tokens" (Section 4.3). This is plausible: a larger model has more capacity to learn the residual dependencies between distant tokens that parallelization ignores, compensating for the independent sampling approximation.

Side-by-side efficiency comparison at equivalent quality (Table 2):

PAR-XXL-4× (1.4B) achieves FID 2.35 in 6.84s — faster than LlamaGen-3B (FID 2.18 in 12.41s) at slightly lower quality but roughly half the latency.
PAR-3B-16× achieves FID 2.88 in 1.31s — dramatically faster than almost any method in Table 2 (only GANs and VAR are faster, but GANs have worse FID and VAR requires specialized tokenizers), and competitive with diffusion models like DiT-XL/2 (FID 2.27 in 11.97s) despite being 9.1× faster.

Context against non-autoregressive methods:

MaskGIT [5] achieves FID 6.18 in 8 steps (0.13s) — faster but substantially worse quality. PAR-3B-16× (FID 2.88, 51 steps) represents a fundamentally different point on the speed-quality Pareto frontier: much higher quality at still-fast speeds, without abandoning the autoregressive framework.
VAR [53] achieves FID 1.97 in 10 steps (0.27s) — better quality and faster. This is the paper's most challenging comparison. VAR outperforms PAR on both axes, but the paper argues (Section 4.2.1) that "our method maintains a simpler framework with fewer tokens per image and preserves the pure autoregressive nature, making it more flexible for multi-modal integration." This is a qualitative argument about ecosystem compatibility rather than a quantitative superiority claim — VAR is genuinely stronger on ImageNet FID, and the paper does not contest this.

What the FID numbers obscure: precision and recall. Table 2 also reports Precision and Recall [24]. Precision measures whether generated images are realistic (high precision = few unrealistic samples), while Recall measures whether the model covers the diversity of the training distribution (high recall = no mode collapse). PAR-3B-4× achieves Precision 0.82 and Recall 0.58, compared to LlamaGen-3B's Precision 0.81 and Recall 0.58. These are essentially identical, indicating that parallel generation preserves both sample fidelity and distribution coverage. IS (Inception Score) shows a similar pattern: PAR-3B-4× achieves 255.5 vs. LlamaGen-3B's 263.3 — a small difference consistent with the slight FID gap.

Video Generation: Larger Speedups with Minimal Degradation

The video generation results in Table 3 demonstrate that PAR's approach transfers to a more demanding domain with even greater relative speedups. PAR-4× achieves FVD 99.5 with 323 steps and 11.27s, compared to the sequential baseline PAR-1× at FVD 94.1 with 1,280 steps and 43.30s — a 3.96× step reduction and 3.84× speedup with only 5.4 FVD increase. PAR-16× further accelerates to 95 steps and 3.44s (12.6× speedup) with FVD 103.4, a 9.3 FVD increase over the sequential baseline.

The sequential baseline itself is competitive: PAR-1× at FVD 94.1 outperforms MAGVIT-v2-AR [71] at FVD 109 (both autoregressive, same tokenizer), suggesting that the paper's training setup is strong independent of the parallelization method. The parallelization cost (FVD increase from 94.1 to 99.5 for 4×, and to 103.4 for 16×) is modest relative to the speedup — less than 10% FVD increase for a 3.8× speedup, and less than 10% for 12.6×.

The gap between algorithmic step reduction and actual speedup is slightly larger for video than for images: 3.96× step reduction yields 3.84× speedup (images: 3.9× → 3.6×). This is expected because video sequences at 1,280 tokens create longer KV-caches and larger attention matrices, making each parallel step proportionally more expensive than in the image case.

Comparison with non-autoregressive MAGVIT-v2: MAGVIT-v2 [71] achieves FVD 58 using masked token prediction — substantially better quality than PAR's best FVD of 94.1 (sequential baseline). However, the masked approach is a fundamentally different modeling paradigm. PAR-16× at FVD 103.4 and 3.44s represents a different point in the design space: an autoregressive method that can generate 17-frame clips in under 4 seconds, making interactive video generation plausible for the first time within the autoregressive framework.

Ablation Studies and Robustness Checks

All ablations in Table 4 use the PAR-XL model (775M) with parallel group size n=4 on the ImageNet 256×256 validation set, unless otherwise specified.

Initial sequential token generation (Table 4a): Removing Stage 1 anchors degrades FID from 2.61 to 3.67 (1.06 increase), while saving only 3 steps (144 vs. 147). This is a striking result: adding just 3 sequential steps (2% overhead) provides 1.06 FID improvement. The paper connects this to the qualitative failure mode in Figure 5 (middle row) — without sequential anchors, regions make incompatible structural commitments. The IS also drops substantially (221.36 vs. 259.17), indicating that global coherence affects both realism and diversity. The 3-step cost is negligible because M² is small (4 for PAR-4×, 16 for PAR-16×), making this a "cheap" sequential overhead.

Number of parallel predicted tokens (Table 4b): Increasing parallel group size from n=1 to n=4 to n=16 produces a speed-quality tradeoff, but the relationship is not linear. This ablation uses PAR-XXL (1.4B) to isolate the effect of parallelization degree from model scale:

n=1 (sequential baseline): 576 steps, FID 2.34.
n=4 (PAR-4×): 147 steps (3.9× reduction), FID 2.35 — essentially no quality loss (0.01 FID increase).
n=16 (PAR-16×): 51 steps (11.3× reduction), FID 3.02 — 0.68 FID increase.

The step reduction from n=1 to n=4 is approximately 4× (as expected), while the quality impact is negligible. The step reduction from n=4 to n=16 is an additional 2.88× (147→51), but the quality cost is disproportionately larger (0.01→0.68 FID). This suggests that the "easy" parallelization gains come from grouping tokens across 2×2 regions (where distances are moderate and dependencies are genuinely weak); pushing to 4×4 regions groups tokens that are, on average, closer together (since the image is divided into smaller regions, tokens at corresponding positions are less distant) and have somewhat stronger residual dependencies. The paper's design principle — identify weakly dependent tokens — is supported by this non-linear degradation: when the grouping starts to include tokens with moderate dependencies, quality degrades faster.

Attention pattern within parallel groups (Table 4c): Enabling full (bidirectional) attention within groups reduces FID from 3.64 to 2.61 compared to causal masking, with no change in step count (147 steps in both cases). This 1.03 FID improvement is large — it essentially makes the difference between a severely degraded model (FID 3.64) and a competitive one (FID 2.61). The causal masking variant restricts each token in group j to see only tokens in group j-1 that are at its position or earlier in the sequence, cutting the available context by roughly a factor of M² (4 for PAR-4×). The full attention variant allows each token to access the entire previous group, providing richer conditioning information. This ablation validates that the context impoverishment failure mode (Innovation 2, point 3) is real and that bidirectional within-group attention is the correct remediation.

Token ordering and prediction pattern (Table 4d): The choice of which tokens to parallelize — not just that parallelization is happening — determines quality. This is the critical ablation that validates the paper's central claim:

Raster scan order, single-token prediction: FID 2.62, IS 244.08, 576 steps. This is the standard autoregressive baseline — strong quality, maximum steps.
PAR's cross-region ordering, single-token prediction: FID 2.64, IS 262.72, 576 steps. The reordered sequence, when generated token-by-token, achieves essentially identical quality to raster scan. This confirms that the reordering itself does not harm the model's ability to learn — the 2D RoPE preserves spatial relationships, and the transformer can adapt to the non-standard sequence order.
Raster scan order, multi-token prediction (4 at a time): FID 5.64, IS 265.46, 147 steps. This is a catastrophic failure — FID more than doubles relative to single-token prediction. Generating adjacent tokens in parallel violates the strong local dependencies, producing the distorted patterns shown in Figure 5 (bottom row).
PAR's cross-region ordering, multi-token prediction (4 at a time): FID 2.61, IS 259.17, 147 steps. This matches the single-token quality (FID 2.64 vs. 2.61, essentially equivalent within noise) while achieving the 3.9× step reduction.

The raster-scan multi-token failure (FID 2.62 → 5.64) proves that naive parallelization — the obvious approach of "just predict the next k tokens together" — is deeply flawed for visual data. The PAR ordering's success (FID 2.64 → 2.61, no degradation) proves that the cross-region grouping identified by the pilot study and entropy analysis correctly isolates tokens with weak enough dependencies for safe parallel prediction.

Model scaling analysis (Table 4e): Larger models handle parallel prediction better, with the quality gap narrowing as parameters increase. With n=4 parallel tokens:

343M: FID 3.76
775M: FID 2.61
1.4B: FID 2.35
3.1B: FID 2.29

The improvement from 343M to 775M (FID 3.76 → 2.61) is larger than the improvement from 775M to 3.1B (2.61 → 2.29), suggesting diminishing returns but consistent monotonic improvement. The paper interprets this as evidence that larger models can better model the joint distribution of parallel tokens — i.e., a bigger model compensates for the independent sampling approximation by learning richer conditioning representations. This is a positive result for practical deployment: if you are already planning to train a large model for quality reasons, the parallelization quality penalty shrinks.

Critical Assessment

Claim 1: PAR achieves 3.6× to 9.5× speedup with comparable or minimal quality degradation.

The evidence supports this claim with important nuance about "comparable." The 3.6× speedup claim (PAR-3B-4× at 3.46s vs. LlamaGen-3B at 12.41s) is solid: the 0.11 FID difference (2.29 vs. 2.18) is small in absolute terms, and the precision/recall numbers (Table 2) confirm no meaningful diversity or fidelity loss. The 9.5× speedup claim (PAR-3B-16× at 1.31s) is also well-supported, but the 0.7 FID increase (2.88 vs. 2.18) is more substantial — whether this counts as "minimal" depends on application. For real-time creative tools where users are iterating quickly, 2.88 FID at 1.31s might be preferable to 2.18 FID at 12.41s. For offline batch generation where quality is paramount, the 0.7 FID gap may be unacceptable.

The video results (Table 3) present a similar pattern: PAR-4× at FVD 99.5 (5.4 increase over baseline 94.1) is a genuine tradeoff — the quality degradation exists but is modest relative to the 3.8× speedup. PAR-16× at FVD 103.4 is a larger quality sacrifice.

What the experiments do not test: the paper does not evaluate PAR at intermediate parallelization levels (e.g., n=8 or n=9 using non-square partitions). The jump from n=4 to n=16 is a 4× increase in parallelization with a 29× increase in region count (4→16 regions), while the quality impact changes from ~0 to 0.7 FID. It is likely that n=8 or n=9 (3×3 regions) would provide an attractive intermediate point on the speed-quality curve, but this is not explored.

Additionally, all speedup measurements use batch size 1. PAR's parallel groups process M² tokens per step, making each step more computationally intensive than a single-token step. At larger batch sizes, the relative overhead of wider attention patterns may change — the paper does not explore how the speedup scales with batch size.

Claim 2: The feasibility of parallel generation depends on token dependencies, not architectural constraints.

Strongly supported by Table 4d. The raster-scan multi-token failure (FID 5.64) vs. PAR's cross-region success (FID 2.61) — using identical architecture, training, and number of parallel tokens — is a clean demonstration that token selection governs parallelization feasibility. The conditional entropy analysis (Figures 11, 12) provides a theoretical justification, showing quantitatively that cross-region tokens have higher conditional entropy (weaker dependencies) than adjacent tokens.

A missing piece: the paper does not compare against a random token grouping baseline. Random grouping would test whether the benefit comes from any non-adjacent grouping or specifically from the structured cross-region grouping that PAR uses. Intuition suggests that random grouping would perform between raster (terrible) and PAR (excellent): some random pairs would be weakly dependent (good), some moderately dependent (worse), and the model would struggle with the inconsistent dependency strengths within each group. The absence of this ablation leaves open the question of how precisely dependency strength must be controlled — can any "distant enough on average" grouping work, or does PAR's structured equidistance within groups matter?

Claim 3: Initial sequential tokens are essential for global structural coherence.

Supported by Table 4a (FID 2.61 vs. 3.67) and Figure 5 (middle row). The ablation is clean — 3 additional steps for 1.06 FID improvement — and the qualitative visualization makes the failure mode interpretable (duplicated body parts, misaligned structures).

What is not tested: the paper always generates exactly one anchor per region and always generates them in raster-scan order across regions. Two questions are left open:

Is raster-scan anchor order optimal, or would center-out or learned ordering perform better? The first anchor (top-left region) sets the initial structural commitment; a different ordering might change how conflicts are resolved. The paper provides no evidence that the order matters — or that it doesn't.
Does every region need an anchor? For the easiest images (simple, centered objects), perhaps one or two anchors would suffice, saving a few more sequential steps. The paper does not explore adaptive anchor allocation based on image complexity.

Claim 4: Group-wise bidirectional attention is necessary for quality.

Supported by Table 4c (FID 2.61 vs. 3.64). The 1.03 FID improvement is clear and the mechanism (context impoverishment under causal masking) is well-motivated in Figure 4b.

What is not explored: the paper uses full bidirectional attention within groups. An intermediate design — e.g., allowing each token to attend to a random subset of its group, or to spatially adjacent regions only — might achieve most of the benefit at lower computational cost. The full bidirectional attention scales as O(M⁴) within each group (every token attends to every other token), which could become a bottleneck for very large M (e.g., PAR-64× with 8×8 regions = 64 tokens per group). The paper does not analyze this scaling or propose sparsification strategies.

Genuine weaknesses in the experimental design:

Single tokenizer family for image experiments. All ImageNet experiments use VQGAN with 16,384 codebook size. The paper's claim that PAR works "with various tokenizers" (abstract) is supported by the video experiments using MAGVIT-v2, but for images, only one tokenizer is tested. Tokenizers with different downsampling ratios, vocabulary sizes, or compression qualities might produce different dependency structures — a tokenizer that preserves finer spatial detail might create stronger long-range dependencies that make cross-region parallelization harder.

No comparison against speculative decoding or Jacobi decoding. The paper argues in Section 2 that these methods "increase model complexity" and "may limit the flexibility of autoregressive models," but this is an architectural-philosophy argument, not an experimental one. A direct comparison — implementing speculative decoding for LlamaGen with a smaller draft model — would quantify the speed-quality tradeoff of PAR against the dominant language-domain acceleration method. The absence of this comparison means readers cannot assess whether PAR's simplicity advantage translates to an actual speedup advantage over draft-model-based approaches.

No ablation on the number of transition tokens. The paper uses n-1 learnable transition tokens (3 for PAR-4×, 15 for PAR-16×). Are they all necessary? Would one token suffice? The paper does not investigate. This is a small detail but represents unexamined design complexity.

Limited video evaluation scale. UCF-101 has only ~9,500 training videos — tiny by modern standards. The video model (792M parameters) is smaller than the largest image models (3.1B). Video results are inherently noisier than image results due to the smaller dataset and higher-dimensional output space, yet no error bars or repeated runs are reported for FVD. The claim that PAR "generalizes across domains" would be stronger with a larger-scale video dataset (e.g., Kinetics-400, Something-Something-v2) or with text-to-video generation benchmarks.

No human evaluation. FID and FVD are proxy metrics that correlate imperfectly with human judgment. For the 0.7 FID degradation in PAR-16×, it would be valuable to know whether humans can perceive the quality difference — if the degradation is imperceptible, the speedup is "free" in practical terms. This is especially relevant for the video results where FVD differences of 5-10 points may or may not be visible.

The FLOPs-matched comparison is absent. Unlike the reference example paper (which compares test-time compute against pretraining compute in a FLOPs-matched framework), this paper does not attempt to answer: "at fixed total FLOPs, does PAR's parallel generation produce better images than simply training a bigger sequential model?" A larger sequential model might naturally close the quality gap (as the scaling analysis in Table 4e suggests), and the total training + inference FLOPs tradeoff is relevant for practitioners deciding between scaling model size vs. adopting PAR.

The computational cost of PAR training is not reported. PAR requires reorganizing token sequences during data preprocessing (a one-time cost) and training models with group-wise attention patterns. The paper does not report whether PAR training is slower or more memory-intensive than training the equivalent sequential model. If PAR training is significantly more expensive (e.g., due to the need to process token groups with bidirectional attention), this would affect the total cost of ownership even if inference is faster.

The test set is standard but small for modern standards. ImageNet validation (50,000 images) is adequate for FID estimation, but the paper does not report generation results on other established benchmarks (e.g., MS-COCO for text-to-image, which would test PAR's compatibility with conditional generation beyond class labels). This limits the generality claims.

6. Limitations and Trade-offs

The Difficulty Estimation / Anchor Allocation Problem Is Not Fully Explored

The assumption or constraint. PAR's two-stage generation procedure depends on designating exactly one initial token per spatial region as a sequentially generated anchor (Stage 1) before parallel generation begins (Section 3.2). The number of regions (M²) and the anchor generation order are fixed hyperparameters — 4 for PAR-4×, 16 for PAR-16× — chosen uniformly regardless of image content. The paper provides no mechanism for determining how many anchors are actually needed for a given image, nor in what order they should be generated. The ablation in Table 4a demonstrates that removing anchors entirely degrades FID from 2.61 to 3.67, but does not test whether, for example, 2 anchors (generating only the first two sequentially and then proceeding with parallel generation for the remaining regions' first tokens) might suffice for simple images.

The consequence. The anchor generation stage introduces a fixed sequential overhead — 4 steps for PAR-4×, 16 steps for PAR-16× — that grows quadratically with M. For PAR-16×, this means 16 of the 51 total steps (~31%) are spent on anchors. On easy images where global structure is simple (e.g., centered objects against uniform backgrounds), many of these anchors are likely redundant — the model does not need 16 structural commitments to generate a coherent image of a single bird on a branch. The fixed anchor allocation thus represents a potentially suboptimal use of the sequential budget, especially at aggressive parallelization levels where every sequential step is precious. More fundamentally, the paper's own diagnostic framework (the three design principles in Section 3.1) identifies anchor generation as the mechanism for establishing global structure, but treats the number of anchors as a spatial partitioning decision (M×M regions → M² anchors) rather than a difficulty-adaptive one. This conflates two distinct choices: how finely to partition the image for parallel generation (M) and how many anchors to generate sequentially (potentially fewer than M²).

What evidence exists in the paper. Table 4a shows that anchors matter — removing them costs 1.06 FID with negligible step savings. Figure 5 (middle row) qualitatively demonstrates the failure mode (duplicated structures) when anchors are absent. But neither experiment varies the number of anchors independently of M. For PAR-4×, the paper does not test generating only 2 of the 4 possible anchors. For PAR-16×, it does not test generating 8 or 4 anchors instead of 16. The conditional entropy analysis (Appendix D, Figure 12c) shows low entropy increases for PAR's ordering, but this analysis assumes exactly M² anchors — it does not measure the marginal benefit of each additional anchor. The paper also does not explore whether anchor generation order matters — the regions are processed in fixed raster-scan order, but different orderings (center-out, based on estimated region importance) might distribute structural information more efficiently.

Mitigation status. The paper does not acknowledge this as a limitation or discuss adaptive anchor allocation. The three design principles (Section 3.1) simply state that "initial tokens for each region" should be generated sequentially without questioning whether every region needs one. The connection between anchor count and content-adaptive difficulty estimation — a natural extension given the paper's emphasis on token dependencies — is not drawn. Future work on dynamic anchor allocation (e.g., generating anchors until a confidence threshold is reached, or learning which regions need anchors from data) could reduce the sequential overhead of PAR-16× substantially.

The Speedup Claims Exclude Tokenizer Cost and Assume Full Parallel Utilization

The assumption or constraint. The headline speedup numbers — 3.6× for PAR-4×, 9.5× for PAR-16× (Section 4.2.1, Table 2) — measure only the autoregressive transformer's forward-pass time on a single A100 GPU with batch size 1, starting from the class token and ending with the final generated token. These measurements do not include: (1) the time to run the VQGAN encoder (if encoding a conditioning image) or decoder (to produce the final pixel output from generated tokens), (2) the overhead of token reordering and grouping (reorganizing the token sequence from PAR order back to spatial order before decoding), or (3) any preprocessing or postprocessing steps. The paper acknowledges that the VQGAN decoder is a separate network but does not factor its cost into the speedup metrics. More critically, the parallelization scheme assumes that the M² tokens in each parallel group can be generated with full parallelism — i.e., the GPU has sufficient compute capacity to process M² output heads simultaneously without additional latency.

The consequence. In practice, the end-to-end generation pipeline for an autoregressive visual model includes tokenizer decode time, which is non-trivial. The VQGAN decoder used in the ImageNet experiments [12] is a convolutional network that upsamples the 24×24 token grid back to a 384×384 (or 256×256) image. For a 3.1B-parameter PAR model, this decode time is likely a small fraction of the total (tens of milliseconds vs. seconds), so the speedup numbers are not substantially inflated. However, for the 16× variant generating 51 transformer steps in 1.31s, the fixed decoder overhead becomes proportionally larger — if decoding takes 50ms, that is ~4% overhead, negligible for practical purposes. The more subtle issue is whether the per-step time for PAR's parallel groups scales linearly with M². Each PAR-4× step predicts 4 tokens simultaneously, requiring 4× the output head computation compared to a single-token step. If the GPU is memory-bandwidth-bound rather than compute-bound, this 4× output computation may not translate to 4× wall-clock increase per step — but the paper does not provide a FLOPs or memory-bandwidth analysis. For larger M (e.g., PAR-64× with 64 tokens per step), the per-step cost could grow superlinearly due to attention patterns (group-wise bidirectional attention within 64 tokens is O(64²) per group), potentially eroding the algorithmic step reduction advantage.

What evidence exists in the paper. Table 2 reports both step count and wall-clock time, allowing comparison of algorithmic vs. actual speedup. For PAR-3B-4×: 3.9× step reduction (576→147) yields 3.6× actual speedup (12.41s→3.46s). The efficiency ratio (3.6/3.9 ≈ 0.92) indicates ~8% overhead from wider per-step computation. For PAR-3B-16×: 11.3× step reduction (576→51) yields 9.5× actual speedup (12.41s→1.31s), with efficiency ratio 9.5/11.3 ≈ 0.84 — a 16% overhead. This increasing overhead with larger M confirms that wider parallel steps cost proportionally more, but the paper does not decompose this overhead into attention vs. output head vs. memory-access components. Appendix B shows that engineering optimizations (PyTorch compilation) can further reduce latency (PAR-3B-4×: 3.46s→1.15s; PAR-3B-16×: 1.31s→0.43s), but these measurements still exclude tokenizer time. The paper does not report FLOP counts per forward pass for PAR vs. LlamaGen, making it impossible to assess whether PAR's speedup comes purely from step reduction or also from reduced total FLOPs (since PAR's reordering does not change sequence length).

Mitigation status. The paper does not discuss tokenizer overhead or provide end-to-end pipeline timing. The speedup metrics are transparently reported as transformer-only time, which is reasonable given that the contribution is about autoregressive step reduction, not end-to-end system optimization. Appendix B demonstrates that PAR's algorithmic speedup is compatible with engineering optimizations, partially addressing the concern that implementation details could erase the gains. However, the paper does not provide a breakdown of where PAR's per-step cost comes from (attention, MLP, output projection), making it difficult for practitioners to estimate how the speedup would scale to different hardware or larger M. For batch inference (processing multiple images simultaneously), the relative speedup of PAR vs. sequential generation is entirely unexplored — if the sequential baseline benefits more from batch parallelism (because its per-step computation is narrower), the reported batch-size-1 advantage may not translate to throughput-optimized deployments.

Single Tokenizer and Benchmark for Image Experiments Limit Generality Claims

The assumption or constraint. All ImageNet experiments use a single VQGAN tokenizer [12] with 16,384 codebook size and 16× downsampling ratio, producing 24×24 token grids (576 tokens per image). The paper states in the abstract that the approach "can be seamlessly integrated into standard autoregressive models without modifying the architecture or tokenizer," implying broad tokenizer compatibility. The video experiments (UCF-101) use a reproduced MAGVIT-v2 tokenizer [71] with different characteristics (64K vocabulary, 8× spatial and 4× temporal compression), providing some cross-tokenizer evidence, but the primary quantitative claims about image generation quality and the scaling analysis (Tables 2 and 4) come from a single tokenizer on a single dataset (ImageNet-1K).

The consequence. The paper's central empirical finding — that spatially distant tokens have weak enough dependencies for safe parallel generation — depends on the spatial correlation structure induced by the tokenizer. VQGAN with 16× downsampling produces tokens that each represent a 16×16 pixel patch, with a receptive field that extends beyond the patch boundaries due to the convolutional encoder. A tokenizer with finer downsampling (e.g., 8×, producing 48×48 grids) would create tokens representing smaller spatial regions with potentially stronger correlations across token boundaries (since each token covers less area, adjacent tokens share more texture/edge information proportionally). Conversely, a tokenizer with coarser downsampling (e.g., 32×) would create tokens with larger receptive fields and potentially weaker between-token dependencies (since each token already captures more context internally). The conditional entropy analysis in Appendix D uses VQGAN features specifically; the entropy maps in Figure 11 and the entropy-increase comparison in Figure 12 are tokenizer-specific. A different tokenizer could produce different dependency structures — potentially making cross-region parallelization either safer (weaker dependencies) or riskier (stronger dependencies at the same spatial distance). The paper's claim that PAR generalizes across tokenizers is supported only by the single video tokenizer experiment, which involves a different modality (video vs. image), different spatial resolution (16×16 vs. 24×24 grids), and different vocabulary size (64K vs. 16K), making it difficult to isolate the effect of tokenizer choice from other confounding factors.

What evidence exists in the paper. Table 3 shows PAR achieving speedups on UCF-101 with MAGVIT-v2, indicating that the approach works with at least one additional tokenizer. However, the video baseline (PAR-1×, FVD 94.1) and parallelized variants (PAR-4×, FVD 99.5; PAR-16×, FVD 103.4) are compared against each other, not against external baselines using different tokenizers at matched quality. The video results show a consistent pattern with the image results — modest quality degradation (5-9 FVD) for large speedups — suggesting that the spatial dependency insight transfers. But the video tokenizer compresses temporally as well as spatially, which changes the information content of individual tokens (a spatiotemporal token represents a 3D volume rather than a 2D patch). The paper does not discuss how temporal compression affects spatial dependency structure — it is possible that MAGVIT-v2 tokens, by aggregating temporal information, have weaker spatial dependencies than VQGAN tokens, making cross-region parallelization easier. The paper also does not evaluate on ImageNet with MAGVIT-v2 or any other tokenizer, leaving open the question of whether the quality-speedup tradeoff curve shifts with tokenizer choice. The absence of text-to-image generation experiments (e.g., on MS-COCO or PartiPrompts) further limits generality — class-conditional generation provides strong global conditioning that may reduce the burden on anchor tokens, making PAR's sequential anchor stage less critical than it would be in a more open-ended generation setting.

Mitigation status. The paper is transparent about its experimental setup — it explicitly states the tokenizer used (Section 4.1) and provides evidence from a second tokenizer in the video domain. However, it does not discuss tokenizer compatibility as a limitation or speculate about how dependency structure might vary with tokenizer design. The abstract's claim about compatibility "without modifying the architecture or tokenizer" is technically accurate (PAR does not require tokenizer modifications) but could be misinterpreted as claiming quality guarantees across all tokenizers. A systematic study varying downsampling ratio and vocabulary size on a fixed dataset would clarify whether PAR's design principles are tokenizer-agnostic or specific to the tested configurations.

The assumption or constraint. The paper's stated motivation for PAR's design is to achieve parallel generation "while maintaining the simplicity and flexibility of standard autoregressive models" (Section 1), and it contrasts this against speculative decoding methods that "employ auxiliary draft models" and Jacobi decoding that requires "iterative refinement" (Section 2). This framing implies that PAR achieves comparable or better speed-quality tradeoffs than these alternative acceleration methods while being simpler. However, the paper provides no experimental comparison against any speculative decoding or Jacobi-style decoding baseline applied to visual autoregressive generation. The claim of superior simplicity is an architectural-philosophy argument, not an empirical one — it tells the reader that PAR requires no auxiliary models, but does not quantify what is lost (or gained) in generation quality or speed by avoiding them.

The consequence. A practitioner choosing an acceleration strategy for their autoregressive visual model faces a decision between PAR (no auxiliary models, cross-region token reordering, group-wise attention) and speculative decoding (train a small draft model, use it to propose candidate token groups, verify with the main model). Speculative decoding has a compelling property that PAR lacks: it is lossless with respect to the main model's output distribution. Because the main model verifies each draft token, the generated samples come from exactly the same distribution as sequential generation — any accepted token was verified to have the same probability under the main model. PAR, by contrast, independently samples tokens within each parallel group, producing samples from an approximate distribution that differs from the sequential distribution in ways quantified (indirectly) by the FID degradation. Whether this approximation is "good enough" depends on the FID gap, but for quality-critical applications, a lossless acceleration method with moderate speedup might be preferable to a lossy method with larger speedup. Without a direct comparison, the paper cannot substantiate the claim that PAR's simplicity justifies its approximation error. Furthermore, speculative decoding can be combined with PAR — using a draft model to propose tokens, then verifying with the main model — potentially recovering some of the quality gap at PAR-16× (0.7 FID) while maintaining most of the speedup. The paper's omission of this comparison leaves the Pareto frontier of speed-quality-simplicity unexplored.

What evidence exists in the paper. None. The paper cites speculative decoding [6, 26, 28] and Jacobi decoding [22, 47] as related work (Section 2) and argues that they "increase model complexity" but does not implement or evaluate any of them. There is no ablation where a small draft model (e.g., PAR-L at 343M) generates candidate token groups that are verified by a larger model (PAR-3B), which would directly test whether speculative decoding can recover the quality loss from parallel sampling while retaining speed gains. The paper also does not compare against blockwise parallel decoding [49] or semi-autoregressive methods [63] that predict contiguous token blocks — approaches that are closer to PAR's cross-region grouping but use different mechanisms for ensuring consistency. The entropy analysis in Appendix D provides a theoretical argument for why cross-region parallelization introduces less uncertainty than adjacent-token parallelization, but this is an argument about which grouping PAR should use, not about whether PAR's sampling approximation is preferable to the rejection-sampling approach of speculative decoding.

Mitigation status. The paper acknowledges speculative decoding as related work (Section 2) but does not treat the absence of comparison as a limitation. The framing throughout the paper is that simplicity and compatibility with standard autoregressive frameworks are ends in themselves — "readily applicable to various visual tasks without specialized architectures or additional models" (Section 2). This is a defensible position if the target audience is researchers building unified multimodal autoregressive models who cannot afford the complexity of auxiliary draft models. But for practitioners focused purely on visual generation speed and quality, the lack of baselines means the paper's central selling point (simple, fast, good) is missing a crucial dimension: is it better than the alternative methods for accelerating autoregressive generation that already exist in the language domain and could be adapted to vision?

The Training Cost and Memory Overhead of Group-Wise Attention Are Not Reported

The assumption or constraint. PAR modifies the standard autoregressive training procedure in two ways that may affect computational cost: (1) token sequences are reorganized into the cross-region grouping order, which changes the input distribution but not the sequence length, and (2) the attention mask is modified to allow bidirectional attention within parallel groups while maintaining causal attention between groups (Section 3.3). The paper reports inference-time speedup metrics (steps, wall-clock time) but does not report training-time metrics — total training FLOPs, GPU-hours, peak memory usage, or throughput (images/second) during training — either in absolute terms or relative to the LlamaGen baseline trained on the same data with the same architecture.

The consequence. The bidirectional attention within groups changes the computational pattern during training. In standard causal autoregressive training, the attention mask is triangular — each position attends to itself and all previous positions — enabling efficient implementations using optimized CUDA kernels (e.g., FlashAttention with causal masking). With group-wise bidirectional attention, the mask is block-diagonal at the group level: within each group of size M², tokens attend to all other tokens in that group, plus all tokens in all previous groups. This is not a simple causal mask — it cannot be represented as a single banded triangular matrix. Depending on implementation, this may require materializing the full attention mask (O(sequence_length²) memory) or using custom attention kernels, potentially increasing memory usage and reducing training throughput compared to the standard causal baseline. For PAR-16× (M=4, group size 16), the group-wise attention block is 16×16 — modest relative to the total sequence length of 576. But the mask is irregular (alternating between causal and bidirectional), which may prevent the use of highly optimized FlashAttention implementations that assume a simple causal pattern. If training PAR takes significantly longer per epoch than training the equivalent sequential model, the total cost of ownership (training + inference) is higher even if inference is faster — a consideration that matters for organizations training models from scratch rather than just deploying pretrained checkpoints.

What evidence exists in the paper. The paper reports training hyperparameters (Tables 5 and 6) including batch size, learning rate, and number of epochs, but does not report training throughput (samples/second), total training time, or GPU memory consumption. The model configurations in Table 1 are identical to those used in LlamaGen [51] (same layers, hidden dimensions, heads), so the architectures are matched, but the paper does not state whether PAR training uses the same hardware configuration or achieves the same throughput. Appendix B shows inference-time latency with and without engineering optimizations, but training cost is never addressed. The paper also does not discuss whether the reordered token sequences affect data loading speed or require additional preprocessing (the tokens are pre-computed using the VQGAN encoder and stored on disk, so the reordering is a one-time offline cost, but this is not stated explicitly).

Mitigation status. The paper does not acknowledge this as a limitation or provide training-cost metrics. For researchers reproducing PAR, the absence of training throughput data means they cannot estimate the total resource requirements before committing to experiments. For practitioners, the total cost of deploying PAR (training a custom model vs. using a pretrained checkpoint) is partially opaque. A fair comparison would report training FLOPs per epoch for PAR and an equivalent sequential baseline, or at minimum note whether group-wise attention required custom kernel implementations and what memory overhead they introduced. This limitation is partially mitigated by the fact that PAR's token reorganization and attention mask changes are relatively simple to implement in standard deep learning frameworks (PyTorch's scaled_dot_product_attention supports custom masks), so the overhead is likely modest — but "likely modest" is not quantified, and the paper should acknowledge the gap.

The Inability to Parallelize Temporal Dimension Caps Video Speedup and Suggests a Fundamental Bound

The assumption or constraint. PAR's extension to video generation (Section 3.3, Table 3) applies parallel generation only along the spatial dimensions (H×W of each frame) while keeping the temporal dimension sequential. The paper explicitly acknowledges this: "we also explored parallel generation along the temporal dimension, we found it less effective than spatial parallelization. This is because temporal dependencies exhibit stronger sequential characteristics that are fundamental to video coherence, making them less suitable for parallel prediction compared to spatial relationships." The exploration of temporal parallelization is mentioned but no results are shown — the paper reports only that it was "less effective" without quantifying the degradation. This means that for video, the parallelization speedup is bounded by the spatial dimensions: even with perfect spatial parallelization (reducing spatial steps to minimum), the temporal dimension still requires T sequential steps (5 for UCF-101's 5-frame latent sequence), placing a hard floor on total generation steps of approximately T × (M² + (HW - M²)/M²). For longer videos (higher T), this temporal bottleneck dominates.

The consequence. For video generation at scale — which is arguably where autoregressive efficiency matters most, since video sequences can be orders of magnitude longer than image sequences — PAR's speedup is fundamentally capped by the temporal sequential requirement. If spatial parallelization reduces per-frame steps from 256 (16×16 grid) to 67 (M=2, as in PAR-4× video) or 19 (M=4, hypothetical), the total steps for a T-frame video are approximately 67T or 19T, respectively. As T grows (longer clips, higher frame rates), the total step count scales linearly with T regardless of spatial parallelization. PAR-16× for video achieves 95 steps for a 5-frame clip (19 steps per frame in the spatial dimension), but for a 30-frame clip it would require approximately 570 steps — back to the range of the sequential image baseline. The paper's finding that temporal dependencies are "fundamentally" sequential suggests that this bound is not an implementation limitation but a structural property of video data — adjacent frames are strongly causally dependent (frame t+1 is a direct consequence of frame t's motion and content), making independent parallel sampling across frames as risky as independent sampling of adjacent spatial tokens. If this is correct, PAR cannot achieve order-of-magnitude speedups for long video generation — the speedup factor asymptotically approaches the spatial reduction factor, not the total token reduction factor.

What evidence exists in the paper. The paper reports video results at T=5 (UCF-101's latent frame count) with spatial-only parallelization: PAR-4× at 323 steps (3.8× speedup over sequential 1,280 steps) and PAR-16× at 95 steps (13.5× speedup). No experiments vary T, so the linear scaling of steps with temporal length is not empirically demonstrated but follows directly from the sequential-temporal design. The paper does not report the FVD of temporal parallelization attempts or characterize why temporal parallelization failed — whether it produced flickering, motion artifacts, or complete temporal incoherence. The conditional entropy analysis (Appendix D) is purely spatial, analyzing dependencies within single images; there is no equivalent analysis for temporal token dependencies. The paper also does not explore hybrid approaches — e.g., generating pairs of temporally adjacent frames in parallel with a specialized consistency mechanism, or using optical flow or motion vectors as auxiliary conditioning to reduce temporal dependencies. The discussion is limited to a single sentence noting that temporal parallelization was "less effective" and leaving exploration as future work.

Mitigation status. The paper identifies temporal parallelization as a limitation and future work direction explicitly (Section 3.3 and Section 5), which is appropriate. However, the depth of analysis is shallow — a single unreported experiment with no quantitative results or failure characterization. For a paper whose core insight is about analyzing token dependencies to determine parallelization feasibility, the absence of a parallel dependency analysis for the temporal dimension is a notable gap. A conditional entropy analysis analogous to Figure 12 but for temporal token pairs (same spatial position, adjacent frames vs. distant frames) would clarify whether the temporal sequential constraint is absolute or could be partially relaxed with the right grouping strategy (e.g., generating every other frame in parallel, or generating frames within shot boundaries in parallel). The paper's contribution would be stronger if it applied its own analytical framework to characterize the temporal dependency structure, even if the conclusion remained that temporal parallelization is not currently feasible.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual shift in how the field thinks about accelerating autoregressive visual generation. The shift is not a paradigm change — PAR does not propose a new modeling family like diffusion or masked modeling — but rather a reframing of the acceleration problem from an architectural challenge to a data-structure challenge. Before this work, the dominant question was: "How do we modify the generation mechanism to produce multiple tokens per step?" The answers involved draft models (speculative decoding), iterative refinement (Jacobi decoding), or abandoning autoregression entirely (MaskGIT, VAR). All implicitly accepted that the standard autoregressive transformer, as is, cannot generate multiple tokens per step without quality collapse. PAR's reframing asks a different question: "Which tokens have weak enough mutual dependencies that they can be independently sampled without modifying the generation mechanism?" The answer — spatially distant tokens from different regions — is discovered through analysis of visual token dependency structure rather than through architectural innovation.

This reframing has three concrete consequences for the field:

First, it redirects research attention from architecture design to token organization. The paper demonstrates that an 11.3× step reduction (576 → 51) with only 0.7 FID degradation (Table 2, PAR-XXL-16× vs. LlamaGen-XXL) is achievable purely through input reordering and attention mask modification — no new layers, no auxiliary networks, no training objective changes. This suggests that the frontier of efficient autoregressive generation is not primarily architectural; substantial gains remain available through better understanding of token dependency structure. The conditional entropy analysis framework (Appendix D, Figures 11-12) provides a quantitative lens for exploring this space that was previously absent. Researchers can now ask: given a new tokenizer, a new modality, or a new generation task, what is the dependency structure? Which tokens can be safely parallelized? The entropy estimation methodology (Algorithm 1) makes these questions empirically tractable without training full generative models for every candidate grouping.

Second, it reconciles a tension between quality and compatibility that previously forced an either-or choice. VAR [53] achieves better FID than PAR (1.97 vs. 2.29) with fewer steps (10 vs. 147), but requires a specialized multi-scale tokenizer and longer token sequences, breaking compatibility with the standard autoregressive next-token prediction framework that underlies unified multimodal models [52, 65]. MaskGIT [5] achieves extreme speed (8 steps, 0.13s) but with substantially degraded quality (FID 6.18) and a non-autoregressive paradigm. Before PAR, the field's implicit assumption was that meaningful speedup required either sacrificing quality (MaskGIT) or sacrificing standard-AR compatibility (VAR). PAR demonstrates that this tradeoff is not fundamental — substantial speedup with quality preservation is possible within the standard AR framework, provided tokens are organized by dependency strength rather than spatial proximity. Table 4(d) is the key evidentiary result here: raster-scan parallel prediction degrades FID from 2.62 to 5.64, while cross-region parallel prediction maintains 2.61. The fact that both conditions use identical architecture, training, and number of parallel tokens proves that token selection — not the autoregressive framework — is the binding constraint.

Third, it establishes token dependency analysis as a first-class design tool for generation systems, not just a post-hoc explanation. Prior work sometimes invoked "token dependencies" qualitatively to explain why certain parallelization schemes failed, but did not measure or optimize over dependency structure. PAR's three design principles (Section 3.1 — sequential anchors for global structure, sequential-within-region for local coherence, parallel-cross-region for weak dependencies) are derived from dependency analysis and then validated through ablations that isolate each principle (Table 4a for anchors, Table 4c for attention context, Table 4d for grouping strategy). Each principle corresponds to a distinct failure mode with a distinct visual signature (Figure 5: duplicated structures for missing anchors, distorted local patterns for adjacent-token parallelization). This transforms token dependency from a vague intuition into an operational design constraint that future systems can satisfy, verify, and optimize.

What becomes more attractive as a research direction: tokenizer co-design, where the tokenizer's compression characteristics (downsampling ratio, receptive field size, vocabulary structure) are optimized jointly with the parallel generation grouping to produce maximally "parallelizable" token sequences. The paper shows that VQGAN and MAGVIT-v2 produce dependency structures compatible with PAR's approach, but does not explore whether tokenizers could be explicitly designed to weaken cross-region dependencies further, enabling more aggressive parallelization without quality loss.

What becomes less attractive: purely architectural approaches to visual generation acceleration (adding auxiliary draft models, building specialized non-autoregressive decoders) that treat token dependencies as an implementation detail rather than a central design constraint. PAR demonstrates that understanding the data's structure yields speedups comparable to or exceeding what architectural modifications provide, with dramatically lower complexity. This does not mean architectural approaches are obsolete — VAR's superior FID proves there is still value in specialized designs — but it raises the bar: a new architecture must now justify its additional complexity against the simpler PAR baseline.

Follow-Up Research This Work Enables

Adaptive difficulty-conditioned anchor allocation. PAR currently generates exactly M² anchors (one per spatial region) for every image, regardless of content complexity. The ablation in Table 4a shows anchors matter (FID 2.61 → 3.67 without them), but does not test whether simple images (centered objects, uniform backgrounds) need fewer anchors than complex scenes (cluttered compositions, multiple interacting objects). A strong follow-up would train a lightweight "anchor necessity predictor" that, given the class token or first few generated tokens, estimates how many anchors are needed and where they should be placed. The experiment: on ImageNet, for each image, vary the number of anchors from 1 to M², measure the FID degradation per image as a function of reduced anchor count, and train a predictor to minimize anchor count subject to an FID budget. This would reduce the fixed sequential overhead of PAR-16× (16 anchors out of 51 total steps, 31% of the budget) for images where it is unnecessary, pushing the speed-quality Pareto frontier further. The paper already provides the necessary infrastructure: the two-stage generation procedure cleanly separates anchor generation (which can be made adaptive) from cross-region parallelization (which follows mechanically once anchors are placed).

Combining PAR with speculative decoding to recover lossless parallel generation. PAR sacrifices the exact autoregressive distribution — tokens within each parallel group are sampled independently, producing an approximation whose error manifests as the FID gap (0.11 for PAR-4×, 0.7 for PAR-16×). Speculative decoding [6, 26] is lossless: a draft model proposes candidate tokens, the main model verifies them, and any rejected tokens are resampled from the correct conditional distribution. A natural synthesis would use PAR's cross-region grouping as the proposal mechanism for speculative decoding: the main model (e.g., PAR-3B) serves as both the drafter and verifier, proposing M² tokens in parallel via independent sampling, then verifying each token against the correct conditional distribution (which conditions on all previous tokens, including same-group tokens that were sampled independently). If verification rejects a token, it is resampled autoregressively. This would recover the PAR-4× quality (FID 2.18, matching LlamaGen-3B) while retaining most of the speedup, since most cross-region tokens have weak dependencies and would be accepted. The experiment: implement token-level speculative verification within PAR's parallel groups, measure acceptance rate per group (expected high for distant regions, lower for regions with stronger correlations), and report the speed-quality curve against PAR and LlamaGen baselines. This directly addresses the "simplicity vs. exactness" tradeoff that the paper leaves unresolved.

Conditional entropy-guided tokenizer design for maximal parallelizability. The paper's entropy analysis (Appendix D, Algorithm 1) provides a method for estimating token dependency strength from a dataset of encoded images without training a full generative model. A natural extension is to treat the conditional entropy of cross-region token pairs as a training signal for the tokenizer itself. The experiment: train a VQGAN variant with an auxiliary loss that penalizes low conditional entropy (high dependency) between tokens in the same cross-region position group, encouraging the tokenizer to produce representations where distant tokens are maximally independent. Then train PAR on this tokenizer's outputs and measure whether the FID gap at PAR-16× shrinks (ideally approaching the PAR-4× gap, since the tokenizer has been optimized to weaken the residual dependencies that cause the 0.7 FID degradation). This would validate whether token dependency structure is a property that can be engineered rather than just observed, and could produce tokenizers specifically optimized for efficient autoregressive generation — a new dimension of tokenizer evaluation beyond reconstruction fidelity.

Scaling PAR to text-conditioned generation and open-ended tasks. All experiments in the paper use class-conditional generation (ImageNet classes, UCF-101 action categories). Class conditioning provides strong global guidance that may partially compensate for the structural coherence that parallelization sacrifices — the class label tells the model "this is a goldfish," which constrains the global layout enough that cross-region inconsistencies are less likely. For text-to-image generation (e.g., on MS-COCO or PartiPrompts), the conditioning signal is richer but also more complex, requiring the model to compose multiple objects, attributes, and spatial relationships from a natural language description. A critical stress test: train PAR with a text-conditioned autoregressive model (e.g., LlamaGen's text-to-image variant) and measure whether the FID gap between PAR-4× and sequential generation widens relative to class-conditional generation. Specifically, test on prompts requiring precise spatial relationships ("a red cube to the left of a blue sphere") — do cross-region parallel groups sometimes generate the red cube in the wrong region because the anchor tokens did not fully coordinate the layout? This experiment would test the limits of the "weak dependencies" assumption: when the conditioning signal demands specific spatial configurations, do cross-region tokens become more strongly dependent than in class-conditional generation?

Investigating the temporal dependency structure for video parallelization. The paper reports that temporal parallelization "was less effective" (Section 3.3) but provides no quantitative analysis. A direct follow-up would apply the conditional entropy estimation framework (Algorithm 1) to video token sequences: for a spatiotemporal token grid (T×H×W), measure H(v_t+1,x,y | v_t,x,y) — the conditional entropy of a token given the token at the same spatial position in the previous frame — and compare it to H(v_t,x',y' | v_t,x,y) for spatially distant tokens within the same frame. If temporal conditional entropy is substantially lower than cross-region spatial entropy (indicating stronger dependencies), this would quantitatively validate the paper's claim that temporal dependencies are "fundamentally sequential." But if temporal entropy is comparable to spatial entropy at moderate frame gaps (e.g., every 3rd frame rather than adjacent frames), it would suggest that skip-frame parallelization (generating frames t, t+k, t+2k, ... in parallel, then interpolating) is viable. The experiment: train PAR with parallel generation across temporally distant frames (rather than spatially distant regions), measure FVD on UCF-101, and characterize the failure modes if quality degrades. This would establish whether the temporal sequential constraint is absolute or can be partially relaxed with the right grouping strategy.

PAR as a component in unified multimodal autoregressive models. The paper repeatedly emphasizes PAR's compatibility with standard autoregressive frameworks as a key advantage, but does not demonstrate integration into a multimodal model that handles text and images jointly. A compelling follow-up would embed PAR into a model like Chameleon [52] or Emu3 [65] that generates both text and image tokens autoregressively. The experiment: take a pretrained unified multimodal model, fine-tune it with PAR's token reordering and group-wise attention on the visual token portions (keeping text token generation sequential), and measure whether the text generation quality is affected by the modified visual token representations. The key question is whether PAR's cross-region grouping and 2D RoPE introduce distribution shift that degrades the model's language capabilities when visual and text tokens interact through shared attention layers. If successful, this would demonstrate that efficient visual generation can be retrofitted into existing multimodal models without sacrificing text quality — a high-impact practical result.

Practical Applications and Downstream Use Cases

Real-time interactive image generation for creative tools. The 9.5× speedup of PAR-3B-16× (1.31s per image vs. 12.41s for LlamaGen-3B, Table 2) crosses a critical psychological threshold: sub-2-second generation enables interactive workflows where a user iteratively refines a design, adjusts a prompt, or explores variations without breaking their creative flow. At 1.31s, a user can generate 45 images per minute — fast enough for a real-time preview stream. With engineering optimizations (PyTorch compilation, Appendix B), PAR-3B-16× reaches 0.43s per image, enabling 140 images per minute — essentially real-time for human interaction. This makes autoregressive visual generation viable for applications like concept art exploration (generate dozens of variations of a character design in seconds), real-time style transfer (resample an image in a new style while maintaining content), and live content creation for streaming or social media. The modest quality cost (FID 2.88 vs. 2.18 baseline) is likely acceptable in these settings, where speed and interactivity outweigh pixel-perfect fidelity — especially since users can always run a final high-quality pass (PAR-4× or sequential) once they have identified the desired composition.

Cost-efficient large-scale batch image generation. For organizations generating millions of images (synthetic data for training, asset creation for games or films, data augmentation), the total inference cost scales with (steps × FLOPs_per_step × number_of_images). PAR-4× reduces steps from 576 to 147 (3.9× reduction) with essentially no quality loss (FID 2.29 vs. 2.18 for PAR-3B-4× vs. LlamaGen-3B, a gap of 0.11). For a batch of 1 million images, this means roughly 3.9× lower GPU-hours and 3.9× lower electricity cost, with image quality that is indistinguishable for most downstream applications. If slightly lower quality is acceptable, PAR-16× reduces steps to 51 (11.3× reduction) — generating 1 million images in the GPU time that LlamaGen would spend on ~90,000. The specific economics depend on GPU pricing, but at typical cloud rates ($1-2 per A100-hour), 11.3× cost reduction on million-image batches translates to thousands of dollars saved per job. This application is well-supported by the paper's results: Table 2 provides clear quality-speed tradeoffs at each parallelization level and model scale, enabling practitioners to select the operating point that matches their quality budget.

On-device or edge deployment of video generation for low-latency applications. The video generation results (Table 3) demonstrate that PAR-16× can generate a 17-frame video clip (128×128 resolution) in 3.44 seconds — compared to 43.30 seconds for the sequential baseline. While 3.44 seconds is not yet real-time (30fps would require 0.57 seconds for a 17-frame clip), it is within range for applications like video preview generation in editing software (show a quick preview of an effect or style transfer before committing to full-quality rendering), real-time video stylization for streaming (generate stylized frames with a 3-4 frame latency), or interactive character animation (generate short motion sequences in response to user input). The PAR-4× variant at 11.27 seconds provides higher quality (FVD 99.5 vs. 103.4) for applications where quality matters more than latency. As GPU hardware improves and engineering optimizations accumulate (Appendix B already shows 3× further latency reduction from simple compilation), sub-second video clip generation with autoregressive models becomes plausible, opening use cases in gaming, virtual production, and real-time video communication.

Integration into unified multimodal models for on-device assistants. The paper's emphasis on preserving the standard autoregressive framework without architectural modifications is directly motivated by the trend toward unified multimodal models that handle text, images, and video within a single next-token prediction architecture [52, 65]. For an on-device assistant that needs to generate both text responses and visual content with low latency, PAR provides a way to accelerate the visual generation component without requiring a separate visual generation architecture or draft model — both of which would increase the on-device memory footprint. A 3.1B-parameter PAR model (similar to PAR-3B) could plausibly run on a high-end mobile device or laptop GPU, generating images in 1-3 seconds and short video clips in 5-10 seconds. The key advantage over alternative acceleration methods is that PAR requires no additional models — the same weights serve both text and visual generation, switching from standard causal attention (for text) to group-wise attention (for visual tokens) via mask modification. This unified deployment scenario is speculative (no multimodal results are presented in the paper), but is the natural motivation for PAR's design philosophy of "maintaining the simplicity and flexibility of standard autoregressive models."

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Document Parsing Unveiled: Techniques, Challenges, and Prospects for Structured Information Extraction2024· vision← in-domain predecessor
- Building and better understanding vision-language models: insights and future directions2024· vision← in-domain predecessor
ELT: Elastic Looped Transformers for Visual Generation2026· vision← similar topic
- Scaling Latent Reasoning via Looped Language Models2025· architecture← LLM-judged similar
- WildDet3D: Scaling Promptable 3D Detection in the Wild2026· vision← in-domain predecessor

What it influenced

Later papers in the same chain. Direction reversed.

Locality-aware Parallel Decoding for Efficient Autoregressive Image Generation2025· inference-optimization→ LLM-judged similar
- BitDance: Scaling Autoregressive Generative Models with Binary Tokens2026· uncategorized→ LLM-judged similar
- Less Is More: Training-Free Sparse Attention with Global Locality for Efficient Reasoning2025· inference-optimization→ in-domain predecessor
- Fast Inference from Transformers via Speculative Decoding2022· inference-optimization→ similar topic
- SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference2023· inference-optimization→ similar topic
- Break the Sequential Dependency of LLM Inference Using LOOKAHEAD DECODING2024· inference-optimization→ similar topic
SeedVR: Seeding Infinity in Diffusion Transformer Towards Generic Video Restoration2025· vision→ in-domain predecessor
- TurboDiffusion: Accelerating Video Diffusion Models by 100-200 Times2025· inference-optimization→ LLM-judged similar
- SphereDiff: Tuning-free 360° Static and Dynamic Panorama Generation via Spherical Latent Representation2025· vision→ in-domain predecessor
- Swin Transformer: Hierarchical Vision Transformer Using Shifted Windows2021· vision→ similar topic
- SeedVR2: One-Step Video Restoration via Diffusion Adversarial Post-Training2025· vision→ similar topic
- Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum2025· vision→ similar topic
HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoising2026· vision→ similar topic
- DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning2026· vision→ in-domain predecessor
- DC-VideoGen: Efficient Video Generation with Deep Compression Video Autoencoder2025· vision→ similar topic
- EgoX: Egocentric Video Generation from a Single Exocentric Video2025· vision→ similar topic
- Mode Seeking meets Mean Seeking for Fast Long Video Generation2026· vision→ similar topic
ELT: Elastic Looped Transformers for Visual Generation2026· vision→ similar topic
- Lyra 2.0: Explorable Generative 3D Worlds2026· vision→ in-domain predecessor
- PixelDiT: Pixel Diffusion Transformers for Image Generation2025· vision→ similar topic