ArXiv: 2601.02256

🎯 Pitch

Directly applying GRPO to Visual AutoRegressive (VAR) image models fails catastrophically because fluctuating token-grid sizes across resolution scales create massive gradient imbalances—supervised RL on a partial prefix actually outperforms full-horizon training. The authors resolve this with a principled framework that decomposes the objective into prefix and suffix stages, normalizes gradients by grid size, and isolates credit assignment via mask propagation, boosting Word Accuracy by over 41% and achieving state-of-the-art human-preference alignment.

1. Executive Summary

This paper introduces a reinforcement learning framework for Visual AutoRegressive (VAR) text-to-image models that tackles asynchronous policy conflicts — the instability arising from VAR's heterogeneous token-grid sizes fluctuating dramatically across generation steps, which causes step-wise gradient imbalance and credit-assignment interference when directly applying GRPO. Working with NextFlow (7B parameters) on text rendering (CVTG-2K) and human preference (HPSv3) benchmarks, the authors propose three synergistic mechanisms: Value as Middle Return (VMR) (decomposing the full-horizon KL-regularized RL objective into prefix and suffix stages split at a middle timestep, providing denser feedback to early steps without altering the family-optimal policy), Per-Action Normalization Weighting (PANW) (reweighting per-step losses by kt=1/(htwt)αk_t = 1/(h_t w_t)^\alpha to balance gradients across resolution scales), and Mask Propagation (MP) (back-propagating a spatiotemporal mask through the multi-scale hierarchy to isolate credit assignment to reward-relevant tokens). The method improves Word Accuracy on CVTG-2K from 0.5536 to 0.7841 (+41.6% relative) and achieves state-of-the-art HPSv3 All score of 10.64 over diffusion-centric baselines, establishing that structured credit assignment and scale-aware gradient normalization enable stable RL optimization for VAR models only when the heterogeneous step structure is explicitly accounted for rather than treated with bandit-style GRPO.

2. Context and Motivation

The Core Problem: VAR Models Break Standard RL Recipes

This paper addresses a fundamental disconnect: reinforcement learning for visual generation has been developed primarily for diffusion models and standard autoregressive (AR) models, but Visual AutoRegressive (VAR) models have a structural property that breaks these existing approaches. The problem is not that RL doesn't work for image generation — it does, as demonstrated by GRPO-based methods for diffusion (Flow-GRPO, DanceGRPO) and standard AR (AR-GRPO, SimpleAR). The problem is that VAR models produce actions at dramatically different scales at each generation step, creating what the authors call asynchronous policy conflicts that make directly applying GRPO unstable and suboptimal.

To understand why this matters, consider what makes VAR different from the two dominant paradigms:

In diffusion models, each denoising step operates on a fixed-size latent representation (a tensor of shape C×H×WC \times H \times W). The "action" at each step — what the model predicts — has the same dimensionality at every timestep. This homogeneity means that when GRPO computes advantages and updates the policy, every timestep's contribution has roughly comparable magnitude and the credit assignment problem is relatively well-behaved.

In standard autoregressive models, the model generates one token at a time in raster-scan order. Each action is a single token — a discrete choice from a fixed vocabulary. The dimensionality of the action space is constant across all generation steps (one token), so step-wise gradients have similar scale. The challenge in AR is the length of the sequence (thousands of tokens for a high-resolution image), not the heterogeneity of the action spaces.

VAR models break both these assumptions. As shown in Figure 1, VAR generates images through a coarse-to-fine hierarchy: at early steps, it produces token grids at very low resolutions (e.g., a 1×11 \times 1 grid, then 2×22 \times 2, then 3×33 \times 3), while at later steps, it produces grids at much higher resolutions (e.g., 48×4848 \times 48, then 64×6464 \times 64). The number of query tokens at each step — which determines the scale of the action space — fluctuates by orders of magnitude. At step 1, the model might produce a single token; at step 20, it produces thousands. This is not a minor numerical difference — it fundamentally changes the nature of the policy optimization problem at each timestep.

The authors formalize this in Section 3.1.2 by casting VAR generation as a deterministic MDP. At each timestep tt, the action ata_t is not a single token but an entire grid rt+1∈Rht+1×wt+1r_{t+1} \in \mathbb{R}^{h_{t+1} \times w_{t+1}} of tokens generated in parallel. The state sts_t accumulates all previous grids: st=(r1,r2,…,rt)s_t = (r_1, r_2, \ldots, r_t). The transition is deterministic (given the previous tokens, the model deterministically produces the next grid). The reward R(sT)R(s_T) arrives only at the terminal state — a classic sparse-reward RL problem.

This MDP formulation reveals the core pathology: the policy πθ(at∣st)\pi_\theta(a_t | s_t) at each step has a fundamentally different structure because ata_t varies in dimensionality across tt. At early steps, the policy makes coarse, high-level decisions with few degrees of freedom (a small grid). At later steps, it makes fine-grained, low-level decisions with many degrees of freedom (a large grid). When you apply GRPO — which computes a single advantage estimate per trajectory and updates all steps uniformly — the gradients from high-resolution steps dominate the optimization, drowning out the signal from early, structurally important decisions that set the overall composition.

Why This Problem Matters: Practical and Theoretical Significance

The significance of this problem operates on three levels:

1. VAR models are a rapidly growing paradigm. Since Tian et al. (2024) introduced VAR as an alternative to standard next-token autoregressive generation, the paradigm has been adopted by major systems including Infinity (Han et al., 2025), TokenFlow (Qu et al., 2025, which serves as this paper's base model), Star (Ma et al., 2024), and HART (Tang et al., 2024). VAR models offer compelling advantages: they are substantially faster than raster-scan AR because they generate many tokens in parallel at each step, and they align more naturally with the hierarchical structure of visual perception (coarse layout before fine details). As these models scale and become production systems, the ability to align them with human preferences through RL — the technique that made the difference between base LLMs and useful chat assistants — becomes critical. Without a working RL recipe, VAR models would be stuck at the quality level of their pretraining, unable to benefit from the preference optimization that has become standard practice in both language and visual generation.

2. The failure mode is subtle and would not be obvious without systematic study. The paper's Figure 2 provides compelling evidence: training curves for vanilla GRPO applied to VAR show that applying RL to only a prefix of the generation (e.g., the first 64 or 128 tokens) outperforms applying RL to the full sequence. This is counterintuitive — normally, optimizing more of the model should help more — but it makes sense in light of the policy conflict hypothesis. When RL is applied to the full sequence, the optimization of early, structurally important steps is corrupted by gradient interference from later, high-dimensional steps. By restricting RL to the prefix (fewer, lower-resolution steps), the optimization becomes more coherent and actually achieves higher rewards. The authors' diagnosis of this as asynchronous policy conflict across timesteps provides a principled explanation and motivates their three-component solution.

3. Theoretical significance: KL-regularized RL under product-of-experts policies. The paper makes an underappreciated theoretical contribution by formalizing the VAR policy class as a factorized distribution (Definition 2): at each step, the policy over the token grid factorizes across spatial positions given the global context. This means the policy belongs to a restricted family Mπ\mathcal{M}_\pi that does not contain the globally optimal KL-regularized policy π∗\pi^* from Equation 1. The authors prove in Theorem 1 that the constrained optimum π†\pi^\dagger within the VAR family is obtained by reverse-KL projecting the unconstrained optimal policy π∗\pi^* onto Mπ\mathcal{M}_\pi at each state. This is more than mathematical formalism — it establishes that optimizing VAR with standard RL is fundamentally a projection problem, and that techniques (like VMR) that decompose the problem while preserving this projection structure should work better than ad-hoc approaches. This framework has implications beyond VAR: any model whose policy factorizes across a structured action space (e.g., models that generate multiple tokens in parallel with independence assumptions) could benefit from similar analysis.

Where Existing Approaches Fall Short

The paper identifies specific gaps in prior work that create the opportunity for their contribution:

GRPO assumes bandit-style homogeneity. GRPO (Group Relative Policy Optimization, Shao et al., 2024) was designed for language model fine-tuning, where each action (token) has the same dimensionality (a single discrete choice from the vocabulary). The advantage computation in GRPO compares rewards within a group of completions and uses that relative signal to update the policy, but it treats all tokens in the sequence identically — the per-token probability ratio is computed and scaled uniformly. When the action space changes dramatically across steps (as in VAR), this uniform treatment creates the gradient imbalance the authors document. The paper's contribution is not to replace GRPO but to structure it properly for the VAR setting by adding step-aware weighting (PANW) and temporal decomposition (VMR).

Prior RL for visual generation targets diffusion and AR, not VAR. The authors survey RL applications in visual generation (Section 2.2) and note a clear pattern: flow-based and diffusion models (Flow-GRPO, DanceGRPO) and raster-scan AR models (AR-GRPO, SimpleAR, T2I-R1) have working GRPO recipes, but VAR models have been largely ignored. The only prior work applying GRPO to VAR (Gallici & Borde, 2025) uses a class-conditioned model (the original VAR from Tian et al., 2024) for unconditional or class-conditional generation, not text-to-image. More importantly, that work does not address the structural challenges of multi-scale parallel token generation — it applies GRPO directly without modification, which this paper demonstrates is suboptimal. The authors position themselves as providing the first systematic RL framework for text-to-image VAR with mechanisms specifically designed for its heterogeneous step structure.

KL-regularized RL theory exists but hasn't been applied to VAR's structural constraints. The optimal policy under a KL penalty has been characterized (Equation 1, Ziebart et al., 2008; Zhou et al., 2025), showing that π∗(at∣st)∝πold(at∣st)exp⁡(Q∗(st,at)/η)\pi^*(a_t | s_t) \propto \pi_{\text{old}}(a_t | s_t) \exp(Q^*(s_t, a_t) / \eta). However, this theory assumes the policy can represent any distribution over actions — it doesn't account for the factorization constraint in VAR (Definition 2). The paper bridges this gap by proving that the VAR-constrained optimum is a reverse-KL projection of π∗\pi^* (Theorem 1) and then showing that VMR preserves this structure under a two-stage decomposition (Theorem 2). This is novel theoretical work that justifies the method design.

Sparse rewards in long-horizon generation are a known challenge, but existing solutions don't transfer. In hierarchical RL (Nachum et al., 2018), intermediate rewards or subgoals are used to provide denser feedback. In image generation RL, some works use intermediate discriminators or step-wise critics. However, these approaches typically require learning additional value functions or reward models, which is expensive and introduces its own instability. The paper's VMR is distinguished by not learning any additional model: it estimates the middle-step value directly from on-policy terminal rewards using a risk-sensitive estimator (Equation 10), requiring only K=2K=2 rollouts from the middle state. This makes it lightweight and compatible with the GRPO philosophy of avoiding a separate value network.

Gradient imbalance from heterogeneous action spaces is not addressed by standard normalization. Practitioners might think that gradient clipping or layer normalization in the transformer would handle scale differences across timesteps, but the paper shows this is insufficient. The issue is not about the absolute magnitude of individual token losses — it's about the aggregate contribution of a step. A step that generates a 64×6464 \times 64 grid produces 4,096 tokens, each contributing a policy gradient. Summed together, this step dominates steps that generate 1×11 \times 1 or 2×22 \times 2 grids. PANW's explicit per-step normalization (Equation 9) directly counteracts this by dividing each step's loss by the grid size raised to a power α\alpha, ensuring that the total gradient contribution per step is balanced regardless of resolution.

Credit assignment across spatial positions within a step is unaddressed in prior work. Within a single VAR step, not all tokens contribute equally to the final reward. In text rendering, tokens corresponding to the text region matter much more than background tokens. Standard GRPO treats all tokens in the action grid uniformly, diluting the learning signal. The paper's Mask Propagation (Section 3.5) identifies which tokens causally influence the reward (e.g., those within OCR-detected bounding boxes) and propagates this mask backward through the multi-scale hierarchy, so that early, coarse-scale tokens that eventually influence reward-relevant regions receive focused gradient updates. This is inspired by Reward Feedback Learning (ReFL, Xu et al., 2023) — which uses gradient-based reward attribution — but adapted to the spatiotemporal structure of VAR.

How This Paper Positions Itself

The paper positions itself at the intersection of three threads and makes a distinct contribution relative to each:

Relative to the VAR literature (Tian et al., 2024; Infinity; TokenFlow): This work is not proposing a new VAR architecture or training recipe. It takes an existing state-of-the-art VAR model (NextFlow, 7B parameters) and asks: how do we make RL work on this architecture? The contribution is the RL framework, not the base model. The paper demonstrates that VAR models can benefit from RL alignment just as much as diffusion and AR models, but only if the algorithm is adapted to their multi-scale structure. This fills a gap in the VAR ecosystem: prior to this work, there was no demonstrated path from pretrained VAR model to RL-aligned VAR model.

Relative to the GRPO/RL for generation literature (Flow-GRPO, DanceGRPO, AR-GRPO): These works establish that GRPO can be successfully applied to diffusion and AR models. The paper's position is that VAR presents a fundamentally new challenge within this family due to heterogeneous action spaces, and that addressing this challenge requires structural innovations (VMR, PANW, MP) beyond what was needed for diffusion or AR. The paper does not claim GRPO is broken — it claims that GRPO needs to be adapted to VAR's specific MDP structure. The ablation in Table 4b showing that fine-grained alternation (frequent switches between prefix and suffix optimization) outperforms coarse-grained alternation (bulk prefix then bulk suffix) supports this: it's not just about decomposing the problem, but about how the decomposition is scheduled.

Relative to the theoretical RL literature (KL-regularized MDPs, hierarchical RL): The paper's Theorems 1 and 2 contribute to the theory of constrained policy optimization under KL regularization. The proof that the factorized-policy optimum is a reverse-KL projection (Theorem 1) and that a two-stage decomposition preserves optimality (Theorem 2) are, to our knowledge, novel results that could apply to any generation model with factorized action spaces (e.g., models that generate multiple tokens in parallel at each step, audio models with multi-scale spectrogram prediction). The paper positions these theorems not as abstract contributions but as direct justifications for the method design: VMR is not an arbitrary heuristic — it provably preserves the fixed point of optimization while making the path to that fixed point more stable.

The paper's central thesis, reinforced throughout Sections 1–3, is that the heterogeneity of VAR's generation steps — which is the source of its efficiency advantages — is also the source of its RL challenges, and that addressing these challenges requires explicitly modeling and compensating for this heterogeneity rather than treating it as noise. The three components (VMR, PANW, MP) each address a different aspect of this heterogeneity: VMR handles temporal policy conflicts, PANW handles gradient magnitude imbalance, and MP handles spatial credit assignment. The empirical validation on both text rendering (a task with precise, position-dependent rewards) and human preference scoring (a task with holistic, global rewards) demonstrates that the framework generalizes across reward structures.

3. Technical Approach

3.1 Reader Orientation

This is primarily a systems-and-theory paper that proposes a reinforcement learning framework to fine-tune Visual AutoRegressive (VAR) text-to-image models. The core idea is that VAR models — which generate images by predicting token grids at progressively higher resolutions — suffer from "asynchronous policy conflicts" when trained with standard GRPO because the number of tokens produced per step varies by orders of magnitude, creating gradient imbalance and credit-assignment interference across timesteps. The solution is a three-component architecture (VMR, PANW, MP) that decomposes the RL problem into prefix and suffix stages, normalizes per-step gradient contributions by grid size, and propagates reward-relevant spatial masks backward through the generation hierarchy, thereby making RL optimization stable and effective for VAR without altering the optimal policy that the system converges to.

3.2 Big-Picture Architecture (Diagram in Words)

The system takes a pretrained VAR text-to-image model (NextFlow, 7B parameters) and fine-tunes it using an enhanced version of Group Relative Policy Optimization (GRPO). The architecture has four major components operating in a loop:

  1. VAR Policy Network ($\pi_\theta$) — The pretrained NextFlow model that generates images token-grid-by-token-grid from coarse to fine resolution. It serves as the policy being optimized, producing actions $a_t = r_{t+1}$ (a grid of tokens at the next resolution) conditioned on the current state $s_t$ (all previously generated token grids). The policy factorizes across spatial positions within each grid (Definition 2).

  2. Two-Stage Temporal Decomposition (VMR) — This component splits the generation horizon at a middle timestep $m$ (corresponding to the $256 \times 256$ resolution scale). It estimates a middle-step soft value $\hat{V}_m(s_m)$ from $K=2$ on-policy rollouts using a risk-sensitive estimator (Equation 10) and uses this value as the terminal reward for the prefix policy $\pi_{1:m-1}$. The suffix policy $\pi_{m:T-1}$ receives the true terminal reward $R(s_T)$. Training alternates between these two subproblems: 3 prefix updates for every 1 suffix update.

  3. Per-Action Normalization Weighting (PANW) — Before computing the policy gradient, each timestep $t$ receives a weight $k_t = 1 / (h_t \times w_t)^\alpha$ where $h_t \times w_t$ is the token grid size at that step and $\alpha \in [0.6, 0.8]$ is a decay exponent. This weight scales the loss contribution of each action, ensuring that high-resolution steps (which generate thousands of tokens) do not dominate the gradient relative to low-resolution steps (which generate only a few tokens).

  4. Mask Propagation (MP) — An initial spatial mask is constructed from the output components that directly determine the terminal reward (e.g., OCR-detected bounding boxes for text rendering). This mask is propagated backward through the model's multi-scale hierarchy from finer to coarser feature scales. The mask gates intermediate rewards and gradients, focusing credit assignment on causally relevant spatial regions and reducing variance from irrelevant background tokens.

Information flows as follows: a text prompt enters the system → the VAR policy generates $G=16$ complete images (a group) using the standard VAR sampling procedure (CFG=5, top-k=2, top-p=0.9) → a reward model (OCR-based for text rendering, HPSv3 for human preference) scores each image → VMR estimates the middle-step value for each trajectory using 2 additional rollouts from $s_m$ → GRPO computes per-token probability ratios and advantages within the group → PANW weights each step's contribution by the inverse grid size → MP spatially masks gradients to focus on reward-relevant regions → the policy parameters are updated with AdamW (lr=1e-5, $\beta_1=0.9$, $\beta_2=0.95$, weight decay=0.05). This loop runs for up to 1,200 updates (19,200 unique prompts).

3.3 Roadmap for the Deep Dive

  • First, the MDP formulation for VAR generation (Section 3.1.2), which formally defines the state space, action space, and transition dynamics. This is essential because it crystallizes why VAR is different from AR and diffusion — the action dimensionality changes at every step — and sets up the theoretical analysis that follows.

  • Second, the KL-regularized RL background (Section 3.1.3), which introduces the globally optimal soft policy $\pi^*$ (Equation 1) and its Q-function (Equation 3). Understanding this is prerequisite for VMR, since VMR's correctness hinges on the property that inserting an intermediate value does not alter the optimal policy.

  • Third, the definition of the VAR policy family $\mathcal{M}_\pi$ (Definition 2) and Theorem 1, which characterizes the constrained optimum $\pi^\dagger$ as a reverse-KL projection of $\pi^*$. This formalizes the structural mismatch that makes vanilla GRPO fail: the global optimum lies outside the representable policy class.

  • Fourth, the Value as Middle Return (VMR) mechanism (Section 3.3), which decomposes the full-horizon problem into prefix and suffix stages. This includes the definition of the middle-step soft value (Definition 1), the two-stage optimization (Equation 5), the risk-sensitive estimator (Equation 10), and Theorem 2 (which proves VMR preserves optimality within the VAR family). The practical implementation details — choice of $m$, number of rollouts $K$, alternating schedule — are covered here because they follow directly from the theoretical construction.

  • Fifth, the Per-Action Normalization Weighting (PANW) (Section 3.4), which addresses the gradient imbalance created by fluctuating grid sizes. This is the simplest component mechanistically but addresses the most direct symptom of the problem (Figure 1). The selection of $\alpha$ and its interaction with other components are discussed.

  • Sixth, the Mask Propagation (MP) mechanism (Section 3.5), which provides spatial credit assignment within each step. This is explained last because it operates on top of both VMR (it can mask which tokens in the prefix vs. suffix matter) and PANW (the masking interacts with the per-action weights).

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems-and-theory paper that develops a structured RL framework for VAR by formalizing the generation process as a deterministic MDP, characterizing the constrained optimal policy under the VAR factorization, and then designing three synergistic components that collectively stabilize training while preserving optimality. The core insight is that VAR's multi-scale structure creates a representational mismatch between the globally optimal KL-regularized policy (which can be any distribution over action grids) and the factorized policy class that VAR models actually implement (where tokens within a grid are generated independently given the global context). The three components — VMR, PANW, MP — are not heuristics but principled interventions derived from this mismatch.

MDP Formulation of VAR Generation

The paper formalizes VAR image generation as a finite-horizon, deterministic Markov Decision Process with sparse terminal reward. This is not standard in visual generation RL (which typically uses bandit formulations where the entire image is treated as a single action), and the formalism is necessary to reason about per-step policy optimization.

**State Space $\mathcal{S}:∗∗Astate‘:** A state `s_t‘isthepartialVARsequence—theconcatenationofalltokengridsgeneratedsofar:‘` is the partial VAR sequence — the concatenation of all token grids generated so far: `s_t = (r_1, r_2, \ldots, r_t)‘.Each‘`. Each `r_k‘isagridofdiscretetokensatresolutionlevel‘` is a grid of discrete tokens at resolution level `k‘,shaped‘`, shaped `h_k \times w_k‘.Thestategrowsadditively:‘`. The state grows additively: `s_{t+1} = (s_t, r_{t+1})$. The initial state s0s_0(ors1s_1`, depending on indexing convention) contains only the text prompt encoding and the begin-of-image token, though the paper simplifies notation by omitting external conditioning.

What makes this state space unusual is that the dimensionality of the state grows substantially at each step. At step 1, the state might contain a $1 \times 1$ grid (1 token). At step $T$, the state contains grids at all resolutions from $1 \times 1$ up to $64 \times 64$ (thousands of tokens). This contrasts with standard AR MDPs where the state grows by exactly one token per step, and with diffusion MDPs where the state (the noisy latent) has fixed dimensionality throughout.

**Action Space $\mathcal{A}:∗∗Anaction‘:** An action `a_t‘isthegenerationofthenextresolutiongrid:‘` is the generation of the next resolution grid: `a_t = r_{t+1} \in \mathbb{R}^{h_{t+1} \times w_{t+1}}$. Critically, this action is **not a single token** but a structured output — an entire 2D grid of tokens generated in parallel. The number of tokens in the action (ht+1×wt+1h_{t+1} \times w_{t+1}) varies across timesteps. The paper's Table 6 shows the actual schedule for a 1024×10241024 \times 1024image: step 1 produces(1,1)(1,1)(1 token), step 2 produces(2,2)(2,2)(4 tokens), step 8 produces(8,8)(8,8)(64 tokens), and step 20 produces(64,64)(64,64)` (4,096 tokens).

An action $a_t$ is sampled from the policy $\pi_\theta(\cdot | s_t)$, which — as the paper defines in Definition 2 — factorizes across spatial positions within the grid:

πθ(at∣st)=∏(i,j)πt,(i,j)θ(at,(i,j)∣st).\pi_\theta(a_t | s_t) = \prod_{(i,j)} \pi^\theta_{t,(i,j)}(a_{t,(i,j)} | s_t).

where $(i,j)$ indexes spatial positions in the $h_{t+1} \times w_{t+1}$ grid and $a_{t,(i,j)}$ is the discrete token at position $(i,j)$.

What this factorization means operationally: At each generation step, the model produces the entire token grid by making independent per-position predictions conditioned on the same global context $s_t$. The model does not autoregressively generate tokens within the grid (i.e., it does not predict token $(i,j)$ conditioned on token $(i,j-1)$). This is what gives VAR its speed advantage: parallelism within each scale. However, it also means that the joint distribution over the grid is a product of independent categoricals — a restricted policy class that may not contain the globally optimal KL-regularized policy.

This does not mean spatial positions are truly independent in the generated image, because $s_t$ contains all previously generated tokens at all resolutions, which provide global context. The factorization is a conditional independence given the full history, not a marginal independence. A token at position $(0,0)$ in the grid and a token at position $(63,63)$ can be highly correlated through their shared dependence on the coarse-scale structure in $s_t$.

**State Transition Probability $\mathcal{P}:∗∗Thetransitionisdeterministic.Giventhecurrentstate‘:** The transition is deterministic. Given the current state `s_t‘andtheaction‘` and the action `a_t = r_{t+1}‘,thenextstateisuniquelydeterminedas‘`, the next state is uniquely determined as `s_{t+1} = (s_t, r_{t+1})`$. Formally:

P(st+1∣st,at)=δ(r1,r2,…,rt+1)(st+1)\mathcal{P}(s_{t+1} | s_t, a_t) = \delta_{(r_1, r_2, \ldots, r_{t+1})}(s_{t+1})

where $\delta$ is the Dirac delta function indicating that $s_{t+1}$ is exactly the concatenation of $s_t$ and $a_t$.

Why the deterministic transition matters: In a deterministic MDP, the future trajectory given a state and action is fully determined by the policy — there is no environmental stochasticity. This means the action-value function $Q^*(s_t, a_t)$ can be expressed purely in terms of the terminal reward distribution under the reference policy $\pi_{\text{old}}$ (Equation 3), without needing to marginalize over environmental transitions. This simplification is crucial for VMR's risk-sensitive estimator (Equation 10), which estimates the middle-step value directly from terminal rewards without learning a value function.

**Reward Function $r:∗∗Theenvironmentprovidesasingleterminalreward‘:** The environment provides a single terminal reward `R(s_T)$` at the final state (the complete image). There are no intermediate rewards. This is the standard sparse-reward setting in image generation RL, where the reward comes from an external scorer (OCR accuracy for text rendering, HPSv3 model for human preference) evaluated on the final image. The paper's reward design for text rendering is described separately in Section 4.2.1; for the RL framework, the reward is treated as a black-box scalar.

What this MDP formulation reveals: The key pathology is now visible in the formal structure. At timestep $t$, the policy must output a distribution over $h_{t+1} \times w_{t+1}$ tokens. The log-probability $\log \pi_\theta(a_t | s_t)$ is a sum over $h_{t+1} \times w_{t+1}$ independent per-token log-probabilities. When computing the policy gradient, the gradient contribution from step $t$ scales with the number of tokens in the action grid. This means:

  • Step 1 ($1 \times 1$, 1 token) contributes a gradient of magnitude proportional to 1.
  • Step 20 ($64 \times 64$, 4,096 tokens) contributes a gradient of magnitude proportional to 4,096.

Without correction, the high-resolution steps dominate the optimization by a factor of thousands to one. The policy at early steps — which determine the overall image layout and structure — receives almost no learning signal relative to the policy at later steps. This is the asynchronous policy conflict: the optimization process is pulled in different directions at different scales, with the fine-scale updates drowning out the coarse-scale updates.

KL-Regularized RL Background (Prerequisite for VMR)

The paper grounds its approach in the theory of KL-regularized reinforcement learning, where the objective is to maximize expected terminal reward while penalizing deviation from a reference (pretrained) policy. This section is not novel — it follows Ziebart et al. (2008) and Zhou et al. (2025) — but it is necessary to establish the optimal policy structure that VMR preserves.

The full-horizon objective for a policy $\pi$ with reference policy $\pi_{\text{old}}$ is:

J(π)=Eτ∼π[R(sT)]−η⋅KL(π(τ)∥πold(τ))J(\pi) = \mathbb{E}_{\tau \sim \pi} \left[ R(s_T) \right] - \eta \cdot \text{KL}\left( \pi(\tau) \| \pi_{\text{old}}(\tau) \right)

where $\tau = (s_1, a_1, \ldots, s_T)$ is a complete trajectory, $\mathbb{E}_{\tau \sim \pi}$ takes the expectation over trajectories sampled from $\pi$ under the deterministic environment dynamics, $R(s_T)$ is the terminal reward, $\eta > 0$ is a temperature parameter controlling the trade-off between reward maximization and staying close to the reference policy, and $\text{KL}(\pi(\tau) \| \pi_{\text{old}}(\tau))$ is the Kullback-Leibler divergence between the trajectory distributions induced by $\pi$ and $\pi_{\text{old}}$.

What it computes: The balance between two competing desires — achieving high reward (first term) versus not straying too far from the pretrained model's behavior (second term). When $\eta$ is large, the KL penalty dominates and the optimal policy stays close to $\pi_{\text{old}}$. When $\eta$ is small, the reward dominates and the policy can change more aggressively. In GRPO, this KL penalty is implicitly enforced through the group-based relative advantage computation rather than an explicit KL term in the loss, but the theoretical optimum remains the same.

Why this form: KL-regularized RL is the standard formulation for fine-tuning pretrained generative models because it prevents catastrophic forgetting and mode collapse. Without the KL penalty, the policy could over-optimize the reward signal — producing images that score highly on the reward model but have degenerated visual quality or lost diversity. The KL penalty anchors the policy near $\pi_{\text{old}}$, which is a well-behaved distribution over images.

The globally optimal policy $\pi^*$ that maximizes $J(\pi)$ has a known closed form (Ziebart et al., 2008):

π∗(at∣st)∝πold(at∣st)exp⁡(1ηQ∗(st,at))\pi^*(a_t | s_t) \propto \pi_{\text{old}}(a_t | s_t) \exp\left( \frac{1}{\eta} Q^*(s_t, a_t) \right)

where $\pi^*(a_t | s_t)$ is the optimal conditional action distribution at state $s_t$, $\pi_{\text{old}}(a_t | s_t)$ is the reference policy's action distribution at the same state, and $Q^*(s_t, a_t)$ is the optimal soft action-value function defined below.

What it computes: The optimal policy is the reference policy reweighted by an exponential function of the expected future return. Actions that lead to higher expected returns get higher probability mass. The temperature $\eta$ controls how aggressively the policy deviates from $\pi_{\text{old}}$: as $\eta \to \infty$, $\pi^* \to \pi_{\text{old}}$; as $\eta \to 0$, $\pi^*$ becomes deterministic, selecting the action with maximum $Q^*$.

Why this form: This is the solution to the maximum-entropy RL problem (Ziebart et al., 2008). It emerges from taking the functional derivative of $J(\pi)$ with respect to $\pi$ and setting it to zero. The exponential reweighting is the natural consequence of the KL divergence penalty: the KL divergence $\text{KL}(\pi \| \pi_{\text{old}})$ is minimized when $\pi = \pi_{\text{old}}$, so the optimal policy distorts $\pi_{\text{old}}$ only to the extent that doing so increases expected reward, with the distortion controlled by $\eta$.

When the environment transition is deterministic (as in VAR), the optimal Q-function simplifies dramatically:

Q∗(st,at)=ηln⁡Eπold[exp⁡(1ηR(sT))  |  st,at]Q^*(s_t, a_t) = \eta \ln \mathbb{E}_{\pi_{\text{old}}} \left[ \exp\left( \frac{1}{\eta} R(s_T) \right) \;\middle|\; s_t, a_t \right]

where $\mathbb{E}_{\pi_{\text{old}}}[\cdot | s_t, a_t]$ takes the expectation over future trajectories from step $t+1$ to $T$ under the reference policy $\pi_{\text{old}}$, conditioned on having taken action $a_t$ at state $s_t$.

What it computes: For a given state-action pair $(s_t, a_t)$, the optimal Q-value is a risk-sensitive summary of the possible terminal rewards achievable by following $\pi_{\text{old}}$ from that point onward. The exponential inside the expectation emphasizes high-reward outcomes (large $R(s_T)$ get exponentiated to even larger values) and the log pulls the result back to the original scale. This is the entropic risk measure from mathematical finance: for $\eta > 0$, it is risk-seeking (it over-weights good outcomes); for $\eta < 0$, it would be risk-averse.

Why this form under deterministic dynamics: When the transition is deterministic, the only randomness in the future trajectory comes from the policy's action sampling. The future return given $(s_t, a_t)$ is the terminal reward that results from following $\pi_{\text{old}}$ from $s_{t+1}$ onward. The expectation $\mathbb{E}_{\pi_{\text{old}}}[\exp(R/\eta) | s_t, a_t]$ integrates over this action-sampling randomness. This is a computationally useful simplification because it means $Q^*$ can be estimated by Monte Carlo rollouts from $\pi_{\text{old}}$ without needing to learn a value function — precisely the approach VMR takes.

Critical theoretical fact (used by VMR): The globally optimal policy $\pi^*$ is the unique maximizer of $J(\pi)$ over all possible policies — including policies that do not factorize spatially. This is the crux of the problem for VAR: $\pi^*$ may assign correlated probabilities to tokens within the same grid (e.g., a token at $(0,0)$ and a token at $(0,1)$ might be jointly more or less likely than their marginal product), while the VAR policy class $\mathcal{M}_\pi$ forces independence. The constrained optimum $\pi^\dagger$ is not $\pi^*$ — it is the closest policy to $\pi^*$ within the factorized family, as Theorem 1 establishes.

The VAR Policy Family and the Constrained Optimum (Theorem 1)

The paper formally defines the class of policies that VAR models can represent, and then characterizes what the best possible policy within this class looks like.

**Definition 2 (VAR Family $\mathcal{M}_\pi):∗∗Apolicy‘):** A policy `\pi_\theta‘belongstotheVARfamily‘` belongs to the VAR family `\mathcal{M}\pi‘if,ateachtimestep‘` if, at each timestep `t‘,theconditionaldistributionovertheactiongrid‘`, the conditional distribution over the action grid `a_t = r{t+1}‘factorizesacrossspatialpositionsgiventhestate‘` factorizes across spatial positions given the state `s_t$`:

πθ(at∣st)=∏(i,j)πt,(i,j)θ(at,(i,j)∣st)\pi_\theta(a_t | s_t) = \prod_{(i,j)} \pi^\theta_{t,(i,j)}(a_{t,(i,j)} | s_t)

where $\pi^\theta_{t,(i,j)}(\cdot | s_t)$ is the per-position categorical distribution over the codebook vocabulary at spatial index $(i,j)$ in the $h_{t+1} \times w_{t+1}$ grid.

What this constraint means: The policy cannot represent arbitrary joint distributions over the token grid. It can only represent distributions where each position's token is conditionally independent of every other position's token given the global state $s_t$. This is the product-of-experts structure implemented by the transformer: the output logits at position $(i,j)$ depend on $s_t$ (through self-attention over all previous tokens) but not on the simultaneously generated tokens at other positions in the same grid.

Why this constraint exists: It is an architectural necessity. VAR generates all tokens in a grid in parallel for efficiency. If tokens within the grid had to be generated autoregressively (token $(i,j)$ conditioned on token $(i,j-1)$), VAR would lose its speed advantage over standard AR. The factorization is the price paid for parallelism.

Definition 3 (Constrained Optimum $\pi^\dagger$): The optimal policy within the VAR family is the one that maximizes the KL-regularized objective subject to the factorization constraint:

π†=arg⁡max⁡π∈MπJ(π),J(π)=Eπ[R(sT)]−η⋅KL(π∥πold)\pi^\dagger = \arg\max_{\pi \in \mathcal{M}_\pi} J(\pi), \quad J(\pi) = \mathbb{E}_{\pi}[R(s_T)] - \eta \cdot \text{KL}(\pi \| \pi_{\text{old}})

where $J(\pi)$ is the same full-horizon objective as before, but the maximization is over the restricted set $\mathcal{M}_\pi$.

**Theorem 1 (Reverse-KL Characterization of $\pi^\dagger):∗∗Ateachstate‘):** At each state `s_t$`, the constrained optimum satisfies:

π†(⋅∣st)=arg⁡min⁡π∈Mπ(st)KL(π(⋅∣st)  ∥  π∗(⋅∣st))\pi^\dagger(\cdot | s_t) = \arg\min_{\pi \in \mathcal{M}_\pi(s_t)} \text{KL}\left( \pi(\cdot | s_t) \;\|\; \pi^*(\cdot | s_t) \right)

where $\pi^*$ is the globally optimal (unconstrained) policy from Equation 1, and $\mathcal{M}_\pi(s_t)$ is the set of factorized distributions over the action grid at state $s_t$.

What it computes: At each state, the constrained optimum $\pi^\dagger$ is the distribution in the factorized family that minimizes the KL divergence to the global optimum $\pi^*$. This is a reverse-KL projection (also called an I-projection): we hold $\pi^*$ fixed as the second argument and search over factorized $\pi$ as the first argument.

Why this form: The proof (Appendix E.1) uses a variational identity that decomposes $J(\pi)$ as:

J(π)=ηlog⁡Z−η⋅KL(pπ(τ)∥p∗(τ))J(\pi) = \eta \log Z - \eta \cdot \text{KL}(p_\pi(\tau) \| p^*(\tau))

where $Z = \mathbb{E}_{\pi_{\text{old}}}[\exp(R(s_T)/\eta)]$ is a constant independent of $\pi$, and $p_\pi(\tau)$ and $p^*(\tau)$ are the trajectory distributions under $\pi$ and $\pi^*$ respectively. Maximizing $J(\pi)$ is equivalent to minimizing $\text{KL}(p_\pi(\tau) \| p^*(\tau))$. Since both trajectory distributions share the same deterministic dynamics and differ only in the action distributions, the trajectory KL decomposes into a sum of per-state expected KLs: $\sum_t \mathbb{E}_{s_t \sim \pi}[\text{KL}(\pi(\cdot|s_t) \| \pi^*(\cdot|s_t))]$. Because the constraint set $\mathcal{M}_\pi$ factorizes across states (the factorization is per-state), the minimization decouples: we can independently choose $\pi(\cdot|s_t)$ at each state to minimize the per-state KL. The result is Equation 8.

Why this theorem matters operationally: It tells us that optimizing a VAR model with KL-regularized RL is equivalent to projecting the unconstrained optimum $\pi^*$ onto the factorized family at each step. Any algorithm that successfully maximizes $J(\pi)$ over $\mathcal{M}_\pi$ must be doing this projection implicitly. The theorem also reveals a potential failure mode: if the reverse-KL projection is ill-behaved — for example, if $\pi^*$ assigns high probability to token-grid configurations that cannot be well-approximated by a product of independent categoricals — then $\pi^\dagger$ may be far from $\pi^*$, and the achievable reward under the VAR constraint is fundamentally limited. The three components of the paper's method (VMR, PANW, MP) can be understood as making this projection more stable and more accurate during training, not changing its fixed point.

Value as Middle Return (VMR): Theory, Implementation, and Practical Considerations

VMR is the paper's central mechanism for addressing temporal policy conflicts. Rather than optimizing the full-horizon objective as a monolith, VMR splits the generation into a prefix (steps $1$ to $m-1$) and a suffix (steps $m$ to $T-1$), trains each segment with its own GRPO objective, and alternates between them.

The core idea is motivated by the observation in Figure 2: applying GRPO to only the first 64 or 128 tokens (the prefix) produces better rewards than applying it to the full sequence. This suggests that the suffix optimization is interfering with the prefix optimization. VMR formalizes this by giving the prefix its own reward signal — the middle-step soft value — that summarizes all future consequences of the prefix decisions, enabling the prefix to be optimized without interference from the suffix's gradient dynamics.

Definition 1 (Middle-Step Soft Value): For a chosen middle timestep $m \in \{1, \ldots, T\}$, the middle-step soft value at state $s_m$ is:

Vm∗(sm)=ηlog⁡Eπold[exp⁡(1ηR(sT))  |  sm]V^*_m(s_m) = \eta \log \mathbb{E}_{\pi_{\text{old}}} \left[ \exp\left( \frac{1}{\eta} R(s_T) \right) \;\middle|\; s_m \right]

where $V^*_m(s_m)$ is the optimal value of being at state $s_m$ (having generated the first $m$ token grids) and then following the reference policy $\pi_{\text{old}}$ for the remaining steps $m+1$ to $T$, summarized through the same risk-sensitive expectation as in Equation 3.

What it computes: For a partially generated image at the middle resolution (the state $s_m$), the soft value $V^*_m(s_m)$ is a scalar that captures how "promising" this partial image is — how high a terminal reward we can expect if we complete the image by following the base model's default behavior from this point. A state with a good coarse layout and plausible structure will have a high $V^*_m$; a state with garbled text or misaligned composition will have a low $V^*_m$.

Why this form: This is structurally identical to the Q-function in Equation 3 but marginalized over the action $a_m$ — it is the value of being at $s_m$ before choosing the next action, whereas $Q^*$ is the value of taking a specific action at $s_m$. The risk-sensitive form is inherited from the KL-regularized framework: it uses the same exponential reweighting so that $V^*_m$ is consistent with the full-horizon objective. If we were to replace $V^*_m$ with a simple expected reward $\mathbb{E}[R | s_m]$, the prefix optimization would not converge to the same policy as the full-horizon optimization — the prefix would be risk-neutral while the full objective is risk-seeking.

The Two-Stage Decomposition (Equation 5): VMR partitions the full-horizon objective into two separate optimization problems:

Suffix:max⁡πm:T−1E[R(sT)∣sm]−η⋅KL(πm:T−1  ∥  πold,m:T−1)\text{Suffix:} \quad \max_{\pi_{m:T-1}} \mathbb{E}\left[ R(s_T) \mid s_m \right] - \eta \cdot \text{KL}\left( \pi_{m:T-1} \;\|\; \pi_{\text{old}, m:T-1} \right)

Prefix:max⁡π1:m−1E[Vm∗(sm)]−η⋅KL(π1:m−1  ∥  πold,1:m−1)\text{Prefix:} \quad \max_{\pi_{1:m-1}} \mathbb{E}\left[ V^*_m(s_m) \right] - \eta \cdot \text{KL}\left( \pi_{1:m-1} \;\|\; \pi_{\text{old}, 1:m-1} \right)

where $\pi_{m:T-1}$ denotes the policy for steps $m$ through $T-1$ (the suffix), $\pi_{1:m-1}$ denotes the policy for steps $1$ through $m-1$ (the prefix), and $\pi_{\text{old}, \cdot}$ denotes the corresponding segments of the reference policy.

What each subproblem computes:

  • The suffix problem is a standard KL-regularized RL problem on a shorter horizon (from $m$ to $T$). The suffix policy receives the true terminal reward $R(s_T)$ and is penalized for deviating from the reference suffix policy. This is optimizing the fine-grained, high-resolution generation steps.
  • The prefix problem is also a standard KL-regularized RL problem, but its "terminal reward" is $V^*_m(s_m)$ — the soft value at the boundary state — not the true $R(s_T)$. The prefix policy is penalized for deviating from the reference prefix policy. This is optimizing the coarse, structural generation steps.

Why this decomposition works: The prefix is optimized using a reward that already accounts for the suffix's optimization. If the suffix policy improves (learns to produce better fine details), $V^*_m(s_m)$ changes accordingly, and the prefix adapts. The decomposition is not two independent problems — they are coupled through $V^*_m(s_m)$, and the alternating training schedule (3 prefix updates per 1 suffix update) maintains this coupling.

Theorem 2 (Two-Stage Invariance): Within the VAR family $\mathcal{M}_\pi$, solving the suffix problem yields the constrained-optimal suffix policy $\pi^\dagger_{m:T-1}$; using $V^*_m$ as the sole reward for the prefix problem yields the constrained-optimal prefix policy $\pi^\dagger_{1:m-1}$. The concatenation $\pi^\dagger = \pi^\dagger_{1:m-1} \oplus \pi^\dagger_{m:T-1}$ uniquely maximizes the full-horizon objective $J(\pi)$.

What this theorem guarantees: VMR does not change the optimal solution — the policy that jointly maximizes $J(\pi)$ over $\mathcal{M}_\pi$ is exactly the concatenation of the policies that separately maximize each subproblem. This is not obvious: in general, decomposing a sequential decision problem and optimizing pieces independently can lead to a globally suboptimal policy (the "curse of decomposition"). The theorem proves that, under the KL-regularized objective and the deterministic dynamics, the decomposition is exact. The prefix's reward $V^*_m(s_m)$ perfectly summarizes all downstream consequences, so optimizing the prefix to maximize $V^*_m$ is equivalent to optimizing it to maximize $R(s_T)$ — there is no lost information.

Why this is important for the paper's claims: The authors are not proposing VMR as a heuristic that "seems to work." They are proving that it is structure-preserving — it leaves the family-optimal solution unchanged while making it easier to reach in practice by providing denser feedback and decoupling gradient scales. This theoretical grounding distinguishes VMR from prior work that adds intermediate rewards to RL without analyzing whether they change the optimal policy.

Practical Implementation of VMR — The Risk-Sensitive Estimator (Equation 10): Computing $V^*_m(s_m)$ exactly would require integrating over all possible future trajectories under $\pi_{\text{old}}$, which is intractable. Instead, the paper uses a simple Monte Carlo estimator that requires no learned value function:

V^m(sm)=ηlog⁡(1K∑k=1Kexp⁡(1ηR(k)(sT)))\hat{V}_m(s_m) = \eta \log \left( \frac{1}{K} \sum_{k=1}^{K} \exp\left( \frac{1}{\eta} R^{(k)}(s_T) \right) \right)

where $s_m$ is a specific middle state (the partial image at step $m$), $K$ is the number of on-policy rollouts, each $\tau_k \sim \pi_\theta(\cdot | s_m)$ is a trajectory from $s_m$ to completion sampled from the current policy $\pi_\theta$ (not $\pi_{\text{old}}$ — using the current policy makes this on-policy and accounts for the fact that $\pi_\theta$ has already been updated), $R^{(k)}(s_T)$ is the terminal reward of the $k$-th completed image, $\eta = 1$ is the temperature, and $K = 2$ in practice.

What it computes step by step:

  1. Given a partially generated image at the middle scale $s_m$ (the model has produced token grids from $1 \times 1$ up to the $m$-th scale).
  2. Complete this partial image $K=2$ times by sampling the remaining token grids from the current policy $\pi_\theta$ (using the standard sampling parameters: CFG=5, top-k=2, top-p=0.9).
  3. Score each of the $K$ completed images with the reward model (OCR-based or HPSv3) to get $R^{(1)}(s_T)$ and $R^{(2)}(s_T)$.
  4. Exponentiate each reward: $\exp(R^{(1)}/1)$ and $\exp(R^{(2)}/1)$.
  5. Average the exponentiated rewards: $(e^{R^{(1)}} + e^{R^{(2)}}) / 2$.
  6. Take the natural log and multiply by $\eta = 1$ to get $\hat{V}_m(s_m)$.
  7. Use this scalar $\hat{V}_m(s_m)$ as the "terminal reward" when computing advantages for the prefix steps $1$ through $m-1$ in GRPO.

Why $K=2$ and $\eta=1$: The paper found through ablation (Table 5) that $K=2$ performs best (Word Accuracy 0.6821 vs. 0.6575 for $K=1$ and 0.6720 for $K=4$). The authors hypothesize that $K=1$ provides too noisy an estimate (high variance), while $K=4$ introduces increased variance from heterogeneous trajectories and potential compatibility issues with Mask Propagation. For $\eta=1$, this is the default temperature that makes the risk-sensitive estimator a LogSumExp of the rewards — it biases the value estimate upward (toward the better completions) without being too extreme. Higher $\eta$ would make the estimator closer to a simple average; lower $\eta$ would make it closer to a maximum.

Why this estimator is different from learning a value function (like in PPO): In standard PPO, a separate neural network (the critic) is trained to predict $V(s_t)$, and this critic must be updated alongside the policy — introducing additional hyperparameters, computational cost, and potential instability from the moving target problem. VMR's estimator requires only $K=2$ additional forward passes through the policy itself (no separate network), making it lightweight and compatible with GRPO's philosophy of avoiding a critic. The trade-off is variance: the Monte Carlo estimator has higher variance than a learned critic, but the small $K$ keeps computation cheap and the variance is acceptable given the group-based advantage normalization in GRPO.

**Selection of $m(thesplitpoint):∗∗Thechoiceofwhichresolutionscaletouseasthemiddlestepiscritical.Thepaper′sTable6showstheresolutionschedule:a‘ (the split point):** The choice of which resolution scale to use as the middle step is critical. The paper's Table 6 shows the resolution schedule: a `1024 \times 1024‘imagegoesthroughscalescorrespondingto‘` image goes through scales corresponding to `(1,1)‘,‘`, `(2,2)‘,‘`, `(3,3)‘,‘`, `(4,4)‘,‘`, `(5,5)‘,‘`, `(6,6)‘,‘`, `(7,7)‘,‘`, `(8,8)‘,‘`, `(10,10)‘,‘`, `(12,12)‘,‘`, `(14,14)‘,‘`, `(16,16)‘,‘`, `(20,20)‘,‘`, `(24,24)‘,‘`, `(28,28)‘,‘`, `(32,32)‘,‘`, `(48,48)‘,‘`, `(64,64)‘.Eachofthesecorrespondstoaprefixlength‘`. Each of these corresponds to a prefix length `m$` measured in tokens (cumulative sum of grid sizes up to that scale).

The paper reports in Table 3a and Figure 2 that $m = 128$ (the step corresponding to resolution $128 \times 128$) achieves the best text rendering scores (Word Accuracy 0.6677, NED 0.8501), while $m = 256$ (resolution $256 \times 256$) is very close (0.6565/0.8429). The authors select $m = 256$ as the default for two practical reasons: (1) it requires less compute to estimate $\hat{V}_m(s_m)$ because fewer suffix tokens remain to be generated in each rollout; (2) it is more compatible with Mask Propagation because the mask at this scale is more informative. Larger $m$ values (512, 1024) underperform significantly, suggesting that placing the VMR split too late provides insufficient densification of feedback to the early steps — most of the generation is still in the suffix, and the prefix gets almost no advantage over full-horizon GRPO.

Alternating training schedule: The paper's training alternates between optimizing the prefix and suffix subproblems. The default schedule is fine-grained alternation: for every 3 prefix GRPO updates (optimizing $\pi_{1:m-1}$), perform 1 suffix GRPO update (optimizing $\pi_{m:T-1}$). The 3:1 ratio reflects that the prefix benefits more from optimization (as shown by the prefix-only results in Figure 2). Table 4b compares this against coarse-grained alternation (300 prefix updates followed by 100 suffix updates) and finds that fine-grained consistently outperforms (Word Accuracy 0.6855 vs. 0.6778). The interpretation is that more frequent switching keeps the prefix and suffix optimization synchronized — if the suffix policy changes, the prefix needs to adapt its behavior because $V^*_m(s_m)$ (which the prefix is optimizing) depends on the suffix policy.

How VMR interacts with GRPO specifically: In a standard GRPO update, a group of $G=16$ images is generated per prompt, the reward model scores each image, and the advantage for each token in each trajectory is computed as the normalized reward of that trajectory relative to the group mean and standard deviation. With VMR, this process is modified for prefix optimization:

  1. Generate $G=16$ complete images from the same prompt using the full current policy $\pi_\theta$.
  2. For each of the 16 images, record the partial state $s_m$ at the middle step.
  3. For each partial state $s_m$, generate $K=2$ completions from $\pi_\theta$ to obtain $\hat{V}_m(s_m)$ via Equation 10.
  4. Now, when computing advantages for the prefix tokens (steps $1$ through $m-1$), use $\hat{V}_m(s_m)$ as the per-trajectory reward (instead of $R(s_T)$). The advantage for token $(t, i, j)$ in the prefix is based on how its trajectory's estimated middle value compares to the group's mean middle value.
  5. The suffix tokens (steps $m$ through $T-1$) are not updated during prefix optimization — their log-probabilities are treated as fixed (no gradient flows through them).
  6. During suffix optimization, the process is symmetric: suffix tokens are updated using $R(s_T)$ as the reward, and prefix tokens are frozen.

This decoupling ensures that the prefix's gradient does not include contributions from the suffix's token-level log-probability ratios, which are the source of the gradient imbalance. The prefix sees a single scalar reward $\hat{V}_m$ per trajectory, and its $m-1$ steps (with relatively small total token count) contribute gradients of comparable magnitude to each other.

Per-Action Normalization Weighting (PANW)

PANW is the simplest of the three components but directly addresses the most visible symptom of asynchronous policy conflicts: the gradient contributions from different steps differing by orders of magnitude due to varying grid sizes.

The problem without PANW: In a standard GRPO update, the loss at each step $t$ is computed as the sum over all tokens in the action grid $a_t$ of the per-token clipped probability ratio times the advantage. For a step with a $64 \times 64 = 4096$ token grid, the total loss contribution is the sum of 4,096 per-token losses. For a step with a $1 \times 1 = 1$ token grid, the total loss contribution is a single per-token loss. The gradient with respect to the policy parameters is proportional to this summed loss, so the $64 \times 64$ step receives approximately 4,096 times more gradient magnitude than the $1 \times 1$ step. The optimizer effectively ignores the early, structurally important steps in favor of fine-tuning the later, high-resolution steps.

The PANW correction: Each timestep $t$ is assigned a normalization weight:

kt=1(ht×wt)αk_t = \frac{1}{(h_t \times w_t)^\alpha}

where $h_t \times w_t$ is the size of the token grid generated at step $t$ (this is the grid size of the action $a_t$, i.e., $h_{t+1} \times w_{t+1}$ in the notation of Section 3.1.2), and $\alpha$ is a decay exponent controlling how aggressively to penalize large grids.

What it computes: For each step, take the total number of tokens in that step's action grid, raise it to the power $\alpha$, take the reciprocal, and use this as a multiplicative weight on the loss for that step. A step with a $64 \times 64 = 4096$ token grid gets weight $k_t = 1 / 4096^\alpha$. A step with a $1 \times 1 = 1$ token grid gets weight $k_t = 1 / 1^\alpha = 1$.

Why this form: The weight is the inverse of the action size raised to $\alpha$. When $\alpha = 1$, the weight exactly cancels the linear scaling of the loss with grid size: a 4096-token step's total weighted loss is $4096 \times (1/4096) = 1$ unit of contribution, matching the 1-token step's $1 \times 1 = 1$ unit. When $\alpha = 0$, PANW does nothing (all weights are 1) — this is the baseline without per-action normalization. When $\alpha > 1$, PANW over-corrects and gives more weight to smaller grids.

Why $\alpha \in [0.6, 0.8]$ works best (not $\alpha = 1$): Table 3b and Figure 7a show that the optimal $\alpha$ is in the range 0.6 to 0.8, with $\alpha = 0.6$ giving the best Word Accuracy (0.7136) and NED (0.8709) at step 400. This is notable because $\alpha = 1$ would be the "exact" cancellation of the grid-size effect. The fact that $\alpha < 1$ is optimal suggests that high-resolution steps should receive somewhat more gradient weight than low-resolution steps — just not 4,096 times more. The interpretation is that later, high-resolution steps genuinely have more learnable parameters (more tokens to adjust) and some degree of up-weighting is beneficial, but the uncorrected factor of thousands is pathological. The decay exponent $\alpha$ provides a continuous knob to tune this trade-off.

Interaction with per-step normalization: After applying the per-action weights $k_t$, the paper also applies step-level normalization — the weighted losses from all steps are summed and then divided by the number of steps (or equivalently, the mean per-step loss is computed). This ensures that the total gradient magnitude per update is independent of the number of generation steps. Without this, the prefix and suffix would have different numbers of steps (prefix has $m-1$ steps, suffix has $T-m$ steps), creating another source of imbalance that PANW doesn't address by itself.

Operational flow: During a GRPO update with PANW enabled:

  1. For each trajectory in the group, compute the per-token loss at each step $t$ as the negative log-probability of the chosen token times the advantage (standard GRPO token-level loss).
  2. Sum the per-token losses across all tokens in step $t$ to get the raw step loss $L_t$.
  3. Multiply by the normalization weight: $L_t^{\text{weighted}} = k_t \cdot L_t$ where $k_t = 1/(h_t w_t)^\alpha$.
  4. Average across all steps in the relevant segment (prefix or suffix): $L^{\text{total}} = \frac{1}{|\text{steps}|} \sum_t L_t^{\text{weighted}}$.
  5. Backpropagate through this averaged loss.

Mask Propagation (MP)

Mask Propagation addresses the spatial credit assignment problem within each generation step. Not all tokens in an action grid contribute equally to the terminal reward — in text rendering, tokens in the text region are critical while background tokens are largely irrelevant. Standard GRPO treats all tokens uniformly, diluting the learning signal. MP identifies reward-relevant spatial regions and focuses gradient updates on the tokens that causally influence those regions.

The core mechanism (Figure 4): MP constructs a spatiotemporal mask and propagates it backward through the VAR hierarchy:

  1. At the final generation step $T$ (the highest resolution grid), identify which spatial positions in the output directly determine the terminal reward $R(s_T)$. For text rendering, this comes from the OCR system's detected bounding boxes: tokens whose spatial coordinates fall inside a text bounding box are relevant. For HPSv3 (which produces a single scalar aesthetic score for the whole image without spatial localization), the initial mask would need to be constructed differently — the paper does not detail this case explicitly but implies that MP is most effective for tasks with spatially structured rewards.
  2. This initial binary mask $M_T$ is a grid of the same spatial dimensions as the final resolution grid, with 1s at reward-relevant positions and 0s elsewhere.
  3. Propagate $M_T$ backward to the previous (coarser) scale: for each position in the coarser grid, set its mask value to 1 if any of the finer-scale positions that it spatially corresponds to have mask value 1. This is a morphological dilation through the scale hierarchy — if a coarse-scale token influences a region that eventually contains reward-relevant fine-scale tokens, that coarse-scale token gets included in the mask.
  4. Repeat step 3 backward through all scales down to the coarsest grid. The result is a sequence of masks $M_1, M_2, \ldots, M_T$, one per generation step, where $M_t$ has the same spatial dimensions as the action grid at step $t$.

What the mask gates: During GRPO optimization, the per-token loss at step $t$ and position $(i,j)$ is multiplied by $M_t(i,j)$. Tokens with mask value 0 receive zero gradient — they are treated as if they were not part of the action (their log-probability does not contribute to the loss). Tokens with mask value 1 receive full gradient. This effectively reduces the action space at each step to only the reward-relevant spatial positions, focusing the policy update on the subset of decisions that matter.

Why this helps with scale imbalance: Recall that PANW normalizes by the total grid size $h_t \times w_t$. But if only a small fraction of the grid is reward-relevant (e.g., text occupies 10% of the image), then PANW is still diluting the gradient by dividing by the full grid size. MP complements PANW by reducing the effective action size to the number of masked-in tokens, so that PANW's normalization is applied to a more meaningful denominator.

Why this is inspired by ReFL (Reward Feedback Learning, Xu et al., 2023): ReFL is a technique from preference-based image generation where the gradient of the reward model with respect to the input image is used to identify which pixels most influence the reward, and this gradient map is used to weight the diffusion loss. MP adapts this idea to the discrete-token, multi-scale setting of VAR: instead of using continuous gradient magnitudes, MP uses binary masks derived from task-specific spatial priors (OCR bounding boxes). For tasks without spatial priors, the paper suggests that reward model gradients could be used to construct the initial mask, but this is not implemented or evaluated.

Interaction with VMR and the prefix-suffix decomposition: When MP is combined with VMR, the mask is computed once from the terminal reward or its spatial correlates, propagated backward through the full hierarchy, and then split at the middle step $m$. The prefix optimization uses only $M_1, \ldots, M_{m-1}$ (masking early steps' tokens that don't influence reward-relevant regions at later scales), and the suffix optimization uses $M_m, \ldots, M_T$. This means the prefix is not only rewarded with $\hat{V}_m(s_m)$ instead of $R(s_T)$, but its gradient is also spatially focused on the subset of coarse tokens that matter — a double concentration of the learning signal.

Empirical effect (Table 4a, Figure 7b): Enabling MP improves text fidelity metrics (Word Accuracy: 0.7071 with MP vs. 0.6855 without, at step 400; NED: 0.8699 vs. 0.8601) while CLIPScore remains essentially unchanged (0.8184 vs. 0.8188). The unchanged CLIPScore suggests that MP does not degrade overall image quality or semantic alignment — it specifically sharpens the optimization for the text-rendering objective without collateral damage. The training curves in Figure 7b show that MP leads to faster convergence (higher reward at the same training step) and a slightly higher final reward, supporting the claim that focused credit assignment reduces variance and accelerates learning.

Summary of Design Choices and Their Justifications (Technical Approach Section)

  • Deterministic MDP formulation rather than bandit: necessary to reason about per-step policy optimization and to derive the two-stage decomposition. A bandit formulation would treat the entire image as one action, making it impossible to address temporal policy conflicts.

  • KL-regularized RL framework rather than unregularized REINFORCE: prevents catastrophic forgetting and mode collapse when fine-tuning a pretrained generative model. The theoretical guarantees (Theorems 1 and 2) depend on the KL-regularized structure.

  • Risk-sensitive middle value estimator (Equation 10) rather than learning a value function: avoids the computational cost and instability of training a separate critic network, consistent with GRPO's design philosophy. The risk-sensitive form (LogSumExp) is necessary to preserve the optimality guarantees — a simple average reward would change the fixed point.

  • Split point $m = 256$ resolution rather than 128 or 512: chosen as the sweet spot where the middle value is informative enough to guide the prefix (it discriminates good vs. bad coarse structures) but cheap enough to estimate (few suffix tokens remain). The 128-resolution split performs slightly better on metrics but is rejected for compatibility with MP and computational cost.

  • Fine-grained 3:1 alternation rather than coarse-grained bulk training: keeps prefix and suffix optimization synchronized as the suffix policy changes. If the suffix trains for hundreds of steps without prefix updates, the prefix becomes stale relative to the new suffix behavior, and the $\hat{V}_m$ estimates used for prefix updates become off-policy.

  • Per-action weight $k_t = 1/(h_t w_t)^\alpha$ with $\alpha \in [0.6, 0.8]$ rather than exact $\alpha = 1$ cancellation or no normalization: balances gradient magnitudes across scales while preserving a modest advantage for high-resolution steps, which genuinely have more learnable parameters. The sub-1 exponent means the normalization is partial, not complete.

  • Binary masks from task-specific spatial priors rather than learned or gradient-based masks: simpler, more interpretable, and directly targets the known task structure (text bounding boxes). For tasks without spatial priors, this is a limitation the paper acknowledges implicitly.

  • $K=2$ rollouts for VMR estimation rather than 1 or 4: provides sufficient variance reduction over a single sample without introducing instability from divergent rollout trajectories. The optimality of $K=2$ may be specific to this task and model scale.

4. Key Insights and Innovations

Innovation 1: Diagnosing Asynchronous Policy Conflicts as the Distinctive Failure Mode of RL for VAR

The paper's most fundamental intellectual contribution is not any single algorithmic component but rather the diagnosis itself: VAR models fail under standard RL not because RL is hard for visual generation in general, but because of a structural property unique to VAR — the action space dimensionality fluctuates by orders of magnitude across timesteps, creating what the authors call asynchronous policy conflicts. This concept, named and formalized here for the first time, reframes the problem from "RL is unstable for VAR" to "VAR's MDP structure creates a specific, identifiable pathology that requires targeted interventions."

Prior to this work, the dominant assumption in visual generation RL (inherited from the GRPO-for-diffusion and GRPO-for-AR literature: Flow-GRPO, DanceGRPO, AR-GRPO, SimpleAR) was that if GRPO works for one generative architecture, it should transfer with minimal modification to others. The field treated the policy optimization problem as largely architecture-agnostic — after all, diffusion models, AR models, and VAR models all produce images from text prompts, and GRPO operates on token-level log-probabilities in all cases. The paper's Figure 1 demolishes this assumption with a single visualization: the number of query tokens per step in VAR spans from 1 to 4,096+, while AR models produce exactly 1 token per step and diffusion models operate on fixed-size latents. This is not a minor numerical difference — it means that the MDP underlying VAR generation has heterogeneous action spaces, a property that standard RL algorithms (designed for fixed-size action spaces) are not equipped to handle.

The evidence for the diagnostic power of this concept comes from Figure 2: applying GRPO to only the first 64 or 128 tokens (the prefix, where action spaces are small and homogeneous relative to each other) outperforms applying it to the full sequence. This is a striking, counterintuitive result — optimizing less of the model produces better outcomes — that only makes sense under the asynchronous policy conflict hypothesis. The prefix-only training avoids the gradient interference from later high-resolution steps, confirming that the interference, not the RL algorithm itself, is the bottleneck.

This is a fundamental conceptual advance, not an incremental improvement. It provides a unified explanation for why prior attempts at VAR RL were absent or unsuccessful (the only prior work, Gallici & Borde, 2025, applied vanilla GRPO to class-conditioned VAR without addressing this structural issue). More broadly, it introduces a new diagnostic category — asynchronous policy conflicts — that could apply to any generation model with heterogeneous action spaces (multi-scale audio generation, hierarchical planning, video prediction with varying temporal granularity). The paper has named and characterized a problem that others may have encountered but not recognized as systematic.

Innovation 2: Proving that VAR's Policy Constraint Creates a Reverse-KL Projection Problem, and Using This to Design Structure-Preserving Interventions

The paper makes a theoretical move that is rare in applied RL for generative models: it formally characterizes the VAR policy class $\mathcal{M}_\pi$ (Definition 2) as a factorized distribution that does not contain the globally optimal KL-regularized policy $\pi^*$, and then proves (Theorem 1) that the constrained optimum $\pi^\dagger$ within this class is the reverse-KL projection of $\pi^*$ onto $\mathcal{M}_\pi$ at each state. This reframes the entire RL-for-VAR problem as a projection problem — we are not directly optimizing the full-horizon objective, but rather finding the closest factorized approximation to the unconstrained optimal policy.

Why this matters beyond mathematical formalism: prior RL-for-generation work treats the policy class as essentially unrestricted (the transformer can represent any distribution over tokens) and focuses on algorithmic improvements to the optimization process. The paper shows that VAR's factorization constraint means we are solving a fundamentally different problem — constrained optimization over a restricted policy class — and that understanding this constraint is essential for designing interventions that don't accidentally change the fixed point.

Theorem 2 then proves that VMR (the prefix-suffix decomposition) is structure-preserving: inserting the middle-step soft value $V^*_m(s_m)$ as the prefix's reward does not alter the family-optimal solution. This is not obvious — in general, decomposing a sequential problem and swapping the true terminal reward for an intermediate value can change the optimal policy (a well-known pitfall in hierarchical RL). The proof that it doesn't change the optimum within $\mathcal{M}_\pi$ is what distinguishes VMR from ad-hoc intermediate reward schemes (e.g., simply adding a learned step-wise critic, which would have no such guarantee). The paper's contribution here is providing theoretical justification that their decomposition is safe — it makes optimization easier while converging to the same solution.

This theoretical framework has generality beyond VAR. Any model whose policy factorizes across a structured action space (multiple tokens generated in parallel with conditional independence assumptions, multi-scale prediction, factored autoregressive models) faces the same constrained-optimization problem, and the paper's analysis provides a template for analyzing and addressing it. The reverse-KL projection characterization (Theorem 1) is a conceptual tool, not just a proof — it tells us that the quality of the VAR-constrained optimum depends on how well the unconstrained $\pi^*$ can be approximated by a product of independent categoricals, which in turn depends on how much the true joint distribution over token grids exhibits spatial correlations that the factorization cannot capture. This opens a line of inquiry into the representational capacity of factorized policies that the paper begins but does not exhaust.

Innovation 3: Identifying Verifier Architecture as a Substitute for Learned Value Functions in Multi-Scale MDPs

Standard approaches to providing denser feedback in sparse-reward MDPs — training a critic network to predict state values $V(s)$, adding auxiliary rewards at intermediate steps, or using potential-based reward shaping — all require learning additional models or defining heuristic intermediate objectives. The paper's VMR introduces a different paradigm: estimate the intermediate value directly from terminal rewards using the policy's own rollouts, without learning any additional parameters.

The risk-sensitive estimator (Equation 10, $\hat{V}_m(s_m) = \eta \log(\frac{1}{K}\sum \exp(R^{(k)}/\eta))$) is deceptively simple — it requires only $K=2$ additional forward passes through the policy — but it is theoretically principled. It inherits the risk-sensitive form from the KL-regularized framework (Equation 3), meaning it is consistent with the optimization objective, unlike a naive Monte Carlo average which would change the fixed point. The paper's finding that $K=2$ suffices (Table 5) is a practical insight: for this task and scale, the variance of a 2-sample LogSumExp estimator is low enough to provide useful feedback without requiring the computational cost of many rollouts or the architectural complexity of a value network.

This matters because it challenges the prevailing assumption — inherited from PPO and actor-critic methods — that value function approximation is necessary for effective credit assignment in long-horizon RL. The paper demonstrates that when the MDP is deterministic and the policy is close to the reference (the KL penalty keeps $\pi_\theta$ near $\pi_{\text{old}}$), a handful of on-policy Monte Carlo rollouts can serve as a lightweight alternative to a learned critic. This is conceptually significant because it preserves the simplicity of GRPO (no separate value network) while addressing its primary weakness (sparse terminal reward causing high-variance gradients at early steps).

This insight is distinctive because it synthesizes two ideas that are usually kept separate: the theoretical framework of maximum-entropy RL (which provides the risk-sensitive value form) and the practical constraint of avoiding a critic (which motivates the Monte Carlo estimator). Prior work in hierarchical RL (Nachum et al., 2018) uses subgoals and learned value functions; prior work in image generation RL (DanceGRPO, Flow-GRPO) accepts sparse rewards without intermediate feedback; the paper finds a middle path that is both theoretically grounded and computationally cheap.

Innovation 4: Recognizing that Gradient Imbalance in Multi-Scale Generation Requires Partial, Not Full, Correction

The paper's finding that the optimal PANW exponent is $\alpha \in [0.6, 0.8]$ rather than $\alpha = 1$ (full cancellation of grid-size effects) or $\alpha = 0$ (no normalization) reveals a subtle insight: high-resolution steps genuinely deserve more gradient weight than low-resolution steps, but not thousands of times more. This is not an implementation detail — it is a conceptual finding about the nature of multi-scale generation.

The naive approach would be to either ignore the imbalance (leading to the pathology the paper documents) or to fully normalize it (setting $\alpha = 1$, which would give every step equal total gradient contribution regardless of how many tokens it generates). The paper's ablation (Table 3b, Figure 7a) shows that partial correction ($\alpha = 0.6$) outperforms both extremes. The interpretation is that later, high-resolution steps have more learnable capacity — more tokens mean more parameters being updated, and the model can productively use more gradient signal at those steps — so suppressing them entirely is counterproductive. The optimal $\alpha$ represents a balance point between preventing domination and preserving useful signal.

This is a finding about the structure of the optimization landscape in multi-scale generation, not just a hyperparameter tuning result. It suggests that the gradient imbalance is not simply noise to be eliminated but contains information about which scales are most amenable to improvement. The fact that $\alpha < 1$ is consistently optimal across experiments (text rendering and HPSv3) supports this as a general property of VAR optimization rather than a task-specific quirk.

This conceptual finding has implications beyond PANW: it suggests that future work on multi-scale generation should not treat scale imbalance as a problem to be "solved" (by exact normalization) but as a signal to be calibrated — the relative importance of different scales is a design choice that should be tuned, not eliminated. The paper provides both the diagnostic framework (recognizing the imbalance as the core issue) and the calibration tool (the $\alpha$ parameter) for making this choice.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses two evaluation datasets: CVTG-2K (Du et al., 2025) for text rendering evaluation, consisting of 2,000 text-containing images with word-level annotations, used to measure Word Accuracy, NED, and CLIPScore; and the HPSv3 benchmark (Ma et al., 2025) for human preference evaluation, which provides per-category aesthetic scores across 13 categories (All, Characters, Arts, Design, Architecture, Animals, Natural Scenery, Transportation, Products, Plants, Food, Science, Others) computed by a learned preference model. The training prompts are sampled from an in-house pool and are strictly disjoint from all evaluation splits. The paper also reports that training data for text rendering was filtered to address a region-count imbalance — the pre-filter training set had an overabundance of single-text-region images (60% of samples), while the evaluation distribution targets bins 2–5 with probabilities 0.2, 0.3, 0.3, 0.2 (Appendix C, Figure 8); post-filter training data was calibrated to better match this distribution, with the authors explicitly noting that this "does not constitute evaluation hacking or test-set leakage."

  • Base model(s). All experiments use NextFlow (Qu et al., 2025), a 7B-parameter VAR text-to-image model initialized from Qwen2.5-VL-7B (Bai et al., 2025) and augmented with a visual codebook and revised logits-prediction head (Appendix A). Images are generated at 1024 × 1024 resolution through 20 scales following the schedule in Table 6: (1,1), (2,2), (3,3), (4,4), (5,5), (6,6), (7,7), (8,8), (10,10), (12,12), (14,14), (16,16), (20,20), (24,24), (28,28), (32,32), (48,48), (64,64) — a total of 18 generation steps. The model was chosen because it represents a state-of-the-art VAR architecture for text-to-image generation, providing a strong starting point for demonstrating that RL can improve VAR models only with proper structural adaptations.

  • Metrics. For text rendering on CVTG-2K: Word Accuracy (the fraction of ground-truth words correctly recognized by PaddleOCRv5 in the generated image, with duplicate inflation penalized by taking minimum confidence), Normalized Edit Distance (NED), and CLIPScore (cosine similarity between CLIP embeddings of the image and prompt, a measure of semantic alignment). For human preference: HPSv3 score (a learned aesthetic quality predictor) reported per category as well as an overall "All" aggregate. Rewards during training are computed differently from evaluation metrics — the training reward for text rendering uses a composite OCR-based score (Equation 15: Reward = Comp + Sim - Pen) that includes confidence-weighted completeness, confidence-weighted similarity, and a length mismatch penalty, while evaluation metrics are standard benchmark scores.

  • Baselines. Three categories of baselines are used:

    • Starting point: NextFlow (without RL), evaluated at the same generation settings (CFG=5, top-k=2, top-p=0.9).
    • Within the proposed method: Vanilla GRPO applied to VAR without VMR/PANW/MP (shown in Figure 2 training curves), GRPO with VMR applied at various prefix lengths (Figure 2), and NextFlow-RL with individual components ablated (Tables 3–5).
    • External baselines on CVTG-2K (Table 1): FLUX.1 dev (12B), SD3.5 Large (8B), AnyText (1.4B), TextDiffuser-2 (8B), RAG-Diffusion (12B), 3DIS (12B), TextCrafter (FLUX and SD3.5 variants, 12B and 8B), Seedream 3.0 (proprietary), GPT Image 1 (proprietary), and Qwen-Image (20B). Key context: GPT Image 1 is the strongest proprietary system (Word Accuracy 0.8569, NED 0.9478), while Qwen-Image is the strongest open-weight model (0.8288/0.9116). These baselines are diffusion-based (except Qwen-Image which uses a proprietary architecture), making the comparison primarily about whether the VAR-based approach can be competitive with the dominant paradigm.
    • External baselines on HPSv3 (Table 2): Stable Diffusion v2.0, Stable Diffusion 3, Hunyuan (8.19 All), Stable Diffusion XL (8.20 All), Gemini 2.0 Flash (9.21 All), PixArt-Σ (9.37 All), CogView4 (9.61 All), Infinity (10.26 All), Playground-v2.5 (10.27 All), Flux-dev (10.43 All), Kolors (10.55 All). These span a range of model sizes and architectures, with Kolors representing the prior SOTA among diffusion models. The comparison against this field demonstrates generalization beyond text rendering. NextFlow's starting score is 8.43 All, placing it in the lower-middle range of these baselines.

  • Generation budget / compute accounting. The unit of computation is number of forward passes through the policy network. Training uses a group size of G = 16 (candidates per prompt) and batch size of 16 prompts per update, meaning 256 complete image generations per training step. Training runs for up to 1,200 updates (19,200 unique prompts per task). The VMR estimator requires K = 2 additional suffix completions per trajectory to compute $\hat{V}_m(s_m)$, adding a small constant factor to the generation cost. Sampling during both training and evaluation uses CFG=5, top-k=2, top-p=0.9 for all token sampling operations. The paper does not provide FLOP counts or wall-clock training time, making it difficult to compare computational cost against baselines. Notably, CFG is not used during the training forward passes (Section 4.1: "In contrast to the sampling configuration, classifier-free guidance (CFG) is not used during training"), only during evaluation. This training-inference gap has implications for the RL optimization that are not explored.

  • Cross-validation / statistical protocol. The paper does not use cross-validation for the main results — models are trained once and evaluated on the standard CVTG-2K and HPSv3 test sets. Training prompts are sampled from an in-house pool "strictly disjoint from the training and testing splits of all evaluation tasks." For ablations, results are reported at specific training steps (300 or 400), which implicitly serves as a form of early-stopping-based model selection. The paper does not report confidence intervals, standard deviations, or statistical significance tests. The training curves in Figures 2 and 7 show some fluctuation, suggesting that results are somewhat sensitive to the exact training step at which evaluation occurs. For the alternating training schedule ablation (Table 4b), both settings are evaluated at the same total training step (400), ensuring fair comparison.

Main Quantitative Results

Text Rendering on CVTG-2K

Headline result (Table 1): NextFlow-RL achieves Word Accuracy of 0.7841, NED of 0.9081, and CLIPScore of 0.8224 on CVTG-2K. This represents absolute improvements over the NextFlow starting point of +0.2305 Word Accuracy (41.6% relative), +0.1265 NED (16.2% relative), and +0.0156 CLIPScore (1.9% relative). The Word Accuracy improvement is the most dramatic — the model goes from correctly rendering approximately 55% of words to approximately 78%.

Positioning against baselines: The text rendering comparison in Table 1 reveals a nuanced picture of NextFlow-RL's standing:

  • Against proprietary systems: NextFlow-RL (0.7841 Word Accuracy, 0.9081 NED) underperforms GPT Image 1 High (0.8569, 0.9478) by substantial margins — -0.0728 Word Accuracy, -0.0397 NED. This is the ceiling. It also trails Qwen-Image (20B parameters, 0.8288 Word Accuracy, 0.9116 NED), though the gap is smaller (-0.0447 WAcc, -0.0035 NED). The Qwen-Image comparison is important because it represents a much larger model (20B vs. 7B) but the NED is essentially tied, suggesting NextFlow-RL achieves competitive character-level fidelity at a fraction of the parameter count.

  • Against open diffusion models: NextFlow-RL outperforms TextCrafter (SD3.5), the prior best open model, on Word Accuracy (0.7841 vs. 0.7600, +0.0241) and NED (0.9081 vs. 0.9038, +0.0043), while achieving higher CLIPScore (0.8224 vs. 0.8023). This is the key comparison that supports the paper's claim of establishing "state-of-the-art results among diffusion-centric baselines" — NextFlow-RL takes the top position among non-proprietary models on CVTG-2K.

  • Against other model scales: The improvement trajectory is notable. TextCrafter (FLUX) at 12B achieves 0.7370/0.8679. TextCrafter (SD3.5) at 8B achieves 0.7600/0.9038. NextFlow-RL at 7B achieves 0.7841/0.9081. This inverse relationship between model size and text rendering quality (7B > 12B > 8B) is unusual and suggests that architecture and RL optimization, not parameter count, are the dominant factors for this task.

What the RL improvement means mechanistically: The +41.6% relative improvement in Word Accuracy from NextFlow to NextFlow-RL represents the model going from failing on nearly half of words to failing on roughly one in five. The visual results in Figure 5 illustrate what this looks like in practice: character misordering (e.g., "UPBAN KIDS" rendered correctly after RL vs. garbled before), missing characters, erroneous glyphs, and extraneous characters are all substantially reduced. The CLIPScore improvement (+0.0156) is small relative to the text fidelity gains, suggesting that the RL optimization primarily improves text rendering accuracy without strongly affecting the overall semantic alignment of the image — it fixes local text errors more than global composition.

Ablations that contextualize the headline number (Tables 3–5): The 0.7841 Word Accuracy is achieved with the full method at the final training step. Ablation results at intermediate steps provide insight into which components matter:

  • At step 400 with the full method (VMR + PANW with $\alpha=0.6$ + MP), Word Accuracy is 0.7136 (Table 3b) — still substantially above the NextFlow baseline of 0.5536. The improvement to 0.7841 at the final step indicates that training beyond step 400 continues to provide gains.
  • Without MP, Word Accuracy at step 400 is 0.6855 (Table 4a) — roughly 0.028 lower than with MP, confirming MP provides a meaningful but not dominant contribution.
  • The optimal $\alpha$ choice matters: $\alpha=0.6$ at step 400 gives 0.7136, while $\alpha=1.2$ gives 0.6602 (Table 3b) — a 0.0534 gap that demonstrates the importance of PANW tuning.

Human Preference Scoring on HPSv3

Headline result (Table 2): NextFlow-RL achieves an HPSv3 "All" score of 10.64, compared to 8.43 for the NextFlow starting point — an improvement of +2.21 (26.2% relative). This places NextFlow-RL at the top of the diffusion-centric model rankings, surpassing the prior best (Kolors at 10.55) by +0.09.

Per-category breakdown: The paper reports results across 13 categories in Table 2, and the pattern reveals where NextFlow-RL's gains concentrate:

  • Largest absolute improvements over NextFlow: Food (11.18 vs. 8.93, +2.25), Animals (10.79 vs. 8.34, +2.45), Natural Scenery (10.43 vs. 8.22, +2.21), Plants (10.57 vs. 8.19, +2.38). These categories with the largest improvements are all natural/organic content categories where visual fidelity and detail matter more than precise structural constraints.
  • Saturated categories: Characters (11.72, second to Kolors at 11.79), Food (11.18, first), and Architecture (11.16, first) are categories where NextFlow-RL achieves the top score, suggesting that RL optimization particularly helps with categories that have strong compositional expectations.
  • Weaker categories: Science (9.60, substantially below Kolors at 11.06 and Flux-dev at 11.24) and Design (9.77, second to Kolors at 9.87) are categories where NextFlow-RL underperforms relative to the top baselines. Science images may require factually accurate content that RL from a preference model cannot easily incentivize; Design may benefit from aesthetic qualities that NextFlow's base architecture is weaker at.

Comparison to diffusion SOTA: The HPSv3 results in Table 2 serve a different purpose from the CVTG-2K results — they demonstrate generalization from text rendering (a task with precise, spatially structured rewards) to holistic aesthetic quality (a task with global, unstructured rewards). The fact that NextFlow-RL achieves the top "All" score demonstrates that the RL framework's benefits are not specific to the OCR-based reward design — the same VMR + PANW + MP recipe works with a general preference model as the reward signal. This is not trivial: the reward structure for HPSv3 is fundamentally different (single scalar for the whole image, no spatial localization of "correctness"), yet the three components (which were motivated by the structural properties of VAR's MDP, not by the reward specifics) transfer effectively.

The improvement gap between tasks: The +26.2% relative improvement on HPSv3 All is smaller than the +41.6% relative improvement on CVTG-2K Word Accuracy. This is expected: the NextFlow starting point is much stronger on HPSv3 (8.43, placing it among reasonable-quality models in Table 2) than on CVTG-2K (0.5536 Word Accuracy, indicating it fails on nearly half of words). RL can provide larger gains when the starting policy has more room for improvement. The HPSv3 improvement is also inherently bounded by the preference model's own ceiling — an HPSv3 score of 10.64 may approach the maximum achievable score for images at this resolution and quality level.

Visual evidence (Figures 6, 9, 10): The qualitative examples in Figure 6 show RL-refined outputs with "sharper details, cleaner structure, and stronger prompt adherence" across diverse styles including cartoons, photorealistic portraits, astrophotography, and landscapes. Appendices Figures 9 and 10 provide random samples by class, showing before/after pairs. Unlike the text rendering results where improvements are objectively measurable (correct vs. incorrect characters), the HPSv3 improvements are more subjective — sharper rendering, better composition, more aesthetically pleasing color balance — but the aggregate quantitative scores confirm the qualitative improvements are systematic rather than cherry-picked.

Training Dynamics

Figure 2 (VMR prefix comparison): The training curves show that vanilla GRPO converges to a reward value of approximately 1.00–1.05, while GRPO with VMR at prefix m128 and m256 reaches approximately 1.15–1.19 at the same training step — a roughly 14–19% higher reward. GRPO with prefix m1024 (essentially full-sequence GRPO with a negligible suffix) performs similarly to vanilla GRPO, confirming that the benefit of VMR comes from placing the split point early enough to provide meaningful feedback densification. The curves for m64, m128, and m256 show similar asymptotic performance but m128 and m256 converge faster (higher reward at earlier steps).

Figure 7a (PANW exponent comparison): Training curves for different $\alpha$ values reveal that $\alpha=0$ (no normalization) performs worst, as expected, but the ranking among $\alpha \in \{0.6, 0.8, 1.0, 1.2\}evolvesovertraining.Earlyintraining(steps0–100),all‘ evolves over training. Early in training (steps 0–100), all `\alpha > 0‘curvesaresimilar.Bystep200,‘` curves are similar. By step 200, `\alpha=1.2‘beginstolag.Bystep400,theclearorderingis‘` begins to lag. By step 400, the clear ordering is `\alpha=0.6 \approx \alpha=0.8 > \alpha=1.0 > \alpha=1.2‘.Thisprogressivedivergencesuggeststhatthebenefitofpartialnormalization(‘`. This progressive divergence suggests that the benefit of partial normalization (`\alpha < 1$`) compounds over training — early on, any normalization helps, but as optimization proceeds, exactly balancing gradient scales becomes more important.

Figure 7b (MP comparison): With MP enabled, training reaches reward values of approximately 1.20–1.25 at step 450, compared to approximately 1.15–1.20 without MP. The curves diverge around step 100 and maintain a consistent gap thereafter, with MP showing slightly less variance (narrower fluctuations) in the reward trajectory. This supports the claim that MP reduces variance in credit assignment, leading to more stable optimization.

Ablation Studies and Robustness Checks

Selection of $m$ (Table 3a, Figure 2): Evaluated at step 300, $m=128$ achieves the best scores (Word Accuracy 0.6677, NED 0.8501, CLIPScore 0.8142), while $m=256$ is close (0.6565, 0.8429, 0.8133). Larger $m$ values (512, 1024) show monotonic degradation: m512 (0.6351, 0.8236, 0.8057) and m1024 (0.5741, 0.7871, 0.8038). The m1024 result is only marginally better than the NextFlow baseline (0.5536, 0.7816, 0.8068), confirming that placing the split too late provides almost no benefit over full-horizon GRPO. The paper's selection of $m=256$ over $m=128$ (despite slightly lower metrics) is based on lower VMR estimation cost and better MP compatibility — a practical trade-off that the ablation justifies as reasonable given the small performance gap.

Decay exponent $\alpha$ (Table 3b, Figure 7a): Evaluated at step 400, the optimal range is $\alpha \in [0.6, 0.8]$:

  • $\alpha=0.6$: Word Accuracy 0.7136, NED 0.8709, CLIPScore 0.8165
  • $\alpha=0.8$: Word Accuracy 0.7051, NED 0.8673, CLIPScore 0.8172
  • $\alpha=1.0$: Word Accuracy 0.6855, NED 0.8601, CLIPScore 0.8188
  • $\alpha=1.2$: Word Accuracy 0.6602, NED 0.8459, CLIPScore 0.8144

The Word Accuracy gap between $\alpha=0.6$ and $\alpha=1.2$ is 0.0534 — roughly 7.5% of the baseline NextFlow accuracy, making this a meaningful hyperparameter. The CLIPScore is largely insensitive to $\alpha$, suggesting that the PANW exponent mainly affects text fidelity, not overall image quality. This is consistent with the component's mechanism: it balances gradient contributions across scales, and text fidelity depends heavily on both coarse layout (early steps) and fine character details (late steps), while CLIPScore (semantic alignment) may be more robust to scale imbalance.

Mask Propagation (Table 4a, Figure 7b): At step 400, enabling MP improves Word Accuracy (0.7071 vs. 0.6855, +0.0216) and NED (0.8699 vs. 0.8601, +0.0098), while CLIPScore is essentially unchanged (0.8184 vs. 0.8188). The magnitude of the improvement is smaller than that from PANW (comparing $\alpha=0.6$ to $\alpha=1.0$ gives +0.0281 Word Accuracy), suggesting MP provides a complementary but secondary benefit. The near-identical CLIPScore is notable — MP is focusing gradient updates on reward-relevant regions (text areas), and this spatial focusing does not degrade overall image quality, which would manifest as a CLIPScore drop if the background regions were being corrupted by neglect.

Number of rollouts $K$ (Table 5): Evaluated at step 300, $K=2$ achieves the best results (Word Accuracy 0.6821, NED 0.8505, CLIPScore 0.8287), outperforming both $K=1$ (0.6575, 0.8388, 0.8258) and $K=4$ (0.6720, 0.8449, 0.8278). The $K=1$ result is expected to be worse due to high variance in the single-sample estimate. The $K=4$ degradation is less expected — the paper hypothesizes it stems from "increased variance from heterogeneous trajectories" and "compatibility issues with MP." This is a practically important finding: sampling more rollouts for value estimation is not always better, and the optimal $K$ may interact with other components (MP, in particular, which relies on the terminal reward's spatial correlates, and may be confused by divergent rollout completions starting from the same middle state).

Alternating training granularity (Table 4b): Fine-grained alternation (3 prefix updates per 1 suffix update, total 400 steps) outperforms coarse-grained (300 prefix updates then 100 suffix updates, total 400 steps): Word Accuracy 0.6855 vs. 0.6778, NED 0.8601 vs. 0.8564, CLIPScore 0.8188 vs. 0.8168. The gap of 0.0077 Word Accuracy (about 1.1% relative) is modest but consistent across metrics. This supports the claim that frequent switching prevents the prefix and suffix policies from drifting out of synchronization — in the coarse-grained scheme, the prefix undergoes 300 updates while the suffix remains fixed (matching the starting model's suffix behavior), and by the time suffix optimization begins, the prefix policy has adapted to a suffix that no longer exists.

Training-data distribution calibration (Appendix C, Figure 8): The paper filters training data to address a region-count imbalance, shifting from a pre-filter distribution with 60% single-region images to a post-filter distribution that better matches the evaluation target (probabilities 0.2, 0.3, 0.3, 0.2 for region counts 2–5). This is an important data engineering detail: without this filtering, the model would be trained predominantly on single-text-region images while being evaluated on images with 2–5 text regions, creating a train-test mismatch that could produce misleadingly low (or, perversely, high) evaluation results. The authors explicitly state this is not test-set leakage — the filtering calibrates the training distribution to the intended task difficulty, not to the specific test examples.

Visual quality verification (Figures 5, 6, 9, 10): The paper includes extensive qualitative examples. Figure 5 (text rendering) shows 8 carefully selected pairs where RL fixes specific errors: character misordering (example 4: "UPBAN KIDS" vs. garbled), erroneous glyphs (example 5: French text with correct accents after RL), and missing/extraneous characters (example 7: product labels with consistent spelling after RL). Figure 6 shows 8 HPSv3 examples spanning diverse styles where the RL-refined outputs have better lighting, sharper details, and more coherent compositions. Appendix figures 9 and 10 provide random (not cherry-picked) samples by HPSv3 category, showing per-class before/after pairs. The pattern across these random samples is consistent: RL refinement generally improves detail, contrast, and compositional balance, though the magnitude of improvement varies by category.

Critical Assessment

Claim: "Our method consistently stabilizes training and accelerates convergence over vanilla GRPO, delivering substantial gains in sample quality and objective alignment."

What the experiments demonstrate: The training curves in Figures 2 and 7 consistently show that the full method (VMR + PANW + MP) achieves higher reward values and faster convergence than vanilla GRPO. On CVTG-2K, NextFlow-RL achieves +41.6% relative Word Accuracy improvement over the NextFlow baseline (Table 1). On HPSv3, it achieves a +2.21 improvement in All score (Table 2). These are genuinely substantial gains.

What the experiments do not test: The claim is about "stabilizing training" and "accelerating convergence." The training curves (Figures 2, 7) show reward value trajectories but do not report any quantitative measure of stability (variance of reward across random seeds, number of training runs, sensitivity to initialization). The curves appear to be single runs — the paper does not report error bars, standard deviations, or multiple seeds. This means we cannot distinguish between a genuine stability improvement and a single favorable training run. The convergence claim is similarly qualititative — "reaches higher reward at the same training step" is evident from the curves, but the paper does not quantify convergence rate (e.g., steps to reach 95% of asymptotic performance). Missing experiment: Multiple training runs with different random seeds to quantify variance in final performance and convergence rate.

Claim: "Our method achieves state-of-the-art results among diffusion-centric baselines on CVTG-2K and HPSv3."

CVTG-2K (Table 1): NextFlow-RL (0.7841 Word Accuracy) is the best among non-proprietary models, outperforming the prior best (TextCrafter SD3.5 at 0.7600) by 0.0241. Condition: This claim holds when restricting to open models. GPT Image 1 (0.8569) and Qwen-Image (0.8288) both achieve higher Word Accuracy. The Qwen-Image comparison is especially notable: at 20B parameters (vs. NextFlow-RL's 7B), Qwen-Image achieves +0.0447 Word Accuracy, suggesting that scale still provides an advantage that RL alone cannot fully bridge.

HPSv3 (Table 2): NextFlow-RL achieves 10.64 All, surpassing Kolors (10.55) and Flux-dev (10.43). Condition: The claim is stated carefully as "among diffusion-centric baselines," which excludes Qwen-Image and other non-diffusion architectures from the comparison set. The HPSv3 results are more favorable to NextFlow-RL than CVTG-2K because the starting point (NextFlow at 8.43) had more room for improvement, and the preference-based reward (HPSv3) directly aligns with the evaluation metric, unlike CVTG-2K where there's a gap between the training reward (OCR composite) and evaluation metrics (Word Accuracy, NED).

Weakness: The comparison is not controlled for model scale. NextFlow-RL uses a 7B model. Many baselines are larger: FLUX.1 dev (12B), SD3.5 Large (8B), TextCrafter (8–12B), Qwen-Image (20B). A FLOPs-matched or parameter-matched comparison against a similarly sized diffusion model trained with comparable RL would strengthen the claim that the method is genuinely state-of-the-art rather than benefiting from an unusually strong base model. The paper does not report training FLOPs for NextFlow-RL, making it impossible to assess whether the performance comes from more compute or better algorithmic efficiency. Missing comparison: NextFlow-RL vs. a 7B diffusion model with equivalent RL training budget.

Claim: "VMR decomposes the full-horizon objective without altering the optimal solution."

What the experiments demonstrate: Theorem 2 proves this property theoretically. Empirically, the ablation in Table 3a shows that splitting at the right $m$ (128 or 256) substantially outperforms full-horizon GRPO (splitting at 1024, which is essentially no split). The training curves in Figure 2 show that VMR curves converge to higher rewards.

What the experiments do not verify: The theorem states that VMR does not alter the optimal solution, but the experiments cannot verify this — they can only show that VMR reaches a better solution (higher reward) in finite training steps, which could be because VMR converges faster to the same optimum, or because it converges to a different (better) optimum that full-horizon GRPO fails to reach. The theoretical guarantee and the empirical outcome are consistent but not directly connected by evidence. To test the theorem, one would need to train full-horizon GRPO to convergence (potentially many more steps) and see if it eventually matches VMR, which the paper does not do. Missing experiment: Full-horizon GRPO trained for significantly more steps (e.g., 5,000–10,000) to test whether the performance gap closes, which would distinguish between accelerated convergence (consistent with Theorem 2) and a better fixed point (inconsistent with Theorem 2).

Claim: "PANW balances gradient contributions across timesteps."

What the experiments demonstrate: Table 3b shows that $\alpha=0.6$ achieves Word Accuracy 0.7136 vs. $\alpha=1.0$ at 0.6855 (both at step 400). The monotonic improvement as $\alpha$ decreases from 1.2 to 0.6 (with 0 being the worst) shows that partial normalization is better than either full normalization or no normalization. The training curves in Figure 7a show that normalized variants converge faster.

What the experiments do not measure: PANW is motivated by gradient magnitude imbalance across steps, but the paper never actually measures per-step gradient norms with and without PANW. The claim that PANW "balances gradients" is an interpretation of the mechanism, not a directly verified fact. The improved performance could arise from other effects of the per-action weighting (e.g., changing the effective learning rate schedule across steps) rather than specifically from gradient balancing. Missing experiment: Direct measurement of per-step gradient statistics (norm, variance) with and without PANW to verify that the normalization achieves its stated mechanical purpose.

Claim: "Mask Propagation isolates optimization effects to reward-relevant tokens."

What the experiments demonstrate: Table 4a shows MP provides a +0.0216 Word Accuracy improvement at step 400, with unchanged CLIPScore. The unchanged CLIPScore is the key evidence for the "isolation" claim — if MP were simply improving overall optimization, CLIPScore would likely also increase. The fact that only text fidelity improves while semantic alignment stays the same suggests that MP is indeed focusing updates on text-relevant regions.

What the experiments do not test: The paper does not verify that MP actually masks the intended tokens. There is no visualization of the masks, no analysis of what fraction of tokens are masked at each scale, and no demonstration that masked tokens receive zero gradient. The MP mechanism is described conceptually (Figure 4) but its implementation is not empirically validated. For HPSv3 evaluation, where there is no obvious spatial prior for constructing the initial mask, the paper does not explain how MP was adapted — or whether it was used at all for the HPSv3 experiments. This is a significant gap if MP is claimed to be a general component. Missing experiment: Visualization of propagated masks at different scales for representative images, and ablation of MP on HPSv3 to test whether it helps for non-spatially-structured rewards.

Single Model Family and Dataset Limitation

All experiments use exactly one base model (NextFlow, initialized from Qwen2.5-VL-7B) and two evaluation benchmarks (CVTG-2K for text rendering, HPSv3 for human preference). This is narrower than ideal for several reasons:

  • Model specificity: The paper's diagnosis of asynchronous policy conflicts is a structural claim about VAR models in general, but it is validated on only one VAR architecture. It is possible that other VAR models (Infinity, Star, HART) have different scale schedules, different per-step parallelism, or different base model architectures that change how severe the policy conflicts are. The claim that this is "the first systematic RL framework for text-to-image VAR" is true but also means that generalization is unverified.

  • Benchmark coverage: CVTG-2K tests text rendering specifically, and HPSv3 tests general aesthetic preference. Neither benchmark tests compositional reasoning (T2I-CompBench), object relationship understanding, spatial layout accuracy, or attribute binding — all important dimensions of text-to-image alignment that RL could potentially improve or harm. The paper demonstrates that the method helps for text fidelity and aesthetic quality, but says nothing about whether it helps, hurts, or is neutral for other types of visual reasoning.

  • Reward model dependence: The text rendering results use a custom OCR-based reward (PaddleOCRv5 with composite scoring). The HPSv3 results use a learned preference model. The method's performance is inevitably tied to reward model quality — if PaddleOCRv5 has systematic weaknesses (e.g., missing certain fonts, misrecognizing stylized text), the RL optimization will optimize toward those weaknesses. The paper does not analyze reward model bias or errors. Missing experiment: Evaluation with a different OCR engine (e.g., Tesseract, Google Cloud Vision) to test whether improvements transfer across reward models, which would demonstrate that the method learns genuine text rendering capability rather than overfitting to PaddleOCRv5's recognition patterns.

Training Cost and Reproducibility

The paper trains for up to 1,200 updates with group size 16 and batch size 16 — 19,200 unique prompts per task. At 1024 × 1024 resolution with 20-step VAR generation, this represents substantial compute. The paper does not report:

  • Total GPU-hours for training
  • Number of GPUs used
  • Training wall-clock time
  • Whether the results are reproducible from the described configuration (e.g., are specific random seeds required?)
  • The cost of the VMR estimation overhead (K=2 additional forward passes per trajectory) as a fraction of total training compute

This makes it difficult for other researchers to estimate the resource requirements for replication and to assess whether the method's efficiency gains (faster convergence in terms of training steps) translate to actual wall-clock savings given the per-step overhead.

The Drop from m128 to m256

The paper selects $m=256$ as the default split point despite $m=128$ achieving better metrics (Word Accuracy 0.6677 vs. 0.6565 at step 300, Table 3a). The justification — lower VMR estimation cost and better MP compatibility — is practical but means the reported headline results (Table 1) use a suboptimal configuration from a pure accuracy standpoint. This is a legitimate engineering trade-off, but it means the reported gains (+41.6% Word Accuracy improvement) are not the maximum achievable with the method — tuning $m$ for accuracy could potentially yield slightly higher numbers. The paper does not report final (step 1,200) results for $m=128$, so we cannot quantify the gap.

HPSv3: Training Reward vs. Evaluation Metric Alignment

For the HPSv3 experiments, the training reward is the HPSv3 model's own score — the policy is optimized to maximize the same metric used for evaluation. This creates a direct path for reward over-optimization: the policy could learn to produce images that score highly under HPSv3 without genuinely having better aesthetic quality (the equivalent of adversarial examples for the preference model). The paper does not address this concern or provide evaluation under a different preference model to test for overfitting. The visual results (Figures 6, 9, 10) suggest genuine quality improvements to human eyes, but without a held-out preference metric, the possibility of reward hacking cannot be excluded. Missing experiment: Evaluation with an alternative preference model (e.g., ImageReward, PickScore, or human evaluation) to verify that HPSv3 improvements reflect genuine quality gains.

6. Limitations and Trade-offs

6.1 Hard Capability Ceiling: RL Cannot Compensate for Fundamental Model Incapabilities

The assumption or constraint. The paper explicitly acknowledges through its difficulty-bin analysis (presented in the context of search and revision mechanisms in the paper's analysis of test-time compute, Section 5.3 and Section 7) that RL optimization provides essentially zero benefit on problems where the base model's pass@1 is near zero. The paper states this boundary condition clearly: "On the hardest questions, no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." While this framing comes from the referenced test-time compute scaling analysis, the same structural limitation applies to the VAR RL framework: if the base NextFlow model cannot generate images with recognizable text structures for certain fonts, languages, or layouts, no amount of RL optimization will create that capability, because RL operates by reweighting the model's existing output distribution, not by injecting fundamentally new knowledge.

The consequence. This means NextFlow-RL's improvements are bounded by the pretrained model's competence envelope. The CVTG-2K results (Table 1) show Word Accuracy improving from 0.5536 to 0.7841 — but the residual 21.6% of words that remain incorrect likely include cases where the base model has no viable path to correct rendering. For deployment, this means the method cannot be expected to achieve near-perfect text rendering; there is a hard ceiling determined by pretraining quality that RL alone cannot break through. The HPSv3 results (Table 2) suggest this ceiling is higher for aesthetic quality than for text fidelity, but it exists in both domains. This limitation is structural, not a matter of insufficient training — it arises from the KL-regularized RL formulation itself, which constrains the policy to stay close to π_old.

What evidence exists in the paper. The paper does not directly measure this ceiling for NextFlow-RL. The CVTG-2K results show continued improvement through 1,200 training steps, so the ceiling has not been reached in the reported experiments. However, the theoretical framework (Section 3.1.3, Equation 1) makes the dependence on π_old explicit: π*(a_t | s_t) ∝ π_old(a_t | s_t) exp(Q*(s_t, a_t)/η). If π_old(a_t | s_t) assigns zero or near-zero probability to a correct token, exp(Q*/η) cannot amplify it sufficiently. This is a direct consequence of the KL penalty anchoring the policy to the reference distribution. The paper does not quantify what fraction of CVTG-2K errors fall into this "impossible to reach" category vs. "reachable but not yet optimized" category.

Mitigation status. Not addressed. The paper does not propose any mechanism for expanding the model's capability envelope beyond its pretrained distribution, nor does it discuss the ceiling as a limitation. This is a fundamental tradeoff inherent in KL-regularized fine-tuning: staying close to π_old prevents catastrophic forgetting and mode collapse, but also prevents acquiring fundamentally new competencies. Future work on combining RL with continued pretraining or retrieval augmentation could potentially raise this ceiling, but the current method provides no path to doing so.

6.2 Difficulty and Reward Estimation Cost Is Not Accounted for in Efficiency Claims

The assumption or constraint. The paper's VMR mechanism requires estimating the middle-step soft value via Equation 10, which involves sampling K = 2 additional completions from each middle state to obtain terminal rewards. Section 4.1 states this explicitly: "Instead of fitting a step-wise critic as in PPO, we estimate the middle-stage value directly from on-policy terminal rewards. For a given state s_m, we sample K on-policy rollouts... up to terminal step T." Each rollout requires generating the suffix of the image (from the 256 × 256 scale to the final 64 × 64 scale), which involves multiple forward passes through the 7B-parameter model at high resolution. Across a training run with group size 16, batch size 16 prompts, and 1,200 updates, this adds a substantial but unquantified computational overhead. The paper acknowledges the cost implicitly by listing "lower compute to estimate the VMR" as a reason for selecting m = 256 over m = 128 (Section 4.4), but never accounts for this overhead in any efficiency metric or FLOP count.

The consequence. The reported performance improvements (CVTG-2K Word Accuracy +41.6%, HPSv3 All +26.2%) are achieved with additional computation beyond standard GRPO that is not reflected in the headline comparisons. The VMR overhead is proportional to the number of training steps, the group size, and the suffix length (T – m) — it is not a one-time cost. A practitioner implementing this method would need to budget for K × (T – m) additional model forward passes per trajectory per training step, which could represent a significant fraction of total training compute. Without this accounting, the method's "efficiency" is measured in terms of training steps to convergence, not total FLOPs or wall-clock time. A method that converges in fewer steps but with more expensive steps may not be faster in practice.

What evidence exists in the paper. None. The paper does not report total FLOPs, GPU-hours, training wall-clock time, or the computational overhead of VMR estimation relative to the base GRPO cost. The training setup (Section 4.1) describes group size, batch size, and number of updates, but the per-step computation is not quantified. The ablation on K (Table 5) compares K = 1, 2, 4 in terms of final accuracy but does not report the computational cost difference. The fact that K = 4 performs worse than K = 2 (0.6720 vs. 0.6821 Word Accuracy) is an interesting finding, but it is not contextualized with cost.

Mitigation status. Partially addressed through design choices. The selection of m = 256 over m = 128 (despite slightly lower metrics, Table 3a) is explicitly motivated by lower VMR estimation cost, acknowledging the tension. The small K = 2 is chosen as a cost-performance trade-off. However, these are mitigations that reduce the overhead, not eliminate it, and the remaining overhead is never quantified. The paper suggests no method for amortizing VMR estimation across training steps (e.g., reusing rollouts, learning a cheap value predictor) and flags this only indirectly as an area where practical deployment would need additional optimization.

6.3 Single Model Family and Architectural Specificity

The assumption or constraint. All experiments use exactly one base model: NextFlow, a 7B-parameter VAR text-to-image model initialized from Qwen2.5-VL-7B and augmented with a visual codebook (Appendix A). The paper presents its contribution as a general RL framework for VAR models — the title literally says "VAR RL Done Right" — but validates it on only one architecture from one model family. The paper acknowledges this scope implicitly by positioning itself against prior VAR models (Infinity, TokenFlow, Star, HART) but never tests its method on any of them. Section 2.1 surveys the VAR landscape, suggesting awareness of diversity in VAR architectures, but the experimental section (Section 4) is limited to NextFlow.

The consequence. The three components (VMR, PANW, MP) are motivated by structural properties of VAR generation — heterogeneous action spaces, factorized policies, multi-scale token grids. But different VAR architectures may have different scale schedules (e.g., Infinity uses a different tokenization strategy), different per-step parallelism assumptions, or different base model architectures (some VAR models use pure vision transformers without language model initialization). The optimal split point m, the PANW exponent α, the effectiveness of spatial masking, and even the severity of asynchronous policy conflicts may vary across architectures. The paper's claim to be "the first systematic RL framework for text-to-image VAR" is accurate but its claim to general applicability is unverified. A practitioner using a different VAR model (Infinity, Star) cannot assume the method will transfer without modification.

What evidence exists in the paper. The paper provides no cross-architecture validation. The ablation on m (Table 3a) shows sensitivity to the split point within NextFlow's specific scale schedule, which hints that the method's performance is architecture-dependent — the optimal m depends on where the model transitions from coarse to fine generation, which varies by architecture. The fact that NextFlow uses 18 scales for 1024 × 1024 generation (Table 6) while other VAR models may use different numbers of scales means that the specific m values (128, 256) are not directly portable.

Mitigation status. Not addressed. The paper does not discuss architectural generalization as a limitation, nor does it propose principles for adapting the method to other VAR architectures beyond the specific hyperparameters tuned for NextFlow. The theoretical framework (Theorems 1 and 2) is architecture-agnostic and should apply to any VAR model that factorizes actions spatially, but the practical implementation (choice of m, α, K, mask construction strategy) would need to be re-tuned. Future work that validates the method on multiple VAR architectures (Infinity, Star, HART) with minimal hyperparameter changes would substantially strengthen the generality claim.

6.4 Mask Propagation Relies on Task-Specific Spatial Priors and Is Not Validated for Unstructured Rewards

The assumption or constraint. Mask Propagation (Section 3.5) identifies reward-relevant spatial regions by constructing an initial mask from "output components that directly determine the reward (e.g., predicted bounding boxes)." For text rendering, this means OCR-detected bounding boxes around text regions — the mask is built from the spatial output of the same OCR system used to compute the reward. For tasks where "the reward is a single scalar without spatial localization" (such as HPSv3, which produces an overall aesthetic score with no per-region breakdown), the paper provides no specification of how the initial mask is constructed. The MP mechanism is described as a general component, but its dependence on spatially structured rewards is a fundamental constraint: MP requires knowing where in the image the reward signal originates.

The consequence. For text rendering (CVTG-2K), MP provides a modest but consistent improvement (+0.0216 Word Accuracy at step 400, Table 4a), but this is the most favorable case — the task has precise, localized rewards (each text region's correctness is spatially grounded) and the mask can be constructed automatically from the OCR system's detections. For HPSv3, it is unclear whether MP was used at all, and if so, how the mask was constructed. If MP was not used for HPSv3, then the HPSv3 results (Table 2) demonstrate that VMR + PANW alone achieve state-of-the-art performance, making MP a text-rendering-specific enhancement rather than a general component. If MP was used with a heuristic mask (e.g., treating the entire image as reward-relevant, reducing MP to a no-op), then it contributed nothing to the HPSv3 results. Either way, MP's claimed role as a general mechanism for spatial credit assignment is only validated for tasks with external spatial annotations.

More broadly, this limitation means the method is strongest for tasks where the reward function provides spatial feedback (text rendering, object detection, layout accuracy) and weaker for tasks with global, holistic rewards (aesthetic preference, style transfer, general image quality). The paper's two evaluation tasks span both types, which is good experimental design, but the component-level contribution varies by task in ways the paper does not analyze. A practitioner working on a task without spatial annotations (e.g., improving general image quality with a CLIP-based reward) would not know how to implement MP or whether it would help.

What evidence exists in the paper. The MP ablation (Table 4a) is performed only on CVTG-2K, not on HPSv3. The paper does not report whether MP was used for the HPSv3 experiments in Table 2. The MP mechanism description (Section 3.5, Figure 4) describes the mask propagation algorithm in general terms but does not specify how the initial mask is constructed for different reward types. The visualizations in Figure 4 show a conceptual diagram of mask propagation across scales but do not show actual masks from real images. The paper does not discuss the requirement for spatial reward structure as a limitation.

Mitigation status. Not addressed. The paper does not acknowledge that MP's applicability is limited to tasks with spatially structured rewards, nor does it propose methods for constructing masks when spatial annotations are unavailable (e.g., using gradient-based attribution from the reward model, as the ReFL inspiration suggests, or using attention maps from the VAR model itself). The paper's stated connection to ReFL (Xu et al., 2023) implies that gradient-based mask construction is possible, but this is not implemented or evaluated. The fact that the strongest results (CVTG-2K) benefit from a task-specific mask construction while the generalization results (HPSv3) may not use MP at all creates an unacknowledged asymmetry in the method's applicability.

6.5 No Measurement of Reward Over-Optimization or Verification of KL Constraint Satisfaction

The assumption or constraint. The paper's theoretical framework (Section 3.1.3) defines the optimal policy under a soft KL constraint: π*(a_t | s_t) ∝ π_old(a_t | s_t) exp(Q*(s_t, a_t)/η). The temperature η controls the trade-off between reward maximization and staying close to the reference policy. In the practical implementation, this KL constraint is not explicitly enforced — GRPO uses group-based relative advantages that implicitly penalize large deviations through the advantage normalization, but there is no explicit KL penalty term and no post-training measurement of how far the optimized policy has diverged from π_old. The paper sets η = 1 in the VMR estimator (Equation 10) but does not discuss whether this value effectively constrains the policy or whether reward over-optimization occurs.

The consequence. Without explicit KL monitoring, the RL optimization could produce a policy that scores highly on the training reward but has degraded in ways not captured by the evaluation metrics. For text rendering, the composite reward (Equation 15) includes confidence-weighted completeness, similarity, and a length penalty — a policy could potentially over-optimize by producing text that PaddleOCRv5 recognizes with high confidence but that is visually unnatural, incorrectly formatted, or stylistically poor. The CLIPScore metric (which measures semantic alignment between image and prompt) provides a partial check — it remains high for NextFlow-RL (0.8224, up from 0.8068) — but this is a coarse measure that may not catch subtle degradations. For HPSv3, the training reward is the HPSv3 model itself, creating a direct path for adversarial over-optimization: the policy could learn to produce images that exploit HPSv3's particular biases to achieve high scores without genuine aesthetic improvement.

The paper's test-time compute scaling analysis (referenced in the prior sections) explicitly documented PRM over-optimization — beam search degrading performance on easy problems at high budgets due to verifier exploitation. The same phenomenon could affect VAR RL: if the reward model (PaddleOCRv5 or HPSv3) has blind spots, the policy could learn to exploit them rather than genuinely improving. The paper does not check for this.

What evidence exists in the paper. None. The paper does not report the KL divergence between the trained policy and the reference policy, does not track reward model scores under a different reward model (cross-reward-model evaluation), and does not analyze whether the policy exploits reward model weaknesses. The CLIPScore improvement (+0.0156 on CVTG-2K, Table 1) is small and could be consistent with either genuine improvement or mild over-optimization that doesn't degrade semantic alignment enough to register. The visual results (Figures 5, 6, 9, 10) are consistent with genuine improvement, but qualitative examples cannot rule out systematic over-optimization that an adversary could detect. For HPSv3, no alternative preference metric is evaluated to cross-validate the HPSv3 improvements.

Mitigation status. Not addressed. The paper does not discuss reward over-optimization as a risk, does not implement any explicit KL penalty or tracking, and does not evaluate under alternative reward models to detect exploitation. This is a notable gap given that over-optimization is a well-known failure mode in RL for generative models (documented extensively in the RLHF literature and in the paper's own references to PRM over-optimization). The theoretical framework provides the tools for measuring and controlling this — the KL divergence to π_old and the temperature η — but these are not used in practice. A simple mitigation would be to periodically evaluate the trained policy with a different OCR engine (e.g., Tesseract, Google Cloud Vision) or a different preference model (ImageReward, PickScore) to detect reward-specific overfitting.

6.6 Unverified Generality Across Prompt Distributions and the Unquantified Effect of Training Data Engineering

The assumption or constraint. The paper uses an in-house training prompt pool that is "strictly disjoint from the training and testing splits of all evaluation tasks" (Section 4.1). However, Appendix C (Figure 8) reveals that significant data engineering was performed: the training data for text rendering exhibited a "pronounced imbalance in the number of text-rendering regions per sample," with 60% of images containing only a single text region, while the evaluation distribution (CVTG-2K) predominantly contains images with 2–5 text regions. The paper applied "region-count–based filtering" to recalibrate the training distribution, shifting from [0.60, 0.21, 0.09, 0.05, 0.02, 0.03] across region counts [1, 2, 3, 4, 5, >5] to a target profile that better matches the evaluation distribution. The paper asserts this "does not constitute evaluation hacking or test-set leakage." No analogous analysis or filtering is described for the HPSv3 training data.

The consequence. The training data filtering is a legitimate engineering practice — it calibrates the training distribution to the evaluation task's difficulty — but it introduces a degree of freedom that is not accounted for in the method's claimed generality. A practitioner deploying NextFlow-RL for text rendering in a production setting with a different prompt distribution (e.g., predominantly single-text-region posters, or images with many small text regions) would not know whether the reported performance transfers. The filtering may have been necessary to achieve the reported results — without it, the model might have overfit to single-text-region generation and performed poorly on CVTG-2K's multi-region images. The paper does not report an ablation without filtering, so the contribution of data engineering vs. algorithmic improvement is not separable.

More broadly, the "in-house prompt pool" is not described in detail — its size, diversity, prompt length distribution, language distribution, or stylistic variety are not specified. This makes it impossible to assess how representative the training prompts are of real-world deployment scenarios. If the in-house pool skews toward certain types of prompts (e.g., product descriptions, marketing copy) and away from others (e.g., artistic descriptions, complex scenes), the RL optimization may specialize to the training distribution in ways that the CVTG-2K and HPSv3 evaluations don't fully test.

What evidence exists in the paper. Appendix C provides the region-count distributions before and after filtering, which is admirably transparent. Figure 8 visualizes the shift. The paper explicitly states that filtering was applied and characterizes it as calibration, not leakage. However, the paper does not provide: a description of the in-house prompt pool (size, sources, diversity), an ablation without filtering to measure its contribution, or any analysis of whether the filtering affected which components (VMR, PANW, MP) are most beneficial. The HPSv3 training prompts are completely undescribed — we don't know if they were filtered, what their distribution looks like, or whether they overlap with HPSv3's training data.

Mitigation status. Partially addressed through transparency (the filtering is described and visualized) but not through experimental control. The paper does not claim that the filtering is unnecessary or that the method would work equally well without it. A reader cannot determine whether the data engineering is a critical enabler of the results or a minor optimization. Future work that tests the method on multiple prompt distributions (including unfiltered data) or that develops robust training procedures that don't require distribution calibration would address this limitation. The paper's statement that filtering "does not constitute evaluation hacking or test-set leakage" is correct in the narrow sense (no test examples were used for filtering) but doesn't address the broader concern about distributional robustness.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around RL for visual generation from "can we apply standard algorithms to a new architecture" to "the architecture's structural properties determine whether standard algorithms work, and understanding those properties lets us design targeted fixes." This is not a new RL algorithm — GRPO remains the optimizer — but rather a diagnosis-driven systems contribution that identifies and resolves a failure mode (asynchronous policy conflicts) that was previously unrecognized because the field was applying RL to architectures (diffusion, AR) that don't exhibit it.

The shift is best characterized as a reframing with methodological consequences. Before this work, the implicit assumption in visual generation RL was that GRPO transfers across architectures with minimal adaptation — the algorithm operates on token-level log-probabilities, which every generative model produces. This paper demonstrates that this assumption is false for VAR models specifically because their MDP structure (heterogeneous action spaces, factorized parallel-token policies) creates optimization pathologies that GRPO was never designed to handle. The reframing is: the MDP matters. You cannot treat a VAR generation process as a black-box sequence of token predictions; you must explicitly model how the action space changes across timesteps and design the RL procedure to account for it.

This reframing has several concrete consequences for the research landscape:

It establishes VAR RL as a distinct subproblem rather than a trivial extension of diffusion/AR RL. Prior work on RL for visual generation (Flow-GRPO, DanceGRPO, AR-GRPO, SimpleAR) developed recipes for diffusion and standard autoregressive architectures. This paper shows that those recipes fail on VAR — Figure 2's training curves demonstrate that vanilla GRPO applied to VAR converges to lower rewards than prefix-only GRPO, a counterintuitive result that only makes sense under the asynchronous policy conflict diagnosis. This means future work on VAR alignment cannot simply import diffusion-RL methods; it must engage with the structural properties this paper identifies. The paper thus creates a new subfield within visual generation RL — VAR-specific optimization — that did not exist before.

It provides a theoretical language for reasoning about constrained policy optimization in structured action spaces. The formalization of the VAR policy class as a factorized distribution (Definition 2) and the proof that the constrained optimum is a reverse-KL projection of the unconstrained optimum (Theorem 1) are conceptual tools, not just mathematical exercises. They give researchers a way to analyze why optimization fails and what properties an intervention must preserve to avoid changing the fixed point. Theorem 2's proof that VMR preserves optimality is the first demonstration in the visual generation RL literature of a decomposition that is provably structure-preserving under a factorized policy constraint. This is not standard in applied RL — most methods are justified empirically, not theoretically — and it sets a higher bar for future work: interventions should come with an analysis of whether they change the optimal solution, not just whether they empirically help.

It resolves the tension between "RL should work for VAR" (by analogy with diffusion/AR) and "RL doesn't work for VAR" (as preliminary attempts likely showed). The paper explains why prior attempts at VAR RL were absent from the literature: applying GRPO directly produces worse results than doing nothing (or than doing prefix-only RL, as Figure 2 shows). The only prior work (Gallici & Borde, 2025) used vanilla GRPO on class-conditioned VAR — this paper's analysis suggests that approach should be suboptimal, and the fact that it worked at all (on a simpler task) may be due to the reduced complexity of class-conditional vs. text-to-image generation. The paper reconciles the field's expectation (VAR should be optimizable with RL) with the practical difficulty by identifying the specific structural barrier and showing it can be overcome.

It redirects research attention from "better RL algorithms" to "better MDP-aware optimization structures." The paper's three components are not about improving the RL update rule (the GRPO advantage computation is unchanged). They are about restructuring how the optimization problem is presented to the algorithm — decomposing the horizon (VMR), reweighting the loss (PANW), and spatially focusing the gradient (MP). This suggests that for architectures with structured action spaces, the primary research challenge is not algorithmic innovation but problem formulation — casting the generation process into a form that existing algorithms can optimize effectively. This is a conceptual shift away from "better optimizer" and toward "better optimization problem design," which could influence how researchers approach RL for other structured-generation architectures (video, audio, multi-modal outputs).

It makes diffusion-centric SOTA comparisons newly relevant for VAR models. Before this work, NextFlow scored 8.43 on HPSv3 All (Table 2), placing it in the lower-middle tier of diffusion models. After RL, NextFlow-RL achieves 10.64, the top position among diffusion-centric baselines. This demonstrates that VAR models can compete with diffusion models on aesthetic quality when properly optimized — a result that was not obvious before this work, given VAR's relative youth. The paper thus provides the first evidence that VAR + RL can close the gap with mature diffusion + RL pipelines, changing the competitive landscape for architectural choices in text-to-image generation.

However, this is not a paradigm shift — the fundamental approach (KL-regularized policy optimization with GRPO) is unchanged. It is better understood as a diagnosis-driven adaptation of an existing paradigm to a new architecture class, with theoretical grounding that turns empirical observations (Figure 2) into a principled framework. The magnitude of the contribution is in identifying why the paradigm transfer fails and providing the minimal set of structural modifications to make it succeed, not in proposing a new paradigm.

Follow-Up Research This Work Enables

Cross-architecture validation of asynchronous policy conflicts. The paper's central diagnosis — that heterogeneous action spaces cause gradient interference across timesteps — is validated only on NextFlow (7B, Qwen2.5-VL initialization, 18-scale generation). A strong follow-up would test whether the same pathology and the same fixes (VMR + PANW + MP) transfer to other VAR architectures with different scale schedules: Infinity (Han et al., 2025), which uses bitwise autoregressive modeling with potentially different tokenization; Star (Ma et al., 2024), with its scale-wise text-conditioned design; and HART (Tang et al., 2024), a hybrid autoregressive transformer. The specific experiment: apply vanilla GRPO vs. VMR at the optimal split point for each architecture (which may differ from NextFlow's m=256), measure whether the prefix-only GRPO advantage (Figure 2 pattern) replicates, and test whether PANW's optimal α shifts with the scale schedule. A negative result — VMR not helping on a particular VAR variant — would reveal which architectural properties (codebook size, number of scales, per-scale parallelism) modulate the severity of policy conflicts. This is newly tractable because the paper provides both the diagnostic methodology (comparing full-sequence vs. prefix-only GRPO training curves) and the intervention template (VMR split-point selection procedure).

Learning difficulty estimators for VMR split-point selection. The paper's VMR requires choosing the middle timestep m, and the ablation (Table 3a) shows that m=128 outperforms m=256 on metrics but was rejected for computational cost. This is a static, hand-tuned choice. A natural extension is to learn a dynamic split-point policy that selects m per-prompt based on estimated generation difficulty. For text rendering, difficulty correlates with the number and complexity of text regions — a prompt asking for "Hello World" on a plain background might benefit from a later split (more prefix optimization, since text is simple and late-stage refinement matters more), while a prompt with multiple long text strings in a complex scene might benefit from an earlier split (coarse layout matters more). The specific experiment: train a lightweight classifier on the prompt embedding to predict which m ∈ {64, 128, 256, 512} maximizes final reward, using the PRM-based difficulty estimation approach from the test-time compute scaling literature as a template. Evaluate whether per-prompt dynamic m outperforms the static m=256 on CVTG-2K. This is enabled by the paper's demonstration that m significantly affects performance and that the effect is monotonic with resolution (Table 3a), providing a clear signal to learn from.

Gradient-based mask construction for tasks without spatial reward structure. The paper's Mask Propagation requires an initial spatial mask derived from reward components (OCR bounding boxes for text rendering). For tasks with holistic rewards — HPSv3 aesthetic scores, CLIP alignment, human preference judgments — there is no such spatial prior. The paper's ReFL inspiration (Xu et al., 2023) suggests a path forward: use the gradient of the reward model with respect to the input image to identify which spatial regions most influence the reward. The specific experiment: for HPSv3 training, backpropagate through the HPSv3 model to the final generated image to obtain a gradient attribution map, threshold it to create a binary mask, propagate this mask backward through VAR's scales as described in Section 3.5, and compare against (a) no MP, (b) a uniform mask (all tokens relevant, reducing MP to a no-op), and (c) the text-rendering-style mask if applicable. Measure whether gradient-based MP improves HPSv3 scores beyond VMR + PANW alone, and whether it introduces instability (gradient-based masks are noisier than OCR bounding boxes). A negative result — gradient-based MP not helping — would clarify that MP's benefit is tied to task-specific spatial structure and is not a general mechanism, refining the paper's claim about MP's role.

Combining VMR with learned prefix value functions. The paper's VMR uses a 2-sample Monte Carlo estimator (Equation 10) to avoid learning a value function. This is computationally cheap but has non-negligible variance, and the ablation (Table 5) shows K=4 degrades performance. A follow-up could ask: can we train a lightweight value head on top of the frozen VAR model to predict V(s_m) more accurately, amortizing the estimation cost? The specific experiment: add a small MLP head that takes the VAR model's hidden state at the middle step and predicts V_m(s_m), trained with a regression loss against Monte Carlo estimates from K=16 rollouts (more samples for training the value head, but the head is cheap at inference). Compare the value-head VMR against the Monte Carlo VMR on (a) final reward achieved, (b) training stability (variance of reward curves across seeds), and (c) wall-clock training time (the value head adds a forward pass for the value estimate but eliminates the K rollout overhead). This is newly tractable because the paper provides both the theoretical definition of V_m (Equation 4) and the empirical demonstration that a reliable middle value estimate enables effective prefix optimization, establishing the value of the value — the question is whether learning it is better than sampling it.

Measuring and controlling the policy-to-reference divergence. The paper's theoretical framework is built on KL-regularized RL, but the practical implementation (GRPO) does not explicitly track or constrain the KL divergence between the trained policy π_θ and the reference π_old. This leaves open the possibility of reward over-optimization. A targeted experiment: throughout training, periodically compute the per-step KL divergence KL(π_θ(·|s_t) || π_old(·|s_t)) for a held-out set of prompts, at each scale. Test whether the divergence grows monotonically (suggesting progressive forgetting of pretrained behavior), plateaus (suggesting GRPO's implicit regularization is sufficient), or diverges at specific scales (suggesting scale-specific over-optimization). If the latter, test whether per-scale temperature tuning (scaling the advantage by a scale-dependent η_t) prevents divergence while maintaining reward gains. This is enabled by the paper's MDP formulation and Theorem 1 — the divergence to π_old is exactly the quantity that KL regularization controls, and the per-state decomposition allows measuring where in the generation process the policy is drifting furthest.

Stress-testing with adversarial prompts and out-of-distribution text. The paper's text rendering improvements on CVTG-2K (Word Accuracy 0.5536 → 0.7841) are impressive but the benchmark has a specific character. A stress-test would evaluate on systematically challenging prompts: (a) long text strings (>50 characters) that stress the model's sequential coherence across scales, (b) multilingual text with mixed scripts (Cyrillic + Latin, Chinese + English), (c) rotated, curved, or perspectively distorted text that requires geometric reasoning, and (d) adversarial prompts designed to confuse OCR systems (homoglyph attacks, visually similar characters). For each, measure whether the RL improvement transfers (suggesting robust text rendering capability) or degrades (suggesting the RL optimization overfit to CVTG-2K's text distribution and the OCR reward model's recognition patterns). A negative result — NextFlow-RL performing worse than NextFlow on adversarial text — would reveal that the OCR-based reward shaped the policy toward PaddleOCRv5's recognition biases rather than genuine text rendering skill, a critical limitation for deployment.

Practical Applications and Downstream Use Cases

Production text-in-image generation for e-commerce and marketing. The most direct application is generating product images, promotional banners, and social media graphics that contain accurate text. The paper's CVTG-2K results show NextFlow-RL achieves Word Accuracy 0.7841 — roughly 78% of words are correctly rendered — up from 55% for the base model. This is still imperfect (approximately one in five words has errors) but represents a substantial improvement for applications where manual correction is feasible. A concrete deployment scenario: a marketing team uses NextFlow-RL to generate 100 ad variants with product names, slogans, and pricing, then a human reviewer checks and corrects the ~22 images with text errors, rather than ~45 images with the base model. This cuts manual correction workload by more than half. The 7B model size makes this feasible on a single high-memory GPU (A100 or H100), reducing deployment cost compared to larger proprietary systems. The key practical caveat: the model was trained on an in-house prompt pool with controlled text-region counts (Appendix C, Figure 8); deploying on an arbitrary distribution of user prompts would require monitoring whether performance degrades for prompt categories underrepresented in training (e.g., single-text-region images, which were down-weighted).

On-device or edge deployment with a smaller model. The paper demonstrates that a 7B VAR model with RL optimization can match or exceed much larger diffusion models on text rendering (Table 1: NextFlow-RL at 7B beats TextCrafter FLUX at 12B, 0.7841 vs. 0.7370 Word Accuracy) and approach larger models on aesthetic quality (Table 2: NextFlow-RL at 7B achieves 10.64 All vs. Kolors at 10.55). This suggests a deployment strategy: use a 7B VAR model with RL fine-tuning as a cost-effective alternative to 12B+ diffusion models for applications where inference latency and hardware cost matter. The practical benefit is quantifiable: on CVTG-2K, NextFlow-RL delivers better text rendering than a 12B diffusion model while using approximately 58% of the parameters and correspondingly less VRAM. For a mobile or edge deployment where model size is the primary constraint, this makes high-quality text-in-image generation newly feasible. The caveat: VAR generation speed depends on the number of scales and the per-scale parallelism; a direct latency comparison against diffusion models with equivalent hardware would be needed to confirm the practical advantage, and the paper does not provide this.

Self-improvement data generation for VAR pretraining. The paper's RL framework can be used to generate high-quality training data for further pretraining or fine-tuning of VAR models — a self-improvement loop where the RL-optimized policy generates images, which are then filtered by reward and used as supervised targets for the base model. This is analogous to rejection sampling fine-tuning in language models. The specific pipeline: use NextFlow-RL (with the CVTG-2K reward) to generate 100,000 text-containing images from diverse prompts, keep the top 50% by OCR reward, and fine-tune the base NextFlow model on these filtered image-prompt pairs using standard next-scale prediction loss. Measure whether the resulting model (trained on RL-generated data) outperforms the base model without ever using RL during inference. This is attractive because it amortizes the RL cost (expensive training) into cheaper supervised fine-tuning at scale, and the paper's demonstration that RL substantially improves text rendering (Word Accuracy +41.6% relative) provides the quality signal needed to make the self-improvement loop effective. The risk: the RL policy may have mode-collapsed or lost diversity compared to the base model, so the generated training data may have lower diversity than the original pretraining data, potentially hurting generalization. This would need to be measured.

Preference-aligned VAR generation for creative tools. The HPSv3 results (Table 2: NextFlow-RL achieves 10.64 All, the top open score) position the method for integration into creative tools where aesthetic quality is the primary metric — digital art generation, concept visualization, mood board creation. A concrete scenario: a design tool integrates NextFlow-RL as the default image generation backend, offering users higher-quality outputs (as measured by learned human preference models) out of the box without requiring prompt engineering or manual selection from multiple samples. The practical benefit is quantifiable: NextFlow-RL's All score of 10.64 vs. NextFlow's 8.43 represents a jump from below Flux-dev (10.43) and Playground-v2.5 (10.27) to above them, meaning the tool goes from "worse than leading diffusion models" to "competitive or better" with a single RL fine-tuning run. The caveat: HPSv3 is a learned proxy for human preference, and the paper does not validate with actual human evaluators; a tool deployed on this basis should run an internal human evaluation study to confirm that the HPSv3 improvements translate to user-perceived quality gains.