FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling

ArXiv: 2604.06916

🎯 Pitch

Quantized FP4 rollouts can match BF16’s alignment performance while accelerating diffusion RL convergence up to 4.64×, provided the policy is only trained on high-precision regenerations of the contrastive samples. The paper resolves the longstanding tension between scalable exploration and training stability by decoupling these stages completely, demonstrating that intra-group reward ranking remains consistent enough under quantization to safely guide selection without degrading final reward.

---

1. Executive Summary

This paper introduces Sol-RL (Speed-of-light RL), a novel FP4-empowered Two-Stage Reinforcement Learning framework that resolves the efficiency-stability dilemma in diffusion model alignment by decoupling exploration from optimization — using high-throughput NVFP4-quantized rollout to filter a massive candidate pool for highly contrastive samples (the top-k and bottom-k by reward), then regenerating only those selected seeds in BF16 precision for the actual policy update. Evaluated on FLUX.1, SANA, and SD3.5-Large across multiple reward models (ImageReward, CLIPScore, PickScore, HPSv2), Sol-RL achieves up to 4.64× convergence speedup while maintaining alignment fidelity on par with the standard BF16 rollout pipeline — for instance, matching the BF16 baseline’s performance with a 2.4× reduction in rollout time and only a marginal accuracy gap (≤1% on HPSv2), establishing that quantized exploration can serve as a reliable proxy for intra-group reward ranking only when the policy optimization phase is strictly shielded from low-precision corruption.

2. Context and Motivation

The Core Problem: Scaling Rollouts in Diffusion RL Creates an Efficiency Bottleneck

The central challenge this paper addresses emerges from a tension between algorithmic effectiveness and computational cost in Reinforcement Learning for diffusion models. Rollout scaling — generating many candidate outputs per prompt and selecting the best ones for policy optimization — has proven to be highly effective for aligning text-to-image models with human preferences. Recent work, particularly DanceGRPO (Xue et al., 2025), demonstrates that using only the most contrastive samples from a large candidate pool (e.g., the top-k and bottom-k by reward) provides more reliable gradient signals for policy updates. This "selective training" paradigm scales the number of rollouts while keeping the training overhead constant, yielding faster convergence and superior alignment performance.

However, this approach introduces a profound practical bottleneck: the computational cost of generating the candidate pool itself now dominates the entire training pipeline. The paper quantifies this in Section 3.1 and Figure 3a — under a typical 24-in-96 rollout setting (selecting 24 contrastive samples from 96 generated candidates), the BF16 generation phase consumes the majority of the iteration time. For FLUX.1-dev, naive BF16 rollout takes 184 seconds out of a 274-second total iteration; for SD3.5-Large, the cost is 451 seconds out of 691 seconds. The problem is compounded by the inherent algorithmic redundancy of the approach: because only a small fraction of generated samples are ultimately used for gradient updates, the vast majority of expensive high-precision compute is effectively wasted.

This bottleneck matters enormously in practice. Diffusion RL post-training has become a standard pipeline for improving text-to-image foundation models — models that are themselves massive (FLUX.1 has 12 billion parameters). Making this post-training step practical at scale requires either accepting prohibitive computational costs or finding ways to accelerate the rollout phase without sacrificing alignment quality. The paper's framing of this as an "efficiency-stability dilemma" captures the core tension: you can scale rollouts for better alignment, but doing so naively becomes computationally unsustainable.

Why This Problem Matters

The significance extends beyond practical engineering concerns into several interconnected domains:

1. The economics of foundation model deployment. Modern text-to-image foundation models are expensive not only to pretrain but also to align. As the field moves toward larger models (SD3.5-Large, FLUX.1) and more sophisticated alignment techniques, the cost of post-training can rival or exceed pretraining costs in certain deployment scenarios. If rollout scaling — which has been empirically validated as highly effective across multiple works — is computationally bottlenecked, the community faces an artificial ceiling on alignment quality that is purely a function of hardware budgets, not algorithmic capability. The paper explicitly connects this to the broader trend of test-time compute scaling seen in LLMs, where there is "substantial room for further alignment gains" that remains inaccessible due to cost.

2. The democratization of model alignment. Large-scale rollout scaling with BF16 precision effectively gates powerful alignment techniques behind expensive hardware clusters. Organizations without access to large GPU fleets cannot afford to run 96-sample rollouts with FLUX.1 repeatedly for thousands of training iterations. By reducing the computational barrier, acceleration techniques like quantization could make state-of-the-art alignment pipelines accessible to a broader research and practitioner community.

3. The theoretical understanding of rollout dynamics in diffusion RL. From a research perspective, the paper sits at the intersection of two active areas: the algorithmic design of diffusion RL methods (GRPO variants, forward-process optimization) and the system-level challenge of efficient inference. Understanding why quantized rollouts work as ranking proxies — and where they fail as direct optimization targets — requires careful analysis of how numerical precision interacts with the specific structure of diffusion model sampling (ODE determinism, the mapping from noise seeds to semantic layouts). This problem is not unique to this paper's solution; it reflects a broader gap in the community's understanding of how precision-accuracy tradeoffs play out in reinforcement learning pipelines.

Where Prior Approaches Fall Short

The paper identifies limitations across two categories of existing solutions:

Category 1: Direct Quantization of RL Rollouts

The most straightforward approach to accelerating rollout generation is to quantize the model to lower precision (e.g., FP4 or FP8). Several recent works have explored this direction:

• FlashRL (Liu et al., 2025) and QeRL (Huang et al., 2025) demonstrated substantial speedups by using quantized inference for LLM rollouts.
• FP8-RL (Qiu et al., 2026) uses importance ratios between quantized inference and BF16 training to correct for the distribution shift.
• QuRL (Li et al., 2026) proposes adaptive clipping mechanisms to prevent divergence between the quantized actor and the high-precision policy.
• Jet-RL (Xi et al., 2026) advocates for a unified FP8 precision flow across both training and rollout phases, fundamentally eliminating the off-policy gap.

However, these approaches face fundamental limitations when applied to diffusion RL, as documented in Section 3.2:

The off-policy gap problem. When trajectories are sampled using a quantized policy, they exhibit an inherent distribution shift from the high-precision target policy. As concurrent studies (Yao et al., 2025; Xi et al., 2026) have highlighted, this shifts the optimization process into an off-policy setting, which can induce "severe numerical discrepancies" and training instabilities. Evolution and value estimation in RL are both sensitive to distribution shift, and quantized rollouts fundamentally break the on-policy assumption that most modern RL algorithms (including GRPO and its diffusion variants) rely on.

The continuous state space exacerbation. Diffusion RL introduces a problem that is less severe in discrete-token LLM settings. Mainstream "forward-process" diffusion RL algorithms — including Advantage Weighted Matching (AWM, Xue et al., 2025) and DiffusionNFT (Zheng et al., 2026) — formulate their objectives based on denoising score matching loss, treating the rollout samples as direct regression targets. When these targets are corrupted by low-bit quantization noise, the high-precision policy is forced to mimic distorted, low-fidelity semantics. As the paper puts it in Section 3.2:

"the numerical noise forces the high-precision policy to imitate distorted, low-fidelity semantics. Consequently, this naive substitution inherently caps the achievable alignment quality of the model, finally neutralizing the benefits of rollout scaling."

This is visualized in Figure 3b, where directly integrating FP4 rollout into the RL pipeline leads to severe instability and performance degradation compared to the BF16 baseline. The degradation is not minor — the training curves diverge substantially, with the quantized variant failing to approach the BF16 pipeline's alignment ceiling.

The fundamental tension. In summary, prior approaches to quantized rollout face an inherent tradeoff: you can accelerate generation (by lowering precision), but you pay for it with degraded optimization (because the policy trains on corrupted targets). The paper's framing of this as an "efficiency-stability dilemma" makes explicit what prior work implicitly struggled with — the two goals (speed and quality) appear to be in direct conflict when quantization is naively applied to the entire RL pipeline.

Category 2: Diffusion RL Algorithms Without Quantization

On the algorithmic side, several recent advances have pushed the state of the art in diffusion RL without addressing the computational bottleneck:

• Flow-GRPO (Liu et al., 2025) and DanceGRPO (Xue et al., 2025) adapt the GRPO formulation from LLMs to diffusion models, demonstrating that group-relative advantages work effectively for image generation.
• DiffusionNFT (Zheng et al., 2026) provides a forward-process interpretation of GRPO for diffusion, improving sample efficiency.
• AWM (Xue et al., 2025) places forward-process optimization on firmer policy-optimization footing using the ELBO as a proxy for policy likelihood.
• BranchGRPO, TreeGRPO, and related work (Li et al., 2025; Ding et al., 2025; Ge et al., 2025) explore structured sampling strategies (trajectory branching, tree-structured advantages) to improve the efficiency of exploration within the GRPO framework.

These methods all share a common property: they improve the algorithmic efficiency of reinforcement learning (better sample utilization, better gradient signals), but they do not address the raw computational cost of generating the candidate pool itself. As rollout sizes scale from 24 to 96 or beyond, the generation phase dominates the training pipeline regardless of how cleverly the samples are subsequently used. DanceGRPO's selective training paradigm — using only contrastive samples — makes the algorithmic redundancy explicit but does not solve it: the waste is in generating and discarding samples, not in how they contribute to the gradient.

The paper positions itself as complementary to these algorithmic advances. Section 3.4 emphasizes that the decoupled framework "integrates the algorithmic mechanisms of rollout scaling and selective training with the system-level throughput gains of NVFP4." In other words, Sol-RL does not replace the algorithmic innovations — it builds on them, specifically leveraging the selective training insight (that only contrastive samples matter) to justify why exploring in low precision is algorithmically sound.

How This Paper Positions Itself

The paper's central thesis — articulated in the introduction and formalized in Section 3 — is that the efficiency-stability dilemma is a false dichotomy created by conflating two distinct phases of the RL pipeline: exploration (generating many candidates to find which are promising) and optimization (computing gradients from the selected candidates to update the policy).

The key observation that enables this decoupling is that FP4-quantized rollouts, while unsuitable as direct optimization targets, serve as reliable proxies for intra-group reward ranking. This is not a trivial claim — it requires empirical validation that the coarse semantic layout and structural outcome dictated by the initial noise seed is preserved under low-precision computation, even though pixel-level fidelity is lost. The paper devotes significant analysis to this claim in Section 3.3 and Appendix C, demonstrating that the NVFP4 proxy rewards maintain high correlation with BF16 ground truth (Spearman's ρ averaging 0.927, Top-4 match rate exceeding 96%) and that the ranking information concentrates in the extreme quantiles (top-k and bottom-k) where the selective training paradigm operates.

The two-stage architecture flows naturally from this observation:

1. Stage 1 (FP4 Exploration): Use high-throughput NVFP4 inference with reduced denoising steps (6 instead of 10) to rapidly generate and score a large candidate pool (96 samples per prompt). Extract only the most contrastive seeds (top-12 and bottom-12 by proxy reward).

2. Stage 2 (BF16 Regeneration): Regenerate only the selected 24 seeds in full BF16 precision with the standard number of denoising steps, producing high-fidelity training targets that the policy can safely optimize against.

This decoupling resolves the dilemma by structurally separating the concerns: speed comes from Stage 1 (where quantization is safe because we only need relative ranking), and quality comes from Stage 2 (where high precision is necessary because the policy learns from these samples as regression targets). The paper explicitly contrasts this with the naive approach where a single precision level must serve both purposes simultaneously, inevitably compromising one or the other.

The positioning relative to prior work is clear: this is not a new RL algorithm (the policy update follows DiffusionNFT's objective), a new quantization technique (NVFP4 uses standard NVIDIA Transformer Engine tooling), or a new reward model. It is a systems-level architecture innovation — a specific way of composing existing components (GRPO-style selective training, NVFP4 quantization, ODE-based diffusion sampling) that exploits their complementary properties. The contribution lies in identifying when quantization is safe (for exploration/ranking) and when it is dangerous (for optimization targets), and designing a pipeline that respects this boundary.

This framing also connects to the broader theme of hardware-algorithm co-design that has become increasingly important as foundation models push against hardware limits. The paper's use of "NVFP4" specifically (rather than generic FP4) signals awareness that the viability of 4-bit quantization depends on hardware support — NVIDIA's Blackwell architecture provides dense FP4 operations with up to 4× the TFLOPs of BF16 — and that the algorithm design is shaped by what the hardware can efficiently execute. The decoupled architecture is not just an algorithmic choice; it reflects an understanding that in modern ML systems, the cost model (relative throughput of FP4 vs. BF16 on specific hardware) fundamentally constrains what architectures are practical.

3. Technical Approach

3.1 Reader Orientation

Sol-RL is a two-stage training framework that accelerates reinforcement learning for text-to-image diffusion models by using low-precision (FP4) computation to quickly explore which image candidates are promising, then using high-precision (BF16) computation only for the small subset of candidates that actually matter for training. The fundamental problem it solves is the efficiency-stability dilemma: scaling up the number of candidate images generated per prompt dramatically improves alignment quality, but generating all those images in high precision creates a computational bottleneck that makes the approach impractical — yet naively switching to low-precision generation corrupts the training targets and degrades the final model quality.

3.2 Big-Picture Architecture (Diagram in Words)

The Sol-RL system has four major components connected in a two-stage pipeline:

1. FP4 Inference Engine — A quantized copy of the diffusion model compiled with NVIDIA Transformer Engine's NVFP4 backend. Its responsibility is high-throughput image generation: for each training prompt, it produces a large pool of candidate images (96) using aggressive acceleration (reduced denoising steps and 4-bit quantization) and scores them with a reward model to produce proxy reward estimates.

2. Seed Selector — Not a learned component but a filtering logic. Given the pool of 96 proxy-ranked candidates from Stage 1, it selects the top-K/2 (best) and bottom-K/2 (worst) noise seeds — the initial random vectors that produced the highest- and lowest-scoring images — and passes only these seeds to Stage 2. The key insight is that the seed (not the pixel-level execution) determines the coarse semantic structure of the image.

3. BF16 Regeneration Loop — The original high-precision diffusion model. For each of the K selected seeds (e.g., 24), it regenerates the image from scratch in full BF16 precision with the standard number of denoising steps, producing high-fidelity training targets that the policy can safely learn from.

4. Policy Update Module — Standard diffusion RL training using the DiffusionNFT objective. It computes gradients exclusively from the BF16-regenerated high-contrastive samples and updates the model weights. After each update, the weights are re-quantized into NVFP4 and synchronized to the Stage 1 inference engine (without recompilation), preparing for the next iteration.

Information flows sequentially: a prompt batch enters → Stage 1 generates 96 FP4-proxy images → reward model scores them → seed selector identifies top-12 and bottom-12 seeds → Stage 2 regenerates those 24 seeds as BF16 images → policy updates from the 24 BF16 samples → weights are re-quantized → cycle repeats.

3.3 Roadmap for the Deep Dive

• First, the rollout scaling bottleneck (Section 3.4.1) — why scaling the candidate pool creates a computational bottleneck, how selective training (using only contrastive samples) makes this redundancy explicit but does not solve it, and what the cost breakdown looks like in practice.

• Second, why naive quantized rollouts fail (Section 3.4.2) — the off-policy gap problem, why diffusion models are particularly vulnerable due to their continuous state space and regression-style objectives, and what specifically breaks when the policy trains on FP4-generated targets.

• Third, the proxy ranking property (Section 3.4.3) — the core empirical insight that enables the whole framework: FP4 rollouts, despite pixel-level degradation, preserve intra-group reward rankings well enough to identify contrastive samples, why ODE determinism and initial-noise dominance make this possible, and the quantitative evidence for ranking fidelity.

• Fourth, the two-stage decoupled architecture (Section 3.4.4) — the full Sol-RL pipeline in detail: how Stage 1 runs FP4 exploration at scale, how seeds are filtered, how Stage 2 regenerates in BF16, and how weight synchronization works across stages, including all hyperparameters.

• Fifth, the theoretical justification (Section 3.4.5) — the Extreme Value Theory analysis that proves why FP4 ranking works even at large scale: the quantization error is a bounded constant while the contrastive signal grows as $\sqrt{2\log N}$, ensuring that at sufficient rollout size, the signal dominates the noise.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems-level architecture paper whose core idea is that FP4-quantized inference is safe for exploration (relative reward ranking) but dangerous for optimization (regression targets), and that a two-stage pipeline that decouples these concerns can recover the throughput benefits of quantization without sacrificing training integrity.

3.4.1 The Rollout Scaling Bottleneck and Selective Training

The paper builds directly on the observation from prior work — particularly DanceGRPO (Xue et al., 2025) — that scaling the number of rollout samples per prompt consistently improves alignment performance. This is not a new finding, but the paper's contribution begins with characterizing why it creates a bottleneck and how severe that bottleneck becomes.

How GRPO uses rollouts. In Group Relative Policy Optimization (GRPO), introduced originally for LLMs in Shao et al. (2024) and adapted to diffusion models by Flow-GRPO and DanceGRPO, the policy generates a group of N independent candidate outputs for each prompt. Each candidate receives a reward score from a reward model (e.g., ImageReward, HPSv2). The advantage for each candidate is computed by standardizing its reward against the group's mean and standard deviation. The policy is then updated using a PPO-style clipped surrogate objective that encourages increasing the probability of candidates with positive advantage (better than average) and decreasing the probability of those with negative advantage (worse than average).

The key mathematical form is the advantage computation from Equation 3 in the paper. For a group of N candidates with rewards {$R(x^{(1)}), \ldots, R(x^{(N)})\}$:

$A_i = \frac{R(x^{(i)}) - \mu_R}{\sigma_R}$

where $\mu_R = \frac{1}{N} \sum_{j=1}^N R(x^{(j)})$ is the group mean reward and $\sigma_R = \sqrt{\frac{1}{N} \sum_{j=1}^N (R(x^{(j)}) - \mu_R)^2}$ is the group standard deviation.

What it computes: For each candidate in the group, this produces a scalar advantage that measures how much better (positive) or worse (negative) that candidate's reward is compared to the group average, normalized by the group's spread. A candidate with reward exactly at the mean gets advantage zero; a candidate one standard deviation above the mean gets advantage +1.

Why this form: The group-relative normalization eliminates the need for a learned value function (critic), which PPO requires to estimate baseline rewards. Instead, the group itself provides the baseline — the mean reward serves as an empirical estimate of expected reward under the current policy, and the standard deviation adaptively scales the advantage based on how much variance the reward model assigns to the group. Critically, this makes the quality of the policy update dependent on the group composition: larger N provides more stable estimates of $\mu_R$ and $\sigma_R$, and a wider reward range within the group creates stronger gradient signals for policy improvement.

The selective training insight. DanceGRPO introduced an additional refinement: rather than training on all N candidates, train only on the most contrastive subset — specifically, the top-k highest-reward candidates and the bottom-k lowest-reward candidates. The rationale (Section 3.1) is that "the most contrastive samples provide more reliable and informative learning signals for policy optimization, while other samples provide limited gradient due to the near-zero advantages." A candidate whose reward is close to the group mean receives an advantage near zero, which means its contribution to the policy gradient is negligible regardless of whether it is technically "correct" or "incorrect." Only the extreme candidates — those clearly better or clearly worse than average — provide meaningful training signal.

The bottleneck emerges. This selective training paradigm creates a computational asymmetry: you must generate the full N-candidate pool to find the extremes, but you only train on a fraction $K \ll N$. As the paper states in Section 3.1:

"the scaling of rollout shifts the computational bottleneck from policy optimization to candidate generation"

The quantitative evidence comes from the time breakdown data in Table 5 and Figure 3a. For SD3.5-Large under the 24-in-96 setting (N=96 candidates, K=24 selected), the naive BF16 rollout takes 451 seconds, while the total end-to-end iteration including gradient updates takes 691 seconds. That means the rollout phase alone consumes 65% of the total iteration time. For FLUX.1-dev, the numbers are 184 seconds for rollout out of 274 seconds total (67%). The "algorithmic redundancy" the paper refers to is stark: 72 of the 96 generated samples (75%) are generated in expensive BF16 precision and then immediately discarded because they provide near-zero advantage.

This cost breakdown establishes the quantitative motivation for the entire framework. The goal is not to speed up an arbitrary part of the pipeline — it is to specifically target the rollout generation phase, which has become the primary cost and which contains substantial redundancy that the selective training paradigm makes explicit but does not eliminate.

3.4.2 Why Naive Quantized Rollouts Fail as Direct Training Targets

The natural solution to the cost problem is to quantize the model during rollout generation — run the diffusion model in FP4 instead of BF16, exploiting the ~4x throughput advantage of FP4 hardware. However, the paper presents detailed evidence that this naive approach catastrophically degrades alignment performance.

The off-policy gap. The first problem is structural to reinforcement learning. The policy gradient theorem depends on the expectation being taken over trajectories sampled from the current policy $\pi_\theta$. When you quantize the policy to low precision and use it to generate rollouts, you are effectively sampling from a different distribution — call it $\tilde{\pi}_\theta$, the quantized approximation. The gradient estimator becomes:

$\nabla_\theta J(\theta) \approx \mathbb{E}_{x \sim \tilde{\pi}_\theta(\cdot|c)}\left[\nabla_\theta \log \pi_\theta(x | c) A(x, c)\right]$

where the expectation is taken under $\tilde{\pi}_\theta$ but the log-probability is evaluated under $\pi_\theta$. This is an off-policy estimator, and off-policy RL is known to be significantly less stable than on-policy methods. The importance ratio between the two distributions is not accounted for (unlike in methods like FP8-RL, which explicitly compute and use it), leading to biased gradient estimates. The paper cites concurrent work (Yao et al., 2025; Xi et al., 2026) that documents these "severe numerical discrepancies" and training instabilities.

The continuous state space problem (diffusion-specific). For diffusion models, the issue is compounded by a mechanism that does not exist in discrete-token LLMs. The paper explains in Section 3.2 that mainstream forward-process diffusion RL algorithms — specifically Advantage Weighted Matching (AWM) and DiffusionNFT — formulate their training objectives using denoising score matching loss. In these formulations, the rollout sample $x_0$ (the final generated image) serves as a direct regression target for the denoising process.

The training procedure works as follows. During the forward diffusion process, clean images $x_0$ are corrupted by adding Gaussian noise at various timesteps $t$ to produce noisy latents $x_t$. The policy $\pi_\theta$ (the diffusion model's denoiser) is trained to predict the noise component or the clean image $x_0$ given $x_t$ and the timestep $t$. The loss is typically a mean squared error between the prediction and the ground truth. When $x_0$ is a high-precision BF16 image, the regression target is clean; when $x_0$ is a quantized FP4 image, the regression target contains quantization noise.

The paper states this directly in Section 3.2:

"when corrupted by low-bit quantization (e.g., FP4), the numerical noise forces the high-precision policy to mimic distorted, low-fidelity semantics"

The consequence is that the policy learns to reproduce the quantization artifacts — it converges to a solution that generates images with FP4-level quality, defeating the purpose of alignment training. The policy is optimizing its parameters to match targets that are systematically degraded by the precision format, not just randomly noisy. This is worse than simple label noise because the degradation is deterministic given the quantized forward pass; the policy learns a consistent bias toward lower-fidelity outputs.

Empirical evidence. Figure 3b provides the direct experimental confirmation. When the paper runs the standard Diffusion RL pipeline with FP4-generated rollouts as training targets, the training curve shows "severe instability and performance degradation compared to the BF16 baseline." The figure shows the HPSv2 evaluation score over training steps: the BF16 baseline climb steadily, while the naive FP4 variant diverges downward. The paper does not provide exact numerical values for the divergence in the main text, but the visual evidence in Figure 3b makes clear that naive quantization is not a viable option — the degradation is not a small constant offset but a failure to converge to the same alignment quality.

Why this matters for the two-stage design. This empirical result is the critical negative finding that motivates the decoupled architecture. If naive quantization worked (even with some small acceptable degradation), there would be no need for a two-stage pipeline — you could simply quantize the entire rollout phase and train directly on the results. The paper's identification that this fails is what justifies the extra complexity of regeneration: the FP4-generated images are too degraded to serve as training targets, so they must be regenerated in high precision before the policy can safely learn from them. The question then becomes: can FP4-generated images serve some other useful purpose that doesn't require pixel-level fidelity?

3.4.3 The Proxy Ranking Property: Why FP4 Rollouts Preserve Relative Reward Ordering

The paper's central empirical insight is that while FP4 rollouts fail as direct optimization targets, they succeed as proxy reward rankers — that is, they can reliably identify which noise seeds will produce high-reward images and which will produce low-reward images when regenerated in BF16. This insight is what enables the decoupled architecture, and the paper devotes significant evidence to validating it.

The ODE determinism argument. Modern diffusion models use ODE (ordinary differential equation) solvers for sampling. Under an ODE formulation, the mapping from initial noise $z$ to final image $x_0$ is deterministic: given the same initial noise vector, the same model parameters, and the same solver configuration (step size, order), you always get the same output image. Stochasticity enters only through the initial noise — all subsequent steps in the ODE solve are deterministic transformations.

The paper's key observation is that this deterministic mapping is structurally robust to precision changes. The coarse semantic layout of the image — the spatial arrangement of objects, the large-scale composition, the dominant colors, the presence or absence of prompt-relevant features — is determined by the initial noise seed $z$, not by the numerical precision of the ODE solver steps that follow. As long as the FP4 solver does not catastrophically diverge from the BF16 solver's trajectory, the high-level semantic structure of the image is preserved.

Figure 6 provides the visual evidence. The paper shows pairs of images generated from the same seed in NVFP4 and BF16. The qualitative observation is that "despite minor localized deviations, the NVFP4 quantized rollouts maintain the overall semantic layout and structure." In a specific example, both images show the same subject in the same pose in the same general scene configuration — but the FP4 version might have slightly different texture details, edge sharpness, or color saturation. The differences are in rendering quality, not in semantic content.

Table 4 provides quantitative confirmation. Across three base models (FLUX.1, SANA, SD3.5-Large), the Inception Score (IS) and CLIPScore of NVFP4 generations are nearly identical to the BF16 baseline. For FLUX.1, IS is 16.84 (BF16) vs. 17.85 (NVFP4); CLIPScore is 27.44 (BF16) vs. 27.10 (NVFP4). These are within normal evaluation variance, confirming that the semantic information content — at least as measured by these metrics — is preserved under quantization.

Why semantic preservation matters for ranking. Reward models like ImageReward, HPSv2, CLIPScore, and PickScore evaluate images based primarily on semantic alignment: does the image contain the objects described in the prompt? Are they in the right spatial relationships? Does the lighting, composition, and overall style match what is expected? These models are not designed to be sensitive to subtle rendering artifacts — they operate on features extracted by pretrained vision encoders (CLIP, BLIP) that are themselves robust to minor pixel-level variations.

Because the reward model's judgment is dominated by semantic content, and semantic content is preserved under FP4 quantization (as Table 4 and Figure 6 demonstrate), the proxy reward $\tilde{R}$ computed from an FP4-generated image should be highly correlated with the true reward $R$ that would be computed from a BF16-regenerated image generated from the same seed. The paper formalizes this in the Lipschitz continuity argument in Appendix A (Section 3.4.5), but the key empirical validation is in Section 3.3 and Appendix C.

The ranking fidelity evidence (Figure 3c and Table 8). The paper presents two forms of evidence that FP4 proxy rewards are reliable for ranking.

First, Figure 3c shows a conditional probability density map comparing BF16 true rank percentiles against NVFP4 proxy rank percentiles. For a given sample whose true rank percentile is $x$ under BF16 rewards, the vertical slice at $x$ shows the probability distribution of its NVFP4 proxy rank percentile. The density is heavily concentrated along the diagonal — meaning that for most samples, the FP4 rank is close to the true BF16 rank. Critically, the concentration is even stronger in the extreme regions (the top-k and bottom-k quadrants), which are exactly the regions that matter for selective training. This means the proxy ranking is most reliable precisely where we need it to be.

Second, Appendix Table 8 provides quantitative correlation metrics. Across four reward models, the paper reports:

• Kendall's τ (tau): A rank correlation coefficient measuring ordinal association based on concordant vs. discordant pairs. The overall average is 0.798, with individual models ranging from 0.752 (CLIPScore) to 0.827 (HPSv2). A τ above 0.70 is generally considered to indicate highly consistent pairwise orderings.

• Spearman's ρ (rho): A rank correlation coefficient measuring monotonic relationship strength. The overall average is 0.927, with individual models ranging from 0.900 (CLIPScore) to 0.943 (HPSv2). A ρ above 0.80 is widely regarded as a very strong positive correlation.

• Top/Bottom-k Match rates: The exact intersection rate of the highest and lowest k items selected under BF16 vs. NVFP4. For k=4, the top match rate averages 96.9% and the bottom false-inclusion rate averages 3.9%. For k=12 (the actual setting used in Sol-RL), the top match rate is 93.3% and the bottom false-inclusion rate is 7.5%.

Interpreting these numbers. The 93.3% Top-12 match rate means that of the 12 seeds that are truly the best under BF16, the FP4 proxy correctly identifies approximately 11.2 of them on average. The 7.5% Bottom-12 false-inclusion rate means that of the 12 seeds selected as "worst" by the FP4 proxy, approximately 0.9 of them are not actually among the true bottom-12 — they are false negatives. These error rates are low enough that the selected subset preserves the contrastive signal needed by GRPO. More importantly, the overall ranking consistency (ρ = 0.927) means that even if the exact boundaries of the top-12 and bottom-12 sets are slightly fuzzy, the seeds selected by FP4 are genuinely from the extremes of the reward distribution, not from the middle.

Why this is sufficient for GRPO. The GRPO objective relies on having a group of candidates with a meaningful spread of rewards — enough to compute a reliable mean and standard deviation for advantage normalization. The exact identity of every candidate in the top-k and bottom-k sets matters less than ensuring that the selected candidates genuinely represent high-reward and low-reward regions of the distribution. As long as the ranking error rate is modest (less than ~10%), the contrastive signal is preserved: the high-reward candidates still have positive advantages and the low-reward candidates still have negative advantages. The absolute reward values will be recomputed from the BF16-regenerated images anyway, so any bias in the FP4 proxy rewards (e.g., systematic underestimation or overestimation) is completely eliminated in Stage 2.

3.4.4 The Two-Stage Decoupled Architecture in Detail

The Sol-RL framework instantiates the decoupling principle as a concrete training pipeline with specific configurations and hyperparameters. The architecture is designed around NVIDIA's NVFP4 format, which encodes values using 1 sign bit, 2 exponent bits, and 1 mantissa bit, with block-level micro-scaling where 16 elements share a single E4M3 scaling factor. This format achieves approximately 4× the TFLOPs of BF16 on Blackwell-architecture GPUs (B200) while maintaining sufficient numerical fidelity for semantic-level operations.

Stage 1: Accelerated Exploration at Scale via FP4.

The exploration stage operates as follows, with all hyperparameters specified from Table 7:

1. Policy weight quantization. The current policy weights (in BF16) are quantized into NVFP4 format using the NVIDIA Transformer Engine. This produces a quantized inference model that is pre-compiled (JIT-compiled) to avoid per-iteration compilation overhead. The quantization uses the standard block-level scaling scheme: for each group of 16 contiguous elements in the weight tensor, a shared E4M3 scale factor is computed, and each element is projected to the nearest representable FP4 value. The quantization formula is:

$\tilde{x} = Q(x) = S \cdot \Pi_{\text{FP4}}\left(\frac{x}{S}\right)$

where $S$ is the shared scaling factor for the block and $\Pi_{\text{FP4}}$ is the projection function that rounds to the nearest FP4-representable value.

What it computes: This converts each block of 16 high-precision weights into 16 low-precision weights that share a common exponent range. The scale factor $S$ captures the overall magnitude of the block, while the FP4 values capture fine-grained relative differences within that range.

Why this form: The block-level scaling is what makes FP4 viable. Without shared scales, FP4's extremely limited dynamic range (4 exponent values, 2 mantissa values per sign) would make it impossible to represent the wide range of magnitudes that appear in neural network weights. By grouping elements that tend to have similar magnitudes (spatially adjacent weights often do) under a common scale, the format effectively extends its dynamic range while preserving relative precision within each block.

2. Batch-level noise sampling. For each training prompt in the current batch, the system samples $N = 96$ independent initial noise vectors $\{z^{(i)}\}_{i=1}^N$ from the standard Gaussian distribution. The paper uses 48 prompts per epoch across 8 GPUs (Table 7), meaning each GPU processes 6 prompts per iteration and generates 96 × 6 = 576 FP4 images.

3. FP4 accelerated generation. Each noise vector $z^{(i)}$ is fed into the NVFP4-compiled model with an ODE solver configured for significantly fewer denoising steps than the standard rollout. The paper uses $T = 6$ exploration steps (compared to $T = 10$ for the BF16 regeneration stage and for the naive BF16 baseline). The ODE solver is the same variant used for the full-precision rollout — Euler for flow-based models (SANA), DPM-Solver-2 for diffusion-based models (FLUX.1, SD3.5-Large) — but executed in NVFP4 arithmetic. This produces $N$ candidate images $\{\tilde{x}_0^{(i)}\}_{i=1}^N$ in FP4 precision.

The choice of 6 steps (rather than, say, 2 or 8) is validated in the ablation in Table 2. Using 4 steps gives suboptimal alignment scores (HPSv2 = 0.3650 vs. 0.3686 at 6 steps) because "the coarse semantic layouts are insufficiently formed, leading to inaccurate intra-group ranking." Using 8 steps provides no further improvement (0.3659), indicating the ranking capability saturates at 6 steps. This saturation point is important for efficiency: 6 steps strikes the right balance between generation quality (enough to form recognizable semantics) and speed (fewer steps than the standard 10).

4. Proxy reward computation. Each FP4-generated image $\tilde{x}_0^{(i)}$ is scored by the reward model $R(\cdot)$ — which runs in standard precision, since reward model inference is cheap relative to diffusion generation — producing proxy rewards $\{\tilde{R}_i\}_{i=1}^N$. The reward model is not quantized; only the diffusion model is.

5. Seed filtering. The system ranks the $N$ candidates by their proxy rewards and selects the top-$K/2$ and bottom-$K/2$ seeds, where $K = 24$. Specifically, the 12 seeds with the highest proxy rewards and the 12 seeds with the lowest proxy rewards are retained; the remaining 72 seeds are discarded. Only the seed vectors $z^{(i)}$ themselves (the initial noise) are preserved — the FP4 images are discarded after ranking because they are too degraded to use as training targets.

The selection of K=24 (out of N=96) is a design choice that balances several factors. Larger K means more training samples, which provides more gradient signal and reduces variance — but it also increases Stage 2 regeneration cost and may include samples with near-zero advantages that contribute little to training. Smaller K means cheaper Stage 2 but risks missing important contrastive information. The paper does not provide a direct ablation of K vs. N, but Table 3 shows that holding K=24 fixed and scaling N from 24 to 96 consistently improves alignment (HPSv2 increases from 0.3569 to 0.3686), confirming that the primary benefit comes from the broader exploration, not from training on more samples.

Stage 2: High-Fidelity Regeneration and Policy Update.

The regeneration stage operates as follows:

1. Seed-driven BF16 regeneration. For each of the $K = 24$ selected seeds, the system regenerates the image from scratch using the unquantized BF16 policy model with the full inference step budget of $T = 10$ denoising steps. The same ODE solver configuration is used (Euler or DPM-Solver-2 depending on the model), and the same deterministic schedule — but now executed entirely in BF16 arithmetic without any quantization. This produces $K$ high-fidelity images $\{x_0^{(i)}\}_{i=1}^K$.

Critically, the regeneration uses the standard 10-step schedule, not the reduced 6-step schedule from Stage 1. This is because Stage 2 images are training targets: they need to be at the quality level that the policy is actually capable of producing in BF16. Using fewer steps would produce lower-quality targets that the policy would then learn to reproduce, defeating the purpose of alignment training. The 6-step FP4 exploration was a proxy — good enough for ranking — but the 10-step BF16 regeneration is the ground truth that the policy should optimize toward.

2. Reward computation and advantage normalization. Each BF16-regenerated image is scored by the reward model to obtain true rewards $\{R_i\}_{i=1}^K$. These are the rewards used for GRPO advantage computation (Equation 3), not the proxy rewards from Stage 1. The group mean and standard deviation are computed over the K selected candidates, and advantages are assigned accordingly.

This is where the decoupling pays off: the advantages are computed from high-fidelity rewards on BF16 images, so they reflect the true quality of the policy's outputs under optimal generation conditions. The FP4 proxy rewards were used only to select which seeds to regenerate, not to compute the optimization signal.

3. Policy update. The policy is updated using the DiffusionNFT objective (Equation 4), which combines a clipped surrogate loss with a KL-divergence penalty toward a reference model:

$\mathcal{L}_{\text{GRPO}}(\theta) = \mathbb{E}_{\pi_{\text{old}}}\left[\frac{1}{K} \sum_{i=1}^K \left(\min\left(r_i(\theta) A_i, \text{clip}(r_i(\theta), 1-\epsilon, 1+\epsilon) A_i\right) - \beta D_{\text{KL}}(\pi_\theta(\cdot|c) \| \pi_{\text{ref}}(\cdot|c))\right)\right]$

where $r_i(\theta) = \frac{\pi_\theta(x^{(i)}|c)}{\pi_{\text{old}}(x^{(i)}|c)}$ is the probability ratio of the updated policy to the old policy for candidate $i$, $A_i$ is the group-relative advantage, $\epsilon$ is the clipping parameter (typically 0.2 in PPO, though the paper does not specify the exact value), and $\beta$ is the KL penalty coefficient.

What it computes: This objective performs a constrained policy improvement step. For each candidate $i$, the policy is encouraged to increase $\pi_\theta(x^{(i)}|c)$ if $A_i > 0$ (good candidate) and decrease it if $A_i < 0$ (bad candidate). The clipping $\min(r_i A_i, \text{clip}(r_i, 1-\epsilon, 1+\epsilon)A_i)$ prevents the policy from changing too much in a single update — if the probability ratio $r_i$ would exceed $1+\epsilon$ for a positive-advantage sample, the gradient is clipped. The KL penalty $\beta D_{\text{KL}}$ further regularizes the policy toward the reference model, preventing catastrophic drift.

Why this form: The PPO-style clipped objective is the standard approach for stable policy optimization in RL. The clipping prevents destructively large policy updates that could cause the model to collapse. The KL penalty toward a reference model (typically the pretrained base model or an EMA of the policy) is especially important for diffusion models, where aggressive fine-tuning can quickly degrade generation quality outside the alignment objective. The DiffusionNFT adaptation makes this objective compatible with forward-process diffusion training, where the likelihood $\pi_\theta(x|c)$ is approximated through the denoising score matching loss rather than computed directly.

4. Weight re-quantization. After the gradient update produces new policy weights in BF16, these weights are re-quantized into NVFP4 format using the same block-level scaling procedure as in Step 1, and copied in-place into the pre-compiled NVFP4 inference engine. The paper emphasizes that this is done "without recompilation" — the JIT-compiled inference graph is reused across iterations, and only the quantized weight values are updated. The re-quantization overhead is described as "merely a 2% computational overhead" in Figure 2.

Training configuration details. The paper uses LoRA (Low-Rank Adaptation) with rank $r = 32$ and scaling factor $\alpha = 64$ across all models, applied to specific attention projection layers (to_{q,k,v,out} for SANA and FLUX.1, additional attn.{to,add}_{q,k,v,out} for SD3.5-Large, as specified in Table 7). The optimizer is AdamW with learning rate $3 \times 10^{-4}$, betas $(0.9, 0.999)$, weight decay $1 \times 10^{-4}$, and epsilon $1 \times 10^{-8}$. Mixed precision training is BF16 throughout policy optimization. KL penalty coefficient $\beta_{\text{kl}} = 1 \times 10^{-4}$, advantage clipping at 5, and an EMA decay of 0.9 with a linear ramp (rate 0.001, cap 0.5) for the old model used in probability ratio computation.

Per-model differences. The models use different image resolutions (SANA and SD3.5 at 1024×1024, FLUX.1 at 512×512), different ODE solvers (Euler for SANA's flow matching, DPM-Solver-2 for FLUX.1 and SD3.5's diffusion), and different per-GPU micro-batch sizes to fit in GPU memory (16 for SANA, 12 for FLUX.1, 4 for SD3.5-Large). FLUX.1 uses classifier-free guidance with a guidance embedding of 1.0, while SANA and SD3.5 disable CFG. These differences reflect the practical engineering of fitting large models on B200 GPUs rather than algorithmic choices.

The efficiency gain mechanism. The speedup comes from three compounding effects:

1. FP4 throughput advantage: NVFP4 dense operations run at approximately 4× the TFLOPs of BF16 on B200 hardware. Stage 1 generates 96 images in FP4 with this throughput advantage.

2. Reduced denoising steps: Stage 1 uses 6 steps instead of 10 — a 1.67× reduction in per-image computation.

3. Selective regeneration: Only 24 of 96 seeds (25%) are regenerated in Stage 2. The remaining 75% of seeds are explored cheaply in FP4 and discarded without ever incurring BF16 generation cost.

The paper quantifies the net effect in Table 5. For SD3.5-Large, the naive BF16 approach (96 samples, 10 steps each) takes 451 seconds for pure rollout. Sol-RL's two-stage approach (96 FP4 samples at 6 steps + 24 BF16 samples at 10 steps) takes 187 seconds — a 2.41× speedup in rollout time. The end-to-end iteration speedup is 1.62× (691s vs. 427s), because the policy update phase (which is identical in both approaches) is not accelerated.

The speedup is model-dependent. SANA shows only a 1.41× rollout speedup because SANA is a smaller, more efficient model (1.6B parameters) where the FP4 advantage is less pronounced relative to other overhead. FLUX.1 (12B parameters) and SD3.5-Large (8B+ parameters) show larger speedups (2.33× and 2.41× respectively) because their larger model sizes make the throughput advantage of FP4 more impactful.

3.4.5 Theoretical Justification: Why FP4 Ranking Works at Scale

Appendix A provides a theoretical analysis that explains why the proxy ranking is reliable and, critically, why it becomes more reliable as the rollout pool scales up. This is not just empirical cherry-picking — it is a mathematical argument that the decoupling strategy is structurally sound.

Step 1: Bounding the per-sample reward error.

The analysis begins with the ODE formulation of diffusion sampling. Let the high-precision (BF16) trajectory satisfy the ODE:

$\dot{x}_t = v_\theta(x_t, t)$

where $v_\theta$ is the vector field (the diffusion model's denoising prediction) parameterized by $\theta$, $x_t$ is the latent state at time $t$, and the dot denotes the time derivative.

The low-precision (FP4) trajectory satisfies a perturbed ODE:

$\dot{\tilde{x}}_t = v_\theta(\tilde{x}_t, t) + e_t$

where $e_t$ denotes the effective perturbation induced by FP4 rounding errors and low-precision solver arithmetic at time $t$. This perturbation captures both the direct quantization error in evaluating $v_\theta(\tilde{x}_t, t)$ in FP4 and any accumulation of errors from the solver's numerical integration in reduced precision.

Assumption 1 (Lipschitz vector field): The vector field $v_\theta(\cdot, t)$ is $L_v$-Lipschitz continuous with respect to $x$. This means that small changes in the latent state produce bounded changes in the predicted velocity: $\|v_\theta(x, t) - v_\theta(y, t)\| \leq L_v \|x - y\|$. This is a standard assumption for neural ODEs and holds in practice for well-trained diffusion models.

Under this assumption, the final sample deviation can be bounded using Grönwall's inequality (a standard ODE comparison tool):

$\|x_0 - \tilde{x}_0\| \leq e^{L_v T} \int_0^T \|e_s\| ds$

where $x_0$ is the BF16 final image, $\tilde{x}_0$ is the FP4 final image, $T$ is the total integration time, and $e_s$ is the instantaneous perturbation at integration time $s$.

What it computes: This inequality provides an upper bound on how far the FP4 image can deviate from the BF16 image, given the same initial noise seed. The bound has two factors: an exponential term $e^{L_v T}$ representing error amplification through the ODE dynamics (how much a small perturbation early in the trajectory gets magnified by the end), and the cumulative perturbation $\int_0^T \|e_s\| ds$ representing the total FP4-induced noise integrated over the sampling process.

Why this form: The exponential factor is inherent to the ODE dynamics — it cannot be eliminated by better quantization because it comes from the Lipschitz constant of the model itself. The cumulative perturbation term is what quantization controls: better precision formats (more bits) reduce $\|e_s\|$ at each step, while more integration steps reduce the per-step error at the cost of more total steps.

Assumption 2 (Lipschitz reward model): The reward model $R(x)$ is $L_R$-Lipschitz continuous. This means that small changes in the image produce bounded changes in the reward: $|R(x) - R(y)| \leq L_R \|x - y\|$. This holds for neural network-based reward models (CLIP, BLIP-based scorers) because they are smooth functions of their inputs.

Combining the two Lipschitz assumptions:

$|R(x_0) - R(\tilde{x}_0)| \leq L_R \|x_0 - \tilde{x}_0\| \leq L_R e^{L_v T} \int_0^T \|e_s\| ds =: \Delta$

where $\Delta$ is defined as the maximum per-sample reward discrepancy between FP4 and BF16 evaluation of the same seed.

What it computes: This gives a deterministic worst-case bound on how much the FP4 proxy reward can differ from the true BF16 reward for any fixed noise seed. The bound $\Delta$ depends only on the numerical precision format (through $\|e_s\|$), the model's Lipschitz constants ($L_v$, $L_R$), and the integration time $T$. Crucially, it does not depend on the number of candidates $N$ or which specific seed is used.

Why this form: The key property is that $\Delta$ is a static constant — it does not grow with N. No matter how many candidates you evaluate, each individual FP4-BF16 reward discrepancy is bounded by the same $\Delta$. This means the ranking error is a per-sample phenomenon, not a systemic bias that would accumulate across a large pool.

Step 2: The extreme value argument (why scaling helps).

The analysis then shifts to the range of rewards within a group — the difference between the maximum and minimum rewards, which directly determines the strength of the GRPO contrastive signal.

Assumption 3 (Sub-Gaussian rewards): The true BF16 rewards of candidates generated from the same prompt are modeled as independent draws from a sub-Gaussian distribution, approximated as $R \sim \mathcal{N}(\mu, \sigma^2)$. This assumes that prompt conditioning induces a distribution of possible image qualities (some noise seeds produce better outputs than others), with mean $\mu$ and variance $\sigma^2$.

For a pool of $N$ independent candidates, the expected range between the maximum and minimum rewards follows extreme value scaling:

$\mathbb{E}[W_N^*] = \mathbb{E}[R_{\max}^*] - \mathbb{E}[R_{\min}^*] \approx 2\sigma \sqrt{2 \log N}$

where $R_{\max}^* = \max_i R_i$ is the true maximum reward, $R_{\min}^* = \min_i R_i$ is the true minimum reward, and $W_N^*$ is the true reward range.

What it computes: This is the classical result from extreme value theory for Gaussian samples: the expected maximum of $N$ i.i.d. standard normals grows as $\sqrt{2 \log N}$, and the expected minimum is symmetric (approximately $-\sqrt{2 \log N}$), so the range grows as $2\sqrt{2 \log N}$. Scaling by the reward standard deviation $\sigma$ gives the reward-space range.

Why this form: The $\sqrt{\log N}$ growth rate is slow (logarithmic), but it is unbounded — as $N \to \infty$, the range $W_N^* \to \infty$ (albeit slowly). This means that scaling the candidate pool inevitably discovers more extreme reward values. The practical implication is that moving from N=24 to N=96 increases $\sqrt{2\log N}$ from approximately $\sqrt{2 \cdot 3.18} \approx 2.52$ to $\sqrt{2 \cdot 4.56} \approx 3.02$ — a 20% increase in the expected extreme spread.

Step 3: Bounding the retained range under proxy selection.

The final step analyzes what happens when we use FP4 proxy rewards to select the extreme candidates. Let $\tilde{R}_i = R_i + \epsilon_i$ be the proxy reward for candidate $i$, where $\epsilon_i$ is the per-sample error bounded by $|\epsilon_i| \leq \Delta$. The system selects the empirical best candidate $\hat{i}_{\max} = \arg\max_i \tilde{R}_i$ and empirical worst candidate $\hat{i}_{\min} = \arg\min_i \tilde{R}_i$.

The true reward $R_{\hat{i}_{\max}}$ of the empirically selected best candidate is at most $\Delta$ worse than the true maximum $R_{\max}^*$, because the proxy ranking could have missed the true best candidate by at most the error bound. Similarly, the true reward $R_{\hat{i}_{\min}}$ of the empirically selected worst candidate is at most $\Delta$ better than the true minimum $R_{\min}^*$. Therefore, the retained range $\hat{W} = R_{\hat{i}_{\max}} - R_{\hat{i}_{\min}}$ satisfies:

$\hat{W} \geq (R_{\max}^* - 2\Delta) - (R_{\min}^* + 2\Delta) = W_N^* - 4\Delta$

What it computes: This lower bound says that the true reward range retained after FP4 proxy selection is at worst the oracle range (if we had perfect BF16 ranking) minus a constant penalty of 4Δ. The factor of 4 comes from the worst case: the empirical best could be Δ below the true best, and the empirical worst could be Δ above the true worst, each contributing 2Δ of range loss (since range is the difference between the two).

Why this form: The key structural insight is that the penalty is additive and constant, while the oracle range $W_N^*$ grows with $N$. Taking the expectation:

$\mathbb{E}[\hat{W}] \geq 2\sigma \sqrt{2 \log N} - 4\Delta$

As $N$ increases, the first term grows unboundedly while the second term stays fixed. Therefore, for sufficiently large $N$, the retained range becomes arbitrarily close to the oracle range in relative terms. The paper states this explicitly:

"As we aggressively scale up the rollout group, the extreme contrastive bounds of the distribution inevitably overpower the constant quantization noise, preserving the critical gradient signals required to unlock oracle alignment."

What this means operationally. The theoretical analysis justifies a design principle: scale $N$ large enough that $2\sigma\sqrt{2\log N} \gg 4\Delta$. At N=96, the paper's empirical results confirm this condition is met — the proxy ranking is reliable enough that the retained range is close to the oracle range. If the theory were wrong or if Δ were too large (e.g., with even more aggressive quantization or fewer denoising steps), the ranking would break down and the two-stage pipeline would fail. The ablation in Table 2 (where T=4 steps degrades to HPSv2=0.3650 vs. 0.3686) confirms that there is a threshold below which the proxy is not reliable enough.

Connection to the architecture. The theoretical analysis directly motivates the decoupled design. The constant penalty $4\Delta$ comes from using FP4 for ranking, but the formula $\mathbb{E}[\hat{W}] \geq 2\sigma\sqrt{2\log N} - 4\Delta$ shows that the reward range is only penalized by a constant offset. After the seeds are selected, the BF16 regeneration stage recomputes the actual rewards, which means the rewards used for optimization have no constant penalty at all — they are the true BF16 rewards of the selected seeds. The only penalty is the selection error (the chance that a suboptimal seed was selected instead of a better one), which the theory shows becomes negligible at large N. The architecture thus exploits the asymptotic property: use FP4 to scale N cheaply, and rely on the extreme value growth to make the selection error irrelevant.

This completes the technical architecture. The remaining sections of the paper (experiments, analysis, limitations) build on this foundation, but the core contribution — the two-stage decoupled design with its theoretical and empirical justification — is fully contained in Section 3.

4. Key Insights and Innovations

Innovation 1: The Efficiency-Stability Dilemma Is a Structural Property of Diffusion RL Pipelines, Not a Quantization Bug

The paper's most fundamental intellectual move is not proposing a solution but diagnosing the problem with a precision that reframes the entire design space. Prior work on quantized RL (FlashRL, QeRL, QuRL, Jet-RL) treated the degradation from low-precision rollouts as an off-policy variance issue that could be mitigated through importance weighting, clipping, or unified precision. The paper demonstrates that for diffusion models specifically, the problem is far more fundamental: it is structural, not statistical.

The diagnosis pivots on a distinction that does not exist in LLM-based RL. In discrete-token generation, the policy gradient depends on the log-probability of sampled tokens under the policy distribution — a scalar that is relatively robust to modest numerical perturbation. But in forward-process diffusion RL (AWM, DiffusionNFT), the training objective is a denoising score matching loss where the rollout samples serve as direct regression targets. The policy is trained to predict clean images from noisy latents, and when those clean-image targets are corrupted by FP4 quantization artifacts, the policy is forced — through gradient descent — to reproduce the corruption. This is not a variance problem that better advantage estimation can fix; it is a systematic bias baked into the loss function itself.

What makes this diagnosis distinctive is that it identifies the continuous state space as the root cause, not an exacerbating factor. The paper's key phrase from Section 3.2 — "the numerical noise forces the high-precision policy to mimic distorted, low-fidelity semantics" — captures the mechanism precisely. The policy doesn't just receive noisy reward signals (which importance weighting could correct for); it receives corrupted training targets, and regression to corrupted targets is fundamentally a different failure mode than off-policy variance. Figure 3b provides the empirical signature: the FP4-naive training curve diverges systematically downward rather than oscillating with high variance. This downward divergence (convergence to a lower alignment ceiling) is the hallmark of bias-dominated failure, not variance-dominated failure.

The significance beyond Sol-RL is that this diagnosis establishes a categorical constraint on any future approach to quantized diffusion RL: you cannot fix this by making the RL algorithm more robust to distribution shift, because the problem isn't the RL algorithm — it's the regression target. This explains why Jet-RL's unified-precision approach (FP8 everywhere) works in LLM RL but would not solve the fundamental issue in diffusion RL (the policy would simply learn the FP8-quality target). It also explains why the paper's own decoupled approach works where prior quantization strategies failed: it doesn't try to fix the regression target problem — it avoids it entirely by never allowing quantized samples to become regression targets in the first place.

This diagnosis is a fundamental conceptual contribution, not an incremental refinement of quantization methods. It reframes the efficiency-stability dilemma from "how do we stabilize off-policy RL under quantization?" to "how do we separate the phase where quantization is safe (exploration) from the phase where it is fatal (optimization)?" The rest of the paper's architecture follows deductively from this reframing.

Innovation 2: The Proxy Ranking Property Identifies a Precision-Asymmetric Information Channel That Enables Decoupling

If Innovation 1 diagnoses why naive quantization fails, Innovation 2 identifies what still works — and this is the conceptual discovery that makes Sol-RL possible. The paper's core empirical finding is that FP4-quantized rollouts, while unsuitable as pixel-level regression targets, preserve intra-group reward ranking with high fidelity. This is not an obvious property, and the paper's careful characterization of it is what distinguishes Sol-RL from prior work that simply abandoned quantization as too lossy.

What makes this insight novel is that it identifies a precision-asymmetric information channel: the relative ordering of candidates (which noise seeds produce better images) survives quantization, even though the absolute quality of the images (their pixel values, textures, fine details) does not. This asymmetry is not a general property of quantization — it is specific to the interaction between ODE-based diffusion sampling and the reward models used for alignment. The paper's analysis reveals why: the coarse semantic layout of an image (object presence, spatial composition, color palette) is dictated by the initial noise seed and the deterministic ODE trajectory, and this trajectory is structurally robust to precision perturbations because the Lipschitz continuity of the vector field bounds error accumulation (Appendix A). Reward models like ImageReward and HPSv2, which operate on features from pretrained vision encoders (CLIP, BLIP), are primarily sensitive to these coarse semantic features and largely invariant to the pixel-level artifacts that quantization introduces.

The quantitative evidence in Section 3.3 and Appendix C is what elevates this from a plausible hypothesis to a reliable design principle. Figure 3c's density map shows that the FP4-BF16 rank correlation is not just high on average — it is specifically concentrated in the extremes, with the top-k and bottom-k quadrants showing the strongest diagonal alignment. This matters because selective training operates exclusively on extremes. The Top-12 match rate of 93.3% (Table 8) means the FP4 proxy correctly identifies over 11 of the 12 best seeds on average, and the Bottom-12 false inclusion rate of 7.5% means fewer than 1 of the 12 selected "worst" seeds is actually a false negative. These error rates are low enough to preserve the contrastive advantage signal that GRPO depends on.

The prior-work contrast is illuminating. Methods like FP8-RL (Qiu et al., 2026) and QuRL (Li et al., 2026) operate on the assumption that quantization introduces noise that must be corrected through importance sampling or clipping — they treat the quantized outputs as degraded versions of the true outputs that need statistical correction. Sol-RL makes a categorically different move: it treats the quantized outputs as perfectly adequate for one task (ranking) and completely unusable for another (regression target), and designs the architecture around this functional asymmetry. This reframes quantization from "an approximation to be corrected" to "a task-specific tool to be selectively deployed."

The theoretical analysis in Appendix A adds another layer of conceptual depth. It proves that the ranking reliability improves with scale: the worst-case reward penalty from proxy ranking is a constant 4Δ, while the true reward range grows as 2σ√(2 log N). As N increases, the signal-to-noise ratio of the ranking grows unboundedly. This means the two-stage architecture is not just efficient — it is asymptotically optimal in the sense that the proxy ranking error becomes negligible relative to the contrastive signal at large N. This theoretical guarantee is what transforms the empirical observation (FP4 ranking works at N=96) into a principled justification (it works because N is large, and would work even better at larger N).

This is a fundamental empirical discovery with a theoretical grounding, not a minor benchmark improvement. It identifies a property of diffusion models under quantization that was previously unknown, characterizes it quantitatively across multiple reward models, and proves that it has the right scaling behavior to support a decoupled architecture. Without this discovery, the two-stage design would be speculation; with it, the design is deduction.

Innovation 3: Decoupling Exploration from Optimization as a Design Pattern, Not Just an Implementation Detail

The two-stage architecture itself — FP4 exploration followed by BF16 regeneration — might appear at first glance to be an implementation trick: "generate cheap candidates, filter them, regenerate the good ones." But the paper's framing reveals a deeper conceptual contribution: it identifies exploration-optimization decoupling as a general design pattern for reinforcement learning with compute-heterogeneous hardware, where different phases of the RL loop have fundamentally different precision requirements.

The field's default assumption — implicit in virtually all prior RL-for-diffusion work — has been that the precision used for exploration (generating rollouts) must match the precision used for optimization (computing gradients). This assumption was not arbitrary; it follows from the on-policy nature of policy gradient methods, where the expectation is taken over trajectories from the current policy. Quantization introduces an off-policy gap that standard theory says should be corrected or avoided. Jet-RL's unified-precision approach (FP8 for both training and rollout) represents the logical endpoint of this assumption: if you can't correct the mismatch, eliminate it by using the same precision everywhere.

Sol-RL breaks this assumption by asking a different question: what if the exploration phase doesn't need to produce training targets at all? By repurposing the FP4 rollout exclusively for ranking — a task that tolerates significant per-sample error as long as relative ordering is preserved — the paper eliminates the need for the exploration distribution to match the optimization distribution. The on-policy assumption only applies to the samples that actually contribute to the gradient (the Stage 2 BF16 regenerations), not to the samples used for filtering (the Stage 1 FP4 explorations). The off-policy gap becomes irrelevant because the gap exists only in a phase of the pipeline that never touches the policy gradient.

This conceptual move has implications beyond Sol-RL and beyond quantization. It suggests a general principle for RL systems with heterogeneous compute: identify phases that require distributional fidelity (optimization targets) and phases that require only coarse signal (filtering, ranking, exploration pruning), and allocate precision accordingly. This pattern could apply to scenarios beyond precision heterogeneity — for example, using smaller draft models for exploration and larger models for final candidate generation, or using approximate reward models for filtering and expensive human evaluations only for the selected candidates.

The architectural innovation is not the two stages themselves — best-of-N filtering with regeneration is hardly new — but the formal justification for why the stages can have different computational properties without violating RL principles. The paper's contribution is making explicit the previously implicit assumption (that exploration precision must equal optimization precision) and then demonstrating a principled way to violate it. This is a conceptual reframing of the precision-allocation problem in RL, not just an efficient implementation of existing ideas.

The evidence that this reframing is robust — and not an artifact of lucky hyperparameter choices — comes from the cross-model consistency in Table 6. Across FLUX.1, SD3.5-Large, and SANA (three models with different architectures, sizes, and ODE solvers), the alignment fidelity gap between the decoupled Sol-RL pipeline and the naive BF16 baseline is at most 1.08% (SD3.5-Large on HPSv2), with FLUX.1 showing a 0.29% gap and SANA actually showing a slight improvement (+0.11%). If the decoupling were fragile or dependent on specific model properties, we would expect much larger variance across models. The consistency suggests the design pattern is robust to the choice of base model.

This is an architectural innovation at the systems-algorithm boundary — not a new RL algorithm, not a new quantization technique, but a new way of composing them that respects their complementary error properties. In a field where methods papers typically propose new loss functions or training objectives, this kind of architecture-level contribution is comparatively rare and arguably more impactful for practitioners who need to deploy these systems on real hardware.

Innovation 4: Characterizing Verifier Over-Optimization as the Bottleneck That Rollout Scaling Addresses — and Quantization Can Circumvent

Section 3.1 of the paper contains a subtle but important conceptual contribution that is easy to overlook: the explicit characterization of why rollout scaling improves alignment in GRPO-based diffusion RL, and what this implies about where the computational budget should be allocated.

The standard narrative from prior work (DanceGRPO, BroRL) is that larger rollout groups provide better exploration, leading to the discovery of higher-reward samples, which in turn provide stronger positive gradient signals. This narrative emphasizes the exploration benefit of scaling. Sol-RL's framing adds a complementary perspective: the benefit comes from widening the reward range within each group, which makes the advantage normalization (Equation 3) produce more reliable gradient signals by increasing the signal-to-noise ratio of the advantage estimates.

This distinction matters because it shifts the focus from "finding the absolute best sample" (which requires high-precision generation to accurately assess quality) to "finding the most extreme samples" (which only requires reliable relative ranking). If the primary benefit of scaling were discovering a single maximally-rewarding image, then any ranking error in the proxy could be catastrophic — missing the true best sample would directly degrade performance. But if the benefit comes from having a wide spread between the best and worst candidates in the training set, then the proxy ranking only needs to be approximately correct. A proxy that ranks the true 1st-best candidate as 3rd-best and the true 2nd-worst as 4th-worst is still selecting genuine extremes from the distribution, and the contrastive signal from these extremes remains largely intact.

The paper's language in Section 3.1 supports this interpretation: "the most contrastive samples provide more reliable and informative learning signals for policy optimization, while other samples provide limited gradient due to the near-zero advantages." The emphasis is on contrastiveness (the magnitude of the advantage signal), not on optimality (the absolute reward value of the best candidate). This is why a 93.3% Top-12 match rate is sufficient — the selected candidates are still highly contrastive, even if they are not exactly the optimal subset.

This reframing has practical consequences that the paper does not fully explore but are implicit in the results. It suggests that rollout scaling in GRPO is primarily addressing a verifier over-optimization problem analogous to the one identified in the LLM test-time compute scaling work (Snell et al., 2024, which the Sol-RL authors do not cite but which analyzes the same phenomenon). In both settings, the reward model (verifier) provides imperfect quality estimates, and the policy update can over-optimize to exploit verifier errors. Rollout scaling mitigates this by providing a wider baseline — when the group mean and standard deviation are estimated from more samples, the advantage estimates are more robust to individual reward model errors. The insight is that you don't need perfect rewards; you need a sufficiently wide reward distribution that the noise in individual rewards is dominated by genuine quality differences.

The further implication — which is the paper's most provocative but understated claim — is that precision reduction can actually be part of the solution rather than a compromise. By enabling larger N (96 candidates instead of 24 at the same cost), FP4 exploration provides wider reward ranges that make the advantage estimates more robust. The 4Δ penalty from proxy ranking error is a constant subtracted from the reward range, but the range itself grows with log N. At N=96, the net effect is positive: the wider range from larger N more than compensates for the small ranking error. The paper provides empirical evidence for this in Table 3, where HPSv2 score increases monotonically with N from 0.3569 (N=24) to 0.3686 (N=96), confirming that the scaling benefit dominates the proxy error.

This is a diagnostic reframing with practical implications, not a new algorithm. It clarifies why the decoupled architecture works — because scaling N is more important than precise ranking within the selected subset — and provides a conceptual framework for reasoning about future improvements. If someone develops a more efficient proxy (e.g., a distilled reward model that runs in 2ms instead of 20ms), the same framework would predict that the benefit comes not from better ranking accuracy but from enabling even larger N, further widening the contrastive signal.

The connection to verifier over-optimization in the test-time compute scaling literature is the paper's own, though it draws the parallel less explicitly than it could have. The idea that "you don't need perfect quality assessment, you need reliable relative ordering at the extremes" is conceptually identical to the finding in Snell et al. that best-of-N with an imperfect process reward model outperforms more sophisticated search algorithms because the search over-optimizes the verifier. Sol-RL makes the same conceptual move but applied to RL training dynamics rather than inference-time search: use a cheap proxy to scale the candidate pool, and rely on the extreme value properties to make the proxy errors asymptotically irrelevant.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses the PickScore dataset (Kirstain et al., 2023) for both training and evaluation prompts. Prompts are sampled from the PickScore training split for RL training, with a separate held-out subset reserved for evaluation. The PickScore dataset contains image captions collected from web-scale text-to-image interaction data, providing diverse and realistic user prompts. Unlike the earlier technical example (which used MATH for math reasoning), this dataset captures the open-ended visual generation tasks for which diffusion alignment is relevant.

• Base model(s). Three state-of-the-art text-to-image diffusion models spanning different architectures, scales, and sampling paradigms: SANA-1.5 1600M (1.6B parameters, flow-matching based, 1024×1024 native resolution), FLUX.1-dev (12B parameters, rectified flow transformer, 512×512 resolution with classifier-free guidance embedding of 1.0), and Stable Diffusion 3.5-Large (SD3.5-L) (8B+ parameters, flow-matching transformer, 1024×1024 resolution). This cross-model selection is deliberate: it tests whether the decoupled architecture's benefits generalize across different ODE solver formulations (Euler for SANA, DPM-Solver-2 for FLUX.1 and SD3.5-L), model sizes (from 1.6B to 12B), and guidance strategies (CFG enabled for FLUX.1, disabled for others).

• Metrics. The paper evaluates alignment using four reward models, each measuring a different facet of human preference: ImageReward (Xu et al., 2023) — a BLIP-based model trained on human preference annotations for overall visual quality and text-image alignment; CLIPScore (Hessel et al., 2022) — cosine similarity between CLIP text and image embeddings, measuring semantic alignment; PickScore (Kirstain et al., 2023) — a preference model trained on the Pick-a-Pic dataset for pairwise image comparison; and HPSv2 (Wu et al., 2023) — Human Preference Score v2, a fine-tuned CLIP-based scorer trained on large-scale human preference data. Each metric produces a scalar score where higher values indicate better alignment. Metrics are computed on the held-out evaluation prompt set. The paper reports absolute scores and, in Table 1, the improvement Δ relative to the non-CFG base model.

• Baselines. The paper compares against four existing diffusion RL algorithms: FlowGRPO (Liu et al., 2025) — adapts the discrete-token GRPO formulation to flow-matching models using multi-step likelihood estimation; DanceGRPO (Xue et al., 2025) — combines DDPO's step-wise formulation with GRPO and introduces selective training on contrastive samples; AWM (Advantage Weighted Matching) (Xue et al., 2025) — frames forward-process diffusion optimization as advantage-weighted denoising score matching, placing the approach on firmer policy-optimization footing via the ELBO; and DiffusionNFT (Zheng et al., 2026) — an NFT-style forward-process version of GRPO that the paper describes as its closest algorithmic counterpart (Sol-RL uses the DiffusionNFT objective for its policy update in Stage 2). The non-CFG base model serves as an additional reference point in Table 1.

• Generation budget / compute accounting. The paper uses two distinct budget frameworks depending on the comparison. For algorithmic comparisons (Table 1, Figures 1, 4), the unit is GPU-hours — total wall-clock time consumed by the full training pipeline on 8 NVIDIA B200 GPUs. This is a holistic measure that accounts for both rollout generation and policy optimization time, making it the appropriate metric for comparing end-to-end training efficiency across methods with different computational profiles. For efficiency breakdown (Table 5), the paper separately reports Rollout Time (pure forward generation phase) and End-to-End Time (rollout + backward gradient updates), measured in seconds per training iteration. The rollout configuration is standardized at 24-in-96 (24 contrastive samples selected from 96 generated candidates) for the efficiency analysis. The paper also reports speedup factors (e.g., 2.33×, 4.64×) defined as the ratio of baseline time to Sol-RL time for reaching equivalent performance.

• Cross-validation / statistical protocol. The paper does not employ explicit cross-validation for model selection. For the main results (Tables 1, 4, 6; Figures 1, 4), the training runs appear to be single-seed evaluations where each method is trained with fixed hyperparameters from a common starting checkpoint and evaluated at regular intervals. The learning curves in Figure 4 show training trajectories over GPU-hours, allowing visual assessment of convergence behavior and final performance. For the ranking fidelity analysis (Table 8, Appendix C), the correlation metrics (Kendall's τ, Spearman's ρ, Top/Bottom-k match rates) are computed as summary statistics over the evaluation prompt set, presumably aggregated across all prompts and seeds. The paper does not report confidence intervals or standard errors for any metric, which is a limitation — particularly given the relatively small per-model evaluation budget (the number of evaluation prompts is not explicitly stated, but the training uses 48 prompts per epoch across 8 GPUs).

Main Quantitative Results

Alignment Performance Under Equal Compute Budget (Table 1, Figures 1 and 4)

The central quantitative claim appears in Table 1: under an identical GPU-hour budget on FLUX.1, Sol-RL achieves the highest scores across all four reward metrics, with improvements over the strongest baseline (DiffusionNFT) ranging from +0.0929 on ImageReward to +0.0075 on HPSv2. The absolute numbers on FLUX.1 for Sol-RL are: ImageReward 1.7636 (vs. 1.6707 for DiffusionNFT, a +5.6% improvement), CLIPScore 0.3089 (vs. 0.2991, +3.3%), PickScore 0.8932 (vs. 0.8852, +0.9%), and HPSv2 0.3688 (vs. 0.3613, +2.1%). All four baselines (FlowGRPO, DanceGRPO, AWM, DiffusionNFT) show substantial improvements over the non-CFG base model (e.g., HPSv2 rises from 0.2566 to 0.3613 for DiffusionNFT, a +0.1047 absolute gain), confirming that RL-based alignment is effective — but Sol-RL pushes the ceiling higher within the same total compute.

The significance of these numbers lies in the compute-matched comparison. DiffusionNFT is the algorithmic backbone of Sol-RL — the Stage 2 policy update uses the identical DiffusionNFT objective. If both methods consume the same total GPU-hours, Sol-RL must be spending its compute more effectively: the FP4 acceleration in Stage 1 frees up budget that can be reinvested into more training iterations or larger exploration pools, and the selective regeneration ensures that every expensive BF16 forward pass contributes meaningful contrastive signal. The gap is largest on ImageReward (+5.6%) and smallest on PickScore (+0.9%), suggesting the benefit may be reward-model-dependent — perhaps because ImageReward is more sensitive to the semantic-level improvements that better exploration provides, while PickScore's training data distribution makes it less responsive.

Figure 4 provides the per-model, per-metric training trajectories that decompose the aggregate improvements in Table 1. Across all nine combinations of base model and reward metric (3 models × 3 metrics: CLIPScore, HPSv2, PickScore), the Sol-RL learning curves (green) consistently lie above the DiffusionNFT baseline (grey) for the majority of training. The key pattern is twofold: (1) faster convergence to equivalent performance, and (2) higher final alignment ceiling. The paper reports convergence speedup factors ranging from 1.91× to 4.64× — these are computed as the GPU-hour ratio at which Sol-RL reaches the baseline's best-achieved score. For instance, on SD3.5-Large with HPSv2, Sol-RL reaches ~0.37 (the DiffusionNFT ceiling) at roughly 30 GPU-hours compared to the baseline's ~90 GPU-hours — a 3.0× speedup. But Sol-RL continues improving beyond this point to ~0.376 at 150 GPU-hours, while the baseline plateaus.

The model-dependent speedup variation is informative. FLUX.1 shows 3.9× speedup on HPSv2, SD3.5-Large shows 3.0×, and SANA shows 3.6×. The paper does not offer an explanation for this variation, but it likely reflects differences in the ratio of rollout cost to total training cost: FLUX.1 (12B parameters, 512×512) has the most expensive generation relative to its policy update cost, making the Stage 1 FP4 acceleration most impactful. SANA (1.6B, 1024×1024) is more efficient at generation to begin with, so the relative speedup is smaller (also reflected in Table 5, where SANA's rollout speedup is 1.41× vs. 2.41× for SD3.5-Large). The convergence curves in Figure 4 visually confirm that Sol-RL's advantage is not merely a constant offset — the gap widens over training, suggesting that better exploration compounds over successive policy updates.

Figure 1 (left panel) provides qualitative validation: images generated by FLUX.1 and SANA fine-tuned with Sol-RL across diverse styles (photorealistic portraits, illustrations, text rendering) demonstrate the visual quality that the quantitative metrics are measuring. The right panel shows ImageReward training curves for FLUX.1, where Sol-RL achieves 2.42× convergence speedup and reaches a final score of ~1.76 vs. the baseline's ~1.67.

Efficiency Breakdown: Where the Speedup Comes From (Table 5, Table 7)

Table 5 decomposes the speedup into its constituent components by measuring pure rollout time and end-to-end iteration time for both the naive BF16 scaling baseline (96 BF16 samples, 10 steps each, training on the top-24) and Sol-RL's two-stage pipeline (96 FP4 samples at 6 steps + 24 BF16 samples at 10 steps). For SD3.5-Large, the rollout time drops from 451 seconds to 187 seconds (2.41× speedup), and the end-to-end iteration time drops from 691 seconds to 427 seconds (1.62× speedup). For FLUX.1, the figures are 184 → 79 seconds rollout (2.33×) and 274 → 169 seconds end-to-end (1.62×). For SANA, the smaller speedup is 65 → 46 seconds rollout (1.41×) and 95 → 76 seconds end-to-end (1.25×).

The gap between rollout speedup (2.41×) and end-to-end speedup (1.62×) for SD3.5-Large is analytically important. It quantifies what fraction of total training time is spent in the rollout phase: under the naive baseline, rollout consumes 451/691 = 65.3% of total time. Under Sol-RL, rollout consumes 187/427 = 43.8%. The policy update phase (which is identical in both approaches) consumes the remaining time — 240 seconds in both cases. The 1.62× end-to-end speedup therefore represents the theoretical maximum improvement if the rollout phase were infinitely accelerated (leaving only the fixed 240-second update cost), scaled by the actual reduction in rollout time. This calculation implies that even if Stage 1 were completely free, the maximum end-to-end speedup would be 691/240 ≈ 2.88× for SD3.5-Large — so the achieved 1.62× represents approximately 56% of the theoretical maximum, with the remaining room for improvement coming from further reducing the 187-second Stage 2 cost (which includes both the 24 BF16 regenerations and any overhead from quantization and seed selection).

Table 7 provides the full training configuration. The rollout hyperparameters are standardized: ODE solver (Euler for SANA, DPM-Solver-2 for FLUX.1 and SD3.5-Large), rollout steps (10 for Stage 2 / naive baseline, 6 for Stage 1 exploration), evaluation steps (40 for SANA, 28 for FLUX.1, 40 for SD3.5-Large). The two-stage configuration specifies N=96 images per prompt and K=24 selected for training. All experiments use 8 NVIDIA B200 GPUs. Per-GPU micro-batch sizes vary by model (16 for SANA, 12 for FLUX.1, 4 for SD3.5-Large) to fit within GPU memory constraints. Gradient accumulation steps adjust accordingly (9, 12, 36 respectively) to maintain effective batch sizes.

Alignment Fidelity Preservation (Table 6)

Table 6 addresses the critical question: does the decoupled architecture preserve the alignment quality of the naive BF16 baseline, or does the proxy ranking error in Stage 1 cause degradation despite the Stage 2 regeneration? The paper compares post-RL HPSv2 scores under identical training steps for the naive scaling baseline (BF16 brute-force 96 samples) and Sol-RL. For FLUX.1, the numbers are 0.3699 (naive) vs. 0.3688 (Sol-RL), a marginal gap of −0.29%. For SD3.5-Large, the gap is −1.08% (0.3803 vs. 0.3762). For SANA, Sol-RL actually slightly outperforms the naive baseline: 0.3686 vs. 0.3682 (+0.11%).

These gaps are described as "at most 1%" and interpreted as evidence that the decoupled pipeline "maintains the alignment fidelity of the naive scaling baseline." The SANA result (+0.11%) suggests that, at least for some models, the proxy ranking might provide a slight regularization benefit — perhaps because the small ranking noise prevents overfitting to the specific top/bottom candidates. However, the −1.08% gap for SD3.5-Large, while small in absolute terms, represents a non-trivial fraction of the total improvement over the base model (0.3762 − 0.2566 = 0.1196, so the gap is roughly 9% of the total gain). The paper does not investigate whether this gap grows with training duration or whether it represents a systematic bias from the proxy ranking rather than random variance.

Cross-Model and Cross-Metric Consistency (Figures 4, 5, 7, 8, 9)

The paper provides extensive qualitative evidence across three base models and four reward objectives. Figures 5, 7, 8, and 9 (Appendix and main text) show visual comparisons between base models and Sol-RL fine-tuned variants. Figure 5 demonstrates SANA before and after Sol-RL optimization across multiple rewards (HPSv2, PickScore, CLIPScore, OCR), with the after images showing improvements in "complex detail rendering and semantic alignment across various prompts" — for example, a "Van Gogh painting of a Tyrannosaurus Rex in Paris" more faithfully captures both the artistic style and the subject after optimization. Figures 7, 8, and 9 compare FLUX.1 base models against Sol-RL, DiffusionNFT, and FlowGRPO fine-tuned variants for PickScore-optimized, HPSv2-optimized, and ImageReward-optimized models respectively. Across all three, the paper claims that Sol-RL produces images with "stronger semantic alignment to the prompt, richer fine-grained details, and more coherent artistic style." These qualitative results are necessarily subjective and cherry-picked (the paper presents 12-16 examples per figure), but they serve to demonstrate that the quantitative metric improvements translate to visually perceptible quality differences.

Ablation Studies and Robustness Checks

The ablation experiments (Section 4.3) investigate two critical hyperparameters of the Stage 1 exploration phase, using HPSv2 score on the evaluation set as the metric and fixing all other settings (K=24, BF16 Stage 2 with 10 steps, FLUX.1 base model from context). The results appear in Tables 2 and 3.

FP4 exploration denoising steps (T): Table 2 sweeps T ∈ {2, 4, 6, 8}, finding that HPSv2 increases from 0.3587 (T=2) to 0.3650 (T=4) to 0.3686 (T=6), then slightly decreases to 0.3659 at T=8. The paper interprets the improvement from 2 to 6 steps as evidence that "coarse semantic layouts are insufficiently formed" at very low step counts, leading to inaccurate intra-group ranking and suboptimal Top-K selection. The slight degradation at T=8 (which remains above T=4 but below T=6) suggests the proxy ranking capability saturates — more steps beyond 6 provide no additional ranking fidelity, and may introduce subtle distributional differences (longer ODE integration under FP4 precision) that slightly harm the ranking's alignment with BF16 rewards. This saturation point is fortuitous for efficiency: it means Stage 1 can operate safely at a substantial step reduction (6 vs. 10) without degrading the quality of seed selection.

Exploration pool size (N): Table 3 sweeps N ∈ {24, 48, 72, 96} while holding K=24 fixed (i.e., always training on 24 contrastive samples, but selecting them from pools of varying size). HPSv2 increases monotonically: 0.3569 (N=24), 0.3622 (N=48), 0.3663 (N=72), 0.3686 (N=96). This monotonic improvement confirms that the selective training paradigm benefits from larger exploration pools even when the number of training samples stays constant. The paper interprets this as evidence that "scaling the FP4 exploration pool size effectively unlocks substantial and consistent alignment gains" — a direct validation that the proxy ranking is reliable enough to make larger pools beneficial despite the constant 4Δ ranking error penalty from the theoretical analysis. The diminishing returns pattern (gains of +0.0053 from 24→48, +0.0041 from 48→72, +0.0023 from 72→96) is consistent with the √(2 log N) growth rate predicted by extreme value theory, where each doubling of N adds progressively less to the extreme range.

NVFP4 image quality preservation (Table 4): Before the main ablations, the paper establishes that NVFP4 quantization does not fundamentally break the images used for proxy ranking. Table 4 reports Inception Score (IS) and CLIPScore for NVFP4-generated images vs. BF16 baselines across all three base models, using (presumably) the standard number of denoising steps for fair comparison. The results show no systematic degradation: FLUX.1 IS is 17.85 (NVFP4) vs. 16.84 (BF16), an unexpected improvement; SANA IS is 15.94 vs. 16.02 (marginally worse); CLIP scores are nearly identical across all models (FLUX.1: 27.10 vs. 27.44; SANA: 29.43 vs. 29.53; SD3.5-Large: 28.34 vs. 28.37). These results support the visual evidence in Figure 6 (NVFP4 and BF16 rollouts sharing the same semantic layout) and justify the paper's claim that NVFP4 preserved "the necessary structural integrity compared to BF16 rollout" for the proxy ranking task.

Global ranking consistency metrics (Table 8, Appendix C): This is not presented as an ablation but as a validation of the core assumption. For context, this table reports Kendall's τ, Spearman's ρ, and Top/Bottom-k match rates between BF16 and NVFP4 reward rankings, computed across the evaluation prompt set for each reward model. The overall averages are: Kendall's τ = 0.798, Spearman's ρ = 0.927, Top-4 match = 96.9%, Bottom-4 false-inclusion = 3.9%, Top-12 match = 93.3%, Bottom-12 false-inclusion = 7.5%. These metrics confirm that the proxy ranking is reliable at the extremes where selective training operates, with the match rate naturally declining as k increases (Top-4: 96.9%, Top-8: 95.0%, Top-12: 93.3%) because the boundaries between adjacent ranks become fuzzier away from the extremes.

Selective training ratio (implicit ablation across configurations): While the paper does not present an explicit ablation of the ratio K/N, varying this ratio is implicit in comparing different experimental setups. The main configuration uses K=24, N=96 (25% retention). The N ablation in Table 3 varies this ratio indirectly: from N=24 (100% retention — all candidates used, which is standard GRPO without selective training) to N=96 (25% retention). The monotonic improvement with decreasing retention ratio (at fixed K) confirms that selective training from larger pools outperforms training on all candidates from smaller pools, even though the same number of gradient updates are performed. This is the key empirical justification for the entire selective training paradigm that Sol-RL builds upon.

Critical Assessment

The paper makes three central claims that warrant evaluation against the experimental evidence: (1) Sol-RL achieves up to 4.64× convergence speedup over baselines while maintaining alignment quality, (2) the decoupled architecture preserves the training fidelity of the BF16 pipeline, and (3) these benefits generalize across diverse models and reward metrics. The experiments provide varying levels of support for each claim.

Claim 1 (4.64× speedup) is supported but the figure requires careful interpretation. The 4.64× number does not appear prominently in the main results but is mentioned in the abstract, introduction, and Figure 4 caption. In Figure 4, the largest speedup I can identify is 3.9× for FLUX.1 on HPSv2 (where Sol-RL reaches the DiffusionNFT ceiling at roughly 25 GPU-hours vs. the baseline's ~97 GPU-hours). The 4.64× may refer to a different metric-model combination not directly plotted in Figure 4, or may be derived from a different baseline comparison. The paper's reporting of convergence speedups in Figure 4 lists ranges (1.91× to 4.64×) without specifying exactly which configuration achieves the maximum. This is a minor but meaningful imprecision — the maximum speedup is a headline number that should be precisely attributable. The more reliably documented speedups are the rollout-time gains in Table 5 (1.41× to 2.41×) and the end-to-end gains (1.25× to 1.62×), which are measured directly rather than inferred from training curves. The gap between these directly-measured speedups (~1.6× end-to-end) and the headline 4.64× reflects the compounding effect of faster iterations over many training steps — the convergence speedup includes both the per-iteration acceleration and any algorithmic benefit from better exploration that reduces the number of iterations needed.

Claim 2 (training fidelity preservation) is supported but with a caveat about SD3.5-Large. Table 6 shows that under identical training steps, Sol-RL matches the naive BF16 baseline's final HPSv2 score within 1% for all three models. However, the comparison is only at equal training steps, not equal GPU-hours. At equal GPU-hours (which is the more practically relevant comparison since Sol-RL's whole purpose is to save time), Sol-RL would have completed more training steps and would presumably outperform the baseline (as seen in Figure 4). The fidelity preservation claim is thus conservative: even without exploiting the speedup for extra steps, Sol-RL loses essentially nothing. The −1.08% gap for SD3.5-Large warrants further investigation — it could be statistical noise (a single training run), could reflect model-specific sensitivity to the proxy ranking error, or could be a systematic effect that grows with more training. The paper does not report standard deviations across multiple seeds, which prevents distinguishing these possibilities.

Claim 3 (cross-model and cross-metric generalization) is the strongest-supported claim in the paper. The three base models span fundamentally different architectures (DiT-based SANA, rectified flow FLUX.1, traditional diffusion SD3.5-Large), scales (1.6B to 12B parameters), and ODE solver types (Euler vs. DPM-Solver-2). The four reward models capture different aspects of alignment (semantic similarity, human preference, aesthetics). That Sol-RL improves over DiffusionNFT across all 12 combinations (3 models × 4 metrics) in Table 1 and all 9 combinations in Figure 4 is compelling evidence that the decoupled architecture is robust to these variations. The variation in speedup magnitude across models (1.9× to 3.9× in Figure 4) is itself informative — it suggests the benefit is proportional to how much of the total training time is spent in the rollout phase, which is exactly what the paper's bottleneck diagnosis predicts.

Genuine weaknesses in the experimental design:

1. Single-seed training runs. None of the main results report error bars, confidence intervals, or standard deviations across multiple random seeds. Given the inherent stochasticity of RL training (noise seed sampling, training dynamics), the reported precision (e.g., HPSv2 to four decimal places in Table 6) overstates the certainty of the measurements. The 0.29% gap for FLUX.1 in Table 6 could easily be within run-to-run variance.

2. Missing FLOPs-based accounting. The paper measures speedup in wall-clock time (GPU-hours), which is appropriate for practical deployment but conflates hardware-specific factors (NVFP4 throughput on B200, memory bandwidth, compiler optimization) with algorithmic efficiency. A FLOPs-based accounting would separate these: how much of the 2.41× rollout speedup in Table 5 comes from FP4's higher TFLOPs vs. from the reduced denoising steps (6 vs. 10)? The 10→6 step reduction alone contributes a 1.67× theoretical speedup. Combined with 4× TFLOPs, the theoretical maximum is 6.67×, but the achieved 2.41× suggests substantial overhead (quantization time, seed selection, GPU kernel launch latency) that the paper does not decompose.

3. Fixed K=24 without systematic study. The paper uses K=24 across all experiments and N scalings, but never ablates K independently. This is a consequential choice: if K were smaller (e.g., 8 or 12), Stage 2 would be cheaper and the speedup would be larger, but the gradient signal from fewer samples would be noisier. If K were larger (e.g., 48), the speedup would be smaller but training might be more stable. Without this ablation, we don't know whether K=24 is near-optimal or whether different models/metics would benefit from different K values.

4. FP4 exploration step count (T=6) saturation claim is model-specific. Table 2 only reports the T ablation for one model (likely FLUX.1 from context). The optimal T may depend on the model's architecture, the ODE solver, and the native resolution — the saturation point at 6 steps might not generalize to SANA or SD3.5-Large, which use different solvers and resolutions. The paper applies T=6 universally across all three models (Table 7) without reporting whether this is equally effective for each.

5. Selective training baseline inconsistency. All the baseline methods in Table 1 (FlowGRPO, DanceGRPO, AWM, DiffusionNFT) presumably use their standard configurations, which may not include the 24-in-96 selective training that Sol-RL exploits. DanceGRPO introduced selective training, but was it configured with the same N and K as Sol-RL? If the baselines use smaller N (e.g., 24 or 48), then Sol-RL's advantage partially comes from better exploration scaling, not from the FP4 decoupling specifically.

6. No comparison against other quantization strategies. The paper argues that naive FP4 quantization fails (Figure 3b) and that decoupling is necessary. But it does not compare against other approaches to quantized diffusion RL that attempt to correct the off-policy gap, such as FP8-RL (importance sampling correction) or Jet-RL (unified precision). These comparisons would strengthen the claim that decoupling is better than correction-based approaches, not just different from naive quantization.

7. The training cost of difficulty estimation (from the reference example) does not apply here, but an analogous concern exists: the paper does not account for the overhead of the NVFP4 weight re-quantization that occurs after every policy update. The paper claims this overhead is "merely a 2% computational overhead" (Figure 2 caption), but this figure is not validated with a dedicated ablation — we don't know whether it is 2% for all models and all GPU configurations.

Missing experiments that would strengthen the paper:

• K ablation: Sweep K ∈ {8, 12, 24, 48} at fixed N=96 to determine whether the selected ratio is optimal and whether the alignment-fidelity gap (Table 6) changes.
• Seed-level comparison: Report the fraction of training iterations where the FP4 proxy selects a different top-12/bottom-12 set than BF16 would have, and whether those iterations show degraded policy improvement.
• Comparison against FP8-decoupled or unified-precision baselines to isolate the benefit of FP4 specifically vs. the decoupling architecture generally.
• Scaling to N > 96 to test the theoretical prediction that ranking error becomes negligible at very large N, and to identify where the proxy ranking eventually saturates or degrades.
• Multi-seed training runs with error bars on Tables 1, 2, 3, 5, 6 to assess statistical reliability.

Conditions under which the claims hold:

The paper's claims are empirically supported under the specific experimental conditions tested: FLUX.1-Dev, SANA-1.5, and SD3.5-Large models; PickScore training/evaluation prompts; ImageReward, CLIPScore, PickScore, and HPSv2 reward models; NVFP4 quantization on NVIDIA B200 GPUs with NVIDIA Transformer Engine compilation; DiffusionNFT as the base RL algorithm; LoRA rank 32 adaptation; and the 24-in-96 selective training configuration with N=96 FP4 exploration at 6 steps followed by K=24 BF16 regeneration at 10 steps. The paper provides no evidence about whether the approach works with: other reward models, other prompt distributions (PickScore prompts may have specific characteristics that make proxy ranking easier), other hardware backends, other quantization formats (INT4, FP8), full fine-tuning instead of LoRA, larger exploration scales (N > 96), or other RL objectives (AWM, DDPO-style formulations instead of DiffusionNFT's forward-process objective). The cross-model consistency across three architectures is encouraging but far from comprehensive — all three models are transformer-based diffusion models with relatively similar ODE sampling semantics.

The most significant unvalidated assumption is that the proxy ranking fidelity demonstrated on FLUX.1 (Tables 4, 8; Figure 6) generalizes to SANA and SD3.5-Large. Table 4 provides IS and CLIPScore for all three models, confirming semantic preservation, but the critical ranking correlation metrics in Table 8 appear to be computed only for FLUX.1 (the paper does not specify, but the context of the appendix suggests the analysis was performed on a single model). If the proxy ranking is less reliable for SD3.5-Large, that could explain its larger −1.08% fidelity gap in Table 6.

6. Limitations and Trade-offs

6.1 The Decoupled Architecture Requantizes Weights After Every Training Step — an Overhead That Is Claimed but Not Measured

The assumption or constraint. The Sol-RL framework requires that after every policy update (every gradient step on the K=24 BF16 regeneration samples), the newly updated BF16 weights be re-quantized into NVFP4 format and synchronized to the Stage 1 inference engine. The paper states in Figure 2's caption that this introduces "merely a 2% computational overhead," but provides no dedicated timing breakdown, no ablation on requantization cost across models, and no discussion of how this overhead scales with model size or quantization block granularity.

The consequence. The claim of "2% overhead" is asserted without measurement. In practice, weight quantization involves computing block-level scaling factors (E4M3 scale for each group of 16 contiguous elements), projecting each element to the nearest FP4-representable value, and copying the quantized weights into the pre-compiled inference engine's memory buffers. For a model with 12 billion parameters (FLUX.1), even with a pre-compiled graph that avoids JIT recompilation, the data movement and scaling-factor computation scales with parameter count. If the true overhead is closer to 5-10% rather than 2%, the reported speedup numbers in Table 5 (1.25-1.62× end-to-end) would be correspondingly lower. Furthermore, the overhead is incurred on the critical path of the training loop — the policy cannot begin Stage 1 exploration for the next batch until the previous iteration's weights are re-quantized and synchronized. In distributed settings across 8 GPUs, the synchronization of quantized weights across devices adds communication overhead that the single-GPU-per-rank timing breakdown in Table 5 may not capture.

What evidence exists in the paper. None beyond the statement in the Figure 2 caption. Table 5 reports Rollout Time and End-to-End Time for naive scaling vs. Sol-RL, but does not decompose Sol-RL's end-to-end time into its constituent phases (Stage 1 FP4 generation, seed selection, Stage 2 BF16 regeneration, policy update, weight re-quantization, synchronization). Without this decomposition, the reader cannot verify the 2% claim or understand whether the re-quantization cost varies across models (SANA's 1.6B parameters vs. FLUX.1's 12B parameters). The paper also does not specify whether the 2% figure is measured relative to total iteration time or specifically relative to the policy update phase.

Mitigation status. Not addressed. The paper asserts the 2% figure without measurement and does not discuss it as a limitation. A practitioner seeking to replicate this architecture on different hardware, with different quantization backends, or with full fine-tuning (which updates more parameters than LoRA and thus requires quantizing a larger fraction of weights) has no empirical basis to estimate what fraction of the speedup will be consumed by weight requantization overhead.

---

6.2 The Framework Is Validated Exclusively on NVIDIA B200 Hardware with NVFP4 — the Results Do Not Transfer to Other Precision Formats or GPU Architectures

The assumption or constraint. Sol-RL is explicitly designed around the NVFP4 format implemented in NVIDIA's Blackwell architecture, which the paper describes as providing "up to 4× the TFLOPs of standard BF16 arithmetic" on B200 GPUs. NVFP4 is a proprietary format (1 sign, 2 exponent, 1 mantissa bit, with E4M3 block-level scaling for groups of 16 elements) that differs from the open OCP MXFP4 standard (which groups 32 elements under an E8M0 scale). All experiments use the NVIDIA Transformer Engine as the quantization backend with model-specific compilation. The paper provides no results using other precision formats (INT4, FP8, NF4), other quantization methods (GPTQ, AWQ, SmoothQuant), or other GPU architectures.

The consequence. The paper's central claims — 2.41× rollout speedup, 1.62× end-to-end speedup, 4.64× convergence acceleration — are contingent on hardware-specific TFLOP ratios that may not hold elsewhere. On GPUs without native FP4 support (e.g., NVIDIA H100, A100, or AMD/Intel accelerators), FP4 inference would either be emulated in higher precision (eliminating the throughput advantage) or unsupported entirely. Even on Blackwell hardware, the 4× throughput advantage is a peak theoretical figure; achieved throughput depends on memory bandwidth, kernel launch overhead, and the specific operation mix in the diffusion model's forward pass. The paper provides no breakdown of achieved vs. peak FP4 utilization.

Furthermore, the proxy ranking property documented in Table 8 and Figure 3c is specific to NVFP4's error characteristics. A different 4-bit format (INT4 with per-channel scaling, or NF4 with learned quantization buckets) would introduce different error distributions — potentially with different effects on semantic preservation and reward ranking fidelity. The paper's theoretical bound in Appendix A depends on the perturbation magnitude Δ (the per-sample reward error bound), which is format-specific. A format with larger per-element error but better outlier handling could produce different ranking correlation metrics. Without replication across formats, the paper's claim that "FP4 exploration works as a proxy ranker" should be read as "NVFP4 exploration on B200 GPUs works as a proxy ranker."

What evidence exists in the paper. None beyond the specific hardware and format configuration. The paper does not compare NVFP4 against INT4, FP8, or any other precision format. It does not measure achieved FP4 TFLOPs as a fraction of theoretical peak. Table 4 validates that NVFP4 specifically preserves semantic integrity (IS and CLIPScore comparable to BF16), but provides no evidence about whether this generalizes to other 4-bit formats. The paper cites related work on FP8 quantized RL (Jet-RL, FP8-RL) but never benchmarks against these approaches to establish whether FP4's additional speedup (relative to FP8) is worth any additional ranking error.

Mitigation status. Not addressed. The paper acknowledges the hardware dependency implicitly by using the "NVFP4" designation, but treats the specific precision format and hardware as implementation details rather than as active constraints on the method's applicability. A practitioner with access only to H100 GPUs or working in a PyTorch-native environment without Transformer Engine cannot implement Sol-RL as described and has no guidance on whether an INT4 or FP8 variant would retain the proxy ranking property.

---

6.3 All Experiments Use LoRA Fine-Tuning with Rank 32 — the Architecture's Behavior Under Full-Parameter Training Is Unknown

The assumption or constraint. Every experiment in the paper — across all three base models, all four reward metrics, and all baselines — uses Low-Rank Adaptation (LoRA) with rank r=32 and scaling factor α=64 applied to specific attention projection layers (to_{q,k,v,out} for SANA and FLUX.1; additional attn.{to,add}_{q,k,v,out} for SD3.5-Large, per Table 7). The paper never experiments with full-parameter fine-tuning or different LoRA configurations (different ranks, different target modules). The policy update in Stage 2 optimizes only the LoRA adapter weights, not the full model parameters.

The consequence. LoRA introduces a structural constraint on the policy update: the learned policy change Δθ is restricted to a low-rank subspace of the full parameter space. This has two interacting effects that are unexplored. First, the off-policy gap problem documented in Section 3.2 — where quantized rollouts cause the policy to learn corrupted targets — may be partially mitigated by LoRA's regularization effect. A low-rank adaptation has fewer degrees of freedom to overfit to quantization artifacts, which could make the naive FP4 baseline (Figure 3b) appear more degraded than it would be under full fine-tuning (if LoRA were actually protecting it). Conversely, Sol-RL's benefit over the naive baseline might be smaller under full fine-tuning if the regularization from LoRA is doing some of the same work as the decoupling architecture.

Second, the weight re-quantization overhead discussed in Limitation 6.1 scales with the number of parameters being quantized. With LoRA, only the adapter weights need re-quantization each iteration — a tiny fraction of the total model parameters. Under full fine-tuning, the entire model's weights would need re-quantization and synchronization after every step, potentially increasing the overhead substantially beyond the claimed 2%. For FLUX.1's 12B parameters, full-model re-quantization at every step could consume a significant fraction of the Stage 2 regeneration time, eroding or eliminating the speedup.

Third, the exploration quality in Stage 1 depends on the FP4 inference engine running the full model. If the policy update only modifies LoRA weights, the base model's FP4 behavior remains largely static across iterations — the quantized forward pass changes only in the adapter's contribution. Under full fine-tuning, the base model's behavior could shift more dramatically, potentially changing the noise-to-semantics mapping in ways that make the proxy ranking from the previous iteration's FP4 model less predictive of the current BF16 model's rewards.

What evidence exists in the paper. None. All experiments use LoRA rank 32 without variation, and the paper does not mention full fine-tuning as a possible configuration. The training hyperparameters in Table 7 list LoRA settings as fixed across models. There is no ablation of LoRA rank, no comparison of LoRA vs. full fine-tuning, and no discussion of how the decoupled architecture would need to change for full-parameter training.

Mitigation status. Not addressed. The paper's default use of LoRA is reasonable for the memory constraints of training 12B-parameter models, but the lack of acknowledgment that all findings are conditional on LoRA-specific regularization and reduced requantization cost is a significant omission. A practitioner seeking to apply Sol-RL with full fine-tuning — which is standard practice for smaller models and may become feasible for larger models with future hardware — has no empirical basis to estimate whether the speedup or alignment fidelity claims hold.

---

6.4 The Speedup Figures Are Reported in Wall-Clock Time on a Specific GPU Configuration, Without Decomposition into FLOPs Reduction vs. Hardware Utilization Effects

The assumption or constraint. The paper measures all speedups in GPU-hours (Table 1, Figures 1, 4) and wall-clock seconds per iteration (Table 5) on 8 NVIDIA B200 GPUs. This is a practical, deployment-relevant metric, but it conflates two fundamentally different sources of acceleration: (1) FLOPs reduction (fewer mathematical operations due to FP4 arithmetic and reduced denoising steps) and (2) hardware utilization effects (higher TFLOPs per second due to NVFP4's 4× throughput advantage on B200 tensor cores). The paper provides no FLOPs-based accounting, no measurement of achieved TFLOPs as a fraction of theoretical peak, and no analysis of how the speedup would change on hardware with different FP4-to-BF16 throughput ratios.

The consequence. The reported speedups are not portable. If a future GPU generation achieves 8× FP4 throughput (rather than 4×), the speedup would increase — but the paper provides no framework for predicting by how much. More importantly, if the FP4 throughput advantage is smaller on a different architecture (e.g., 2× instead of 4×), the relative ranking of Sol-RL vs. alternative approaches could change: an FP8-based decoupled architecture might achieve comparable end-to-end speed at lower implementation complexity, or a full-BF16 pipeline with more aggressive step reduction might be simpler and equally fast.

The conflation also obscures how much of the speedup comes from the algorithmic innovation (decoupling exploration from optimization) vs. from the hardware innovation (NVFP4's 4× throughput). As noted in Section 5's critical assessment, the 10→6 step reduction in Stage 1 alone contributes a 1.67× theoretical speedup independent of precision. Combined with the 24/96 = 25% regeneration ratio in Stage 2, a naive BF16 pipeline with the same reduced exploration steps and selective regeneration would achieve some fraction of Sol-RL's speedup — but the paper never benchmarks this configuration. Without a "BF16 two-stage" ablation (same decoupled architecture, same reduced steps, but all in BF16), we cannot separate the contribution of the architecture from the contribution of the precision format.

What evidence exists in the paper. Table 5 provides wall-clock rollout time and end-to-end time for Naive (full BF16, 96 samples, 10 steps) vs. Sol-RL (96 FP4 at 6 steps + 24 BF16 at 10 steps). No intermediate configurations are timed: we see neither the cost of a "BF16 two-stage" baseline (96 BF16 at 6 steps + 24 BF16 at 10 steps), nor the cost of a "FP4 naive" configuration (96 FP4 at 10 steps, training on selected subset without regeneration). The paper also does not report achieved TFLOPs or GPU utilization percentages.

Mitigation status. Not addressed. The paper reports speedups as empirical facts without attempting to decompose or explain them in implementation-independent terms. This is standard industry practice (wall-clock time is what practitioners care about), but it limits the paper's scientific contribution: we learn that the specific combination of NVFP4, reduced steps, and selective regeneration is faster on B200 GPUs, but we do not learn a general principle for predicting when decoupled exploration will be beneficial on other hardware.

---

6.5 The Proxy Ranking Fidelity Analysis Is Performed on a Single Model (Likely FLUX.1) and Not Validated Across the Three Tested Architectures

The assumption or constraint. The central empirical claim enabling Sol-RL is that NVFP4-quantized rollouts preserve intra-group reward ranking with sufficient fidelity to serve as a proxy for seed selection. The paper devotes Section 3.3 and Appendix C to validating this claim, presenting Figure 3c (conditional probability density of BF16 vs. NVFP4 ranks), Table 8 (Kendall's τ, Spearman's ρ, Top/Bottom-k match rates across four reward models), and Table 4 (Inception Score and CLIPScore preservation across all three base models). However, Table 8 and Figure 3c — which measure the ranking fidelity directly — are presented in a way that strongly suggests they were computed on a single model, most likely FLUX.1 from context. The paper does not explicitly state which model was used for the ranking correlation analysis, and does not report ranking fidelity metrics for SANA or SD3.5-Large.

The consequence. The paper's cross-model claims (that Sol-RL generalizes to SANA, FLUX.1, and SD3.5-Large) are supported by end-to-end training results (Table 6, Figure 4) and semantic preservation metrics (Table 4), but not by direct evidence that the proxy ranking is equally reliable across architectures. This matters because the three models have fundamentally different properties relevant to quantization error: different parameter counts (1.6B vs. 8B+ vs. 12B), different ODE solvers (Euler for SANA flow matching vs. DPM-Solver-2 for FLUX.1 and SD3.5-Large diffusion), different native resolutions (1024×1024 for SANA and SD3.5-Large vs. 512×512 for FLUX.1), and different CFG configurations (enabled for FLUX.1, disabled for others). Any of these factors could affect how FP4 quantization error propagates through the ODE integration and impacts the final image's reward-relevant features.

The alignment fidelity gap in Table 6 provides indirect, circumstantial evidence. SD3.5-Large shows a −1.08% degradation under Sol-RL vs. the naive BF16 baseline, while FLUX.1 shows −0.29% and SANA shows +0.11%. One possible explanation for this pattern is that SD3.5-Large's proxy ranking is less reliable, causing worse seed selection in Stage 1 and thus a larger fidelity gap. But without direct ranking fidelity metrics for SD3.5-Large (the equivalent of Table 8 for that model), this remains speculation — the gap could equally be due to run-to-run variance, different convergence dynamics, or model-specific sensitivity to the selective training ratio.

What evidence exists in the paper. Table 4 provides IS and CLIPScore for all three models, confirming that NVFP4 preserves aggregate semantic quality across architectures. But IS and CLIPScore are distribution-level metrics computed over many images — they measure whether the FP4 model's outputs are on average as good as BF16, not whether the per-seed ranking is preserved. A model could have identical IS and CLIPScore distributions while shuffling the rank ordering of individual seeds (e.g., if FP4 systematically improves some seed categories and degrades others, leaving the mean unchanged). The ranking-specific metrics in Table 8 (τ, ρ, Top/Bottom-k match) are what directly validate the proxy's fitness for selective training, and these are provided for only one model.

Mitigation status. Not addressed. The paper does not acknowledge that the ranking fidelity analysis might not transfer across models, and does not recommend that practitioners validate proxy ranking on their specific model and reward combination before deploying Sol-RL. Given that the entire two-stage architecture depends on this property holding, the lack of per-model ranking validation is a significant gap — particularly for SD3.5-Large, where the fidelity gap is largest.

---

6.6 Hard Prompts Where the Base Model Produces Uniformly Low-Quality Images Receive No Benefit from Rollout Scaling — and the Method Provides No Mechanism for Detecting or Routing These Cases

The assumption or constraint. The extreme value analysis in Appendix A proves that the proxy ranking signal improves with scale N — but only if there is a genuine reward spread to discover. The analysis assumes rewards are sub-Gaussian with variance σ² > 0, meaning that different noise seeds produce meaningfully different image qualities. For prompts where the base model's pass@1 (to borrow the earlier paper's terminology) is near zero — i.e., essentially all generated images are of uniformly poor quality regardless of seed — the true reward range W*_N approaches zero, and the proxy ranking's 4Δ error penalty dominates. The paper's theoretical bound (Equation 10 of Appendix A) predicts: E[Ŵ] ≥ 2σ√(2 log N) − 4Δ. When σ ≈ 0 (no genuine quality variation across seeds), the bound becomes negative — the proxy ranking produces no useful signal, and the selected "contrastive" candidates are effectively random.

The consequence. For hard prompts — those where the base diffusion model fundamentally lacks the capability to produce semantically correct images (e.g., prompts requiring compositional reasoning, rare object combinations, or text rendering that the model has not learned) — Sol-RL's Stage 1 exploration provides no benefit. The FP4 proxy will select seeds that it believes are best and worst, but these selections are driven by quantization noise and reward model imperfections rather than genuine quality differences. Stage 2 will then regenerate these effectively random seeds in expensive BF16, producing training targets that are no more informative than randomly selected samples. The net effect: Sol-RL spends extra computation (the Stage 1 FP4 generation and reward scoring) without gaining any improvement in training signal quality — it is strictly worse than simply using a smaller N of randomly selected BF16 samples.

This limitation is not unique to Sol-RL — it applies to any rollout scaling method — but Sol-RL's decoupled architecture adds a specific failure mode. The naive BF16 baseline, when faced with a hard prompt, generates 96 BF16 images that are all poor, computes their rewards, and selects the "best" and "worst" — but at least those images are genuine BF16-quality samples, and the policy update uses them directly. Sol-RL, by contrast, introduces an additional error source: the FP4 proxy ranking may select seeds based on proxy reward differences that are purely noise, and then regenerates those seeds in BF16 at additional cost. The regeneration cost is wasted because the seeds were not genuinely contrastive.

The paper does not provide per-prompt difficulty analysis. Unlike the earlier technical paper (which binned MATH problems into five difficulty quintiles and showed that test-time compute provides zero benefit on the hardest bin), Sol-RL reports only aggregate metrics across the entire evaluation set. We do not know what fraction of prompts fall into the "hard" regime where no method helps, whether Sol-RL performs worse than the naive baseline on those prompts, or whether the aggregate improvements are driven primarily by "easy" prompts where the base model already produces decent images.

What evidence exists in the paper. None directly. The paper does not break down results by prompt difficulty, does not report per-prompt reward distributions, and does not analyze the relationship between σ (the true reward variance for a prompt) and Sol-RL's improvement over baselines. The theoretical analysis in Appendix A acknowledges that the benefit depends on 2σ√(2 log N) dominating 4Δ, but does not empirically measure σ for the evaluation prompts or identify conditions where σ is too small for the bound to be positive.

Mitigation status. Not addressed. The paper does not discuss hard-prompt degradation as a limitation, does not propose a mechanism for detecting prompts with low σ (which would enable falling back to a different strategy), and does not provide guidance on what prompt characteristics or reward model behaviors indicate that Sol-RL's overhead is justified vs. when a simpler approach would suffice. A practitioner deploying Sol-RL on a diverse prompt distribution — where some fraction of prompts are intrinsically difficult for the base model — has no way to know whether the method is actively harmful on those prompts or merely neutral.

---

6.7 The Paper Studies Only Reinforcement Learning for Image Generation — the Decoupled Architecture's Applicability to Other Diffusion RL Domains (Video, Audio, 3D) Is Unvalidated

The assumption or constraint. All experiments are on text-to-image generation with square aspect ratios (512×512 or 1024×1024) and standard diffusion sampling schedules. The paper's analysis of why FP4 proxy ranking works — the ODE determinism argument, the semantic-vs-texture decomposition in reward models, the Lipschitz continuity bounds — is specific to static image generation with relatively short denoising trajectories (6-10 steps). The paper does not experiment with video generation (which involves temporal consistency constraints), audio generation (different perceptual quality metrics), 3D asset generation (different rendering pipelines and reward models), or other diffusion-based generative tasks.

The consequence. The extension to other domains is not straightforward. In video generation, the ODE trajectory is substantially longer (often 50-100+ steps) and involves temporal dynamics that may be more sensitive to accumulated quantization error — each frame's error propagates to adjacent frames through temporal attention. The Lipschitz constant L_v in the error bound (Equation 6 of Appendix A) could be larger for video models due to the temporal coupling, and the exponential factor e^(L_v T) could amplify FP4 perturbations beyond the point where semantic structure is preserved. In audio generation, reward models operate on spectrograms or auditory features that may be more sensitive to high-frequency quantization artifacts than vision-based reward models.

Even within image generation, the paper's validation is limited to a specific prompt distribution (PickScore dataset) with a specific prompt style (concise, descriptive captions from web-scale interaction data). Prompts with different characteristics — very long, highly compositional, requiring precise spatial relationships, involving text rendering — may show different proxy ranking fidelity because the reward models' sensitivity to semantic structure varies with prompt type.

What evidence exists in the paper. None beyond the three image generation models tested. The paper makes no claims about other modalities or prompt distributions. The theoretical analysis in Appendix A is general enough to apply to any ODE-based generative model with Lipschitz-continuous reward functions, but the constants (L_v, L_R, Δ) are domain-specific and unmeasured for anything beyond the tested image models.

Mitigation status. Not addressed. The paper does not discuss domain generalization as a limitation, does not identify which properties of text-to-image generation are essential for the proxy ranking to work, and does not provide guidance for practitioners seeking to adapt Sol-RL to other diffusion-based generative tasks. This is a standard scope limitation (papers are not expected to validate on every possible domain) but is consequential because "Diffusion RL" in the paper's title and framing implies a generality that the experiments do not establish. A practitioner working on video alignment or audio generation cannot assume the proxy ranking property holds without independent validation.

7. Implications and Future Directions

How This Work Changes the Landscape

Sol-RL introduces a conceptual reframing rather than a paradigm shift: it changes how the field thinks about precision allocation in reinforcement learning pipelines, not what reinforcement learning itself is or does. The magnitude is best described as a diagnostic intervention — the paper identifies a structural confusion in prior work (conflating exploration and optimization precision requirements) and demonstrates that disentangling them yields practical benefits, but the underlying RL algorithms, quantization techniques, and hardware primitives remain unchanged.

The specific reframing is this: prior work on quantized RL — FlashRL, QeRL, QuRL, FP8-RL, Jet-RL — operated under the implicit assumption that the precision used for generating rollouts must be compatible with (or corrected toward) the precision used for gradient computation. This assumption followed naturally from the on-policy nature of policy gradient methods: the expectation in the gradient estimator is taken over the policy's own sampling distribution, and changing the sampling distribution (via quantization) creates an off-policy gap that must be corrected. Sol-RL's contribution is to point out that this assumption is contingent on what the rollouts are used for, not on any fundamental property of RL. When rollouts serve as direct optimization targets (as in AWM's and DiffusionNFT's denoising score matching objectives), precision compatibility is indeed mandatory — the paper's Figure 3b empirically confirms that violation causes catastrophic degradation. But when rollouts serve only to filter candidates (i.e., to estimate relative ranking for selective training), precision compatibility becomes largely irrelevant as long as relative ordering is preserved. The proxy ranking analysis (Table 8, Appendix A) proves that this ordering survives FP4 quantization with high fidelity, and the two-stage architecture exploits this asymmetry by structurally separating the precision-tolerant phase (exploration) from the precision-critical phase (optimization).

This reframing has several downstream effects on how the field should think about system design for RL:

1. It closes the door on "unified precision" as the only viable path. Jet-RL (Xi et al., 2026) advocated for using the same FP8 precision for both training and rollout, fundamentally eliminating the off-policy gap. This approach is clean and theoretically sound, but it forces the entire pipeline — including the expensive backward pass and weight update — into reduced precision, which may limit the achievable model quality even if convergence is stable. Sol-RL demonstrates an alternative: rather than unifying precision, partition the pipeline by function. The backward pass and weight update remain in BF16 where precision matters; only the forward exploration pass is quantized where throughput matters and precision is tolerant. This makes Jet-RL's approach look unnecessarily conservative for settings where selective training is already in use — you don't need to accept BF16-quality rollouts for samples that will be discarded, and you don't need to degrade the optimizer to FP8 when only the exploration phase is the bottleneck.

2. It makes the efficiency gains from quantization structural rather than incremental. Prior quantization approaches to RL offered a tradeoff curve: more aggressive quantization → more speedup but more degradation. The contribution was to push the curve outward (better speedup for the same degradation, or less degradation for the same speedup). Sol-RL changes the nature of the tradeoff by making the degradation conditional on the function of the quantized computation. The Stage 1 FP4 exploration can use extremely aggressive quantization (4-bit, 60% of standard denoising steps) because its error never reaches the policy gradient — it only affects which seeds get selected, and the extreme value analysis proves this selection error becomes asymptotically negligible at large N. This is not "better quantization" in the traditional sense; it is removing quantization from the optimization critical path entirely. The practical implication is that future work on quantized RL should not focus on making quantized rollouts better training targets (a fundamentally difficult problem due to the regression-to-corrupted-targets issue) but on making quantized rollouts better filters (which is a much more tractable problem, as the ranking correlation metrics in Table 8 demonstrate).

3. It reconciles two conflicting observations in the diffusion RL literature. On one hand, rollout scaling consistently improves alignment (DanceGRPO, BroRL) — more candidates → better policy updates. On the other hand, quantized rollouts consistently degrade alignment (Yao et al., 2025; Xi et al., 2026) — lower precision → worse policy updates. These observations appeared contradictory because the natural synthesis — "quantize to scale rollouts" — fails empirically (Figure 3b). Sol-RL provides the resolution: the contradiction is an artifact of assuming that the same precision must serve both purposes. Rollout scaling improves alignment through better exploration (wider reward ranges, more stable advantage estimates); quantization degrades alignment through corrupted targets (the regression objective mimics artifacts). These mechanisms are separable — you can have exploration at scale (via FP4) without using quantized targets for optimization (via BF16 regeneration). The decoupling transforms the contradiction from a deadlock into a design opportunity.

4. It redirects research attention from RL algorithm design to pipeline architecture design. The recent diffusion RL literature has been heavily focused on algorithmic innovations: better policy gradient estimators (AWM's ELBO formulation), better advantage computation (GRPO's group-relative normalization), better sampling strategies (BranchGRPO's trajectory branching, TreeGRPO's structured advantages). Sol-RL's results suggest that at least for the current generation of models and reward functions, the bottleneck is not algorithmic — it is computational. The 3.9× convergence speedup on FLUX.1 HPSv2 (Figure 4) dwarfs the algorithmic improvements from switching between AWM, DiffusionNFT, FlowGRPO, and DanceGRPO (Table 1 shows these methods cluster within ~0.02 HPSv2 of each other). This implies that the highest-ROI investment for improving diffusion RL in practice is not devising new loss functions but redesigning the training pipeline to better utilize heterogeneous compute resources. The paper makes this point implicitly by demonstrating that a simple architectural change (two-stage decoupling) applied to an existing algorithm (DiffusionNFT) yields larger gains than switching between state-of-the-art algorithms.

5. It establishes a new research axis: the precision-safety boundary for different RL sub-tasks. The paper implicitly draws a line: ranking is precision-safe; regression to pixel-level targets is not. But this is a first cut — there are other sub-tasks in the RL pipeline (reward model inference, advantage normalization, KL divergence computation, EMA weight updates) whose precision requirements are unknown. The paper's methodology — measure degradation when a sub-task is quantized, characterize the error mode, determine whether it is structural or correctable — provides a template for mapping the precision-safety landscape of the full RL stack. Future work could extend this mapping to identify additional phases that can be quantized without regeneration.

Follow-Up Research This Work Enables

Precision-safety mapping for the full diffusion RL stack. The paper establishes that exploration (reward ranking) is precision-safe while optimization targets (regression to generated images) are not. But the RL pipeline contains other phases whose precision requirements are uncharacterized: reward model inference (currently run in standard precision, but could be quantized if ranking fidelity is maintained), advantage normalization (computing group mean and standard deviation — trivial arithmetic, but could the normalization be corrupted by quantized reward values?), KL divergence computation (comparing policy distributions — does the log-probability computation degrade under quantization?), and EMA weight updates (accumulating floating-point averages — does low-precision accumulation cause drift?). A natural follow-up would systematically quantize each phase of the DiffusionNFT training loop independently, measure the degradation, and construct a precision-safety map analogous to the exploration-vs-optimization boundary identified in this paper. The methodology is established: for each phase, measure a relevant fidelity metric (ranking correlation for exploration, downstream HPSv2 for optimization targets, KL divergence accuracy for regularization, etc.) under FP4, FP8, and BF16, and determine whether the degradation is structural (requires regeneration) or statistical (could be corrected via importance weighting). This would produce a design handbook for precision-heterogeneous RL systems, with Sol-RL's two-stage architecture as the first entry.

Can the proxy ranking be performed by a smaller, distilled model rather than the quantized full model? Sol-RL uses the quantized same model for exploration — the full FLUX.1 at FP4 — because the hardware supports NVFP4 and the throughput advantage is substantial. But this introduces the weight re-quantization overhead after every policy update (Limitation 6.1). An alternative is to use a permanently-quantized smaller model (e.g., a distilled 600M-parameter student) for the Stage 1 exploration, eliminating the need to re-quantize the main policy's weights entirely. The key question is whether a distilled model's noise-to-reward mapping is sufficiently correlated with the full model's mapping to serve as a proxy ranker. A concrete experiment: train a lightweight reward-predicting head on top of a frozen, quantized student model, measure the Kendall τ and Top-12 match rate between the student's proxy rankings and the teacher's BF16 rankings (the equivalent of Table 8 for a cross-model setting), and determine whether the ranking fidelity crosses the threshold (τ > 0.7, Top-12 match > 90%) needed for effective selective training. If it does, this would eliminate re-quantization overhead, decouple the exploration model's compute from the policy model's size, and potentially enable exploration on cheaper hardware entirely. If it fails, the failure mode (is the ranking correlation uniformly low, or does it degrade specifically at the extremes?) would clarify whether the proxy ranking property depends on weight-space proximity between exploration and optimization models.

Stress-testing the proxy ranking property on prompts with low reward variance. The extreme value analysis in Appendix A proves that the FP4 ranking signal dominates quantization noise only when 2σ√(2 log N) ≫ 4Δ, where σ² is the true reward variance across seeds for a given prompt. The paper does not measure σ empirically or analyze performance conditional on σ. A critical stress-test would: (1) run Sol-RL and the naive BF16 baseline, (2) for each prompt in the evaluation set, compute σ² (the variance of BF16 rewards across, say, 100 seeds), (3) bin prompts by σ quintile, and (4) measure the alignment fidelity gap (Table 6) within each bin. The prediction from the theory is that for low-σ prompts (σ ≈ 0), Sol-RL should underperform the naive baseline because the Stage 1 FP4 cost is wasted — the proxy ranking is selecting effectively random seeds, and the regeneration cost produces no benefit over random selection. For high-σ prompts, Sol-RL should match or exceed the baseline. This experiment would establish boundary conditions for when Sol-RL is beneficial: if the deployment prompt distribution has a heavy tail of low-σ prompts (e.g., prompts where the base model consistently fails regardless of seed), the method's aggregate improvement could mask per-prompt degradation. If the degradation on low-σ prompts is substantial, the experiment would motivate a gating mechanism: use a quick variance estimate (e.g., generate 8 FP4 samples, compute the proxy reward standard deviation, and if it falls below a threshold, skip Stage 2 regeneration and fall back to random sampling or a standard BF16 rollout).

Scaling N to 1000+ with cheaper proxies to test the extreme value asymptotics. The paper shows monotonic improvement in HPSv2 as N scales from 24 to 96 (Table 3), and the theory predicts continued improvement as √(2 log N) grows (albeit slowly). But at what N does the benefit saturate — either because the true reward distribution has finite support (there is a genuine maximum-quality image for each prompt, and once N is large enough to reliably find it, further scaling adds nothing) or because the proxy ranking error Δ becomes the dominant term? A concrete experiment: scale N to 256, 512, 1024 while keeping K=24 fixed, measuring both the HPSv2 score and the Top-12 match rate between FP4 proxy and BF16 ground truth at each N. If the Top-12 match rate degrades at large N (because the FP4 ranking noise causes more false selections among a larger pool), this would identify a scale limit to the decoupling approach — a point beyond which more aggressive quantization (or a better proxy) is needed to maintain ranking fidelity. If the match rate remains stable (as the theory predicts, since Δ is constant while the signal grows), this would validate the asymptotic argument and justify pushing N much higher, potentially enabling exploration pools of thousands of candidates per prompt with minimal Stage 2 cost.

Combining decoupled exploration with structured sampling strategies. Sol-RL uses independent random sampling for the N=96 Stage 1 candidates — each seed is drawn i.i.d. from the Gaussian prior, with no attempt to coordinate exploration across seeds. Recent work on structured exploration in diffusion RL — BranchGRPO (Li et al., 2025), TreeGRPO (Ding & Ye, 2025), Expand-and-Prune (Ge et al., 2025) — proposes trajectory branching, tree-structured advantage computation, and diversity-maximizing pruning to improve the efficiency of exploration per generated sample. These methods are orthogonal to precision: they change which seeds are sampled, not the precision at which they are generated. A natural synthesis would apply structured sampling in Stage 1 (e.g., generate a tree of 96 samples with branching, rather than 96 independent samples) at FP4 precision, use the structured relationships to improve ranking fidelity (e.g., advantage estimates that account for shared prefixes), and then regenerate only the selected leaves in BF16. The key question is whether structured sampling improves proxy ranking fidelity — does the tree structure provide additional signal that compensates for FP4 noise, leading to higher Top-K match rates at the same N? A concrete experiment: implement TreeGRPO's sampling strategy in the Stage 1 FP4 engine, measure the Kendall τ between FP4 proxy rankings and BF16 ground truth (with and without tree structure), and compare the HPSv2 convergence curves against both Sol-RL (independent sampling) and TreeGRPO (full BF16). This would test whether precision reduction and structured exploration are complementary or redundant.

Online difficulty estimation for adaptive regeneration ratios. Limitation 6.6 identifies that hard prompts (where the base model's reward variance σ² is near zero) receive no benefit from rollout scaling — Sol-RL wastes Stage 1 computation on these prompts. An adaptive variant of Sol-RL would estimate σ² from the Stage 1 FP4 rewards themselves (the standard deviation of the 96 proxy rewards) and use this estimate to dynamically adjust the Stage 2 regeneration budget per prompt. For prompts with high proxy reward variance, the system would regenerate the full K=24 contrastive candidates (the current behavior). For prompts with low proxy variance, the system would skip regeneration entirely for that prompt in this iteration (saving Stage 2 cost) or regenerate only a minimal set (e.g., K=4) for gradient computation. The key implementation challenge is ensuring that the proxy reward variance is a reliable estimator of true reward variance — the paper's FP4-BF16 ranking correlation (Table 8) measures ordinal consistency, not variance preservation. A concrete experiment: generate 96 FP4 and 96 BF16 samples for each evaluation prompt, compute the FP4 reward variance and BF16 reward variance, measure their correlation (e.g., Pearson r on the log-variance), and determine whether a threshold on FP4 variance reliably separates prompts where Sol-RL helps from prompts where it doesn't. If the correlation is strong, implement the adaptive policy and measure whether it improves GPU-hour efficiency over fixed Sol-RL (by eliminating wasted regeneration on low-σ prompts) without degrading alignment on high-σ prompts.

Practical Applications and Downstream Use Cases

1. Cost-efficient post-training of large diffusion models for product image generation. Consider a company that has licensed a foundation diffusion model (e.g., FLUX.1) and needs to fine-tune it on their specific product catalog — thousands of SKUs with diverse visual attributes — to improve prompt adherence and aesthetic quality. The naive approach (DiffusionNFT with 96-sample rollouts on 8 GPUs) consumes significant compute: for FLUX.1 at 512×512, Table 5 reports 274 seconds per training iteration, and convergence to full alignment quality (Figure 4) requires roughly 100 GPU-hours. At cloud GPU prices (B200-equivalent instances), this is a substantial line item that recurs with every model update (new products, seasonal refreshes). Sol-RL reduces this to ~62 GPU-hours (1.62× end-to-end speedup, Table 5) while maintaining or slightly improving final alignment quality (Table 6: −0.29% HPSv2 gap for FLUX.1). For a team running weekly model updates, this is a direct cost savings of ~38% with no quality compromise. The integration path is straightforward: the two-stage pipeline is compatible with standard RL training infrastructure (the policy update uses the same DiffusionNFT objective), requires no additional data or reward model changes, and leverages the existing NVIDIA Transformer Engine quantization toolchain.

2. Democratizing alignment research for academic labs and small teams. The computational bottleneck that Sol-RL addresses is not just an industry concern — it gates who can participate in diffusion RL research. A typical academic lab with access to 4-8 consumer GPUs (e.g., RTX 4090s, which lack native FP4 tensor core support entirely) cannot feasibly run 96-sample rollouts with a 12B-parameter model for thousands of iterations; the wall-clock time would stretch from hours to days or weeks. Sol-RL's two-stage architecture, if adapted to a supported quantization format on consumer hardware (INT4 via bitsandbytes or GPTQ, or FP8 on Ada/Ampere architectures), would bring the per-iteration cost into a range where smaller-scale experimentation is possible. The paper does not provide this adaptation itself (Limitation 6.2), but the conceptual framework — decouple exploration from optimization, use low-precision for the former, high-precision for the latter — is hardware-agnostic. A direct follow-up would measure the proxy ranking fidelity of INT4-quantized FLUX.1 on an RTX 4090 (equivalent to Table 8 but for INT4) and, if the ranking correlation crosses the viability threshold, implement the two-stage pipeline using PyTorch's native quantization APIs. This would lower the barrier to entry for diffusion RL research by making the most computationally intensive phase (exploration) accessible on commodity hardware.

3. Accelerating the RL iteration cycle during reward model development and hyperparameter tuning. In practice, deploying diffusion RL involves substantial experimentation: testing different reward models (ImageReward vs. HPSv2 vs. PickScore vs. custom objectives), tuning hyperparameters (KL penalty β, learning rate, LoRA rank, selective training ratio K/N), and debugging training dynamics (collapse, reward hacking, mode dropping). Each experiment requires a full training run to convergence — at 100 GPU-hours per run for FLUX.1, a researcher might complete only 2-3 experiments per day on an 8-GPU cluster, and comprehensive sweeps are infeasible. Sol-RL's 1.6× end-to-end speedup translates directly to 1.6× more experiments per day — from ~2.4 to ~3.8 full training runs — without changing the experimental protocol. More significantly, if the convergence speedup is closer to the headline 4.64× (achieved for specific model-metric combinations in Figure 4), the iteration throughput improvement is even larger. For a team developing a custom reward model for a specific vertical (e.g., medical illustration accuracy, architectural visualization fidelity), the ability to test 4-5 candidate reward models per day instead of 2 is the difference between a one-week and two-week development cycle. The integration requires no changes to the RL training code beyond the rollout pipeline configuration (Table 7), making it a drop-in efficiency improvement for existing workflows.

4. Enabling larger-scale exploration for discovering rare high-quality outputs in creative applications. The paper's scaling analysis (Table 3) shows monotonic improvement as the exploration pool N increases from 24 to 96, and the extreme value theory (Appendix A) predicts continued gains at larger N. For applications where the goal is not just average alignment improvement but the discovery of exceptional individual outputs — a concept artist generating the single best variation of a creature design, an advertising team selecting the most compelling image from thousands of candidates — larger exploration pools are directly valuable. Sol-RL's decoupled architecture makes N=500 or N=1000 exploration pools feasible by generating the bulk of candidates in cheap FP4 and reserving BF16 only for the most promising few. A concrete deployment: an interactive tool where a user enters a prompt, the system generates 1000 FP4 candidates in ~5 seconds (exploiting the 4× TFLOPs advantage and reduced denoising steps), displays the top-20 ranked by proxy reward, and then regenerates the user's 4 selected candidates in full BF16 quality. This workflow would be impossible with naive BF16 scaling (generating 1000 BF16 candidates would take minutes and cost dollars per prompt) but becomes interactive with Sol-RL's approach. The paper does not demonstrate this use case directly (all experiments are in the training setting, not inference-time scaling), but the proxy ranking property validated in Table 8 and Figure 3c is equally applicable to inference-time candidate filtering — the ranking fidelity does not depend on whether the samples are being used for training or for user selection.