ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation

ArXiv: 2304.05977

🎯 Pitch

A reward model trained on only 137k human comparisons can predict text-to-image preferences dramatically better than CLIP or aesthetic scorers, and its score only becomes meaningful after denoising step 30—a finding that unlocks direct gradient-based tuning of diffusion models. This ReFL algorithm beats dataset filtering and loss reweighting by a wide margin, proving that plugging a learned human-preference signal into the final denoising steps is a simple yet powerful alignment strategy.

---

1. Executive Summary

This paper introduces and releases ImageReward, the first general-purpose text-to-image human preference reward model, trained on a systematically collected dataset of 137k expert comparison pairs derived from real user prompts and corresponding model outputs. Using ImageReward's feedback, the authors further propose Reward Feedback Learning (ReFL), a direct fine-tuning algorithm for optimizing latent diffusion models — specifically Stable Diffusion v1.4 — by backpropagating gradients through the reward model's score at randomly selected latter denoising steps (steps 30–40 out of 40), exploiting the key empirical insight that ImageReward scores become reliably distinguishable only after roughly 30 denoising steps. ImageReward outperforms existing text-image scoring methods such as CLIP (by 38.6%), Aesthetic (by 39.6%), and BLIP (by 31.6%) in human preference prediction accuracy, achieving 65.14% pairwise accuracy, while ReFL-tuned models achieve a 58.79% win rate against the baseline in human evaluation — substantially outperforming indirect data-oriented alternatives like dataset filtering and reward-weighted loss reweighting — establishing that human preference feedback can be directly and effectively incorporated into diffusion model optimization through gradient signals from a learned reward model, though the approach is most effective when the reward model's score distribution becomes meaningful only in the final stages of the denoising process.

2. Context and Motivation

The Core Problem: Text-to-Image Models Don't Naturally Align with What Humans Want

The fundamental problem this paper addresses is deceptively simple: state-of-the-art text-to-image generative models, despite their impressive capabilities, systematically fail to produce outputs that align with human preferences. This misalignment manifests across multiple dimensions that the authors carefully catalog in Section 1, and the gap exists because the pre-training objectives of these models (learning to reconstruct the data distribution) differ fundamentally from what humans actually care about when they use these systems.

This matters enormously for both practical and societal reasons. On the practical side, a model that can generate technically plausible images but fails to accurately depict what the user asked for, or produces distorted body parts, or generates aesthetically unpleasing outputs, is a model that requires extensive trial-and-error prompting and cherry-picking to be useful. The "generate-then-filter" paradigm — produce many images and pick the best one — is computationally wasteful and frustrating for users. On the societal side, models that produce toxic, biased, or otherwise harmful content when prompted in certain ways pose genuine risks that cannot be addressed through architectural improvements or larger pre-training datasets alone.

The paper identifies four specific, well-documented failure modes that structured pre-training cannot resolve:

• Text-image Alignment failures: The model fails to accurately depict all the numbers, attributes, properties, and relationships described in the prompt. For example, Figure 1(a)–(b) shows images where the semantic content of the prompt is only partially rendered. This isn't about the model lacking the ability to draw the requested objects — it's about the model not reliably prioritizing faithful text grounding in its generation process.

• Body Problems: Humans and animals appear with distorted, incomplete, duplicated, or anatomically impossible body parts, as shown in Figure 1(e)–(f). This is a particularly salient failure mode because humans are highly sensitive to body structure, and even technically sophisticated images are rendered unusable by obvious anatomical errors. The authors note an average body problem frequency of 21.14% across all prompt categories in their annotation data (Figure 11, Appendix A.4).

• Human Aesthetic deviations: Generated images diverge from mainstream human aesthetic preferences in ways that are hard to specify algorithmically but immediately recognizable to viewers. Figure 1(c)–(d) illustrates this — the images are technically competent but aesthetically off in ways that matter to users.

• Toxicity and Biases: The model can produce harmful, violent, sexual, discriminatory, or psychologically disturbing content. This is not purely a content filtering problem at the prompt level — even seemingly innocuous prompts can trigger problematic generations due to spurious correlations in training data.

The crucial insight here is that none of these problems can be solved solely through improvements in model architecture or pre-training data scale. The pre-training distribution is inherently noisy and differs from actual user-prompt distributions, and the self-supervised objective (denoising score matching or next-token prediction) optimizes for distributional fidelity rather than human preference alignment. As the authors put it:

"These prevalent challenges, however, are difficult to address solely through improvements in model architectures and pre-training data."

The NLP Precedent: RLHF Works for Language, But Not Directly for Images

The paper draws explicit inspiration from the success of Reinforcement Learning from Human Feedback (RLHF) in natural language processing. In NLP, researchers developed a now well-established pipeline: collect human preference judgments on language model outputs (e.g., "which of these two summaries is better?"), train a reward model to predict those preferences, and then use reinforcement learning (typically PPO) to fine-tune the language model to maximize the reward model's score. This approach, pioneered in works on summarization and later scaled to instruction-following in InstructGPT, has proven remarkably effective at aligning language models with human intent — making them more helpful, truthful, and harmless.

The authors see a direct parallel for text-to-image generation. If we could learn a reward model that captures human preferences for generated images, we could use it to guide image generation models toward outputs that people actually want. The problem is that the RLHF recipe does not transfer directly to diffusion models for a fundamental technical reason that the paper identifies in Section 3:

"Unlike language models, latent diffusion models' multi-step denoising generation cannot yield likelihoods for their generations, and thus fail to adopt the same RLHF approaches."

In language models, you can compute the log-probability of generating a particular token sequence — this likelihood is what PPO and similar algorithms use to compute policy gradients and update the model. Latent diffusion models (LDMs), by contrast, generate images through an iterative denoising process that does not provide tractable likelihoods for the final image. You cannot simply "score" a completed image under the generative model and backpropagate through the generation process — the chain of denoising steps is not a differentiable function with respect to the model parameters in the same way that autoregressive language generation is.

Where Existing Approaches Fall Short

Before this work, the few attempts to incorporate human feedback into text-to-image generation fell into two categories, both of which the paper argues are fundamentally limited:

1. Inference-only approaches: classifier guidance. One could conceivably use a trained reward model as a classifier during inference, following the classifier-guidance paradigm introduced by Song et al. and Dhariwal and Nichol. The idea would be: at each denoising step, compute the gradient of the reward model's score with respect to the image and use it to steer the generation. However, this approach has a fatal mismatch: reward models are trained on fully denoised images (because human annotators need to see complete images to express preferences), while classifier guidance requires a classifier trained on noisy intermediate latents — the noised images at various steps of the diffusion process. Training a reward model on noisy images would require humans to annotate them, which is practically impossible because the semantic content of noisy latents is unintelligible. The paper explicitly notes this contradiction:

"It is for inference only and employs a classifier necessarily trained on noisy intermediate latents, which naturally contradicts RMs' annotation where images need to be completely denoised for humans to mark correct preference."

2. Indirect data-oriented approaches. The concurrent work the paper identifies falls into two subcategories:

• Dataset Filtering: This approach, exemplified by concurrent work from Wu et al., uses a reward model to score generated images, then constructs a filtered training dataset by keeping high-scoring images and discarding low-scoring ones. The model is then fine-tuned on this curated dataset using standard diffusion loss. The problem, as the authors argue, is that this is indirect: the reward model's signal influences the training data but not the optimization process itself. The model never receives gradient information about why certain images are preferred — it just sees more examples of "good" images and fewer "bad" ones. This limits how precisely the model can adapt to human preferences and couples the approach tightly to the quality and distribution of the filtering dataset.

• Reward Weighted Loss: This approach, from Lee et al., modifies the training loss by weighting each training sample's contribution according to the reward model's score. High-quality images receive larger loss weights; low-quality ones receive smaller weights. While this seems more direct than dataset filtering, the paper identifies a subtle problem: the reweighting coefficients are constrained to [0, 1], meaning that non-preferred images still contribute to the loss — they're just downweighted rather than actively penalized. As the paper explains:

"This implies that while preferred images are given larger weights and poor images are given smaller weights, the influence of the non-preferred images is not completely eliminated... the failure to eliminate the impact of non-preferred images hinders the effectiveness of the Reward Weighted method."

Both of these approaches share a common limitation: they operate at the data level rather than the gradient level. They decide what the model sees but not how the model should respond to what it sees in an optimization-theoretic sense. The model is still minimizing the standard diffusion loss (MSE on noise prediction), just on a reweighted or filtered dataset. This means the optimization signal is only weakly coupled to the reward model's preferences.

3. No suitable reward model existed. Before ImageReward, there was no general-purpose text-to-image reward model trained on diverse real-user prompts with rigorous annotation standards. Existing scoring methods had fundamental limitations:

• CLIP score computes cosine similarity between text and image embeddings using a dual-encoder architecture. It captures some notion of text-image alignment but is insensitive to image fidelity, aesthetic quality, body correctness, and toxicity. The paper shows in Table 1 that CLIP's ranking of six popular text-to-image models has a Spearman correlation of only 0.77 with human rankings on real user prompts, and near-zero correlation (0.09) when evaluated on MS-COCO.

• Aesthetic predictors, such as the one released by LAION, are trained to predict human aesthetic ratings of photographs — not of AI-generated images and not in the context of text prompts. They capture some notion of visual quality but are oblivious to text alignment and toxicity.

• FID (Fréchet Inception Distance) measures distributional distance between generated images and reference real images (typically from MS-COCO). The paper identifies two critical limitations that make FID unsuitable as a human preference metric: (1) it requires reference real images and can only evaluate at the dataset level, not per-image, meaning it cannot serve as a selector for individual generations; (2) it measures averageness — how close generated images are to the reference distribution — rather than quality or preference. An image that is technically excellent but stylistically unusual can be penalized by FID. Moreover, many modern text-to-image models may have been inadvertently trained on MS-COCO images, making zero-shot FID comparisons potentially unfair due to data leakage.

The concurrent emergence of reward models like HPS and PickScore (which the paper compares against in Appendix D) partially filled this gap, but ImageReward's design emphasizes a rigorous annotation pipeline with explicit quality criteria (alignment, fidelity, harmlessness), a diverse real-user prompt distribution, and systematic annotation management — elements the authors argue are critical for a reward model to serve as a reliable optimization target.

The Broader Vision: Closing the Human Preference Gap in Generative Models

The paper positions itself at the intersection of two major research trends. The first is the rapid advancement of text-to-image generation capabilities — models like DALL-E, CogView, Stable Diffusion, and their successors have made high-quality image synthesis widely accessible. The second is the growing recognition across AI that raw capability without alignment is insufficient and potentially dangerous. The RLHF paradigm demonstrated this for language; this paper argues that the same philosophy must be extended to generative vision.

The stakes are substantial. As the authors note in their annotation analysis, body problems affect over 21% of generated images on average, and toxic content appears across multiple prompt categories. These are not edge cases — they are systematic failures that affect real users. A reward model that can reliably distinguish preferred from non-preferred images, and an optimization method that can use that signal to improve the generator, would address these failures at their source rather than patching them with post-hoc filtering.

The paper's title — "Learning and Evaluating Human Preferences for Text-to-Image Generation" — reflects this dual ambition. ImageReward is designed to serve both as an evaluation metric (replacing or supplementing CLIP score and FID with a metric that actually tracks what humans care about) and as a training signal (enabling direct optimization of diffusion models toward human-preferred outputs). The evaluation function is important in its own right because the field has lacked reliable automatic metrics that correlate with human judgment, making it difficult to compare models or even to select the best output from a batch of generations without manual inspection.

How This Paper Positions Itself

The paper's contributions are presented as filling a specific sequence of gaps that together enable the full RLHF pipeline for text-to-image generation:

1. No suitable dataset of human preferences existed → The paper designs and executes a systematic annotation pipeline, collecting 137k expert comparisons across 8,878 prompts from real users, with explicit criteria covering alignment, fidelity, and harmlessness, and with quality controls including inspector double-checking.

2. No reward model trained on such data existed → ImageReward is trained using a Bradley-Terry preference model with a BLIP backbone, achieving 65.14% pairwise accuracy against human judgments.

3. No way to use such a reward model for direct diffusion optimization existed → ReFL provides a method that backpropagates through the reward model at latter denoising steps where scores become reliable, circumventing the likelihood-free nature of diffusion models.

4. No empirical demonstration of direct reward feedback learning for diffusion models existed → The paper shows ReFL outperforms indirect baselines (dataset filtering, reward weighting, RAFT) in human evaluations.

The paper explicitly draws a contrast with prior work that either (a) attempted to use reward models only at inference time, (b) used indirect data-level methods rather than direct gradient-level optimization, or (c) relied on synthetic template-based prompts rather than real user distributions. The claim is that ImageReward + ReFL together represent the first end-to-end demonstration of direct human-preference-based optimization for text-to-image diffusion models, analogous to what RLHF achieved for language models but adapted to the fundamentally different technical constraints of diffusion.

This positioning is important because it frames the work not as a marginal improvement over existing scoring methods but as an enabling contribution that opens up a new research direction — one where human preference feedback can be directly incorporated into the training loop of generative vision models, potentially leading to systematic improvements in alignment, fidelity, and safety that cannot be achieved through architecture or data scaling alone.

3. Technical Approach

3.1 Reader Orientation

This paper builds two interconnected systems: first, a reward model (ImageReward) that learns to score text-image pairs according to human preferences, trained on a carefully collected dataset of expert comparisons; and second, a direct optimization algorithm (ReFL) that fine-tunes a latent diffusion model (Stable Diffusion) by backpropagating gradients through the reward model's score to maximize human preference, exploiting the empirical finding that the reward model's assessments become reliable only in the final denoising steps. The overall "shape" of the solution is: annotate human preferences at scale → train a model to predict those preferences → use that model as a differentiable critic to directly guide the generator's parameters toward outputs that humans prefer, bypassing the likelihood-free limitation of diffusion models.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of four major components connected in a pipeline, depicted conceptually in Figure 2 of the paper:

1. Data Collection Pipeline (top of Figure 2): Prompts are sampled from DiffusionDB (a large dataset of real user prompts) using a graph-based diversity selection algorithm. For each selected prompt, 4–9 images generated by existing models are collected. Expert annotators then rate each image on alignment, fidelity, and harmlessness, and rank the images from best to worst per prompt. This produces comparison pairs — for every pair $(x_i, x_j)$ from the same prompt, the annotation tells us which image is preferred.

2. ImageReward Model (middle-left of Figure 2): A neural network built on a BLIP backbone that takes a prompt $T$ and an image $x$ as input and outputs a single scalar score $f_\theta(T, x)$ representing how human-preferred that image is for that prompt. The model is trained to maximize the probability that its scores correctly order the annotated comparison pairs, using a Bradley-Terry preference loss formulated as binary classification.

3. ReFL Optimization Loop (bottom of Figure 2): Starting from a pre-trained latent diffusion model (Stable Diffusion v1.4), the algorithm performs iterative fine-tuning where each training step has two phases: (a) pre-training loss on real text-image pairs (standard diffusion denoising objective, keeping the model grounded in realistic image generation), and (b) reward feedback loss on generated images from user prompts, where the reward model scores a partially denoised image and gradients flow back through the score into the diffusion model's parameters.

4. Inference and Evaluation: The fine-tuned model generates images in the standard way (progressive denoising, classifier-free guidance). ImageReward can also be used standalone as an evaluation metric — to rank models by their generations' quality or to select the best image from multiple candidates for a given prompt.

Information flows: Real user prompts → sampling from multiple text-to-image models → human annotation (rating + ranking) → preference pair extraction → ImageReward training (scalar prediction with ranking loss) → ReFL training (ImageReward scores generation at latter denoising steps, gradients update LDM parameters). At deployment: either fine-tuned model generates directly, or base model generates many candidates and ImageReward selects the best.

3.3 Roadmap for the Deep Dive

• First, the annotation pipeline and dataset construction — because the reward model's quality depends entirely on the quality and coverage of the human preference data it learns from. Understanding the annotation design (criteria, trade-offs, quality control) explains why ImageReward generalizes better than simpler scoring methods.

• Second, the ImageReward model architecture and training — including the Bradley-Terry loss formulation, backbone choice (BLIP vs. CLIP), overfitting mitigation strategies, and the specific hyperparameter sweep that produced the final model. This is the "learned critic" that both evaluates and provides gradients.

• Third, ImageReward as an automatic evaluation metric — before using it for optimization, the paper validates that ImageReward actually correlates with human judgments at the model-ranking level and the per-image selection level. The metric properties (distinguishability, human alignment) are what make ReFL possible — if the reward signal were noisy, optimization would fail.

• Fourth, the ReFL algorithm — including the critical empirical insight about when ImageReward scores become reliable during denoising, the random step selection mechanism, the joint pre-training + reward loss formulation, and the specific implementation details (hyperparameters, dataset composition, training configuration). This is the core technical contribution that distinguishes this work from indirect data-oriented approaches.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core contributions are (1) a carefully designed human preference dataset and the reward model trained on it, and (2) a novel algorithm (ReFL) for directly optimizing diffusion models against a learned reward function, enabled by the empirical observation that the reward model's scores become reliable discriminators only in the final stages of denoising.

---

The Annotation Pipeline: Why It Matters and How It Works

The quality of any learned reward model is bounded by the quality of the preference data it's trained on. If the annotations are noisy, inconsistent, or unrepresentative of real user needs, the reward model will be noisy, inconsistent, or misaligned — and using it for generator optimization will amplify those flaws. The paper devotes substantial effort (Section 2.1 and Appendix A) to designing an annotation pipeline that addresses the specific challenges of text-to-image preference collection, which differ significantly from the text-only preference annotation pioneered in RLHF for language models.

Prompt Selection: Ensuring Real-World Representativeness

The starting point is DiffusionDB, a publicly available dataset containing approximately 1.8 million real user prompts collected from the Stable Diffusion Discord community. This is important because the distribution of prompts that actual users type into text-to-image systems differs substantially from the clean, well-specified captions in academic datasets like MS-COCO. Real users use cryptic abbreviations, style modifiers, function words (like "8k, highly detailed, trending on ArtStation"), and sometimes underspecified or ambiguous language. A reward model trained only on clean captions would fail to generalize to the messy reality of deployed systems.

From the 1.8M prompts, the authors select 10,000 for annotation using a graph-based diversity sampling algorithm. The procedure works as follows: each prompt is embedded into a vector using Sentence-BERT, which produces semantically meaningful embeddings. A k-nearest-neighbor graph is constructed where each prompt vertex is connected to its $k=150$ nearest neighbors based on cosine similarity between embeddings. A scoring function is then computed for each vertex based on how many of its neighbors have NOT yet been selected — vertices in underrepresented regions of the embedding space receive higher scores. Selection proceeds greedily: the highest-scoring prompt is selected, its neighbors' scores are updated (since they now have a selected neighbor), and the process repeats until 10,000 prompts are chosen. To make this computationally feasible (the algorithm is quadratic in the number of samples), prompts are first grouped into 100 batches of approximately 20,000 each, and 100 prompts are selected from each batch.

The result is a prompt distribution that spans 12 categories (from the Parti taxonomy): Abstract, Animals, Artifacts, Arts, Food, Illustrations, Indoor Scenes, Outdoor Scenes, People, Plants, Vehicles, and World Knowledge. The distribution (Figure 9) is dominated by common categories like People (3,360 prompts), Arts (2,407), and Outdoor Scenes (1,189), but includes rarer categories like Plants, Illustrations, and World Knowledge as well — ensuring the reward model sees diverse visual concepts.

Prompt Pre-Annotation: Identifying Problematic Inputs

Before rating images, annotators first classify each prompt into one of the 12 categories and identify potential issues: whether the prompt's intent is unclear, whether it contains sexual content, violent content, content defaming protected groups, or personally identifiable information. This pre-filtering step is critical because some prompts are inherently ambiguous ("a brand new medium," "low quality") or toxic, and images generated from such prompts cannot be fairly evaluated against normal criteria. These problematic prompts are flagged during annotation, though they are still included in the dataset — the reward model needs to learn that certain prompt types produce inherently lower-quality or problematic outputs.

Text-Image Rating: Three Dimensions of Quality

The core of the annotation is a three-dimensional rating of each text-image pair on 7-point Likert scales, with separate scores for:

• Image-Text Alignment (1–7): Does the generated image faithfully depict all the objects, attributes, properties, and relationships described in the prompt? A score of 1 means "the picture and the text do not match at all, it does not contain the object described in the text at all." A score of 7 means "the image and text match almost perfectly, including all objects described in the text, and all object properties are also consistent with the text."

• Fidelity (1–7): Is the image free of visual artifacts, anatomically correct, and aesthetically coherent? This captures problems like distorted body parts, repeated generation artifacts, and unrealistic object structures. The annotation document provides concrete examples: Spider-Man should have exactly two legs (extra legs = fidelity error), a unicorn should have exactly one horn (multiple horns = error), people should eat with utensils (eating with hands = error unless specified). A score of 1 means "there is an obvious error in the above image, and seriously affects the degree of beauty or coordination of the picture." A score of 7 means "no image errors, beautiful and coordinated pictures."

• Overall Rating (1–7): A holistic assessment combining alignment, fidelity, harmlessness, and aesthetic appeal. The annotation document frames this operationally: "How happy would you be if you gave an AI assistant this text and received this image result?"

Additionally, annotators check binary flags for specific problems: repeated generation causing unreality, body problems, excessive blur, psychological discomfort, sexual content, violent content, and content defaming protected groups. These binary flags serve both as diagnostic information for understanding model failures and as potential features for future reward model improvements.

Image Ranking: From Ratings to Preferences

After rating each image independently, annotators rank all images for a given prompt from best to worst. The ranking stage uses a 5-slot interface where images are dragged into ordered positions. Up to two images can share a slot (ties), but annotators are encouraged to distinguish quality differences when possible. With $k \in [4, 9]$ images per prompt, this yields up to $\binom{k}{2}$ comparison pairs — for example, 8 images produce up to 28 ordered pairs. These pairs are the fundamental training signal for the reward model: they tell us, for any two images from the same prompt, which one a human prefers.

The key challenge in ranking is handling trade-offs between criteria. What if one image has better text alignment but worse fidelity? What if one is more aesthetically pleasing but contains subtle toxicity? The annotation document provides explicit guidelines for these conflicts, prioritizing them in this order:

1. Harmlessness is generally most important. An image that is more aligned to the prompt but contains toxic or harmful content should be ranked lower than a less-aligned but harmless alternative. The document states: "if an image is more aligned to prompt but also more toxic, the less toxic one should outweigh it since we regard toxicity as a more unacceptable property."

2. Fidelity generally outweighs alignment. For most tasks, an image that is anatomically correct and visually coherent is preferred over one that matches the text better but has obvious quality defects.

3. Exception for clear alignment superiority: If one image matches the text clearly better than another, and the fidelity/harmlessness difference is only slight, and the content doesn't involve "sensitive areas" (like human body correctness), then the more aligned image can be ranked higher.

These trade-off rules are crucial because without them, different annotators would resolve conflicts differently, introducing noise into the preference data. The paper reports inter-annotator agreement of 65.3% (Table 2a) at the individual level and 73.4% for the annotator ensemble (majority vote), establishing a reasonable upper bound for what a learned reward model can achieve — no model can be more consistent than its training signal.

Annotation Management and Quality Control

The annotation was conducted over two months in collaboration with a professional data annotation company. Annotators were recruited with a requirement of at least college-level education (95.8% met this threshold). The process included:

1. Training: Annotators studied the annotation document (reproduced in full in Appendix B) and worked through example cases before beginning production annotation.

2. Entry Testing: Prospective annotators took a qualification test, and their agreement with researcher annotations and the annotator ensemble was computed. Those with low agreement were not hired.

3. Quality Inspection: Every annotation was double-checked by quality inspectors. Invalid annotations (inconsistent ratings, clear rule violations) were reassigned for relabeling.

4. Researcher Validation: For the test set construction, prompts from annotators with higher agreement with researchers were selected (466 prompts), providing a cleaner evaluation signal.

The final dataset comprises 8,878 valid prompts with a total of 136,892 comparison pairs (after filtering invalid annotations). This is the dataset on which ImageReward is trained.

---

ImageReward Model Architecture and Training

Model Architecture: BLIP Backbone with MLP Head

ImageReward uses BLIP ("Bootstrapping Language-Image Pre-training") as its backbone, specifically the ViT-L variant (a Vision Transformer with Large capacity for the image encoder, coupled with a 12-layer transformer for the text encoder). The choice of BLIP over the more commonly used CLIP is data-driven — Table 2(b) shows that at equivalent training set sizes, BLIP achieves consistently higher preference accuracy than CLIP (e.g., 64.71% vs. 61.87% at 4k prompts). The authors hypothesize that BLIP's advantage stems from two architectural differences: (1) BLIP uses bootstrapping during its pre-training, which may produce richer representations, and (2) BLIP's image-grounded text encoder processes text and image jointly through cross-attention, rather than as independent encoders whose outputs are compared via cosine similarity (as CLIP does).

The architecture processes inputs as follows. For a prompt $T$ and generated image $x$:

1. The BLIP image encoder (ViT-L) extracts visual features from $x$.
2. The BLIP text encoder (12-layer transformer) extracts textual features from $T$, with cross-attention to the image features — meaning the text representation is conditioned on the image content, not computed independently.
3. The combined text and image features are passed through a multi-layer perceptron (MLP) head that outputs a single scalar value $f_\theta(T, x)$.

This scalar is the predicted "human preference score" for the image given the prompt. Higher scores indicate more preferred images. The MLP head is randomly initialized according to $\mathcal{N}(0, 1/(d_{model}+1))$, where $d_{model}$ is the feature dimension from the backbone — a standard initialization that prevents the initial output scale from dominating the pre-trained backbone features.

Training Objective: Bradley-Terry Preference Model

The training objective formalizes the annotation data as a ranking problem using the Bradley-Terry model of paired comparisons. The intuition is: when humans compare two images $x_i$ and $x_j$ for the same prompt $T$, the probability that they prefer $x_i$ over $x_j$ should be proportional to the exponential of the score difference:

$P(x_i \succ x_j | T) = \sigma(f_\theta(T, x_i) - f_\theta(T, x_j)) = \frac{1}{1 + e^{-(f_\theta(T, x_i) - f_\theta(T, x_j))}}$

where:

• $\sigma(\cdot)$ is the sigmoid function, mapping any real number to $(0, 1)$
• $f_\theta(T, x)$ is the reward model's scalar output for prompt $T$ and image $x$
• $x_i \succ x_j$ denotes "$x_i$ is preferred over $x_j$"

This formulation has the property that when $f_\theta(T, x_i) \gg f_\theta(T, x_j)$, the predicted preference probability approaches 1; when the scores are equal, the probability is exactly 0.5; and when $f_\theta(T, x_j) \gg f_\theta(T, x_i)$, the probability approaches 0. This matches our intuition about preference: larger score gaps should correspond to more confident predictions.

The training loss is the negative log-likelihood of the observed preferences under this model:

$\text{loss}(\theta) = -\mathbb{E}_{(T, x_i, x_j) \sim \mathcal{D}} \left[ \log\left( \sigma(f_\theta(T, x_i) - f_\theta(T, x_j)) \right) \right]$

where:

• $\mathcal{D}$ is the dataset of comparison triples (prompt, preferred image, dispreferred image)
• The expectation $\mathbb{E}_{(T, x_i, x_j) \sim \mathcal{D}}$ means the loss is averaged over all comparison pairs in the training set
• $f_\theta(T, x_i)$ is the score of the preferred image, $f_\theta(T, x_j)$ is the score of the dispreferred image

What it computes: For each comparison pair where $x_i$ is preferred over $x_j$, the model computes the difference $f_\theta(T, x_i) - f_\theta(T, x_j)$, passes it through a sigmoid to get a predicted probability of preference, and then takes the negative log of this probability. If the model is confident and correct (score difference large and positive → sigmoid near 1 → log near 0 → loss near 0), the loss is small. If the model is wrong (score difference negative → sigmoid near 0 → log very negative → loss very large), the loss heavily penalizes the error. The result is that the model learns to assign higher scores to images that humans consistently prefer.

Why this form: The Bradley-Terry model is the standard choice for learning from pairwise preference data because it (1) converts the ranking problem into a collection of independent binary comparisons, which is statistically efficient and easy to optimize, (2) produces a transitive ordering (if A is preferred over B and B over C, the model will assign scores such that A > C), and (3) the loss is convex in the score differences for linearly parameterized models, making optimization well-behaved. An alternative like training a regressor to predict the raw Likert ratings would be more susceptible to annotator-specific scale biases (one annotator's "5" might be another's "4"), whereas pairwise preferences are more robust to such shifts — what matters is the relative ordering within each prompt, not the absolute rating scale.

Training Details and Overfitting Mitigation

The dataset is split by annotator: 466 prompts from annotators with higher researcher agreement are held out as the test set, while the remaining 8,000+ prompts are used for training. This split strategy is important — if the split were random across prompts, the same annotator's judgments might appear in both training and test, inflating apparent performance because the model would learn annotator-specific biases rather than general preference patterns.

The model is trained on 4 NVIDIA A100 GPUs (40GB each), with a per-GPU batch size of 16 (total effective batch size of 64). The learning rate follows a cosine schedule decaying from an initial value, with a final optimal setting of $1 \times 10^{-5}$ identified through grid search.

Overfitting is the primary training challenge. The paper reports that ImageReward converges rapidly and then overfits, degrading validation performance. This is a common problem in preference learning because the comparison pairs are highly correlated (all pairs from the same prompt share the same underlying prompt semantics), making the effective sample size smaller than the 136k pair count suggests.

The key countermeasure is freezing backbone layers: a specific fraction of the BLIP transformer layers are kept fixed during training, and only the remaining layers (plus the MLP head) are updated. Through grid search, the paper finds that freezing 70% of the transformer layers produces the best validation performance. This is effectively a form of strong regularization that prevents the model from distorting the powerful pre-trained BLIP representations to fit noise in the preference data. The unfrozen layers (the top 30%) are sufficient to adapt the semantic representations toward preference-relevant features (alignment quality, aesthetic assessment, harmlessness detection), while the frozen lower layers preserve general visual and linguistic understanding.

The sensitivity to hyperparameters is explicitly noted: learning rate and batch size interact with the freezing fraction, and the paper performs a careful grid search over these parameters on the validation set. This sensitivity is typical for fine-tuning large pre-trained models on relatively small downstream datasets.

---

ImageReward as an Automatic Evaluation Metric

Before using ImageReward for generator optimization, the paper establishes its validity as an evaluation metric — it needs to correlate with human judgments in ways that existing metrics (CLIP score, Aesthetic predictor, FID) do not.

Model-Level Ranking on Real User Prompts

The paper conducts a human evaluation comparing six popular high-resolution text-to-image models: CogView 2, Versatile Diffusion, Stable Diffusion 1.4, Stable Diffusion 2.1-base, DALL-E 2, and Openjourney. For 100 real-user test prompts, each model generates 10 outputs. The best of 10 is selected (by the metric being evaluated), and human annotators rank the six models' best outputs for each prompt. Win counts are aggregated across all annotators and prompts, yielding the rankings in Table 1.

ImageReward's ranking (Openjourney > SD 2.1-base > DALL-E 2 > SD 1.4 > Versatile Diffusion > CogView 2) achieves a Spearman rank correlation of 1.00 with the human evaluation ranking — perfect rank agreement. In contrast:

• CLIP score ranks SD 1.4 first (which humans rank fourth), yielding a Spearman correlation of 0.60.
• Zero-shot FID on MS-COCO ranks DALL-E 2 first (humans rank it third), yielding a Spearman correlation of only 0.09 — effectively no correlation.

This is a striking result. CLIP score and FID are the most widely used automatic metrics in the text-to-image literature, yet on real user prompts, ImageReward dramatically outperforms both at the task that actually matters: predicting which model's outputs humans will prefer. The failure of CLIP is particularly noteworthy — CLIP is designed to measure image-text similarity, and indeed the SD 1.4 outputs may have high CLIP scores because they are visually simple and prototypical, even when they fail to capture the nuanced content and style of complex real-user prompts.

Sample-Level Distinguishability

Beyond ranking entire models, a useful metric should distinguish between individual samples — telling us which of two images from the same model and prompt is better. Figure 3 shows box plots of normalized score distributions per model for both ImageReward and CLIP. ImageReward's scores exhibit substantially larger interquartile ranges than CLIP's, meaning ImageReward spreads out the quality spectrum within a model's outputs. CLIP scores, by contrast, cluster tightly — for many models, the interquartile range covers only a small fraction of the available score range, making it hard to distinguish a good generation from a mediocre one.

The paper also reports preference accuracy (Table 3): on 6,399 comparison pairs from the 466-prompt test set, ImageReward achieves 65.14% pairwise accuracy — correctly identifying which of two images a human preferred 65.14% of the time. This is 15.14 percentage points above random chance (50%), and substantially outperforms CLIP score (54.82%), Aesthetic (57.35%), and BLIP score (57.76%). The authors note that ImageReward's improvement over BLIP score (7.76% above random → 15.14% above random) represents nearly a doubling of the usable signal.

Recall and Filtering Performance

When asked to select the single best image from a set of 8 (Recall@1), ImageReward succeeds 39.62% of the time, versus 30.73% for BLIP score and 27.22% for CLIP score (Table 3). When asked to identify the worst image (Filter@1), ImageReward succeeds 49.06% of the time. These are per-prompt averaged metrics — for any given prompt, the model ranks all 8 images and the selected top (or bottom) image is compared against the human's choice.

The performance at top-4 selection is more forgiving: ImageReward's Recall@4 of 90.84% means that when the model selects its top 4 images from 8, the human's preferred image is among those 4 selections over 90% of the time. This makes ImageReward practical for the "best-of-N" selection paradigm: from a large pool of candidate images, select the top few for human review, dramatically reducing the inspection burden while rarely missing the best outputs.

Insertion Analysis

The paper also tests whether interpolating between ImageReward and other scorers can improve performance (Figure 17, Appendix C). A linear combination of CLIP score and Aesthetic score improves over either alone, but still falls well short of ImageReward alone. Adding ImageReward to CLIP or Aesthetic provides only marginal gains over ImageReward alone, suggesting that ImageReward already captures the preference-relevant aspects of both text-image similarity and aesthetic quality — these other signals are largely redundant once ImageReward's score is available.

---

ReFL: The Core Algorithm for Direct Diffusion Optimization from Reward Feedback

The ReFL algorithm addresses the central technical challenge identified in Section 3: how do you use a reward model trained on fully denoised images to provide gradient feedback for a diffusion model, when (a) diffusion models don't provide tractable likelihoods for their outputs, and (b) reward models are trained only on completed images, not on noisy intermediate latents?

The Critical Empirical Insight: When Are Reward Scores Reliable?

The key observation enabling ReFL is illustrated in Figure 4 and detailed in the companion analysis of Appendix F.1. For a given prompt, the authors generate multiple images with different random seeds and track how ImageReward scores evolve as denoising progresses. Specifically, at each step $t$ (from 40 down to 1, where 40 is pure noise and 1 is the final image), they take the current noisy latent $x_t$ and predict the final clean image $x_0'$ directly from it using the noise scheduler's formula (this is a standard operation in diffusion models — given the noise prediction at step $t$, you can analytically estimate what the final clean latent would be). They then compute ImageReward's score on this predicted clean image.

The pattern is striking:

• For $t \leq 15$ (steps 1–15, early denoising): All generations receive uniformly low ImageReward scores, regardless of their eventual quality. The predicted images are still too noisy and incoherent for ImageReward to assess meaningfully — the reward model was trained on clean images and cannot extract reliable preference signals from nearly-pure-noise predictions.

• For $15 \leq t \leq 30$ (steps 16–30, middle denoising): High-quality generations begin to separate from low-quality ones, but the separation is noisy and incomplete. You cannot reliably rank all generations' final quality from their scores at these steps — some good generations still look mediocre, and some poor ones haven't revealed their flaws yet.

• For $t \geq 30$ (steps 31–40, final denoising): ImageReward scores become clearly distinguishable and generally predictive of final image quality. The images are sufficiently formed that the reward model can assess alignment, fidelity, and aesthetics in ways that correlate with the fully denoised result.

The Appendix F.1 provides quantitative support: Figure 19 (left) shows the Spearman rank correlation ($\rho$) between ImageReward scores at each step and the final ImageReward scores of the completed images, plotted against the denoising step rate. The correlation rises sharply after step 30, confirming that the reward model's assessments stabilize in the final quarter of the denoising process.

This insight is what makes ReFL possible: you don't need to wait until the very last denoising step to get a useful reward signal — steps 30–40 are sufficient. This means you can backpropagate through the final denoising steps where the images are already fairly clear, compute a reward score, and use that gradient to update the diffusion model. The gradient flows through only the selected step's denoising operation (and the subsequent $x_0$ prediction), which is entirely differentiable.

Algorithm Design: Random Step Selection and Two-Loss Optimization

Algorithm 1 in the paper presents the full ReFL procedure. For each training iteration, the algorithm operates on a batch that mixes two data sources:

1. Pre-training data: a subset of 625k real text-image pairs from LAION-5B, filtered by aesthetic score to ensure reasonable quality. This is the standard pre-training distribution for Stable Diffusion.

2. Prompt set: prompts sampled from DiffusionDB (real user prompts) for which images are generated on-the-fly during training. No pre-existing images are used — the model generates its own images using its current parameters, computes their reward, and updates accordingly.

The training alternates between two loss terms for each batch element:

Loss 1: Pre-training Loss (standard diffusion denoising objective)

$\mathcal{L}_{pre} = \mathbb{E}_{(y_i, x_i) \sim \mathcal{D}} \left( \mathbb{E}_{\mathcal{E}(x_i), y_i, \epsilon \sim \mathcal{N}(0,1), t} \left[ \|\epsilon - \epsilon_\theta(z_t, t, \tau_\theta(y_i))\|_2^2 \right] \right)$

where:

• $(y_i, x_i)$ is a text-image pair from the pre-training dataset $\mathcal{D}$ (where $y_i$ is the prompt/caption and $x_i$ is the real image)
• $\mathcal{E}(x_i)$ encodes the real image $x_i$ into its latent representation using the VAE encoder
• $\epsilon \sim \mathcal{N}(0, 1)$ is random Gaussian noise
• $t$ is a randomly sampled timestep (1 to $T$)
• $z_t$ is the noisy latent at step $t$ (the clean latent mixed with noise $\epsilon$ according to the diffusion schedule)
• $\epsilon_\theta$ is the LDM's noise prediction network (a U-Net) with parameters $\theta$
• $\tau_\theta(y_i)$ is the text encoder output for prompt $y_i$, providing conditioning
• $\|\cdot\|_2^2$ is the squared L2 norm — the mean squared error between predicted noise and actual noise

What it computes: The standard latent diffusion training objective. For a real (prompt, image) pair, the model is given the noisy latent at a random timestep $t$ and must predict what noise $\epsilon$ was added. The loss is the squared error between the prediction and the true noise. This is the same loss used to pre-train Stable Diffusion — it teaches the model the fundamental image generation skill and, crucially, prevents catastrophic forgetting during ReFL fine-tuning. If we only optimized the reward loss, the model would quickly learn to produce images that score highly under ImageReward but may no longer be recognizable as the prompted content — the pre-training loss acts as a regularizer that keeps the model grounded in the real image distribution.

Loss 2: ReFL Loss (reward feedback)

$\mathcal{L}_{reward} = \lambda \mathbb{E}_{y_i \sim \mathcal{Y}} \left( \phi(r(y_i, g_\theta(y_i))) \right)$

where:

• $\mathcal{Y}$ is the set of user prompts sampled from DiffusionDB
• $g_\theta(y_i)$ denotes the image generated by the LDM with current parameters $\theta$ for prompt $y_i$. Importantly, this generation uses the ReFL procedure (described below) — it's not a full 40-step generation but a truncated one starting from noise and denoising to a random step $t \in [30, 40]$, then predicting the final clean image
• $r(y_i, g_\theta(y_i))$ is ImageReward's scalar score for the generated image given the prompt
• $\phi(\cdot)$ is a reward-to-loss mapping function. The paper sets $\phi = \text{ReLU}$, meaning $\phi(r) = \max(0, -r)$ up to the sign convention. More precisely, the paper formulates this as wanting to maximize reward, so the loss is minimized when the reward is maximized. With $\phi = \text{ReLU}$ and $\lambda > 0$, the ReFL loss is directly proportional to the negative reward, clipped below zero
• $\lambda = 1 \times 10^{-3}$ is the reward re-weight scale, a hyperparameter that controls how strongly the reward signal influences training relative to the pre-training loss

What it computes: For each prompt from the user prompt set, the model generates an image using the truncated ReFL procedure, computes ImageReward's score on it, and converts that score to a loss value. Higher ImageReward scores (more preferred images) produce lower ReFL loss, so the gradients push the model parameters toward generating higher-scoring images. The $\phi$ mapping and $\lambda$ scale are crucial hyperparameters — if the reward gradient is too strong, the model overfits to the reward and produces degenerate outputs that "fool" ImageReward but look unrealistic; if too weak, the pre-training loss dominates and no preference alignment occurs.

The ReFL Generation Procedure (Lines 8–15 of Algorithm 1):

This is the heart of the algorithm — how a "generation" is produced during ReFL training to enable gradient flow:

1. Sample a random timestep $t$ from the range $[T_1, T_2]$, where $T_1 = 30$ and $T_2 = 40$ when the total denoising steps $T = 40$. This is the step at which the gradient will be computed. The range covers the final 10 steps out of 40, corresponding to the regime where ImageReward scores become reliable.

2. Sample pure noise $x_T \sim \mathcal{N}(0, I)$ as the starting latent, same as standard diffusion generation.

3. Denoise from step $T$ down to step $t+1$ WITHOUT gradient computation (no grad: in Algorithm 1). These early steps produce an intermediate latent $x_t$ that already contains rough image structure but is still somewhat noisy. The model's parameters are NOT updated based on these steps — they are purely for producing the intermediate latent at step $t$.

4. Perform ONE denoising step from $x_t$ to $x_{t-1}$ WITH gradient computation (with grad: in Algorithm 1). This is the single step through which gradients will flow. It computes $x_{t-1} = \text{LDM}_{w_i}\{x_t\}$, where $\text{LDM}_{w_i}$ denotes the LDM's denoising operation with current parameters $w_i$.

5. Predict the final clean image $x_0'$ from $x_{t-1}$ using the noise scheduler's analytic formula. In diffusion models, given the noise prediction at step $t-1$ and the latent $x_{t-1}$, you can compute an estimate of what the original clean latent would be — this is a standard computation used during DDIM sampling and other fast sampling schemes.

6. Convert the predicted clean latent $x_0'$ to an image $z_i$ by passing it through the VAE decoder (from latent space to pixel space).

7. Compute $\mathcal{L}_{reward}$ on this image using ImageReward, and backpropagate through the entire differentiable chain: ImageReward → VAE decoder → $x_0$ prediction → denoising step $t \to t-1$ → LDM parameters.

The reason for random step selection ($t \leftarrow \text{rand}(T_1, T_2)$) rather than always using the LAST step ($t = 40$) is explicitly motivated: the paper found that using only the gradient from the final denoising step leads to very unstable training and bad results. By randomly selecting from $[30, 40]$, the model receives gradients at multiple stages of the late denoising process, which provides a more robust training signal — it learns to improve the generation across the entire final phase rather than only at the very last operation.

Joint Optimization and Regularization

The total training objective applied at each iteration combines both losses. The exact weighting is controlled by $\lambda = 10^{-3}$ — the ReFL loss is multiplied by 0.001 before being added to the pre-training loss. This small coefficient reflects the fact that the pre-training batch (real images) is the primary signal keeping the model grounded, while the reward signal provides a subtle steering force toward human-preferred outputs. Larger $\lambda$ values would cause the model to rapidly overfit to ImageReward's idiosyncrasies.

The batch composition is also carefully balanced: of the total batch size of 128, half (64) are pre-training samples and half (64) are ReFL samples. This 50/50 split ensures that every training step maintains roughly equal influence from distributional fidelity (pre-training loss) and human preference alignment (ReFL loss).

Why This Works: Gradient Flow Through Partial Denoising

The key mathematical insight is that the denoising operation at step $t$ is a differentiable function of the model parameters $\theta$. When the model computes $x_{t-1}$ from $x_t$ using its noise prediction $\epsilon_\theta(x_t, t)$, the output $x_{t-1}$ depends on $\theta$ through this prediction. The predicted clean image $x_0'$ is computed from $x_{t-1}$ through a deterministic formula (the noise scheduler's reverse process), making it also differentiable with respect to $\theta$. ImageReward's score is a differentiable function of the clean image and prompt.

Therefore, by the chain rule, the gradient $\nabla_\theta \mathcal{L}_{reward}$ exists and can be computed by standard automatic differentiation. The model parameters are updated to increase the reward model's score on the partially-denoised-then-predicted image, which translates to producing more human-preferred images after full denoising (in inference). This is the "direct" optimization that distinguishes ReFL from indirect data filtering or loss reweighting approaches.

Training Configuration and Infrastructure

The ReFL experiments use Stable Diffusion v1.4 as the base model, fine-tuned in half-precision (FP16) on 8 NVIDIA A100 GPUs (40GB each). The total batch size is 128 (64 pre-training + 64 ReFL per step, with per-GPU batch sizes apportioned accordingly). The learning rate is $1 \times 10^{-5}$ (same as ImageReward training), and training proceeds for one epoch over the pre-training dataset subset (625k samples from LAION-AES) with corresponding prompts sampled from DiffusionDB for the ReFL loss.

The pre-training dataset is a 625k subset of LAION-5B, filtered by the LAION aesthetic predictor to select images with high aesthetic scores. This subset is the same as what Stable Diffusion v1.4 was originally trained on — the ReFL fine-tuning starts from the same data distribution, ensuring that the pre-training loss remains on-distribution.

The prompt set $\mathcal{Y}$ for ReFL is sampled from DiffusionDB, the same source used for ImageReward's training data, but drawn from the broader pool to provide diverse real-user-style prompts during training. The specific size of this prompt set for ReFL is described as 20,000 samples total when comparing against baselines (Section 4.2 and Appendix E), though the exact number used for ReFL's own training may differ — the paper states prompts are "sampled from DiffusionDB" without specifying an exact count for ReFL's training set size.

Comparison with Baseline Methods

The paper compares ReFL against three alternative approaches for incorporating reward model feedback into diffusion model training, all implemented with ImageReward as the reward model for fair comparison (Appendix E provides full implementation details):

1. Dataset Filtering: Following the concurrent work of Wu et al., this approach uses ImageReward to score generated images, selects the highest-scoring images for a "preferred" dataset and the lowest-scoring images (paired with a "Weird image" prefix) for a "non-preferred" dataset, then fine-tunes the LDM on these filtered datasets along with pre-training data. The construction uses 20,000 pre-training samples + 20,000 filtered samples (10,000 preferred + 10,000 non-preferred). The reward model is NOT used during fine-tuning — only for dataset construction.

2. Reward Weighted: Following Lee et al., this approach modifies the loss function to weight training samples by their reward model scores. The loss is $\mathcal{L}(\theta) = \mathbb{E}_{(x,z)\sim\mathcal{D}^{model}}[-r_\phi(x,z)\log(p_\theta(x|z))] + \beta\mathbb{E}_{(x,z)\sim\mathcal{D}^{pre}}[-\log(p_\theta(x|z))]$, where $r_\phi(x,z)$ is ImageReward's score mapped to $[0,1]$ via min-max normalization, and $\beta = 0.5$. The dataset includes 20,000 pre-training samples and 20,000 generated samples (2,000 prompts × 10 images each). Unlike ReFL, the reward model influences training only through per-sample loss weights, not through gradients of the reward function itself.

3. RAFT (Reward rAnked FineTuning): Following Dong et al., this iterative method alternates between (a) generating 100,000 images (10,000 prompts × 10 images), (b) using ImageReward to rank them and select the best, and (c) fine-tuning the LDM on the selected images. The process can be repeated for multiple iterations (the paper tests up to 3 iterations).

The paper's human evaluation results (Table 4, Figure 6) show that ReFL substantially outperforms all three baselines: 58.79% win rate against the SD v1.4 baseline on real user prompts, versus 55.17% for Dataset Filtering, 39.52% for Reward Weighted, and 49.86% for RAFT (iteration 1; performance degrades with more iterations). The paper attributes these differences to the indirect nature of the baseline methods:

• Dataset Filtering is constrained by the fact that the model only sees "good" examples but receives no gradient-level signal about what makes them good — it's still just minimizing diffusion MSE on a filtered set.
• Reward Weighted suffers because the $[0,1]$ coefficient range means non-preferred images still contribute to training — their influence is muted but not eliminated, introducing interference.
• RAFT overfits as iterations increase because the generated dataset becomes self-reinforcing — the model trains on its own outputs, which can amplify reward model exploits rather than genuine quality improvements.

ReFL, by computing gradients through the reward model at the exact denoising step where its assessments become reliable, provides a direct signal about how to modify the generation process to increase human preference, without the indirection of dataset filtering or the interference of non-preferred samples.

Inference Configuration

For evaluation, the ReFL-fine-tuned model uses the same inference settings as the baseline SD v1.4: PNDM noise scheduler, classifier-free guidance scale of 7.5, 40 denoising steps. The full 40-step generation is used at inference time (unlike training, where only steps 30–40 flow gradients). This means the model's improvement comes from parameter updates that affect the entire denoising trajectory, even though the training signal was computed only at the final steps — the gradient through step $t$ updates parameters that are also used at earlier steps.

4. Key Insights and Innovations

Innovation 1: Human Preference Evaluation for Text-to-Image Is a Distinct Problem with Its Own Annotation Criteria

The paper's most foundational contribution isn't the reward model architecture or the optimization algorithm — it's the systematic identification and operationalization of what "human preference" actually means for text-to-image generation and the construction of an annotation pipeline that translates this multifaceted concept into a learnable signal. Before ImageReward, the field had no coherent framework for collecting text-to-image preference data at scale, and the available automatic metrics (CLIP score, FID, aesthetic predictors) each captured only a narrow slice of what humans care about, often in ways that contradicted actual human rankings.

What makes this contribution distinctive at the idea level is that the paper treats annotation design as a first-class research problem rather than a logistical afterthought. The authors don't just collect labels — they decompose human preference into three explicit, competing dimensions (alignment, fidelity, harmlessness), anticipate the inevitable contradictions between them, and codify a trade-off hierarchy in the annotation document (Appendix B) that resolves those conflicts consistently. The prioritization rule — harmlessness over fidelity over alignment, with exceptions for clear alignment superiority in non-sensitive domains — is a conceptual framework for thinking about what users actually value, not just an annotation instruction. It reflects a philosophical stance: that a faithful but disturbing image is worse than a slightly inaccurate but benign one, and that anatomical correctness matters more than semantic completeness in most contexts.

This contrasts sharply with the dominant assumptions in prior work. Previous reward models for text-to-image (HPS, PickScore, which emerged concurrently) relied on noisy online user click-through data[21, 61] — signals like "which image did the user download?" or "which image did they click on?" — which capture revealed preference but without consistent standards across raters, without explicit criteria for resolving conflicts, and without the ability to disentangle why an image was preferred. The paper's explicit annotation criteria make the preference signal interpretable and auditable in ways that click-through data cannot match. When the paper reports in Appendix A.4 that body problems affect 21.14% of images on average, or that the Abstract prompt category receives the lowest scores across all three dimensions, these are not just statistics — they are diagnostic insights enabled by the annotation design that would be invisible in an end-to-end preference signal without dimensional decomposition.

The significance extends beyond this specific reward model. By releasing the full annotation document (Appendix B) and describing the pipeline in detail (prompt pre-annotation → multi-dimensional rating → ranked comparison with trade-off rules), the paper establishes a template for future text-to-image preference data collection that subsequent work can adopt, adapt, or critique. The annotation document's handling of edge cases — what to do when a prompt is ambiguous, when nudity appears in an artistic context, when cultural norms differ — surfaces the normative judgments embedded in any preference dataset and makes them explicit rather than hiding them behind a black-box labeling process. This is a methodological contribution to the emerging field of AI alignment for generative vision, analogous to the role that the InstructGPT annotation methodology played for language model alignment.

The evidence that these annotation choices matter is in the results: ImageReward's 65.14% pairwise accuracy substantially exceeds what any individual annotator achieves with another annotator (65.3%, Table 2a — meaning ImageReward nearly matches human inter-annotator agreement), and its Spearman correlation of 1.00 with human model rankings (Table 1) versus 0.60 for CLIP and 0.09 for FID demonstrates that the dimensional decomposition and trade-off rules capture real human consensus that simpler metrics miss entirely. The annotation pipeline is not just a dataset — it's an operationalization of a normative framework for evaluating AI-generated images, and that framework is what enables both the reliable reward signal for ReFL and the metric properties that make ImageReward useful as an evaluator.

---

Innovation 2: Reward Model Scores Become Reliable Only in Late Denoising — and This Is the Enabling Insight for Direct Optimization

The paper's central technical insight is deceptively simple but opens up a previously closed optimization pathway: a reward model trained on clean images can provide meaningful gradient feedback if you only compute scores after the diffusion process is mostly complete — specifically, after step 30 of 40. This isn't a theoretical result but an empirical diagnostic (Figure 4, Appendix F.1) that the authors investigate systematically and then exploit algorithmically.

To understand why this is innovative, consider the prior landscape. The standard approach for using a trained classifier with diffusion models is classifier guidance, introduced by Dhariwal and Nichol[9] and extended by Song et al.[54]: train a classifier on noisy images at every noise level, then at inference time, compute the classifier's gradient with respect to the noisy image and use it to steer the denoising trajectory. This works beautifully for class-conditional generation (e.g., "generate more dog-like images") because you can train the classifier on noised versions of labeled data. But for a reward model trained on human preferences, this approach is fundamentally impossible — you cannot ask humans to express preferences about noisy, unintelligible intermediate latents. The reward model must be trained on clean images, and the classifier guidance paradigm requires classifiers trained on noisy images. This is the contradiction the paper identifies in Section 3.

What the paper discovers instead is a pragmatic compromise that sidesteps the contradiction: you don't need to score noisy images if you partially denoise first without gradients, then score the predicted clean image at a late step. The key empirical finding (Figure 4) is that ImageReward scores for different generations are indistinguishable and uniformly low at early steps, begin to separate at middle steps, and become clearly predictive of final quality only after step 30. This is not an obvious property — one might have expected the reward model to be completely useless at any intermediate step, or surprisingly, to be useful even at early steps due to learned correlations. The actual pattern — useless early, noisy in the middle, reliable late — is a specific empirical finding that enables the ReFL algorithm design.

The conceptual move here is reframing the problem from "how do we make the reward model work at all denoising steps?" (the classifier guidance approach) to "at which steps is the reward model already good enough, and can we build an optimization algorithm around only those steps?" This is a fundamentally different approach — it accepts the limitation (reward models can't score noisy images) and works around it rather than trying to overcome it. The random step selection mechanism (sampling $t \in [30, 40]$ uniformly rather than always using $t = 40$) is a further refinement that the paper justifies empirically: using only the final step gradient is unstable, while sampling across the reliable range provides more robust training.

This insight matters beyond the specific algorithm because it characterizes a general property that any future work on reward-guided diffusion optimization will need to account for. The finding that reward model scores become meaningful only in the final ~25% of denoising steps is likely specific to ImageReward + Stable Diffusion at 40 steps, but the methodology of measuring score reliability as a function of denoising progress (tracking Spearman correlation or score distinguishability across steps) is generalizable. The Appendix F.1 analysis showing the correlation between step-level and final scores provides a template for diagnosing when any reward model becomes reliable for any diffusion model configuration — a diagnostic tool that the field previously lacked.

The significance is that this insight converts an impossibility result into a practical optimization method. Before this work, one could reasonably have concluded that direct reward-based optimization of diffusion models was fundamentally infeasible because reward models and diffusion models operate in incompatible spaces (clean images vs. noisy latents). The paper shows that the incompatibility is not absolute — it's concentrated in the early denoising steps, and by restricting gradient flow to the late steps where the spaces converge, you can get a useful training signal. This is a conceptual breakthrough rather than an architectural one: it changes what researchers should believe is possible.

---

Innovation 3: Direct Gradient-Level Reward Feedback Fundamentally Differs from Data-Level Preference Incorporation

The paper develops a taxonomy of approaches for using reward models to improve generators — and then provides empirical evidence that direct gradient feedback (ReFL) is qualitatively better than indirect data-level methods (dataset filtering, reward-weighted loss, RAFT). This is not just a "our method gets higher numbers" claim but a claim about the nature of the optimization signal and why indirect methods have structural limitations that direct methods avoid.

The distinction the paper draws (Section 3 and the analysis in Section 4.2) is between:

• Data-level methods: Use the reward model to filter, reweight, or select training examples, then train the generator using its standard loss (diffusion MSE). The reward model influences what the model sees but not how the model should respond in an optimization sense. The generator still minimizes the same pixel-level reconstruction error — it just does so on a curated dataset.

• Gradient-level methods: Compute the gradient of the reward model's score with respect to the generator's parameters and use it directly in the parameter update. The reward model influences the direction of parameter movement, telling the generator not just "this is a good image" but "change your parameters in THIS specific way to produce better-scoring images."

The paper argues that the data-level methods have structural limitations that gradient-level methods transcend:

Dataset Filtering is constrained by the fact that the model only sees filtered examples but receives no information about why they were selected or what the reward model values. The fine-tuned model still minimizes diffusion MSE on the filtered set — it learns to better reproduce the selected images, but those images are a finite sample from a high-dimensional space, and the model can only interpolate within their distribution, not generalize reward-model-driven improvements to unseen prompts. The paper's result that Dataset Filtering achieves only 55.17% win rate against the baseline despite using the same ImageReward for filtering (Table 4) supports this interpretation — it helps, but less than direct gradient feedback.

Reward Weighted is more subtle in its failure mode. Because the loss weights are constrained to [0, 1], every image still contributes to training — bad images are just downweighted rather than actively penalized. The paper's diagnosis is that this introduces interference: the model is simultaneously pulled toward high-quality images (high weight) and still partially toward low-quality ones (low but non-zero weight), which muddies the optimization signal. With real user prompts, even the "best" generated images in a batch may be mediocre, and the model has no mechanism to actively avoid producing the failure modes it sees. The 39.52% win rate — worse than the baseline — suggests this interference is not just theoretical but practically damaging.

RAFT degrades with iterations (49.86% → 30.85% → 20.97% over three iterations, Table 4) because the self-training loop amplifies reward model exploits: the generator produces images that score highly under ImageReward but aren't actually better, those images become the training data for the next iteration, and the cycle compounds. This is a concrete demonstration of reward over-optimization in image generation — analogous to the reward hacking phenomena documented in RLHF for language, but occurring through the data filtering path rather than through policy gradient updates.

ReFL, by operating at the gradient level, avoids these failure modes: it doesn't filter data (the pre-training data remains unchanged, preserving distributional coverage), it doesn't weight samples (the reward signal is a loss term, not a data weight), and it doesn't iterate on its own outputs (each ReFL step generates fresh images from the current model parameters and computes gradients through them, but those images are not stored or used as future training data). The direct gradient tells the model "modify your denoising process to produce images that score higher" — a signal that can, in principle, generalize to unseen prompts because it's updating the generative process itself, not just the training data distribution.

This innovation is conceptually significant because it establishes a hierarchy of feedback incorporation methods — gradient-level > data-level — that parallels the RLHF vs. supervised fine-tuning distinction in NLP. Just as RLHF was shown to produce better instruction-following than fine-tuning on curated demonstration data alone, ReFL's direct gradient feedback produces better preference alignment than fine-tuning on reward-filtered data. The paper provides empirical evidence for this hierarchy in the image generation domain, supported by the human evaluation results in Table 4 and Figure 6, where ReFL consistently wins against all indirect methods in head-to-head comparisons. This is a fundamental claim about optimization methodology, not just a incremental performance improvement — it changes what researchers should believe about how to best use reward models for generative model training.

---

Innovation 4: ImageReward Demonstrates That CLIP Score and FID Are Fundamentally Misaligned with Human Evaluation of Text-to-Image Models

The paper's evaluation results (Table 1, Figure 3) do more than show that ImageReward is a better metric — they provide strong empirical evidence that the field's standard automatic evaluation metrics are measuring something fundamentally different from what humans care about, and that continuing to rely on them as primary evaluation tools is actively misleading.

This is a diagnostic contribution rather than a methodological one, but its implications are substantial. CLIP score has been the de facto standard for evaluating individual text-to-image generations since DALL-E and Stable Diffusion — it's used to select the best images for papers, to rank models, and to tune hyperparameters. The paper shows that on real user prompts, CLIP score's ranking of six popular models has a Spearman correlation of only 0.60 with human rankings, and it ranks SD 1.4 first when humans rank it fourth — a dramatic misalignment. On MS-COCO, CLIP's correlation drops to effectively zero. This is not a marginal failure — it's evidence that CLIP score is systematically capturing properties that are orthogonal or even opposed to human preference when evaluated on diverse real-world prompts.

The explanation the paper offers is that CLIP measures prototypicality — how close an image is to the "average" representation of the caption in CLIP's embedding space — while humans care about fidelity, alignment, harmlessness, and aesthetic quality in ways that are not well-captured by cosine similarity in a joint embedding space. An image that is simple, typical, and matches the "gist" of the prompt may score highly on CLIP while being compositionally wrong, anatomically distorted, or aesthetically uninteresting. This is visible in Figure 3: CLIP scores cluster tightly because most images for a given prompt are roughly equally "close" to the text embedding, whereas ImageReward scores spread out because ImageReward is making finer-grained quality distinctions that CLIP misses.

Similarly, FID is shown to have essentially no correlation with human model rankings on real user prompts. This is less surprising given FID's well-documented limitations (it measures distributional similarity to reference images, not per-sample quality), but the paper's demonstration that the correlation is 0.09 is still striking — it quantifies just how useless FID is for the evaluation task that actually matters to users and practitioners.

The innovation here is not that CLIP and FID are imperfect — this has been noted before — but that the paper provides a systematic, quantitative comparison against a consistent human baseline across multiple models, showing that the misalignment is severe enough to make these metrics actively misleading. A researcher selecting the "best" model by zero-shot FID would choose DALL-E 2, while humans consistently prefer Openjourney and SD 2.1 — a completely different ranking. This has direct implications for how text-to-image research should be evaluated: the paper argues that human preference metrics like ImageReward (or future improvements) should replace or at least supplement CLIP score and FID as primary evaluation tools, especially for model comparison and hyperparameter selection.

The evidence is anchored in Table 1 (Spearman correlations), Figure 3 (score distribution distinguishability), and the per-sample selection results in Table 3 (preference accuracy, recall, filtering). Together, these results make a compelling case that the evaluation crisis in text-to-image generation — long suspected but rarely quantified — is real, substantial, and has practical consequences for research progress.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary preference training dataset consists of 8,878 prompts with 136,892 comparison pairs collected through the annotation pipeline (Section 2.1). Prompts are sourced from DiffusionDB, a dataset of real user prompts from the Stable Diffusion Discord community. For ImageReward evaluation as a metric, the authors use 100 real-user test prompts for model-level ranking and an additional 371 prompts with 8 images each for recall/filtering metrics. MS-COCO 2014 validation set is used only for FID comparison. The ReFL training uses prompts sampled from DiffusionDB and a 625k subset of LAION-5B filtered by aesthetic score as the pre-training dataset. All test sets are disjoint from training data, with the ImageReward test set specifically drawn from annotators with higher researcher agreement (466 prompts).

• Base model(s). ImageReward uses BLIP (ViT-L image encoder, 12-layer transformer text encoder) as its backbone, initialized from pre-trained weights. For ReFL experiments, the base generative model is Stable Diffusion v1.4 (a latent diffusion model with approximately 860M parameters operating in the latent space of a pre-trained VAE, using 40 denoising steps with PNDM noise scheduler). The paper also evaluates six text-to-image models for metric validation: CogView 2, Versatile Diffusion, Stable Diffusion 1.4, Stable Diffusion 2.1-base, DALL-E 2 (via OpenAI API), and Openjourney.

• Metrics. The authors define and measure:

  • Preference Accuracy (%): the percentage of comparison pairs where the model correctly identifies which image a human preferred, computed as the fraction of pairwise comparisons where $\text{sign}(f_\theta(T, x_i) - f_\theta(T, x_j))$ matches the human preference direction. A random baseline achieves 50%.
  • Recall@k (%): when ranking 8 images per prompt, the fraction of prompts where the human-annotated best image appears in the model's top-k selections (k = 1, 2, 4). Averaged per prompt.
  • Filter@k (%): when ranking 8 images per prompt, the fraction of prompts where the human-annotated worst image appears in the model's bottom-k selections. Averaged per prompt.
  • Win Count / Win Rate (%): for model comparison, the number (or fraction) of human evaluations where one model's output is preferred over another's. Aggregated across prompts and annotators.
  • Spearman rank correlation ($\rho$): between automatic metric rankings and human evaluation rankings of models.
  • ImageReward Score: the scalar output of the trained reward model for a text-image pair, with higher values indicating higher human preference. Normalized per-model by min-max scaling when comparing distributions.
  • Average Score (1-7 Likert): human-annotated ratings for alignment, fidelity, and overall satisfaction, used for analyzing annotation data properties.
  • Human Evaluation Win Rate (ReFL): for ReFL ablation, the fraction of comparisons where human annotators prefer the fine-tuned model's output over the baseline SD v1.4 output for the same prompt.

• Baselines. ImageReward is compared against three existing automatic scoring methods:

  • CLIP Score (Radford et al., 2021): cosine similarity between CLIP text and image embeddings.
  • Aesthetic Score (LAION, Schuhmann et al., 2022): a CLIP-based linear predictor trained on human aesthetic ratings of photographs.
  • BLIP Score (Li et al., 2022): cosine similarity between BLIP text and image embeddings, analogous to CLIP score but using the BLIP backbone.

  For ReFL, the paper compares against three indirect optimization methods implemented with ImageReward as the reward model:

  • Dataset Filtering (Wu et al., 2023): construct filtered training sets by selecting highest and lowest-scoring generated images, then fine-tune on these datasets plus pre-training data.
  • Reward Weighted (Lee et al., 2023): modify the diffusion loss to weight each training sample by its ImageReward score mapped to [0,1], with a penalty term $\beta = 0.5$ on pre-training data.
  • RAFT (Dong et al., 2023): iterative data selection where each iteration generates images, ranks them with ImageReward, fine-tunes on the best, and repeats. Tested at 1, 2, and 3 iterations.

  The unmodified Stable Diffusion v1.4 serves as the baseline for human evaluation comparisons.

• Generation budget / compute accounting. For ImageReward training, the computational cost is measured by training epochs, batch size (64 total across 4 GPUs), and number of comparison pairs (136,892). For ReFL training, the budget is normalized by: (1) total batch size of 128 (64 pre-training + 64 ReFL) across 8 GPUs, (2) training for one epoch over a 625k pre-training subset with corresponding ReFL prompts, (3) all methods use the same pre-training dataset and same training configurations (learning rate $1\times10^{-5}$, half-precision) for fair comparison. For inference-time evaluation, the budget is measured by number of generated images: best-of-10 for model ranking, best-of-64 for Appendix D comparisons with HPS and PickScore. The ReFL training algorithm requires one generation per ReFL batch element during training, making it substantially more expensive per step than data-oriented methods (which use pre-generated images) — but all methods are trained for equivalent epochs.

• Cross-validation / statistical protocol. The ImageReward test set (466 prompts) is constructed by selecting prompts from annotators with higher agreement with researchers, ensuring a cleaner evaluation signal separate from training. For preference accuracy, the paper computes the metric across 6,399 comparison pairs from these 466 prompts. For human evaluation of model ranking, 100 prompts are used with 10 images per model, and annotators rank the best images using the same discipline as the annotation pipeline. For ReFL human evaluation, the paper uses 466 real user prompts from DiffusionDB and 90 challenging prompts from the Multi-task Benchmark (MT Bench), with annotators sorting multiple images per prompt. The 56 human evaluation results in Table 4 are reported with total win counts and win rates. Agreement between annotators is reported with standard deviations in Table 2(a), showing inter-annotator agreement of 65.3% ± 5.6% and researcher-annotator agreement of 65.3% ± 8.5%. No confidence intervals are reported for the main preference accuracy results.

Main Quantitative Results

ImageReward: Human Preference Prediction Accuracy

The headline result for ImageReward is a preference accuracy of 65.14% on the test set of 466 prompts (6,399 comparison pairs), substantially outperforming all baseline scoring methods (Table 3). Specifically:

• ImageReward achieves 65.14%, which is 15.14 percentage points above the 50% random baseline — nearly double the improvement of the next-best method (BLIP score at 57.76%, or 7.76 points above random).
• CLIP score achieves 54.82%, Aesthetic achieves 57.35%, and BLIP score achieves 57.76%.
• The improvement over CLIP is 38.6% relative to the gap from random: $(65.14 - 54.82) / (65.14 - 50) \times 100 \approx 68\%$ of the remaining achievable improvement, but the paper's quoted figure of "38.6%" in the abstract appears to be calculated differently — likely as the relative accuracy improvement over CLIP's error rate above random, or more plausibly as simply the absolute percentage-point gap relative to CLIP's accuracy (65.14 - 54.82 = 10.32, and 10.32 / 54.82 ≈ 18.8%) — the exact calculation methodology for the abstract percentages is not fully specified in the main text but the raw numbers are unambiguous in Table 3.

For the recall and filtering tasks (Table 3), ImageReward demonstrates strong practical utility for best-of-N selection:

• Recall@1: ImageReward identifies the single best image from 8 candidates 39.62% of the time, versus 30.73% for BLIP score, 27.22% for CLIP score, and 30.73% (the identical value reported for Aesthetic and BLIP at @1 appears to be a coincidence in the table).
• Recall@4: When selecting the top 4 from 8 images, ImageReward includes the human-preferred best image 90.84% of the time — meaning in over 9 out of 10 cases, the best image survives a 50% filtering rate.
• Filter@1: ImageReward correctly identifies the single worst image from 8 candidates 49.06% of the time.
• Filter@4: When selecting the bottom 4, the human-annotated worst image is among them 88.95% of the time.

These results appear in Table 3, reporting per-prompt averaged metrics. The recall and filtering metrics use a separate set of 371 prompts with exactly 8 images each, distinct from the 466-prompt preference accuracy test set.

Win rates in direct selection comparison (Figure 5): When ImageReward and baseline methods each select their top-3 images from pools of 9, 25, or 64 generated images, and human annotators rank these selections:

• ImageReward wins against random selection 77.1% of the time on average.
• ImageReward wins against CLIP score selections 69.3% of the time.
• ImageReward wins against Aesthetic score selections 69.8% of the time.
• ImageReward wins against BLIP score selections 65.8% of the time.

These are reported as overall win rates averaged across the different pool sizes, presented visually in Figure 5.

ImageReward as an Automatic Evaluation Metric: Model-Level Ranking

The model-level evaluation (Table 1) demonstrates ImageReward's alignment with human consensus at the system-comparison level:

• Human ranking (by win count on 100 real-user prompts): Openjourney (507 wins) > SD 2.1-base (463) > DALL-E 2 (390) > SD 1.4 (362) > Versatile Diffusion (340) > CogView 2 (74).
• ImageReward ranking (by average score): Openjourney (0.2614) > SD 2.1-base (0.2458) > DALL-E 2 (0.2114) > SD 1.4 (0.1344) > Versatile Diffusion (-0.2470) > CogView 2 (-1.2376).
• Spearman rank correlation between ImageReward and human evaluation: 1.00 — perfect rank agreement on this set of 6 models.

In contrast:

• CLIP score produces a Spearman correlation of 0.60 with human rankings, ranking SD 1.4 first (which humans rank fourth).
• Zero-shot FID on MS-COCO produces a Spearman correlation of only 0.09, ranking DALL-E 2 first (humans rank it third).

Additionally, the paper reports ImageReward scores on 30k MS-COCO generations (Table 1, rightmost columns), where ImageReward ranks DALL-E 2 first (score 0.5387) followed by SD 2.1-base (0.1553), while FID ranks DALL-E 2 first (10.9) followed by SD 1.4 (17.9). The ImageReward MS-COCO ranking is not directly validated against human preferences on MS-COCO captions — the human evaluation is only on real user prompts — so the MS-COCO ImageReward scores are provided as supplementary evidence rather than as a validated metric for that distribution.

Score distribution distinguishability (Figure 3): Box plots of normalized score distributions across 1,000 generations per model show that ImageReward exhibits substantially larger interquartile ranges than CLIP for every model tested. This means ImageReward can distinguish quality differences between individual samples within a model's output distribution, whereas CLIP scores cluster tightly and cannot effectively discriminate between good and mediocre generations from the same model. Additionally, the medians of ImageReward scores across models align roughly with the human ranking from Table 1, while CLIP's medians show no such alignment.

Ablation: Training Dataset Size and Backbone Choice

Table 2(b) reports the effect of training data quantity and backbone architecture on preference accuracy:

Training dataset size: Increasing the number of annotated prompts consistently improves ImageReward's performance:

• BLIP backbone with 1k prompts: 63.07%
• BLIP backbone with 2k prompts: 63.18%
• BLIP backbone with 4k prompts: 64.71%
• BLIP backbone with 8k prompts: 65.14%

The improvement from 1k to 8k prompts is approximately 2.07 percentage points, suggesting that additional annotation data would likely yield further gains — the curve has not saturated.

Backbone comparison: BLIP consistently outperforms CLIP when both are equipped with the same MLP head and trained with the same Bradley-Terry loss:

• CLIP with 4k prompts: 61.87%
• CLIP with 8k prompts: 62.98%
• BLIP with 4k prompts: 64.71%
• BLIP with 8k prompts: 65.14%

The gap between BLIP and CLIP at 8k prompts (2.16 percentage points) is non-trivial, especially given that CLIP was trained on a larger pre-training dataset. The paper attributes this to BLIP's image-grounded text encoder and bootstrapping pre-training procedure. This ablation justifies the architectural choice and suggests that stronger backbone models could further improve reward model performance.

Agreement Analysis

Table 2(a) reports inter-annotator and annotator-researcher agreement on a subset of 40 prompts (778 comparison pairs):

• Researcher-Researcher agreement: 71.2% ± 11.1%
• Annotator-Annotator agreement: 65.3% ± 5.6%
• Annotator Ensemble-Researcher agreement: 73.4% ± 6.2%
• ImageReward-Researcher agreement: 64.5% ± 2.5%
• ImageReward-Annotator Ensemble agreement: 70.5% ± 18.6%

Key observations: ImageReward (64.5%) nearly matches individual annotator agreement with researchers (65.3%), and the annotator ensemble achieves the highest agreement with researchers (73.4%). This establishes that 65–73% is the practical ceiling for preference prediction given the inherent subjectivity of the task — ImageReward's 65.14% test accuracy is close to the inter-annotator ceiling.

The CLIP score, Aesthetic score, and BLIP score all fall significantly below these agreement levels (54.4%–62.0% with the annotator ensemble), confirming that they fail to capture dimensions of preference that humans consistently agree on.

ReFL: Human Evaluation of Fine-Tuning Methods

The headline result for ReFL is a win rate of 58.79% against the SD v1.4 baseline on real user prompts in human evaluation, outperforming all indirect optimization methods (Table 4, Figure 6):

On real user prompts (466 prompts from DiffusionDB):

• ReFL: 1,508 total wins, 58.79% win rate against baseline
• Dataset Filtering: 1,394 total wins, 55.17% win rate
• Reward Weighted: 1,075 total wins, 39.52% win rate
• RAFT (iteration 1): 1,341 total wins, 49.86% win rate
• RAFT (iteration 2): 753 total wins, 30.85% win rate
• RAFT (iteration 3): 398 total wins, 20.97% win rate
• SD v1.4 baseline: 1,315 total wins (baseline comparison point)

On Multi-task Benchmark (MT Bench, 90 challenging prompts):

• ReFL: 808 total wins, 58.49% win rate
• Dataset Filtering: 735 total wins, 51.72% win rate
• Reward Weighted: 585 total wins, 43.33% win rate
• RAFT (iteration 1): 578 total wins, 42.31% win rate
• SD v1.4 baseline: 718 total wins

Head-to-head comparisons (Figure 6 and Figure 20 in Appendix F) show that ReFL is always preferred when compared directly against any other method — the win rate arrows all point in ReFL's favor.

Dataset Filtering provides a modest improvement over the baseline (55.17% and 51.72% on the two test sets), which the paper attributes to the indirect nature of the optimization — the model sees preferred images but receives no gradient-level signal about what makes them preferred.

Reward Weighted performs substantially worse than the baseline (39.52% and 43.33%), which the paper explicitly diagnoses: the [0,1] weight range means non-preferred images still contribute to training, introducing interference that degrades performance. With real user prompts where even the "best" generated images may be imperfect, the model is pulled toward mediocre outputs.

RAFT degrades severely with iterations: the win rate drops from 49.86% (iteration 1) to 30.85% (iteration 2) to 20.97% (iteration 3) on real prompts, and similarly on MT Bench. This is a clear demonstration of reward over-optimization through self-training — each iteration amplifies the model's ability to exploit ImageReward's scoring patterns rather than genuinely improving image quality, and the resulting images become progressively less preferred by humans. The paper notes that RAFT is "constrained by the quality of the constructed dataset" and that "even expert generators have limitations."

Figure 7 provides qualitative examples comparing ReFL outputs to baseline methods, showing that ReFL generates images with better text coherence (e.g., correctly rendering "long pointy ears" when specified, while other methods omit or distort this feature) and overall fidelity. The paper provides additional qualitative comparisons in Appendix F (Figures 21–22).

Comparison with Concurrent Reward Models (Appendix D)

Appendix D compares ImageReward against two concurrent text-to-image reward models: HPS (Wu et al., 2023) and PickScore (Kirstain et al., 2023):

Preference accuracy (Table 5, on the same 6,399 comparison pairs from 466 prompts):

• ImageReward: 65.14%
• PickScore: 62.78%
• HPS: 60.79%

Recall@1 / Filter@1 (Table 5, on the 371-prompt set):

• ImageReward: 39.62% / 49.06%
• HPS: 39.89% / 47.17%
• PickScore: 38.27% / 46.36%

Best-of-64 selection (Table 6, human evaluation): When each reward model selects the best image from 64 candidates per prompt:

• ImageReward: 676 total wins (73.33% win rate) on real prompts, 824 wins (74.42%) on MT Bench
• PickScore: 620 wins (72.16%) on real prompts, 773 wins (72.73%) on MT Bench
• HPS: 572 wins (67.24%) on real prompts, 662 wins (69.15%) on MT Bench

ReFL with different reward models (Table 6): When each reward model is used to train ReFL on SD v1.4:

• ReFL trained with ImageReward: 512 wins (58.38%) on real prompts, 492 wins (58.67%) on MT Bench
• ReFL trained with PickScore: 472 wins (56.91%) on real prompts, 454 wins (55.09%) on MT Bench
• ReFL trained with HPS: 428 wins (52.86%) on real prompts, 426 wins (52.86%) on MT Bench

ImageReward leads across all metrics and both usage modes (selection and training), confirming that the annotation pipeline and BLIP backbone produce a more reliable reward signal than concurrent approaches. Notably, the ReFL results demonstrate that better reward models produce better fine-tuned generators — the performance ranking of reward models (ImageReward > PickScore > HPS) is preserved in the performance ranking of ReFL-trained models using those reward models, validating that reward model quality directly translates to downstream optimization effectiveness.

Ablation Studies and Robustness Checks

• Training dataset size: Increasing annotated prompts from 1k to 8k monotonically improves preference accuracy from 63.07% to 65.14% (Table 2b). The improvement has not saturated at 8k, suggesting further annotation would yield continued gains — a positive result indicating scalability.

• Backbone architecture (BLIP vs. CLIP): BLIP outperforms CLIP by 2.16 percentage points at 8k prompts (65.14% vs. 62.98%, Table 2b). The paper attributes this to BLIP's image-grounded text encoder (cross-attention to image features) versus CLIP's independent dual encoders, and to BLIP's bootstrapping pre-training procedure.

• Freezing backbone layers for overfitting mitigation: The paper's text reports that freezing 70% of BLIP transformer layers produces optimal performance, discovered through grid search. The exact performance at other freezing ratios is not tabulated, but the paper describes the optimization as "a careful grid search based on the validation set to determine optimal values" (Section 2.2). This is described as a critical training technique — without layer freezing, ImageReward "observe[s] rapid convergence and consequent overfitting, which harms its performance." This ablation is qualitative in the main text but is presented as a key empirical finding for practical RM training.

• Interpolation between scorers (Appendix C, Figure 17): Linear interpolation between CLIP score and Aesthetic score improves preference accuracy over either alone (exact values not reported numerically, shown visually in Figure 17), but still falls substantially short of ImageReward alone. Interpolating ImageReward with CLIP score or Aesthetic score provides only marginal improvement over ImageReward alone (less than 1 percentage point visually), suggesting that ImageReward already captures the preference-relevant variance of both text-image similarity and aesthetic quality.

• ReFL reward-to-loss mapping function ($\phi = \text{ReLU}$): The paper states this choice without ablating alternatives within the ReFL context. The $\phi$ function determines how the scalar reward score is converted to a loss value. Using ReLU means that images scoring below zero (negative ImageReward scores) receive a proportionally large loss penalty, while images scoring above zero receive no penalty — the optimization pushes toward positive scores. This creates an asymmetric loss landscape where avoiding bad generations is heavily incentivized, while plateauing at high scores is not penalized. The sensitivity of ReFL to this choice is not explored experimentally.

• ReFL reward re-weight scale ($\lambda = 10^{-3}$): The paper sets this hyperparameter to $1\times 10^{-3}$ without reporting ablation over alternative values. This coefficient controls the balance between the pre-training loss (weight 1) and the reward feedback loss (weight $10^{-3}$). The paper's text notes that larger values lead to overfitting, and the chosen value was determined empirically, but no sweep results are reported. This is a critical hyperparameter — too large and the model over-optimizes the reward signal; too small and ReFL reduces to standard fine-tuning.

• ReFL random step range ($[T_1, T_2] = [30, 40]$ out of 40): The choice of step 30 as the lower bound is motivated by Figure 4 showing that ImageReward scores become distinguishable only after step 30. The paper reports that using only the final step ($t = 40$) leads to "very unstable training and bad results," while sampling uniformly from [30,40] provides more robust training. This is presented as an empirical finding but the comparison is qualitative — no table compares different step ranges.

• RAFT iteration count: The paper tests RAFT at 1, 2, and 3 iterations (Table 4), demonstrating monotonic degradation (49.86% → 30.85% → 20.97% win rate). This ablation is critical for establishing that reward over-optimization through self-training is a genuine failure mode in text-to-image generation, not just a theoretical concern. The severity of degradation (losing 29 percentage points of win rate over 3 iterations) is surprisingly large — suggesting RAFT's self-training loop is particularly vulnerable to reward hacking.

• Comparison of different reward models for ReFL training (Appendix D, Table 6): ReFL trained with ImageReward (58.38% win rate) outperforms ReFL trained with PickScore (56.91%) and HPS (52.86%). This demonstrates that reward model quality directly impacts downstream optimization quality, providing evidence that the choice of reward model is consequential and that ImageReward's annotation quality translates to better generator fine-tuning.

• Problem frequency analysis by prompt category (Appendix A.4, Figures 11, 13): The annotation data reveals systematic failure patterns: body problems affect 21.14% of images on average, with highest frequency in People and Arts categories. Repeated generation affects 3.16% on average. Function words in prompts (like "8k, highly detailed") initially increase then decrease problem frequencies — non-linear effects that the annotation dimensional decomposition makes visible. These are not traditional ablations but serve as robustness checks on the annotation pipeline's diagnostic value.

Critical Assessment

Does ImageReward Actually Predict Human Preferences at a Useful Level?

The paper claims ImageReward "outperforms existing text-image scoring methods... in terms of understanding human preference" (Abstract). The evidence for this is strong within the specific evaluation framework: 65.14% pairwise accuracy (Table 3), Spearman correlation of 1.00 with human model rankings (Table 1), and superior recall/filtering performance (Table 3). However, the 65.14% figure warrants careful interpretation. This means ImageReward is correct on 65% of comparison pairs — wrong on the remaining 35%. The paper frames this as 15.14 percentage points above random, which is substantial, but it also means that in roughly one-third of comparisons, ImageReward disagrees with the human annotator. This error rate is close to the inter-annotator agreement ceiling of 65.3% (Table 2a), suggesting ImageReward has largely saturated the available signal in the annotation data. The practical implication is that ImageReward is more useful for batch selection (Recall@4 of 90.84%) than for precise pairwise discrimination, and using it as a training signal (via ReFL) necessarily amplifies both its genuine insights and its systematic errors.

The Spearman correlation of 1.00 on model ranking (Table 1) is impressive but based on only 6 models and 100 prompts. A larger-scale evaluation with more models and more prompts would provide a more robust assessment of metric reliability. The paper acknowledges this implicitly by noting that their test set is limited and that additional annotation data would improve the reward model.

Does ReFL Actually Provide a Fundamental Improvement Over Data-Level Methods?

The paper's central technical claim is that direct gradient-level feedback (ReFL) is fundamentally better than indirect data-level methods. The human evaluation results (Table 4) support this: ReFL achieves 58.79% win rate versus 55.17% for Dataset Filtering, 39.52% for Reward Weighted, and 49.86% for RAFT (iteration 1). However, several aspects of this comparison warrant scrutiny:

The gap between ReFL and Dataset Filtering is modest. ReFL's 58.79% win rate is only 3.62 percentage points higher than Dataset Filtering's 55.17% on real user prompts. While this difference is directionally consistent across both test sets (real user and MT Bench), the magnitude is smaller than the paper's framing might suggest. The abstract and Section 3 emphasize the "indirect" nature of data-level methods as a fundamental limitation, but the empirical gap to the best-performing indirect method is not overwhelming.

The Reward Weighted baseline is weaker than it needs to be. The paper's implementation of Reward Weighted uses min-max normalization to map ImageReward scores to [0,1] and sets $\beta = 0.5$. The authors correctly diagnose that the [0,1] range means non-preferred images still contribute to training, but this is a specific implementation choice — alternative normalization schemes (e.g., softmax over a batch) or coefficient regimes could produce different results. The paper does not explore whether Reward Weighted's poor performance (39.52%, substantially worse than the baseline) is inherent to loss reweighting or specific to the normalization method chosen.

RAFT's degradation is convincing but the mechanism is underexplored. The paper demonstrates that RAFT's performance collapses with iterations (49.86% → 30.85% → 20.97%), which is compelling evidence of reward over-optimization. However, the paper does not investigate why the images get worse — are they becoming more stereotyped? Exploiting specific ImageReward biases? Losing diversity? Qualitative examples of RAFT degradation are not provided, making it difficult to assess whether the failure mode is reward hacking (producing images that score highly but look unrealistic) or catastrophic forgetting of pre-training knowledge.

All methods use the same hyperparameters and training duration. The paper states that all methods "use the same dataset as the pre-training dataset or generated dataset (both contain 20,000 samples), and train for one epoch with the same training settings (the same learning rate and batch size) for a fair comparison." This is a reasonable fairness strategy, but it means that hyperparameter tuning was done for ReFL and then applied uniformly to baselines that might have different optimal configurations. If Dataset Filtering or Reward Weighted would benefit from different learning rates or training durations, the comparison may be biased in ReFL's favor.

Does the Annotation Pipeline Actually Generalize Beyond DiffusionDB?

The paper's annotation data comes entirely from DiffusionDB prompts, which are sourced from the Stable Diffusion Discord community. This is a specific user population with specific prompt-writing conventions — heavy use of style modifiers ("trending on ArtStation," "8k, highly detailed"), substantial presence of function words, and a bias toward artistic and fantastical content. The paper's ReFL training also uses prompts sampled from DiffusionDB. This creates a potential distributional overfitting problem: ImageReward and ReFL may be optimized for the specific prompt distribution of early Stable Diffusion users, and their performance on prompts from users of other systems (DALL-E, Midjourney users) or on different types of prompts (photorealistic requests, technical diagrams, UI mockups) is unknown. The paper shows in Appendix D that ImageReward's prompt distribution is slightly more evenly distributed than PickScore's, but this is within the DiffusionDB ecosystem — it does not address cross-platform generalization.

The 12-category Parti taxonomy used for annotation (Appendix A.4, Figure 9) shows that Abstract, People, and Arts dominate the prompt distribution, while categories like Illustrations, World Knowledge, and Plants are relatively rare. The reward model may be poorly calibrated for rare categories, and ReFL may not improve (or may even degrade) generation quality for underrepresented prompt types. The paper does not report per-category preference accuracy or ReFL performance.

Can ReFL's Improvements Be Attributed to the Reward Signal, or to Any Fine-Tuning?

The paper uses SD v1.4 as the baseline but does not include a "fine-tuned without reward" baseline — i.e., fine-tuning SD v1.4 on the same pre-training data subset (625k LAION-AES samples) for one epoch without any ReFL loss. This makes it impossible to distinguish improvements caused by the reward feedback from improvements caused by simply continuing pre-training on the curated dataset. The Dataset Filtering and Reward Weighted baselines both fine-tune on additional data, but neither isolates the effect of "more pre-training on the same data distribution." If the 625k LAION-AES subset is already higher quality than the full LAION-5B, then simply fine-tuning on it without any reward signal might account for some (or all) of the observed improvement.

This is particularly relevant because ReFL's training includes 50% pre-training samples (64 per batch of 128) — the model is receiving substantial additional pre-training during ReFL training, beyond what the baseline SD v1.4 received. The paper's argument that the reward signal is responsible for the improvement would be stronger with a "pre-training only" ablation that matches the total number of training steps but omits the ReFL loss.

How Robust Is the "Scores Become Reliable After Step 30" Finding?

The critical insight enabling ReFL — that ImageReward scores are distinguishable only after approximately 30 of 40 denoising steps — is demonstrated for one prompt in Figure 4 and for multiple prompts in Appendix F.1 (Figure 19). The quantitative evidence (Figure 19, left) shows Spearman correlation between step-level scores and final scores rising sharply after step 30. However, the paper does not report whether this threshold generalizes across:

• Different prompt categories: Do scores for Abstract prompts (where the model performs poorly, Appendix A.4) become reliable at the same step as scores for People prompts?
• Different guidance scales: The analysis uses a fixed guidance scale. At different classifier-free guidance scales, the denoising trajectory changes — the "image becomes clear enough" threshold might shift.
• Different numbers of denoising steps: If the model used 100 steps instead of 40, would the threshold still be at 75% of the total steps, or would it shift?
• Different base models: Would a different LDM architecture or a different reward model exhibit a different reliability threshold?

Without this analysis, the ReFL step range [30,40] is a point estimate rather than a characterized property. The random sampling within this range provides some robustness, but if the threshold varies substantially across prompts, some training examples would receive gradients at steps where the reward signal is still noisy, potentially adding variance to the optimization.

Missing Experiments That Would Strengthen the Paper

• Ablation of ReFL step range: Comparing [30,40] to [25,35] or [35,40] would quantify how sensitive ReFL is to the reliability threshold. The paper states that using only step 40 is unstable, but other ranges within the "reliable" window are not tested.

• Ablation of pre-training data ratio in ReFL: The paper uses 50% pre-training samples. Testing different ratios (e.g., 25%, 75%) would reveal whether the pre-training loss is primarily a regularizer or whether the reward signal alone is sufficient with weaker regularization.

• Per-category evaluation of ReFL: Does ReFL improve generation uniformly across prompt categories, or does it improve some categories at the expense of others? The annotation analysis (Appendix A.4) shows that different categories have different failure modes — ReFL might differentially affect these.

• Human evaluation of ImageReward-selected vs. ReFL-generated images: The paper evaluates them separately but never compares ImageReward's best-of-N selection against ReFL's 1-shot generation at equivalent total compute. This would answer the practical question: is it better to fine-tune the model or to just generate more and filter?

• Training curves for ReFL and baselines: The paper reports final performance after one epoch but does not show how performance evolves during training. Reward over-optimization might begin to manifest within a single epoch of ReFL training — training curves would reveal whether early stopping is critical.

• Confidence intervals: No confidence intervals are reported for the main preference accuracy numbers (Table 3), the win rates in human evaluation (Table 4), or the Spearman correlations (Table 1). For a 466-prompt test set with 6,399 pairs, the standard error of the 65.14% accuracy is approximately 0.6%, but this is not reported or discussed.

Summary of Claims vs. Evidence

Claim: "ImageReward outperforms existing scoring methods" — supported with strong evidence across multiple metrics (preference accuracy, recall, filtering, win rates, Spearman correlation) and multiple baselines (CLIP, Aesthetic, BLIP, HPS, PickScore). The performance gap is consistent and substantial. However, the absolute accuracy (65.14%) is close to the inter-annotator agreement ceiling (65.3%), suggesting ImageReward has saturated the achievable signal from the current annotation data rather than being limited by model capacity.

Claim: "ReFL is the first direct tuning method for optimizing diffusion models from scorer feedback" — the novelty claim is supported by the paper's clear differentiation from classifier guidance (which requires noisy-image classifiers) and from indirect data-level methods. The technical mechanism (gradient through partial denoising at late steps) is indeed novel and not present in prior work.

Claim: "ReFL outperforms compared methods" — supported by human evaluation (Table 4, Figure 6) showing ReFL wins against all baselines. The gap to the strongest baseline (Dataset Filtering) is modest (3.62 percentage points) on real user prompts, and the missing "pre-training only" baseline weakens causal attribution of the improvement to the reward signal specifically.

Claim: "ImageReward could serve as a promising automatic text-to-image evaluation metric" — supported by the Spearman correlation of 1.00 with human model rankings (Table 1) and superior distinguishability (Figure 3). The claim is bounded by the 6-model, 100-prompt evaluation scale and the DiffusionDB-centric prompt distribution. Generalization to radically different models or prompt distributions is not established.

Claim: "ReFL's advantages over existing approaches including data augmentation and loss reweighting" — supported for the specific implementations tested. The paper provides a coherent mechanistic explanation for why direct gradient feedback should outperform indirect methods, and the empirical results are directionally consistent with this explanation. However, the failure of Reward Weighted (worse than baseline) and RAFT (degrading with iterations) may reflect suboptimal hyperparameter choices rather than fundamental limitations of those approaches — the paper argues they are fundamental but does not exhaustively search alternative configurations that might close the gap.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Included in the Headline Efficiency Numbers

The paper presents ImageReward as an automatic evaluation metric and selection tool, reporting 4× improvements in selection efficiency (e.g., Recall@4 of 90.84% when selecting 4 from 8 images, Table 3). However, the cost of computing ImageReward scores is never accounted for in any efficiency comparison. Unlike CLIP score or Aesthetic score — which are lightweight cosine similarity or single forward-pass computations — ImageReward requires a full BLIP-based forward pass through both image and text encoders with cross-attention, plus an MLP head, for every text-image pair being scored. For best-of-N selection (evaluating N images per prompt), this means N separate ImageReward inferences. For model-level ranking with 30k generations per model (as done for MS-COCO in Table 1), this means 30k × 6 = 180k ImageReward evaluations. The paper does not report inference latency, FLOP count, or memory requirements for ImageReward relative to the baselines it outperforms.

The consequence is that practitioners cannot determine whether ImageReward's improved selection accuracy justifies its computational overhead compared to cheaper but less accurate alternatives (CLIP score, Aesthetic predictor). In latency-sensitive applications, the cost of scoring candidates may dominate the total pipeline cost, especially when the alternative is simply generating more images and using a cheaper filter. The paper implicitly assumes that scoring cost is negligible relative to generation cost, but for models like Stable Diffusion with relatively fast inference, this may not hold — ImageReward's BLIP backbone is itself a substantial model. The paper provides no evidence on whether ImageReward's selection advantage persists under a total compute budget that includes scoring cost.

The paper does not measure or discuss scoring inference cost anywhere. The efficiency comparisons in Table 3 and Figure 5 treat ImageReward scores as free — they compare selection accuracy at fixed N without accounting for the computational difference between computing an ImageReward score versus computing a CLIP score. The mitigation status is complete absence: the limitation is not acknowledged, and no suggestions for reducing scoring cost (e.g., model distillation, cascaded filtering with a cheaper first stage) are provided.

---

The ReFL Algorithm Requires On-the-Fly Image Generation During Training, Making It Prohibitively Expensive at Scale

ReFL's core mechanism — generating images from the current model parameters during training, scoring them with ImageReward, and backpropagating through the partial denoising trajectory — is computationally intensive in ways that the paper does not quantify. Each training iteration requires running the full 40-step denoising process for 39 steps without gradients, then one step with gradients, plus a forward pass through ImageReward (BLIP backbone) for every ReFL sample in the batch. The paper's training configuration uses a batch size of 128 total (64 pre-training + 64 ReFL) across 8 A100 GPUs, but it does not report the wall-clock time per iteration, total training time, or how training time compares to the baseline methods.

This matters because the paper positions ReFL as a practical alternative to indirect data-level methods, but the computational cost comparison is entirely absent. Dataset Filtering and Reward Weighted both use pre-generated images — the expensive generation step happens once offline, and training proceeds at standard fine-tuning speeds. RAFT requires periodic re-generation between iterations, but the generation is also offline. ReFL, by contrast, requires generation during every training step, which could make it 10–40× more expensive per parameter update depending on the relative cost of generation versus gradient computation. A practitioner deciding whether to adopt ReFL needs to know: does the 3.62 percentage point win rate improvement over Dataset Filtering (58.79% vs. 55.17%, Table 4) justify the additional training cost?

The paper provides no direct evidence on training cost. The methods are compared at equal epochs (one pass through the training data), which is a data-efficiency comparison but not a compute-efficiency comparison. If ReFL training takes 20× longer per epoch than Dataset Filtering, the "fair comparison" at equal epochs is not fair in total compute terms. The mitigation status is: the limitation is not discussed. The paper acknowledges ReFL's generation requirement in the algorithm description but treats it as a methodological detail rather than a practical cost consideration.

---

The Evaluation Is Limited to a Single Model Family, a Single Dataset Ecosystem, and Narrow Prompt Distributions

All experiments — both ImageReward training/evaluation and ReFL optimization — are conducted within a tightly constrained scope. ImageReward is trained exclusively on images generated by Stable Diffusion-family models (the images in DiffusionDB come from the Stable Diffusion Discord community). ReFL is tested exclusively on Stable Diffusion v1.4. The prompt distribution is entirely drawn from DiffusionDB, which captures early Stable Diffusion users' prompting patterns — heavy use of style modifiers, artistic references, and function words like "8k" and "trending on ArtStation." The MS-COCO evaluation in Table 1 is supplementary and not validated against human preferences on MS-COCO captions.

The paper explicitly acknowledges the annotation diversity limitation in Appendix H:

"our current prompts are all sampled from DiffusionDB, which is an abundant collection of human real use but still exists some bias. Despite these prompts may close to many real cases, biases exist since the real application when people use the text-to-image model are far beyond trying strange prompts."

However, the scope limitations extend beyond just the prompt distribution:

• Generator model distribution shift: ImageReward was trained on images from specific Stable Diffusion versions. If deployed to score images from DALL-E, Midjourney, or future models with different visual characteristics, its calibration may degrade. The paper's PRM (Process Reward Model) training on Stable Diffusion outputs means it has never seen the failure modes, aesthetic styles, or artifact patterns of other model families.

• ReFL transferability: ReFL was tested only on SD v1.4. Whether the algorithm works for other diffusion model architectures (e.g., pixel-space diffusion, cascaded models, models with different numbers of denoising steps) is unknown. The step reliability threshold (step 30 of 40) is specific to SD v1.4's 40-step PNDM schedule — different schedulers or step counts would require re-characterizing the threshold.

• Task scope: MATH benchmark-style evaluation is not present here, but the equivalent concern applies: all evaluation is on open-ended image generation from text prompts. Whether ImageReward or ReFL generalize to related tasks (image editing, inpainting, style transfer, multi-modal conditioning) is unexplored.

The consequence is that the paper's claims about ImageReward's general-purpose nature and ReFL's effectiveness are bounded by these distributional constraints. A practitioner using Midjourney or DALL-E cannot assume ImageReward will rank those models' outputs correctly. A researcher applying ReFL to a different diffusion architecture cannot assume the step-30 threshold transfers. The mitigation status is partial: Appendix H acknowledges annotation diversity as a limitation and suggests exploring "more diverse prompts distribution," but the model-family and task-distribution constraints are not discussed.

---

Reward Over-Optimization Is Demonstrated as a Failure Mode but Not Characterized or Mitigated

The paper provides clear evidence that reward-based optimization can backfire — RAFT's performance collapses from 49.86% to 20.97% win rate over three iterations (Table 4), substantially worse than the unmodified SD v1.4 baseline. This is strong evidence of reward over-optimization in image generation. However, the paper provides no analysis of whether ReFL itself is susceptible to the same phenomenon at longer training durations, different loss weights, or different hyperparameter settings.

The ReFL results are reported after a single epoch of training with a fixed $\lambda = 10^{-3}$ and a 50/50 pre-training-to-ReFL batch ratio. The paper does not show training curves that would reveal whether ImageReward scores on generated images are increasing while human preference plateaus or declines — the classic signature of reward over-optimization. The paper does not explore whether ReFL's performance would degrade with additional training epochs, larger $\lambda$ values, or reduced pre-training regularization. The choice of one epoch and the specific $\lambda$ value was determined empirically, but the sensitivity of the result to these choices is not reported.

This matters because reward over-optimization is the central risk of using learned reward models for generator training, and the paper's strongest contribution — the ReFL algorithm — is presented without any characterization of its robustness to this failure mode. The RAFT results demonstrate that the ImageReward signal can be exploited, and ReFL's direct gradient access to the reward model might make it more susceptible to reward hacking than data-level methods, not less. The paper's argument that ReFL avoids RAFT's self-training amplification is plausible but not empirically validated beyond the single-epoch result.

The paper provides no evidence on ReFL's over-optimization behavior. The training protocol (one epoch, fixed hyperparameters) was selected to produce the reported results, but the absence of training dynamics analysis means the reader cannot assess whether the reported ReFL performance represents a stable optimum or a point on a trajectory that would eventually diverge. The mitigation status is: the limitation is not discussed for ReFL specifically, though the RAFT degradation results implicitly raise the question.

---

Human Evaluation Relies on Small Sample Sizes Without Reported Confidence Intervals

The paper's key empirical claims depend on human evaluation studies — model ranking (100 prompts, Table 1), preference accuracy (466 prompts, Table 3), and ReFL comparison (466 + 90 prompts, Table 4). None of these results are reported with confidence intervals, statistical significance tests, or effect size measures. The Spearman correlation of 1.00 in Table 1 is based on 6 models — with 6 data points, the 95% confidence interval for a Spearman correlation spans from approximately 0.66 to 1.00, meaning the true correlation could be substantially lower than perfect. The 58.79% ReFL win rate is based on 1,508 wins out of a total pool of comparisons, but without confidence intervals, the reader cannot determine whether the 3.62 percentage point gap to Dataset Filtering is statistically distinguishable from noise.

The consequence is that the paper's quantitative claims are less precise than they appear. The headline numbers (65.14% accuracy, 1.00 Spearman correlation, 58.79% win rate) are point estimates from finite samples, and the uncertainty around them could materially affect conclusions. For example, if the true Spearman correlation between ImageReward and human rankings is 0.80 (within a plausible confidence interval for n=6), the claim of "perfect alignment" would be misleading. If the ReFL win rate confidence interval overlaps with Dataset Filtering, the claimed superiority would be uncertain.

The paper does report standard deviations for inter-annotator agreement (Table 2a: ±11.1% for researchers, ±5.6% for annotators), which demonstrates awareness of variance in human judgments. However, these measures are not propagated to the downstream metrics that depend on them. The paper provides no standard errors, confidence intervals, or significance tests for any of the main results in Tables 1, 3, or 4. The mitigation status is: the limitation is not acknowledged. The test sets are described as having specific sizes (466 prompts, 100 prompts), but the statistical implications of these sample sizes are not discussed.

---

ImageReward's Architecture and Training Choices Are Empirically Motivated but Theoretically Unexamined

The paper makes several important design choices — BLIP backbone over CLIP, 70% layer freezing, last-step-only score aggregation, ReLU reward-to-loss mapping, $\lambda = 10^{-3}$ — that are determined through grid search or empirical observation without theoretical justification or systematic sensitivity analysis. While empirical tuning is standard practice, the paper does not provide enough information for practitioners to adapt these choices to new settings. A reader wanting to train an ImageReward-like reward model for a different image domain (medical imaging, satellite imagery, fashion) would not know: (a) whether BLIP's advantage over CLIP generalizes, (b) what fraction of layers to freeze for a different backbone or dataset size, (c) how $\lambda$ should scale with dataset size or model capacity.

The specific concern for ReFL is the interaction between the step range $[30, 40]$ and the number of denoising steps. The paper uses 40 steps throughout. If a practitioner uses 20 steps (for faster inference) or 100 steps (for higher quality), the "30–40" range does not directly translate — the threshold is presumably a fraction of total steps, but whether it's exactly 75% (30/40) for all configurations is unknown. The paper's Figure 19 (Appendix F.1) shows the step-reliability correlation for 40-step generation, but no analysis with different step counts.

The consequence is that the paper's prescriptions are point solutions rather than characterized design principles. The community learns that BLIP works better than CLIP for this specific task and dataset, but not why or under what conditions this would change. The community learns that freezing 70% of layers works for the 8k-prompt ImageReward training, but not how this fraction should vary with dataset size or backbone capacity — the paper reports that performance improves from 1k to 8k prompts (Table 2b), suggesting more data would help, but if more data reduces overfitting, the optimal freezing fraction would change.

The paper partially mitigates this through ablations: the backbone comparison (Table 2b) and training data scaling are directly tested. However, the ReFL hyperparameters ($\lambda$, step range, $\phi$ function, pre-training ratio) are not ablated, and the layer freezing fraction is described qualitatively without a sweep table. The paper's suggestion in Appendix H that "more advanced techniques (e.g., parameter-efficient tuning) could be helpful for the problem" acknowledges that the specific regularization approach may be suboptimal, but does not provide guidance for alternatives.

7. Implications and Future Directions

How This Work Changes the Landscape

ImageReward and ReFL together represent a methodological shift in how the field approaches text-to-image alignment, not an incremental improvement over existing scoring methods. The paper's contribution is analogous to what RLHF achieved for language models — it demonstrates that human preference feedback can be systematically collected, modeled, and used as a direct optimization signal for generative models in a domain where this was previously considered infeasible due to technical barriers. The magnitude of this shift is significant but bounded: it opens a new research direction (reward-guided diffusion optimization) that was previously blocked by the likelihood-free nature of diffusion models, but it does not solve all problems within that direction — the paper explicitly documents that hard cases (intrinsically difficult prompts, rare categories) remain largely unaddressed, and the optimization signal is limited by reward model reliability.

The conceptual change is twofold. First, the paper reframes text-to-image evaluation from distribution-matching to preference-prediction. The demonstration that CLIP score and FID have Spearman correlations of 0.60 and 0.09 respectively with human model rankings (Table 1), while ImageReward achieves 1.00, is not just a better number — it exposes a fundamental mismatch between what the field has been measuring and what users actually care about. CLIP score measures semantic similarity in a joint embedding space; FID measures distributional distance to reference images. Neither captures the multidimensional human preference signal (alignment + fidelity + harmlessness) that determines whether a generated image is actually useful. By constructing an annotation pipeline that decomposes preference into explicit, trade-off-aware criteria and then training a model to predict the resulting pairwise judgments, the paper establishes a new evaluation paradigm where the metric is optimized to predict what humans will say, not what mathematical convenience dictates.

Second, the paper demonstrates that direct gradient-level reward feedback is qualitatively different from data-level preference incorporation and provides empirical evidence for this hierarchy in image generation. The ReFL results (58.79% win rate vs. 55.17% for Dataset Filtering, 39.52% for Reward Weighted, and degrading RAFT performance, Table 4) establish that how you use the reward signal matters as much as having it. This parallels the NLP finding that RLHF fine-tuning produces better instruction-following than supervised fine-tuning on curated demonstration data, but adapted to the fundamentally different technical constraints of diffusion models. The gradient-through-partial-denoising mechanism is the enabling innovation — it converts an impossibility (reward models can't score noisy latents) into a practical algorithm (score predicted clean images at late steps where scores become reliable, then backpropagate through that single step).

The paper also resolves a latent contradiction in prior work on human feedback for image generation. Before ImageReward, the few existing approaches fell into two camps: inference-time classifier guidance (impossible because reward models require clean images for training) and indirect data filtering (possible but limited because the optimization signal is decoupled from the reward function's gradients). Concurrent work had explored both but had not articulated why they were structurally limited or provided evidence that a direct gradient approach would outperform them. The paper's clear taxonomy — data-level vs. gradient-level methods — and empirical comparison provides a framework for understanding why some approaches work better than others, moving the discussion from "we tried X and it helped" to "X is structurally limited because it doesn't propagate reward gradients to the generator."

This work redirects research attention in several ways. It makes reward model quality the central bottleneck for further progress — if ReFL's effectiveness is bounded by ImageReward's reliability, then improving reward models (through larger annotation datasets, better backbones, more sophisticated training) is the highest-leverage investment, not developing more complex search or optimization algorithms. The RAFT degradation results (49.86% → 30.85% → 20.97% win rate over three iterations, Table 4) serve as a cautionary tale: more aggressive optimization of a fixed reward model leads to exploitation, not genuine improvement. This implies that reward model robustness — resistance to over-optimization — is as important as reward model accuracy.

Conversely, the paper makes inference-time best-of-N selection with a learned reward model a more attractive paradigm than previously recognized. ImageReward's Recall@4 of 90.84% (Table 3) means that selecting the top 4 from 8 generated images misses the human-preferred best image only ~9% of the time. For applications where generation is cheap and human review is expensive, this is a practical deployment strategy that doesn't require any model fine-tuning — just a good reward model and sufficient generation budget. The paper's finding that ImageReward outperforms CLIP score by ~38.6% (abstract) in preference prediction makes this paradigm substantially more viable than it was with previously available scoring methods.

Follow-Up Research This Work Enables

Characterizing the reward reliability threshold as a function of denoising steps, model architecture, and prompt type. The paper's central enabling insight — ImageReward scores become reliable after step 30 of 40 — is demonstrated for SD v1.4 with 40 PNDM steps on an unspecified set of prompts (Figure 4 shows one prompt; Appendix F.1 Figure 19 shows aggregate correlations). A systematic study would measure the Spearman correlation between step-level and final scores across: (a) different numbers of total denoising steps (20, 40, 100, 1000), testing whether the threshold is a fixed fraction (~75%) or an absolute number of steps from completion; (b) different diffusion architectures (pixel-space diffusion, cascaded models, consistency models), testing whether the threshold depends on the latent space properties; (c) different prompt categories (using the 12-category Parti taxonomy from the paper's annotation data), testing whether Abstract prompts where the model performs poorly have a different threshold than People prompts where performance is stronger; (d) different classifier-free guidance scales, testing whether stronger guidance shifts the threshold earlier or later. The result would be a reliability threshold function that practitioners can use to set ReFL's step range for any diffusion configuration, converting the paper's point observation into a general design principle. A negative result — finding that the threshold varies unpredictably across configurations — would imply that ReFL requires per-configuration calibration, limiting its practical deployability.

Scaling the annotation dataset by an order of magnitude and measuring reward model accuracy saturation. The paper's ablation (Table 2b) shows that preference accuracy improves from 63.07% at 1k prompts to 65.14% at 8k prompts — a 2.07 percentage point gain that has not saturated. The inter-annotator agreement ceiling is 65.3% (Table 2a), suggesting the model has little room to improve on the current annotation distribution. A direct follow-up would collect 80k–137k annotated prompts (10–17× the current scale) using the same pipeline, train progressively larger reward models (BLIP-L, BLIP-2, larger vision transformers), and measure: (a) whether preference accuracy asymptotically approaches the inter-annotator ceiling or surpasses it (the annotator ensemble achieves 73.4% agreement with researchers, suggesting there is headroom above individual annotator accuracy if the model can learn to aggregate multiple annotators' implicit consensus); (b) whether ReFL-trained generators improve monotonically with reward model quality or exhibit diminishing returns, establishing the shape of the "reward model scaling law" for downstream optimization; (c) whether rare prompt categories (Plants, Illustrations, World Knowledge — each with <5% of the annotation data, Figure 9) benefit disproportionately from increased data, or whether preference in those categories is inherently harder to model. This would determine whether the community should invest in larger annotation efforts (if scaling continues) or focus on better architectures and training techniques (if saturation is imminent). The paper's Appendix H explicitly identifies annotation scale as a limitation, making this a natural follow-up.

Testing ReFL against a properly regularized RL baseline for diffusion models. The paper argues that standard RLHF algorithms (PPO) cannot be directly applied to diffusion models because likelihoods are intractable, and presents ReFL as the solution. However, recent work on diffusion model fine-tuning has developed alternative RL formulations — notably, DDPO (Black et al., 2023) uses score function estimators to compute policy gradients for diffusion models without requiring exact likelihoods, and DPOK (Fan et al., 2023) adapts the RLHF framework using denoising process likelihood bounds. A controlled comparison would train all three methods (ReFL, DDPO-style score function RL, DPOK-style likelihood-bound RL) using the same ImageReward model on the same SD v1.4 base, with the same pre-training data and compute budget, and evaluate via the paper's human evaluation protocol. The key question is whether ReFL's gradient-through-partial-denoising mechanism is genuinely superior to alternative gradient estimators for diffusion policy optimization, or whether it's simply the first demonstrated method. If DDPO matched or exceeded ReFL's 58.79% win rate, it would suggest that the direct-vs-indirect taxonomy is less important than simply having any gradient-level optimization method. If ReFL won, it would provide evidence that the partial-denoising gradient has lower variance or better bias properties than score function estimators for this task. The paper's introduction frames ReFL as solving the "cannot yield likelihoods" problem, but doesn't compare against alternative likelihood-free RL approaches — this comparison would contextualize ReFL's contribution within the broader RL-for-diffusion landscape.

Measuring and mitigating reward over-optimization in ReFL training. The paper demonstrates that RAFT catastrophically over-optimizes ImageReward (49.86% → 20.97% over three iterations, Table 4), but does not characterize ReFL's own susceptibility. A systematic study would train ReFL for multiple epochs, tracking both ImageReward scores and human preference win rates at regular intervals, and test whether the gap between reward model scores and human judgments widens over training — the hallmark of reward hacking. Key ablations would include: varying $\lambda$ (the reward loss weight) from $10^{-4}$ to $10^{-1}$, testing whether higher $\lambda$ accelerates over-optimization; varying the pre-training batch ratio from 25% to 75%, testing whether stronger regularization delays over-optimization; and testing whether over-optimization manifests differently across prompt categories (do the model's outputs become more stereotyped? Do they exploit specific ImageReward biases, like generating higher-scoring but less diverse images?). The crucial metric is not just win rate at a single checkpoint but the robustness curve — how quickly performance degrades past the optimal point, and whether the optimal checkpoint is sharp (requiring careful early stopping) or broad (robust to training duration). The paper's finding that RAFT degrades severely with iterations while ReFL performs well at one epoch could mean ReFL is genuinely more robust, or it could simply mean the paper stopped training before over-optimization set in — this follow-up would distinguish these interpretations. A negative result — finding that ReFL also over-optimizes after 2–3 epochs — would not diminish the paper's contribution (the one-epoch result still demonstrates the method works) but would establish important guardrails for practitioners.

Investigating whether ReFL improves or degrades performance on rare and challenging prompt categories. The paper's annotation analysis (Appendix A.4) reveals systematic performance differences across the 12 prompt categories: Abstract prompts receive the lowest average scores across all three dimensions (alignment, fidelity, overall), body problems are concentrated in People and Arts categories (Figure 11), and function words have non-linear effects on quality (Figure 12). A per-category evaluation of ReFL would measure whether the fine-tuned model's improvements are uniform across categories or concentrated in already-strong categories (People, Outdoor Scenes) while potentially degrading performance on weak categories (Abstract, World Knowledge). This would use the paper's own category taxonomy and annotation criteria: generate images for prompts from each category using both SD v1.4 and the ReFL-fine-tuned model, have annotators compare them using the same rating and ranking protocol, and compute per-category win rates. If ReFL improves strong categories but degrades weak ones (a "rich get richer" effect), it would suggest that reward model scores are less reliable for out-of-distribution prompts and that ReFL should be applied selectively or with category-specific regularization. If ReFL improves uniformly, it would strengthen the claim that the reward signal generalizes. This matters practically because deployed systems receive prompts from all categories, and a model that is better on average but worse on rare long-tail prompts may have unacceptable failure modes.

Developing lightweight difficulty estimation for text-to-image prompts to enable adaptive ReFL or best-of-N allocation. The paper's annotation pipeline produces per-prompt statistics (average scores, problem frequencies) that correlate with prompt difficulty — Abstract prompts score lower, prompts with >60% function words produce fewer body problems (Figure 13), certain categories have higher failure rates. A natural extension would train a prompt difficulty predictor — a lightweight model (e.g., fine-tuned BERT or a small language model) that takes only the prompt text as input and predicts the expected ImageReward score or problem probability, trained on the paper's 8,878 annotated prompts with their per-prompt aggregate statistics. This predictor could then be used for adaptive resource allocation: (a) for best-of-N selection, allocate more generation budget to hard prompts where the best image is harder to find, less to easy prompts where few samples suffice; (b) for ReFL training, weight the reward loss by inverse predicted difficulty so the model focuses learning on prompts where it has the most room to improve; (c) for inference, potentially skip ReFL fine-tuning's effects for extremely hard prompts where no amount of test-time optimization helps (analogous to the paper's finding that function word proportion affects quality non-linearly). The evaluation would measure whether adaptive allocation achieves equivalent overall quality with reduced total generation budget — directly extending the paper's annotation analysis into a practical deployment optimization. This is made tractable because the paper's annotation data already contains the per-prompt aggregate labels needed for training.

Practical Applications and Downstream Use Cases

Cost-efficient best-of-N filtering for production text-to-image APIs. Image generation APIs (DALL-E, Midjourney, Stable Diffusion endpoints) currently face a fundamental tension: users want high-quality images on the first try, but single-generation quality is inconsistent. A common workaround is generating multiple images and having the user pick, but this increases compute cost and user friction. ImageReward enables automatic best-of-N selection: generate N images server-side, score them with ImageReward, return only the top k. With Recall@4 of 90.84% (Table 3), a system generating 8 images and returning the top 4 would include the human-preferred best image ~91% of the time while reducing user-facing options by 50%. For an API handling millions of requests daily, cutting average images-per-request from 4 (user selects) to 2 (system pre-filters then user selects) halves generation cost while maintaining output quality. The counter-argument — that ImageReward scoring adds its own inference cost — would need to be evaluated against the generation savings, but for expensive models (DALL-E 2, Imagen) where generation far exceeds scoring cost, the tradeoff is favorable. The paper's preference accuracy of 65.14% and the 69.3% win rate over CLIP-based selection (Figure 5) provide the quantitative justification that ImageReward is reliable enough for this use case to outperform existing lighter-weight scorers.

Improving open-source text-to-image model alignment without full retraining. The paper's ReFL results demonstrate that a single epoch of reward-guided fine-tuning on SD v1.4 yields a 58.79% win rate against the base model in human evaluation (Table 4). For organizations deploying open-source text-to-image models (Stable Diffusion, SDXL, community fine-tunes), ReFL provides a practical fine-tuning recipe that requires: (a) a pre-trained reward model (ImageReward is publicly released), (b) a pre-training dataset subset (the paper uses 625k LAION-AES samples), and (c) a prompt set from the target user distribution (DiffusionDB or the organization's own prompt logs). The 8-A100 training configuration is expensive but feasible for a one-time fine-tuning investment. The resulting model generates measurably more human-preferred images — particularly for prompts with specific attribute requirements (the qualitative example in Figure 7 shows ReFL correctly rendering "long pointy ears" while baseline methods fail). The key practical concern for adopters is the ReFL hyperparameter sensitivity ($\lambda = 10^{-3}$, step range [30,40], 50/50 pre-training ratio) — the paper established these values for SD v1.4 but does not characterize how they generalize to other base models. Organizations adopting ReFL would need to replicate the hyperparameter sweep on their target model, which the paper's public code release facilitates.

Automatic evaluation for text-to-image model leaderboards and benchmarking. The current standard for comparing text-to-image models relies on FID (which the paper shows has Spearman correlation of 0.09 with human rankings on real prompts, Table 1) or CLIP score (correlation 0.60). Both metrics can be gamed — models can be optimized to produce high CLIP scores without improving actual quality — and neither captures the multidimensional preference that users care about. ImageReward offers a drop-in replacement evaluation metric that correlates with human judgments (Spearman 1.00 on the tested models) and is publicly available. A benchmark suite using ImageReward would: sample a diverse prompt set (generalizing beyond DiffusionDB to cover multiple user populations), generate images from candidate models, score all images with ImageReward, and report both average scores and win rates via pairwise comparisons. The paper's demonstration that ImageReward scores have higher distinguishability than CLIP scores (larger interquartile ranges, Figure 3) means the metric can detect quality differences between models that CLIP would miss. The practical limitation is the DiffusionDB-centric training distribution — a benchmark using ImageReward would need to validate that ImageReward's human alignment holds for the benchmark's specific prompt distribution, or retrain/fine-tune ImageReward on benchmark-relevant annotation data. The paper's annotation pipeline design document (Appendix B) provides a template for such retraining.

Improving safety filtering through preference-aware content assessment. The paper's annotation pipeline explicitly rates images on harmlessness and provides binary flags for toxic, violent, pornographic, and psychologically disturbing content (Appendix A.3, B.2). ImageReward was trained to predict overall preference incorporating these harmlessness judgments, meaning its scores implicitly capture safety-relevant features. A deployment scenario would use ImageReward as a learned safety filter: before returning generated images to users, score them with ImageReward and block or flag those below a threshold score, especially for prompts flagged during pre-annotation as potentially problematic. The paper's finding that harmlessness is the highest-priority criterion in the annotation trade-off hierarchy (Appendix B: "for most tasks, fidelity and harmlessness are more important than image-text alignment") means ImageReward is explicitly trained to penalize harmful content even at the expense of alignment quality — precisely the behavior desired in a safety filter. The advantage over simple blocklists or CLIP-based toxicity classifiers is that ImageReward considers harmfulness in context (an anatomically correct nude in a medical illustration context is different from gratuitous sexual content), which the annotation document's handling of edge cases (Appendix B.4: "if the text requires artistic creation then nudity is allowed") explicitly encodes. The limitation is that ImageReward's harmlessness assessment is only as good as the annotation data covering harmful content — the paper reports some toxic/pornographic/violent content in the annotation data (Figure 11, 13) but the absolute frequency is low, suggesting ImageReward may have limited training signal for rare safety-critical cases.