Leveraging Verifier-Based Reinforcement Learning in Image Editing

ArXiv: 2604.27505

🎯 Pitch

Image editing reward models typically give a single holistic score, ignoring the conflicting demands of following instructions versus preserving unedited regions. Edit-R1 introduces a reasoning verifier that explicitly decomposes an instruction into checkable sub-principles, scores each one, and trains editing models via RL—achieving a 15.2% relative gain over Qwen-Edit on motion edits without any task-specific fine-tuning.

---

1. Executive Summary

This paper introduces Edit-R1, a framework for building and leveraging a verifier-based Reasoning Reward Model (RRM) to enhance image editing through Reinforcement Learning from Human Feedback. Rather than treating reward modeling as holistic scoring, the RRM functions as a reasoning verifier—decomposing editing instructions into distinct principles, evaluating edited images against each principle via Chain-of-Thought (CoT) analysis, and aggregating these checks into an interpretable score. The RRM is trained through a two-stage pipeline: a cold-start supervised fine-tuning phase using principle-decomposed data filtered by an external VLM quality-control judge, followed by Group Contrastive Preference Optimization (GCPO)—a novel RL algorithm that refines the pointwise RRM by contrasting groups of winner and loser reasoning trajectories derived from pairwise human preference data (computing cross-group win/loss ratios as per-sample rewards). The resulting 7B RL-RRM achieves 82.2% accuracy on the authors' internal benchmark, surpassing both strong proprietary VLMs like Seed-1.5-VL (79.3%) and the concurrent EditScore-7B on the public EditRewardBench (78.2% vs. 65.9%), while demonstrating a clear scaling trend from 3B to 7B parameters. When deployed as a non-differentiable reward signal within a GRPO-based reinforcement learning loop, Edit-R1 boosts FLUX.Kontext's overall score on GEdit-Bench-EN from 5.77 to 6.24 and yields a 15.2% relative gain on Qwen-Edit's Motion Change category (from 4.01 to 4.62), establishing that verifier-based reasoning rewards can substantially improve downstream editing models even when those models are already highly optimized.

2. Context and Motivation

The Core Problem: Image Editing Lacks Robust Reward Models for RLHF

The fundamental problem this paper tackles is that Reinforcement Learning from Human Feedback (RLHF) has become a standard post-training paradigm for text-to-image generation, but its application to image editing remains largely unexplored. This is not because researchers lack interest—it is because the underlying infrastructure of RLHF, particularly the reward model that provides training signal, is much harder to build for editing than for generation.

To understand why, consider what an image editing reward model must evaluate. In text-to-image generation, the model produces an image from scratch given a text prompt. A reward model for this task mainly needs to judge whether the output matches the prompt and looks visually appealing. This is a "does A match B?" problem. But in image editing, the model starts from an existing image and must modify it according to an instruction while preserving everything not mentioned in the instruction. The evaluation now spans at least three dimensions simultaneously: (1) instruction fidelity—did the edit actually happen as specified? (2) preservation of unedited regions—are background elements, secondary objects, and subtle details still intact? (3) overall visual quality—does the final image look coherent and artifact-free?

This is a fundamentally more demanding evaluation task, and the paper argues that the dominant paradigm in image editing reward modeling—using a general-purpose Vision Language Model (VLM) as a holistic scorer that outputs a single scalar score without explicit verification—is fundamentally inadequate. The authors cite evidence that such scorers produce "biased or hallucinated feedback" (Section 1), citing Gunjal et al. (2025) as supporting evidence. The core failure mode is intuitive: when you ask a VLM to summarize everything about an edit into one number, it inevitably weights some aspects arbitrarily, ignores others entirely, or produces scores that correlate poorly with human judgment because the reasoning behind the score is opaque and unverifiable.

Why Image Editing Specifically, and Why Now?

The timing of this work is not coincidental. Diffusion and flow-based image editing models have advanced dramatically, with systems like FLUX.Kontext (Batifol et al., 2025), SeedEdit 3.0 (Wang et al., 2025), BAGEL (Deng et al., 2025), and Qwen-Image-Edit (Wu et al., 2025) pushing the frontier of what is possible in instruction-guided editing. These models can perform complex operations—changing object materials, relocating subjects, modifying text in images, and applying stylistic transformations—that were infeasible just a few years ago.

However, as these editing models improve, the standard training paradigm remains stuck at pretraining and supervised fine-tuning (SFT). The paper explicitly notes this limitation: "research still centered on pretraining and supervised fine-tuning (SFT)" (Section 1). The implication is clear: we are hitting diminishing returns from scale alone, and the next frontier is aligning these models with human preferences through RLHF—exactly as happened in T2I generation, where RLHF became a "pivotal paradigm" driven by powerful reward models and optimization algorithms. But editing RLHF cannot simply borrow T2I reward models, because the evaluation problem is qualitatively different.

The absence of robust editing-specific reward models creates a bottleneck that affects multiple stakeholders:

• Researchers cannot run stable RLHF training loops for editing models because the reward signal is too noisy or biased.
• Practitioners deploying editing models lack reliable automated metrics to judge model quality, benchmark progress, or filter outputs.
• End users experience edited outputs that may partially satisfy an instruction but fail in subtle ways—a background element slightly distorted, a secondary object's color shifted, or an edit applied with insufficient magnitude.

The paper frames this as an infrastructure gap: RLHF for image editing is waiting for reward models that match the sophistication of the editing models themselves.

Where Existing Reward Models Fall Short

The paper provides a detailed taxonomy of existing reward models in Table 1 and Section 2.1, revealing a fragmented landscape with systematic weaknesses that Edit-R1 is designed to address.

Holistic scorers ignore instruction structure. The dominant paradigm in editing reward modeling is to use a general-purpose VLM to output a single score. For example, Skywork-EditReward (Wei et al., 2025) produces pointwise scores but does not decompose the instruction into sub-tasks or perform explicit verification. OneReward (Gong et al., 2025) applies to editing but is purely a scorer without reasoning. The concurrent EditScore (Luo et al., 2025) treats the reward model as a holistic scorer. These approaches share a common weakness: when the model outputs "7.3" without explaining how it arrived at that number, there is no way to verify whether it correctly distinguished between instruction-following failures and preservation failures, or whether it hallucinated changes that did not actually occur. The paper characterizes this as biased or hallucinated feedback—a claim that is empirically validated when their verifier-based model substantially outperforms these scorers on both internal and public benchmarks (Table 2: Qwen-7B with Think+Verify reaches 75.4% vs. 68.3% for a VIESCORE-prompted baseline).

Pointwise vs. pairwise mismatch. Table 1 reveals a critical design tension that prior work has not resolved. Pointwise methods (ImageReward, VideoAlign, WorldPM) score single outputs independently, which makes them suitable for the kind of continuous reward signal that RLHF optimization algorithms expect—you need a scalar score for each generated image to compute advantages within a group. Pairwise methods (RewardDance, Pairwise RM) compare two outputs to determine which is better, which is often more aligned with how human preference data is collected but cannot provide absolute quality scores for individual outputs. The paper explicitly identifies this as a limitation: "A significant drawback of pairwise approaches is their inability to provide an absolute quality score for a single response, making them ill-suited for direct quality assessment or filtering" (Section 2.1). This is why the paper's GCPO algorithm (Section 3.1.2) is specifically designed to bridge this gap—using pairwise human preference data to train a pointwise verifier—a contribution that addresses a genuine architectural tension in the field.

Reasoning-capable models exist but are not designed for visual editing. The paper acknowledges that several recent reward models have begun incorporating Chain-of-Thought reasoning and explicit principles—notably DeepSeek-GRM (Liu et al., 2025), UnifiedReward (Wang et al., 2025), and VisualQuality-R1 (Wu et al., 2025). These models represent the state of the art in interpretable, verifiable reward signals. However, they are either designed for non-visual tasks (DeepSeek-GRM targets generalist reward modeling), text-to-image generation rather than editing (VisualQuality-R1), or general multimodal evaluation rather than the specific demands of editing evaluation.

The missing combination. The last four columns of Table 1 tell the story. Edit-RRM is the only model that simultaneously supports: (1) pointwise scoring (needed for RLHF optimization of editing models), (2) working as a verifier with explicit principles (needed to decompose complex editing instructions into checkable sub-tasks), (3) Chain-of-Thought reasoning (needed to make the evaluation interpretable and auditable), and (4) reinforcement learning for training the reward model itself (needed to align reasoning with human preferences). No prior work—and importantly, no concurrent work like EditScore—achieves all four simultaneously for image editing. This is the specific gap the paper fills.

Why RLHF Algorithms for Editing Are Stuck

The paper identifies a second, related bottleneck that compounds the reward model problem: existing RLHF algorithms for image editing are incompatible with reasoning-based reward models (Section 1, Section 2.2).

The state of the art in editing RLHF uses algorithms like REFL (Xu et al., 2023) or DPO (Rafailov et al., 2023). REFL, applied in works like ByteEdit (Ren et al., 2024) and OneReward (Gong et al., 2025), requires the reward model to be differentiable—the gradient of the reward with respect to the generated image must flow back through the model. This works when the reward model is a regression head on a VLM that outputs a continuous scalar. But a verifier-based reward model that first generates a multi-step reasoning trace (sampling discrete tokens) and then produces a score is inherently non-differentiable. You cannot backpropagate through token sampling. The paper states this explicitly: "Since the RRM generates an explicit multi-step reasoning trace through discrete token sampling before producing a final score, the process is inherently non-differentiable, rendering REFL-style methods inapplicable" (Section 1).

DPO, on the other hand, directly optimizes the policy on a preference dataset without a separate reward model. While this avoids the differentiability problem, the paper notes that DPO "direct optimization on a preference dataset inherently restricts policy exploration, risking suboptimal convergence" (Section 2.2). Recent work has moved toward GRPO-based approaches (Flow-GRPO by Liu et al., 2025; DanceGRPO by Xue et al., 2025) inspired by DeepSeek-R1's success, but these require verifiable rewards to ensure robust training and prevent reward hacking. The paper explicitly connects this observation to the editing domain: "Yet, defining such rewards for visual generation remains challenging. To address this, we extend the visual GRPO algorithm with a reasoning-based reward model for image editing, offering structural and principle-driven feedback" (Section 2.2).

This creates a chicken-and-egg problem: GRPO needs verifiable rewards to work well, but verifiable rewards require a reasoning verifier to be trained first, and training that verifier requires a method to align it with human preferences—which is itself an RLHF problem. The paper's two-stage pipeline (cold-start SFT → GCPO for the RRM → GRPO for the editing model) is specifically designed to break this circular dependency.

How This Paper Positions Itself

The paper does not claim to invent any single component from scratch. Instead, it positions itself as synthesizing and adapting existing ideas into a coherent framework for a new domain, with one genuinely novel technical contribution (GCPO). This is evident from how the authors describe their work relative to prior art:

Building on DeepSeek-R1's philosophy, not its implementation. The paper repeatedly invokes DeepSeek-R1 (Guo et al., 2025) as inspiration, but the context is crucial. DeepSeek-R1 demonstrated that verifiable rewards enable stable reinforcement learning without reward hacking. The paper's key insight is that this philosophy transfers to image editing if—and only if—you design a verifier that can check editing outputs against decomposed sub-tasks. The paper explicitly states this connection: "A key factor in DeepSeek-R1's success was its reinforcement learning framework with verifiable rewards, which ensured robust training and mitigated the risk of reward hacking. Yet, defining such rewards for visual generation remains challenging" (Section 2.2). The contribution is not the idea of verifiable rewards—it is the specific instantiation of them for the image editing domain through principle decomposition and CoT verification.

Extending GRPO to leverage non-differentiable verifiers. The paper adopts Gru-Policy Optimization (GRPO) for the downstream editing model optimization, but with a crucial adaptation: the reward signal comes from a non-differentiable reasoning model rather than a scalar regressor. This is made possible by the fact that GRPO computes rewards offline (the reward model evaluates images after they are generated) and only requires the scalar score for advantage computation—the reasoning trace that produced the score does not need to be differentiated through. The paper does not modify the GRPO algorithm; it demonstrates that the algorithm is inherently compatible with verifier-based rewards in a way that REFL and DPO are not.

Introducing GCPO as a novel contribution. The GCPO algorithm (Section 3.1.2) is the paper's genuinely novel technical contribution. Standard RLHF methods like DPO or GRPO are designed to optimize a policy given a reward signal. But GCPO optimizes the reward model itself, using pairwise human preference data, when the reward model generates reasoning traces before scoring. This is a meta-RL problem: how do you train a verifier to reason better about image edits when all you have are pairwise preferences (image A is better than image B)? The paper designs a custom solution—cross-group win/loss ratios as per-sample rewards with intra-group advantages—that is distinct from standard DPO (which directly optimizes from paired data) and standard GRPO (which assumes existing verifiable rewards, not pairwise preferences). Table 1's bottom row positions Edit-RRM as filling this specific gap.

Positioning relative to concurrent work. The paper explicitly engages with EditScore (Luo et al., 2025), a concurrent work that also addresses image editing reward models. The comparison is instructive: EditScore advocates for holistic scoring and achieves 65.9% on EditRewardBench. The paper's SFT-only RRM achieves 73.3% on the same benchmark, and adding GCPO pushes this to 78.2% (Table 4). The gap is substantial—over 12 percentage points—which the paper attributes to the verifier paradigm being fundamentally better suited to the multi-faceted evaluation that editing demands. The paper positions EditScore as representing the old paradigm (scoring) against which the new paradigm (verification) is validated.

The Paradigm Shift Claim

The paper frames its contribution as a paradigm shift from scorer to verifier, but this framing requires careful unpacking. What exactly changes?

In the scorer paradigm, the reward model is a function $f(\text{source image}, \text{edit instruction}, \text{edited image}) \rightarrow \text{score}$. The function is typically implemented as a VLM that outputs a scalar. The evaluation is holistic—all aspects of the edit are collapsed into one number. The problem is that different failure modes (wrong color vs. distorted background vs. insufficient edit magnitude) may produce similar scores, making the feedback uninformative for learning.

In the verifier paradigm, the reward model first decomposes the instruction into explicit principles $P = \{p_k\}_{k=1}^{K}$, then evaluates each principle against the edited image (producing a binary or graded judgment per principle), and finally aggregates these judgments into a score. The key difference is that the evaluation is structured, interpretable, and auditable. If the model claims an edit is good, you can inspect which principles it checked and whether those checks were correct. This is what the paper means by "verifier"—the model's reasoning trace serves as evidence that it actually performed the checks it claims, rather than outputting a score based on superficial heuristics.

The paper validates this paradigm shift empirically in multiple ways:

• The "Think+Verify" data construction pipeline (Section 3.1.1, Step 4) uses an external VLM to verify that the reasoning traces are factually consistent with the images, filtering out traces with hallucinated judgments. This filtering step is what makes the verifier reliable—without it, the model might generate plausible-sounding but incorrect reasoning.
• The ablation in Table 2 shows that "Think" alone (68.9% for Qwen-7B) underperforms "Think+Verify" (75.4%), confirming that principled verification filtering is essential.
• The GCPO stage further refines the verifier's reasoning by training against human preferences, reducing hallucinations and improving judgment accuracy on subtle cases (as illustrated in Figure 10 and Appendix G, where the RL-tuned model corrects a hat-color hallucination that the SFT model missed).

The Human Annotation Bottleneck

An important practical consideration that motivates the paper's design is the cost of human preference data for reward model training. The cold-start SFT phase uses 200K samples constructed via automated VLM pipelines—no human annotation required. The GCPO phase uses only 10K human-annotated preference pairs, which the paper explicitly notes is "less than 1% of the SFT-scale training data" (Section 4.1). This is a deliberate design choice: the verifier paradigm enables the RRM to be mostly trained on automatically generated data (since principle decomposition and verification can be automated with strong VLMs), with only a small amount of human feedback needed to correct the residual biases and hallucinations in the automated reasoning. This makes the approach practically scalable in a way that a purely human-annotated reward model would not be.

Summary: The Gap Edit-R1 Fills

The paper addresses a specific, well-defined gap: image editing models need RLHF to improve beyond SFT, but RLHF requires reward models that can evaluate edits across multiple dimensions without hallucination, and existing reward models—whether holistic scorers, pairwise comparators, or reasoning models designed for other domains—all fail at this task in systematically different ways. The verifier paradigm, instantiated through principle decomposition, CoT verification, and the GCPO algorithm for alignment, is proposed as the solution, and the paper validates this claim through both reward model benchmarks and downstream editing model improvements.

3. Technical Approach

This is primarily a systems and methods paper whose core idea is that image editing reward models should function as reasoning verifiers—decomposing editing instructions into explicit principles, checking each principle against the edited image, and aggregating those checks into a score—rather than as holistic scorers that output a single number without interpretable justification.

3.1 Reader Orientation

The paper builds a verifier-based Reasoning Reward Model (RRM) that evaluates image edits by breaking down complex editing instructions into specific, checkable principles, verifying each principle through Chain-of-Thought reasoning, and producing an interpretable score. The system then uses this RRM as a non-differentiable reward signal within a GRPO-based reinforcement learning loop to optimize downstream image editing models. The core problem it solves is that existing editing reward models produce biased or hallucinated feedback because they attempt to evaluate all aspects of an edit (instruction fidelity, preservation of unedited regions, visual quality) simultaneously in a single holistic score—the verifier paradigm addresses this by making the evaluation structured, auditable, and trainable through a novel RL algorithm (GCPO) that aligns the verifier's reasoning with human preferences using only small amounts of pairwise comparison data.

3.2 Big-Picture Architecture (Diagram in Words)

The Edit-R1 framework consists of five major components arranged in a training-then-deployment pipeline:

1. Principle Decomposer — Given a source image and an editing instruction, a powerful external VLM (Seed-1.5-VL API) decomposes the instruction into a set of verifiable principles spanning three categories: "Keep" (elements that should remain unchanged), "Follow" (modifications required), and "Quality" (visual integrity checks). This component produces the structured evaluation criteria that the RRM will later verify.

2. Quadruple Data Generator — For each (source image, instruction, principles) triple, multiple editing models (Flux-Kontext, Bagel, SeedEdit3.0) generate diverse edited candidates, creating approximately 2 million quadruples of the form (edited image, source image, instruction, principles). This provides the raw material for training the RRM.

3. Reasoning Reward Model (RRM) — Built on Qwen-VL-2.5 (3B or 7B), this is the central verifier. It takes a quadruple as input, generates a Chain-of-Thought analysis that evaluates each principle against the edited image, and produces a final scalar score. The RRM is trained in two stages: (a) a cold-start SFT phase where it learns to mimic high-quality reasoning traces selected by an external VLM quality-control judge, and (b) a GCPO reinforcement learning phase where it refines its reasoning to better align with human pairwise preferences.

4. Group Contrastive Preference Optimization (GCPO) Trainer — A novel RL algorithm that optimizes the RRM itself using human-annotated preference pairs (image A is better than image B). It generates N reasoning traces for both the winner and loser images, computes per-sample rewards as cross-group win/loss ratios (how often a winner trace scores higher than loser traces, and vice versa), then computes advantages within each group to update the RRM's parameters via clipped policy gradients.

5. GRPO Editing Model Optimizer — Takes the trained, frozen RRM and uses it as the reward signal in a GRPO-based reinforcement learning loop to optimize downstream editing models (FLUX.Kontext, Qwen-Image-Edit). For each training context, the editing model generates G edited images, the RRM scores them via principle-based CoT verification, and within-group advantage normalization drives policy updates that maximize instruction fidelity and visual quality.

Information flows as follows: source image + edit instruction → Principle Decomposer produces principles → Editing Models generate candidate quadruples → External VLM generates many "Think+Score" CoT traces per quadruple → Quality-Control Judge filters to keep only the most accurate reasoning traces → These traces form the SFT dataset to cold-start the RRM → Human annotators provide pairwise preferences on a small subset (10K pairs) → GCPO Trainer uses these pairs to further align the RRM → The final RL-RRM serves as the frozen reward signal for GRPO-based optimization of editing models.

3.3 Roadmap for the Deep Dive

• First, the cold-start SFT data construction pipeline (Section 3.1.1), because this produces the initial training data and establishes the verifier paradigm—the principle decomposition, CoT reasoning format, and quality-control filtering that define how the RRM operates.
• Second, the GCPO algorithm (Section 3.1.2), since this is the novel technical contribution that bridges the gap between pairwise human preference data and pointwise verifier training, and understanding it requires knowing the RRM's output format from the SFT stage.
• Third, the downstream editing model optimization via GRPO (Section 3.2), because this is where the trained RRM is deployed—its design (especially the choice of GRPO over REFL or DPO) is directly motivated by the RRM being non-differentiable and requiring offline advantage computation.

3.4 Detailed, Sentence-Based Technical Breakdown

Cold-Start SFT: Building the Verifier Training Data

The goal of the first stage is to construct a high-quality supervised dataset that teaches the RRM to function as a verifier—a model that evaluates edits by decomposing instructions into principles, checking each principle, and aggregating the results into a score. The key challenge is that the reasoning traces must be factually accurate (the model should not hallucinate that an edit succeeded when it didn't, or vice versa) and must reflect the kind of structured evaluation that will be useful for downstream RLHF. The paper addresses this through a four-step pipeline that uses strong external VLMs both to generate diverse reasoning traces and then to filter them for accuracy.

Step 1: Decomposing Instructions into Principles

For each reference image $x_{\text{ref}}$ and its corresponding edit instruction $q$, the paper employs the Seed-1.5-VL API to decompose the task into a concise set of verifiable principles. This decomposition uses a carefully designed system prompt (fully reproduced in Appendix A.1) that provides a few-shot example and specifies three categories of evaluation points:

• Instruction Following (3–4 points): checks whether the edit actually happened as specified—"Did the background change to a city landscape?", "Is the object now red instead of blue?"
• Feature Preservation (3–4 points): checks whether unmentioned elements remained unchanged—"Are the petal patterns on the fire hydrants preserved?", "Does the character's pose remain consistent with the original image?"
• Image Quality (2–3 points): checks for generic visual integrity—"Is the generated image free of significant structural problems?", "Is the clarity and overall quality of the generated image good?"

The prompt is used in an in-context learning manner with expert-written decomposition examples. The paper maintains a pool of 60 expert-authored exemplars and randomly samples 4 of them for each query to improve diversity and robustness in principle generation. This sample-wise decomposition effectively factorizes the editing task, structuring the model's reasoning process to distinguish between what to preserve and what to modify based on the specific input.

Formally, the principle set is denoted as $P = \{p_k\}_{k=1}^{K}$ for each (reference image, instruction) pair, where each $p_k$ is a specific verifiable question (e.g., "Is the car now red?"). The decomposition is performed once per (source image, instruction) pair and shared across all edited candidates generated from that pair—this means the principles are grounded in the source image and the instruction, not in any particular edit attempt, making them reusable as a standardized evaluation rubric.

Step 2: Large-Scale Quadruple Generation

For each source image and corresponding edit instruction, a diverse set of edited candidates is generated using multiple image-editing models: Flux-Kontext (Batifol et al., 2025), Bagel (Deng et al., 2025), and SeedEdit 3.0 (Wang et al., 2025). Each edited candidate $x_{\text{edit}}$, together with the reference input image $x_{\text{ref}}$, the instruction $q$, and the principle set $P$, forms a quadruple $(x_{\text{edit}}, x_{\text{ref}}, q, P)$. This process yields a total dataset of approximately 2 million quadruples.

The use of multiple editing models is important because it ensures diversity in the types of edits, error patterns, and success rates present in the training data. The RRM needs to learn to evaluate edits from a wide range of quality levels—perfect edits, partially successful edits, and complete failures—across different editing models with different characteristic strengths and weaknesses.

The paper also curates the source data carefully. It draws 200K samples from a public image-editing benchmark and partitions them into two subsets:

• Random Subset: The first 100K samples are randomly selected from the benchmark to represent a general distribution of edits.
• Hard Subset: The second 100K samples are specifically curated for higher complexity. GPT-4o is used to filter the remaining data and select edit instructions that require multi-step visual modifications, fine-grained detail editing, implicit semantic understanding, or precise spatial control, while rejecting simple single-step edits.

This dual-subset strategy ensures that the RRM is trained on both typical editing scenarios (where simple edits are common) and challenging scenarios (where nuanced evaluation is critical), preventing the model from being biased toward easy evaluation cases.

Step 3: VLM Reasoning and Pointwise Scoring

Each quadruple is processed by Vision-Language Models (VLM Pools) using Chain-of-Thought prompting. The VLM first performs a pointwise verification, assessing the edited image against each principle in $P$. Specifically, for each principle $p_k$, the VLM examines the source image, edited image, and instruction, and produces a binary judgment (0 or 1) along with a brief reasoning explanation. After evaluating all principles, it generates a final scalar score representing the overall quality of the edited image, computed as a weighted aggregate of the principle-wise verification outcomes. This score ranges from 0 to 10.

The system prompt for this evaluation step is extensive and detailed (reproduced in full in Appendix A.2). It specifies numerous edge-case rules, including:

• For positional changes (e.g., "move from left to right"), a significant shift is required; minor moves are insufficient.
• When dealing with positional information, the VLM must output bounding box coordinates in its thought process.
• If an evaluation point contradicts the edit instruction (e.g., "preserve the watch" when the instruction is "lower the hand" which would hide it), the point should be ignored.
• A perfect score on individual evaluation points does not guarantee a perfect final score—the model must also assess whether the edited image is directly usable and whether unmentioned elements changed unreasonably.
• If the edited image is nearly identical to the original (no edit was performed), the score should be 0.
• The final score should consider the "future editing cost" of fixing any unmet points—how much information and pixel modification would be needed.

To enhance dataset diversity, the paper samples multiple thinking CoTs for each quadruple by varying three factors: system prompts, sampling temperatures, and VLM variants (e.g., Seed-1.5-VL and Seed-1.6-VL). This produces multiple "Think + Score" candidates per quadruple, each potentially disagreeing on specific principle judgments or final scores. The VLMs are required to generate the reasoning trace in a fixed format: verify principles in JSON format, and output the final score within <score>...</score> tags. A concrete example of this output format is shown in Appendix B, where the RRM evaluates a "change the style to Chinese style" instruction with 10 principles, producing per-principle scores like [{"question": "Does the generated image change the garage style from modern to Chinese style?", "score": 1}, ...] and a final summary of <score>7</score>.

Step 4: External Verification and SFT Data Selection

This is the critical quality-control step that distinguishes the verifier approach from naive CoT data generation. All "Think + Score" candidates corresponding to the same quadruple are subjected to an external verification process. This is performed by SeedVLM-1.5 acting as a pointwise verifier that re-evaluates each principle in $P$ for every reasoning trace and calculates a verification accuracy.

The verifier prompt (Appendix A.3) instructs the model to act as a "strict image editing verification inspector" that produces gold-standard 0/1 judgments for each evaluation point. Crucially, the verifier receives not only the images, instruction, and principles, but also the "reference intermediate judgments"—the per-point predictions and reasoning summaries from all candidate models. The instructions explicitly state: "The reference intermediate judgments are for reference only and must not be copied. If the references contradict the images and text, the images and text are the ground truth." This means the verifier is checking whether each candidate model's reasoning is factually consistent with the actual image content, not whether it sounds plausible.

For each quadruple, the thinking CoT that achieves the highest verification accuracy (the greatest agreement between its per-principle judgments and the verifier's gold-standard judgments) is selected. The resulting data—comprising the instruction, images, principles, CoT reasoning trace, and final score—constitute the initial Supervised Fine-Tuning (SFT) dataset for the reward model's cold start.

The paper validates the importance of this verification step through ablation. In Table 2, the Qwen-7B model achieves 68.9% accuracy when trained with "Think" only (no external verification filtering), 70.9% with "Verify" only (no CoT reasoning, just direct scoring), and 75.4% with "Think+Verify" (the full pipeline). The 6.5 percentage point improvement from adding verification indicates that many of the initially generated reasoning traces contain errors that the external verifier successfully filters out.

Training Configuration for Cold-Start SFT

The RRM is built on the open-source Qwen-VL-2.5 (Bai et al., 2025) at two scales: 3B and 7B parameters. For the SFT phase, the model is fine-tuned on the selected "Think+Score" trajectories using standard next-token prediction loss—the model learns to generate the CoT reasoning trace followed by the final score, conditioned on the quadruple input. The paper does not report specific SFT hyperparameters (learning rate, batch size, etc.) in the main text or appendix, which is a notable omission for reproducibility. However, the training dynamics are shown in Figure 3a and 3b: SFT loss decreases steadily and converges, and SFT evaluation accuracy improves consistently throughout training, with the 7B model showing faster convergence and higher final accuracy than the 3B model.

Design Choice: Why Generate Multiple Traces and Filter Rather Than Using a Single Strong Model?

A natural alternative would be to use the strongest available VLM to generate a single high-quality reasoning trace per quadruple, rather than generating many and filtering. The paper's approach of generating diverse traces from varying prompts, temperatures, and model variants, then selecting the best, serves several purposes:

• Robustness to model-specific biases: Different VLMs and different sampling temperatures may produce reasoning traces with different characteristic errors (e.g., one model might be overly generous on color change evaluations, another might miss subtle preservation failures). By generating diverse candidates and verifying them against a consistent gold-standard, the SFT data is less corrupted by any single model's idiosyncrasies.
• Coverage of edge cases: Some reasoning traces may catch subtle issues that others miss. The verification step ensures that the selected trace is the most comprehensively accurate one, effectively performing a form of inference-time ensembling.
• Teaching the RRM to handle ambiguity: By seeing multiple valid reasoning traces during training (each selected as the best for its specific quadruple), the RRM learns that there can be multiple valid ways to reason about an edit, but that the factual conclusions must be consistent.

GCPO: Aligning the Verifier with Human Preferences

After the cold-start SFT phase, the RRM possesses effective Chain-of-Thought reasoning capabilities—it can decompose instructions, verify principles, and produce scores that are substantially more accurate than holistic scorers (75.4% on the internal benchmark for Qwen-7B with Think+Verify, per Table 2). However, the paper observes that its judgments can still be fallible in specific ways: the model may exhibit hallucinations (claiming an edit succeeded when it did not) or struggle to accurately assess the magnitude of edits. The paper gives a specific example: "incorrectly verifying a principle 'move to the left of the figure' as successful, but the object has only slightly moved" (Section 3.1.2).

To address these residual errors and further align the model with human preferences, the paper introduces a reinforcement learning phase. This phase faces a fundamental challenge that standard RLHF algorithms are not designed to handle: the RRM first generates a reasoning trace (discrete tokens) before producing a final score, making the scoring process non-differentiable. Standard RLHF for language models typically optimizes a scalar reward directly, but here the "action" is the entire reasoning trajectory plus the score, and the reward for that action must be derived from pairwise human preference data.

Preference Data Collection

For this phase, the paper constructs a preference dataset $\mathcal{D}$ through human annotation. The annotation process works as follows: annotators are presented with a source image $x_{\text{ref}}$, an editing instruction $q$, and a pair of edited images. They are asked to choose which image is better, or to label them as "same" if they are of comparable quality or if a clear preference cannot be established. The primary criteria for judgment are instruction fidelity and overall image quality.

This process yields approximately 10,000 preference pairs $(x_w, x_l)$ for each context $c = (x_{\text{ref}}, q)$, where $x_w$ denotes the preferred (winner) image and $x_l$ the non-preferred (loser) one. The "same" pairs are excluded from GCPO training because the algorithm requires clear preference signals to compute win/loss ratios (a tie provides no gradient signal). The paper explicitly notes that 10K pairs represent "less than 1% of the SFT-scale training data," emphasizing that the gains from GCPO are mainly attributable to better human alignment rather than increased data volume.

Win/Loss Ratio Rewards: The Core Mechanism

The key insight of GCPO is that pairwise human preferences can be converted into per-sample scalar rewards for a pointwise verifier by comparing groups of reasoning traces. For each preference pair $(x_w, x_l)$, the reward model $R_\phi$ (with parameters $\phi$) stochastically generates $N$ distinct reasoning traces and their corresponding scores for each image:

$\tau_j^w = \Phi\left(R_\phi(x_j^w, c, P)\right), \quad \tau_j^l = \Phi\left(R_\phi(x_j^l, c, P)\right)$

where $\Phi(\cdot)$ is an operator that extracts the scalar score from the text output of $R_\phi(\cdot, \cdot, \cdot)$ via rule-based parsing (looking for the <score>...</score> tags), $x_j^w$ and $x_j^l$ denote the $j$-th stochastic generation of reasoning traces for the winner and loser images respectively (the RRM may produce different reasoning and scores each time due to sampling), and $c = (x_{\text{ref}}, q)$ is the context consisting of the source image and instruction, and $P$ is the principle set.

What it computes: For each of the $N$ reasoning traces generated for the winner image, $\Phi$ extracts a scalar score (0–10). The same is done for the loser image. This produces two sets of $N$ scores each: $\{\tau_j^w\}_{j=1}^N$ (winner scores) and $\{\tau_j^l\}_{j=1}^N$ (loser scores). These scores are the raw outputs from the RRM's verification process—in principle, if the RRM were perfectly aligned with human preferences, every winner score would be higher than every loser score, but in practice there is stochastic variation and occasional errors.

The per-sample win/loss ratios are then defined based on exhaustive pairwise comparisons between the two sets of scores, ignoring ties:

$r_j^w = \frac{1}{N}\sum_{k=1}^N \mathbb{1}\left\{\tau_j^w > \tau_k^l\right\}, \quad r_j^l = \frac{1}{N}\sum_{k=1}^N \mathbb{1}\left\{\tau_j^l < \tau_k^w\right\}$

where $\mathbb{1}\{\cdot\}$ is the indicator function (1 if the condition holds, 0 otherwise), $r_j^w$ is the win ratio for the $j$-th winner candidate—the fraction of the $N$ loser candidates that it scores higher than, and $r_j^l$ is the loss ratio for the $j$-th loser candidate—the fraction of the $N$ winner candidates that score higher than it (or equivalently, that it scores lower than).

What it computes operationally: For a specific winner reasoning trace $j$, the model compares its score $\tau_j^w$ against every loser score $\tau_1^l, \tau_2^l, \dots, \tau_N^l$ and counts how many comparisons it wins (its score is strictly greater). That count, divided by $N$, gives $r_j^w$. If a winner trace scores higher than all loser traces, $r_j^w = 1.0$ (perfect win ratio). If it scores lower than all loser traces, $r_j^w = 0.0$ (complete failure). Most traces fall somewhere in between. Similarly, for a loser trace, the model counts how many winner traces score higher than it, divided by $N$, giving $r_j^l$. If the RRM is functioning well, winner traces should have high win ratios (close to 1) and loser traces should have low loss ratios (close to 0, meaning they are consistently scored lower).

Why this form over alternatives: Several alternative reward formulations would fail for this problem:

• Binary pairwise reward (1 if winner score > loser score, 0 otherwise): This would provide only a single scalar reward per pair, completely ignoring the fact that the RRM generates $N$ different reasoning traces with different scores. It would treat all winner traces as equally good and all loser traces as equally bad, providing no signal about which specific reasoning traces are most aligned with human preferences. The cross-group win/loss ratio provides a rich, continuous reward signal per reasoning trace, enabling the model to learn which of its reasoning patterns lead to judgments that agree with human preferences and which lead to disagreements.

• Direct score difference (average winner score minus average loser score): This would be a group-level reward that ignores within-group variation. If two winner traces produce very different scores (one very high, one borderline), both would receive the same reward, providing no incentive to improve the weaker reasoning. The win/loss ratio is computed per-sample, so each individual reasoning trace receives its own reward signal.

• Standard GRPO with scalar rewards: This would require a pre-existing scalar reward function that provides absolute quality scores, which is exactly what the GCPO phase is trying to build. There is a circularity problem: you cannot use GRPO to train the RRM if the RRM is itself the reward source. The win/loss ratio breaks this circularity by deriving rewards from pairwise comparisons rather than absolute scores.

Optimization with GCPO: Intra-Group Advantages

After computing the win/loss ratio rewards $\{r_j^w\}_{j=1}^N$ and $\{r_j^l\}_{j=1}^N$ from cross-group comparisons, the original pairing between samples is disregarded for the optimization step. Instead, advantages are computed independently within each rollout group (preferred or non-preferred). This is the "group contrastive" aspect of GCPO: the rewards come from cross-group comparisons, but the optimization is within-group.

The advantages are computed as follows:

$\bar{r}^w = \frac{1}{N}\sum_{j=1}^N r_j^w, \quad \bar{r}^l = \frac{1}{N}\sum_{j=1}^N r_j^l$

where $\bar{r}^w$ is the mean win ratio across all $N$ winner reasoning traces, and $\bar{r}^l$ is the mean loss ratio across all $N$ loser reasoning traces.

$A_j^w = r_j^w - \bar{r}^w, \quad A_j^l = r_j^l - \bar{r}^l$

where $A_j^w$ is the advantage for the $j$-th winner reasoning trace—positive if this trace has a higher win ratio than the average winner trace (meaning it is better at distinguishing winner from loser than the group average), and negative if it is worse. Similarly, $A_j^l$ is the advantage for the $j$-th loser trace—positive if this trace generates scores that make it easier to distinguish from winners (i.e., it is a "better" loser trace that admits defeat clearly), and negative if it generates scores too close to winner scores.

What this advantage computation achieves: It creates a relative ranking within each group that incentivizes the RRM to:

• Generate reasoning for winner images that produces higher scores (larger margin between winner and loser scores, leading to higher win ratios).
• Generate reasoning for loser images that produces lower scores (clearer distinction from winners, leading to lower loss ratios).
• Reduce variance in its judgments—a trace that produces an outlier score (a winner trace that scores low, or a loser trace that scores high) will have a negative advantage and be penalized.

The key mathematical property is that $\sum_j A_j^w = 0$ and $\sum_j A_j^l = 0$—the advantages are zero-mean within each group by construction. This means the GRPO update is purely redistributive within each group: it pushes probability mass toward reasoning traces with above-average win/loss ratios and away from those with below-average ratios, without changing the overall expected score magnitude.

The GCPO Objective Function

Let $r_{t,j}^w(\phi)$ and $r_{t,j}^l(\phi)$ denote the per-token likelihood ratios for the $j$-th rollout and $t$-th token in the preferred and non-preferred groups, respectively. These are the ratio of the current policy probability to the old policy probability (from before the update) for each token, following the standard importance sampling formulation in policy gradient methods. The objective function is:

$\mathcal{L}_{\text{GCPO}}(\phi) = \mathbb{E}_{\dots\sim\mathcal{D}}\left[\frac{1}{2N}\sum_{j=1}^N \frac{1}{T}\sum_{t=0}^{T-1} \left(\min\left(r_{t,j}^w(\phi) A_j^w, \text{clip}\left(r_{t,j}^w(\phi), 1-\epsilon, 1+\epsilon\right) A_j^w\right) + \min\left(r_{t,j}^l(\phi) A_j^l, \text{clip}\left(r_{t,j}^l(\phi), 1-\epsilon, 1+\epsilon\right) A_j^l\right)\right)\right]$

where $\mathbb{E}_{\dots\sim\mathcal{D}}$ is the expectation over preference pairs sampled from the dataset $\mathcal{D}$, $N$ is the number of reasoning traces generated per image, $T$ is the number of tokens in each reasoning trace, $\epsilon$ is the clipping parameter (standard in PPO/GRPO), $\text{clip}(r, 1-\epsilon, 1+\epsilon)$ clips the likelihood ratio to the interval $[1-\epsilon, 1+\epsilon]$, and the $\min$ operator implements the standard conservative policy iteration objective—if the likelihood ratio would move the policy too far in the direction of the advantage, the update is clipped to prevent destructive large policy changes.

What it computes: For each reasoning trace (both winner and loser), the objective computes a per-token loss that is the product of the likelihood ratio and the advantage, clipped to prevent the policy from changing too rapidly. The inner sum averages over all tokens in the trace (the $1/T$ normalization), the middle sum averages over all $N$ reasoning traces in each group (the $1/(2N)$ normalization, with the factor of 2 accounting for both winner and loser groups), and the outer expectation averages over the preference dataset. The loss is minimized when the policy generates reasoning traces that produce high win ratios for winners and low loss ratios for losers.

Why this form: The $\min(\text{unclipped}, \text{clipped})$ structure is the standard PPO/GRPO clipping mechanism that prevents destructive large policy updates. If the advantage is positive (a good reasoning trace), the objective encourages increasing the likelihood ratio $r_{t,j}(\phi)$ (making these tokens more probable), but clips the increase at $1+\epsilon$ to prevent the policy from collapsing to a single trajectory. If the advantage is negative (a poor reasoning trace), the objective encourages decreasing the likelihood ratio, but clips at $1-\epsilon$ to prevent completely eliminating potentially useful reasoning patterns. The $\min$ operation ensures that the objective is a lower bound on the unclipped objective, making the optimization conservative—it is better to under-update than to over-update and risk policy collapse.

Why no KL divergence term: The paper explicitly states that the objective "omits the KL divergence term" (Section 3.1.2). In standard PPO/GRPO, a KL penalty is added to prevent the policy from deviating too far from a reference policy. The paper's choice to omit this term is not fully justified in the text, but it is likely motivated by the observation that the SFT-initialized RRM already produces reasonable reasoning traces, and the GCPO phase is a fine-tuning step with a small amount of data (10K pairs) relative to the SFT phase (200K quadruples). The clipping mechanism alone may provide sufficient regularization in this regime. Additionally, since GCPO optimizes within-group advantages (which are zero-mean), the expected magnitude of policy updates is naturally constrained—the advantages cannot grow arbitrarily large because they are normalized within each group.

Training Dynamics and Diagnostics

The paper provides several diagnostics of GCPO training in Figure 3:

• Figure 3c: Weighted advantage during GCPO training. Defined as $\frac{1}{G}\sum_{i=1}^{G} \frac{A_i}{L_i}$, where $L$ represents the length of reasoning tokens. This metric is negative, which the paper interprets as indicating that the model "learns to generate longer reasoning traces for correct judgments." The intuition: when the model produces a reasoning trace that correctly distinguishes winner from loser (positive advantage), it is incentivized to produce longer, more detailed reasoning (which takes more tokens). The negative value arises because the advantage is positive but the reasoning length is large, making the per-token advantage small.

• Figure 3d: Training reward during the GCPO phase. Shows stable improvement and clear scalability—the 7B model achieves higher rewards than the 3B model, and both improve over training. This is the mean win/loss ratio reward across training batches, indicating that the RRM's ability to distinguish winners from losers improves throughout GCPO training.

The connection between GCPO and standard GRPO: GCPO can be understood as applying GRPO's advantage normalization logic to a meta-problem. In standard GRPO, the policy generates $G$ outputs, each is scored by a reward model, and advantages are computed by normalizing rewards within the group. In GCPO, the "policy" is the RRM itself, the "outputs" are reasoning traces, and the "rewards" are win/loss ratios derived from cross-group comparisons with traces from the opposite preference group. The advantage computation uses the same within-group mean subtraction, but the rewards come from a fundamentally different source—pairwise comparisons rather than absolute scoring.

GRPO for Downstream Editing Model Optimization

Once the RRM is trained (after both SFT and GCPO), it serves as a frozen, non-differentiable reward signal for optimizing downstream image editing models. The paper uses the GRPO algorithm (Group Relative Policy Optimization) adapted for flow-based editing models via Flow-GRPO (Liu et al., 2025).

Why GRPO instead of REFL or DPO for the editing model?

The paper explicitly addresses this design choice. REFL (Reward Feedback Learning) requires the reward model to be differentiable—the gradient of the reward with respect to the generated image must flow back through the editing model during training. This works when the reward model is a regression head on a VLM that outputs a continuous scalar, because the scalar is differentiable with respect to the VLM's input image embedding, which in turn is differentiable with respect to the editing model's output pixels. However, the RRM generates a reasoning trace through discrete token sampling before producing a score. This sampling operation is non-differentiable—you cannot backpropagate through a categorical sampling step. The paper states this constraint explicitly: "Since the RRM generates an explicit multi-step reasoning trace through discrete token sampling before producing a final score, the process is inherently non-differentiable, rendering REFL-style methods inapplicable."

DPO (Direct Preference Optimization) avoids the reward model entirely by directly optimizing the editing policy on a preference dataset. While this circumvents the differentiability problem, the paper notes that DPO "inherently restricts policy exploration, risking suboptimal convergence" (Section 2.2). This is because DPO updates the policy only on pairs where one output was preferred over another, but the editing model may need to explore regions of the output space that are not represented in a static preference dataset. Additionally, DPO does not provide a mechanism for the model to learn from its own exploration—it is purely an offline algorithm.

GRPO, by contrast, is an online RL algorithm that:

1. Samples a group of $G$ edited images from the current policy for each training context.
2. Scores each image using the frozen RRM (which can be non-differentiable because scoring happens offline, not during backpropagation).
3. Computes advantages by normalizing rewards within the group.
4. Updates the policy to increase the probability of high-advantage outputs and decrease the probability of low-advantage outputs.

The online nature (the model learns from its own generations, not a fixed dataset) and the compatibility with non-differentiable reward sources make GRPO the natural choice for leveraging the RRM.

The GRPO Training Procedure

For each conditioning context $c$ sampled from the dataset $\mathcal{D}$, the flow-based editing model $\pi_\theta(\cdot, c)$ generates a group of $G$ edited images $\{x_0^i\}_{i=1}^G$ along with their corresponding generation trajectories $\{(x_T^i, \dots, x_0^i)\}_{i=1}^G$. Here, $\theta$ denotes the editing model parameters, $x_0^i$ is the final generated edited image (after the full denoising trajectory), and $\{x_T^i\}_{i=1}^G$ are the initial noise samples drawn from a Gaussian distribution—the starting point of the flow-based generation process.

The verifiable reward model $R_\phi(\cdot, \cdot, \cdot)$ (the frozen, trained RRM) verifies and evaluates each generated image $x_0^i$ based on the context $c$ and the corresponding principle set $P$. The reward is:

$\tau_i = \Phi\left(R_\phi(x_0^i, c, P)\right)$

where $\tau_i$ is the holistic reward score for the $i$-th generated image, extracted from the RRM's text output via rule-based parsing (the same $\Phi$ operator used in GCPO), and $R_\phi$ is the frozen RRM that generates a full CoT verification trace before outputting the final score.

What it computes: The RRM takes the generated edited image, the source image and instruction (context $c$), and the pre-decomposed principles $P$, and produces a detailed verification analysis that checks each principle, then aggregates into a final scalar score $\tau_i$ between 0 and 10. This is functionally identical to how the RRM operates during its own training, but now it is used as an evaluator rather than being trained.

Within-Group Advantage Normalization

Within each group of $G$ generated images, the advantage $A_i$ for the $i$-th image is calculated by normalizing its reward against the mean and the standard deviation within the group:

$A_i = \frac{\tau_i - \text{mean}\left(\{\tau_i\}_{i=1}^G\right)}{\text{std}\left(\{\tau_i\}_{i=1}^G\right) + \epsilon_{\text{std}}}$

where $\text{mean}(\{\tau_i\}_{i=1}^G)$ is the average score across all $G$ images in the group, $\text{std}(\{\tau_i\}_{i=1}^G)$ is the standard deviation of scores within the group, and $\epsilon_{\text{std}}$ is a small constant added for numerical stability (to prevent division by zero when all images in a group receive identical scores).

What this normalization achieves: The advantage $A_i$ is positive if image $i$ scores higher than the group average (it is a relatively good edit) and negative if it scores lower than average (it is a relatively poor edit). The division by the standard deviation ensures that the scale of the advantage is invariant to the absolute magnitude of the RRM's scores—if the RRM tends to give scores in a narrow range (e.g., 6–8), the advantages will still be meaningful because they are measured in units of within-group standard deviation. If all images in a group receive identical scores, the standard deviation is zero, and all advantages become zero (no update is performed, because there is no signal to learn from).

Why group-relative over absolute rewards: Absolute reward scores could drift over training (the editing model might produce better images, shifting the RRM's score distribution upward), making it difficult to set a fixed threshold for what constitutes a "good" score. Group-relative advantages are adaptive: they always compare each output to its peers generated from the same context, ensuring that the training signal is always meaningful regardless of absolute score magnitudes. This is particularly important when the RRM is not perfectly calibrated in an absolute sense—it may systematically under-score or over-score certain types of edits, but as long as it ranks outputs correctly within a group, the advantages will be correct.

The GRPO Training Objective

The GRPO objective maximizes the expected advantage while incorporating two regularization mechanisms: a clipped objective function (to prevent excessively large policy updates) and a KL-divergence penalty term (to regularize the policy $\pi_\theta(\cdot, c)$ and keep it from deviating too far from a reference policy $\pi_{\text{ref}}(\cdot, c)$). The paper does not provide the full GRPO objective equation in the main text, but references Flow-GRPO (Liu et al., 2025) for the implementation details and specifies the key hyperparameters: group size $G = 24$ and KL penalty coefficient $\beta = 0.04$.

The reference policy $\pi_{\text{ref}}$ is typically the initial editing model checkpoint (before GRPO fine-tuning), ensuring that the optimized model does not drift into regions of the output space where generations become degenerate or exploit the RRM's weaknesses. This is the same principle as in RLHF for language models—the KL penalty prevents "reward hacking" where the model learns to produce outputs that score highly under the reward model but are actually low-quality.

The Complete Training Loop

Putting it all together, the training loop operates as follows for each step:

1. Sample a batch of contexts $c = (x_{\text{ref}}, q)$ from the training dataset. For each context, retrieve the pre-computed principle set $P$ (decomposed during the SFT data construction phase).

2. For each context, generate $G = 24$ edited images using the current editing policy $\pi_\theta$, starting from random Gaussian noise $\{x_T^i\}_{i=1}^{G}$ and running the full flow-based denoising trajectory.

3. For each generated image $x_0^i$, run the frozen RRM $R_\phi$ to produce a CoT verification trace and extract the final score $\tau_i$. This involves the RRM generating a reasoning trace (checking each principle, producing per-principle judgments, reasoning about the overall quality, and outputting a score in <score>...</score> tags), followed by rule-based parsing to extract the scalar.

4. Compute within-group advantages $A_i$ using the mean and standard deviation of the $G = 24$ scores.

5. Update the editing model parameters $\theta$ using the Flow-GRPO objective: maximize the clipped advantage-weighted log-probability of the generation trajectories, minus the KL penalty $\beta \cdot D_{\text{KL}}(\pi_\theta \| \pi_{\text{ref}})$.

Scaling Behavior and RRM Size

The paper experiments with two RRM sizes: 3B and 7B parameters. The training dynamics in Figure 4 reveal several important patterns:

• First row (training reward): All RRM variants provide stable training signals. The SFT-RRM (trained without GCPO) produces training rewards that are as effective as the Seed-1.5-VL baseline, indicating that even the SFT-only verifier provides useful supervision.

• Second row (evaluation reward): Here the critical distinction between SFT-RRM and RL-RRM (trained with GCPO) emerges. The RL-RRM variants (third and fourth columns) consistently yield higher evaluation rewards than their SFT counterparts. The paper interprets this as evidence that GCPO transforms the RRM into a stricter and more robust evaluator—it provides lower training rewards (it is harder to satisfy) but higher evaluation rewards (its judgments better correlate with downstream editing quality).

• Scaling from 3B to 7B: The 7B SFT-RRM (second column) provides stronger supervision than the 3B SFT-RRM, showing clearer separation between high-quality and low-quality edits. This scaling trend is a meaningful finding because it suggests that the verifier paradigm benefits from increased model capacity—a larger model can maintain more nuanced principle checks and more accurate CoT reasoning.

• GCPO benefits at both scales: Comparing the third and fourth columns (RL-RRM at 3B and 7B) against the first and second columns (SFT-RRM at 3B and 7B), GCPO provides a consistent improvement at both model sizes. At 7B, the RL-RRM produces evaluation rewards that continue to increase throughout training, while the SFT-RRM's evaluation rewards plateau earlier.

Training Efficiency Considerations

The paper addresses a practical concern about the cost of GCPO and GRPO training. GCPO requires generating $N$ reasoning traces per image for both winner and loser candidates in each preference pair, which could be computationally expensive. However, the paper notes that "the overall training cost remains manageable in practice due to the small rollout group size and efficient packed inference" (Section 4.1). The specific value of $N$ (the number of reasoning traces) is not explicitly stated, but the group size for GRPO editing model training is $G = 24$, and the GCPO rollout group is presumably of similar or smaller size.

For the downstream editing model optimization, the main computational cost is generating $G = 24$ edited images per context (the denoising trajectory through the flow model) plus running the RRM inference $G = 24$ times per context. The RRM inference generates a full CoT reasoning trace before producing a score, which adds token generation cost compared to a simple scalar regressor. The paper does not quantify this cost overhead, but the performance gains in Table 3 (e.g., FLUX.Kontext Overall Score improving from 5.77 to 6.24) justify the added computation.

The Two-Level RL Architecture: Why Train the RRM Separately Before Using It for Editing Models?

A natural question is why the paper does not jointly train the RRM and the editing model in a single RL loop. The two-stage approach (first train RRM with SFT + GCPO, then use frozen RRM to train editing models with GRPO) is motivated by several practical and theoretical considerations:

• Differentiability decoupling: As discussed, the RRM is non-differentiable (discrete token sampling), so it cannot be backpropagated through. This means joint training would require some form of REINFORCE-style gradient estimation for both the RRM and the editing model simultaneously, which is known to have high variance and slow convergence. Training separately avoids this complexity.

• Stable reward signal: If the RRM were being updated simultaneously with the editing model, the reward signal would be non-stationary—the same edited image might receive different scores at different points in training as the RRM itself evolves. This non-stationarity can destabilize RL training for the editing model. By freezing the RRM after GCPO, the reward signal is consistent and the editing model can converge reliably.

• Data efficiency for RRM training: The GCPO phase uses only 10K human-annotated preference pairs, which is insufficient to directly train an editing model (which would need orders of magnitude more training data). By focusing the limited human feedback on improving the RRM's reasoning rather than the editing model's behavior, the approach gets more leverage from the expensive human annotations—the improved RRM can then provide high-quality reward signals for arbitrarily many editing model training steps.

4. Key Insights and Innovations

Innovation 1: The RRM as a Reasoning Verifier — A Paradigm Shift from Holistic Scoring to Structured Decomposition

The paper's most fundamental conceptual contribution is not a specific algorithm but a reframing of what a reward model for image editing should be. Before this work, the dominant paradigm treated the reward model as a scorer—a function mapping (source image, instruction, edited image) to a single scalar, typically implemented by prompting a general-purpose VLM to output a number. This approach, used by Skywork-EditReward, OneReward, and the concurrent EditScore, collapses the multi-faceted evaluation problem into an opaque, unverifiable judgment. The scorer might output "7.3" with no explanation of whether the edit succeeded but the background warped, or whether the instruction was followed but with insufficient magnitude. The user—and critically, the RL optimizer receiving this signal—has no way to diagnose why a particular score was assigned.

The verifier paradigm breaks this opacity by insisting that the reward model explicitly decompose the editing instruction into discrete, checkable principles, evaluate each one against the actual image content, and produce an auditable reasoning trace as evidence for the final score. This is not merely an improvement in interpretability—it fundamentally changes what information the reward signal contains. In the scorer paradigm, the RL optimizer receives a single number that conflates instruction fidelity, preservation quality, and visual artifacts. In the verifier paradigm, even though the RL optimizer ultimately uses a scalar score for advantage computation, the process that produces that score is structured to prevent common failure modes: hallucinated successful edits (where the model claims an edit happened but the image shows otherwise) and missed preservation failures (where the background subtly degrades but the overall score remains high).

Table 1 in the paper makes this paradigm shift concrete by taxonomizing reward models across multiple reasoning capability dimensions. Edit-RRM is the only model that simultaneously supports pointwise scoring, principle-based verification, Chain-of-Thought reasoning, and reinforcement learning for the reward model itself. The gap is not in any single capability—DeepSeek-GRM does CoT, VisualQuality-R1 does principles, RewardDance does RL—but in their integration for the specific demands of image editing. The paper's positioning here is significant: it identifies a systematic absence in the literature (no model fills all four columns of Table 1) and fills it with a purpose-built design.

Evidence that this paradigm shift matters comes from both internal and public benchmarks. Table 2 shows that "Think+Verify" (the full verifier pipeline) achieves 75.4% accuracy for Qwen-7B, while "Think alone" (CoT without external verification filtering) reaches only 68.9%. The 6.5 percentage point gap is the marginal value of verification—it represents the correction of errors that a reasoning model would otherwise confidently assert. On the public EditRewardBench (Table 4), the SFT-only RRM already surpasses the scorer-based EditScore-7B by 7.4 percentage points (73.3% vs. 65.9%), directly validating that the verifier architecture is superior for editing evaluation, independent of the GCPO training innovation.

This reframing is fundamental rather than incremental because it changes the input-output relationship of reward models. A holistic scorer can be improved with better training data or larger models, but it remains a scorer—the evaluation structure is implicit and unverifiable. The verifier paradigm restructures the evaluation task itself, factorizing a complex judgment into sub-judgments that can be independently checked and trained. This factorization is what enables the cold-start SFT pipeline (where diverse reasoning traces are generated and filtered for factual accuracy) and the GCPO training (where the verifier's reasoning is aligned with human preferences at the principle level, not just the score level). The paradigm shift is the enabling condition for the technical innovations that follow.

Innovation 2: GCPO — Bridging Pairwise Human Preference Data and Pointwise Verifier Training

The GCPO algorithm is the paper's genuinely novel technical contribution, addressing a problem that prior RLHF methods cannot handle: how to optimize a pointwise reasoning verifier using only pairwise human preference data, when the verifier's output includes a non-differentiable reasoning trace before the final score.

Standard approaches to this problem fall into two categories, both of which fail for the RRM setting. Pairwise reward models (RewardDance, Pairwise RM) directly model preference probabilities from pairs but cannot provide absolute scores for individual outputs, making them incompatible with the GRPO-based editing model optimization that requires per-image scalar rewards. Pointwise reward models trained with scalar regression (ImageReward, HPSv3) require absolute quality labels for individual outputs, which are much more expensive to collect than pairwise preferences (annotators find it easier to say "A is better than B" than to assign calibrated scores). The paper explicitly identifies this tension in Section 2.1: "A significant drawback of pairwise approaches is their inability to provide an absolute quality score for a single response."

GCPO's key insight is that pairwise preferences can be converted into per-sample scalar rewards for a pointwise model by comparing groups of reasoning traces. Rather than asking "is A better than B?" and training a model to predict that binary, GCPO asks: for a winner image and a loser image, generate N reasoning traces for each, and compute per-trace rewards as cross-group win/loss ratios—what fraction of opposite-group traces does this trace score correctly relative to? This transforms a single binary preference label into N continuous reward signals (one per reasoning trace), each reflecting how well that specific trace's reasoning and scoring aligns with the human judgment.

What makes this non-obvious is that the rewards come from cross-group comparisons but the optimization uses intra-group advantages. After computing win/loss ratios, the algorithm discards the original winner-loser pairing and normalizes within each group independently. This means a winner reasoning trace receives a positive advantage not just for scoring higher than loser traces, but for scoring higher than other winner traces. This within-group competition creates pressure for the verifier to maximize the margin between winner and loser score distributions while also reducing variance—a trace that produces an anomalously low winner score or high loser score is penalized relative to its peers.

The significance of GCPO extends beyond its immediate application. It demonstrates a general principle for training generative verifiers: when the model's output includes both reasoning and a score, and you only have pairwise supervision, you can create per-sample scalar rewards from cross-group comparisons and optimize with standard policy gradient methods within each group. This is incremental in mechanism (it uses the established GRPO advantage normalization structure) but fundamental in problem formulation—it solves the meta-RL problem of training a reward model's reasoning when the training signal is at the preference level rather than the reasoning level.

Evidence for GCPO's effectiveness comes from multiple sources. Table 4 shows that adding GCPO to the SFT-trained RRM improves EditRewardBench accuracy from 73.3% to 78.2%—a 4.9 percentage point gain from only 10K human preference pairs (less than 1% of the SFT data volume). Figure 3c shows that GCPO training causes the model to produce longer reasoning traces for correct judgments (the weighted advantage metric is negative, indicating that positively-rewarded traces use more tokens). Figure 3d shows stable training reward improvement and clear scaling from 3B to 7B. Most revealing is Figure 4 (training dynamics for downstream editing models): RL-RRMs (trained with GCPO) provide lower training rewards but higher evaluation rewards than SFT-RRMs, indicating that GCPO produces a stricter, more discriminating evaluator that is harder to satisfy during training but whose judgments better predict downstream quality.

The qualitative analysis in Appendix G (Figure 10) provides a concrete example of what GCPO fixes: an SFT-trained RRM fails to catch a hat-color hallucination (the instruction is "change the shirt to red" but the SFT model's output incorrectly turns the hat red as well). GCPO training corrects this, penalizing the hallucination and rewarding correct preservation. This illustrates that GCPO's benefit is not just improved aggregate accuracy but targeted correction of specific reasoning failures that the SFT phase's external verification filtering missed.

Innovation 3: Non-Differentiable Verifiers as Compatible Reward Sources for GRPO-Based Editing Model Optimization

This innovation is less about a new method and more about identifying and exploiting a compatibility property that the field had not recognized: GRPO-based RL for visual generation models is inherently compatible with non-differentiable, reasoning-based reward models, while the dominant REFL-based approaches are not.

The paper's argument is simple but powerful. REFL (used in ByteEdit, OneReward, and other editing RLHF works) requires backpropagating reward gradients through the reward model into the editing model. This works when the reward model is a regression head outputting a continuous scalar, because that scalar is differentiable with respect to the input image. But a verifier that first generates discrete reasoning tokens before scoring breaks the gradient chain—you cannot differentiate through a categorical sampling operation. DPO avoids the reward model entirely but restricts policy exploration to a static preference dataset, preventing the editing model from learning from its own exploration.

GRPO circumvents both problems because it is an online algorithm where the reward model evaluates outputs offline (after generation, before the policy update). The reward score is treated as a constant during backpropagation—only the policy's log-probabilities are differentiated. This means the reward model can be arbitrarily complex, including generating multi-step reasoning traces with discrete token sampling, as long as it produces a scalar score at the end. The paper does not modify GRPO; it recognizes that the algorithm's design already supports the verifier paradigm in a way that REFL and DPO do not.

This insight is significant because it removes a barrier that might otherwise prevent the adoption of verifier-based rewards in visual RLHF. A practitioner might reasonably assume that sophisticated reasoning-based evaluators are incompatible with gradient-based policy optimization and therefore stick with simple differentiable scorers. The paper demonstrates that this assumption is false for GRPO, opening the door to using arbitrarily complex verifiers (including future models with longer reasoning chains, multi-step verification, or external tool use) as reward sources without modifying the optimization algorithm.

The empirical evidence is that this compatibility translates to practical gains. Table 3 shows that FLUX.Kontext optimized with RL-RRM (7B) outperforms the same model optimized with SeedVLM-1.5 (a strong but non-reasoning baseline) on GEdit-Bench-EN (Overall Score 6.24 vs. 6.03). More tellingly, the RL-RRM (7B) achieves the highest Overall Score among all FLUX.Kontext variants (6.24), surpassing both the SFT-RRM variants and the external SeedVLM baseline. On Qwen-Edit—a model that already benefits from Best-of-N scaling to reach 7.45 Overall—the RL-RRM still provides a modest gain to 7.50, with a particularly notable 15.2% relative improvement on the challenging Motion Change category (from 4.01 to 4.62). This category-specific gain suggests that the verifier's principle-based feedback is especially valuable for edits where the model must produce specific, non-trivial changes (like altering a subject's pose or action) while preserving everything else.

This innovation is incremental in mechanism (no new algorithm is introduced) but fundamental in implication: it establishes that the algorithmic infrastructure for verifier-based RLHF already exists in GRPO, and that the bottleneck is entirely in building better verifiers, not in developing new optimization methods. This reframes the research agenda from "how do we optimize editing models with human feedback?" to "how do we build verifiers that can provide rich, accurate feedback?," directing effort toward the reward model side of the RLHF pipeline.

Innovation 4: Difficulty-Aware Quadruple Construction and the Distinction Between Training the Verifier and Using It

While much of the paper focuses on the RRM architecture and GCPO algorithm, there is a subtler but important conceptual move in how the training data is constructed. The paper does not simply collect a random set of edits and train a verifier on them. Instead, it introduces a deliberate stratification of training data difficulty—partitioning the 200K source samples into a "Random" subset (100K, representing the general edit distribution) and a "Hard" subset (100K, filtered by GPT-4o to select instructions requiring multi-step modifications, fine-grained detail editing, implicit semantic understanding, or precise spatial control). This dual-subset strategy is motivated by a diagnostic observation: a verifier that only trains on typical edits may develop superficial evaluation heuristics that fail on complex cases.

The insight here parallels a finding from the LLM test-time compute literature (see the reference example's discussion of difficulty-dependent strategy selection), where performance varies qualitatively across difficulty tiers. The paper's key move is to bake difficulty awareness into the training data construction rather than the inference strategy. This is different from, say, training a difficulty classifier and routing edits dynamically—instead, the RRM is trained on a distribution that explicitly includes hard cases, forcing it to develop reasoning capabilities that generalize across the difficulty spectrum.

Why is this non-obvious? A simpler approach would be to collect the largest possible dataset of edits regardless of difficulty and trust that scale will cover edge cases. The paper argues—implicitly, through the design—that representation of difficulty in training data matters for verifier robustness. A verifier trained only on simple edits (e.g., "change the color to red") might learn that checking for color presence is sufficient, but fail on edits requiring spatial reasoning ("move the object to the left by 20%") because it never encountered such cases during training. By explicitly curating hard examples and ensuring they appear in the training distribution, the RRM is forced to develop the fine-grained reasoning capabilities that the system prompt in Appendix A.2 specifies—including checking edit magnitude, outputting bounding box coordinates for positional changes, and considering "future editing cost" for unmet principles.

The distinction is also important for understanding the role of the external VLM in the cold-start pipeline. The VLM (Seed-1.5-VL) that generates reasoning traces and the VLM that verifies them are not the same model, and they serve different purposes. The generator produces diverse, potentially noisy CoT traces; the verifier acts as a quality-control filter, selecting the trace with the highest principle-level accuracy. This two-model architecture means the RRM is not merely distilling a single teacher model—it is learning from a curated selection of the best reasoning across multiple models and sampling configurations, which is a form of inference-time ensembling baked into the training data. The fact that this curation step improves accuracy by 4–6 percentage points (Table 2, Think vs. Think+Verify) confirms that the raw VLM generations contain systematic errors that the verification step successfully filters.

This innovation is incremental in mechanism (data curation is standard practice) but fundamental in framing: it positions the RRM not as a model that learns to evaluate edits by mimicking a single oracle, but as one that learns from the best available reasoning across an ensemble, filtered by a separate verification process. This two-stage data construction (generation → verification → selection) is what makes the RRM a verifier rather than a scorer—the verification step ensures that the training data contains factually accurate reasoning, not just plausible-sounding text.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses a curated internal benchmark of 5,000 reference images and instructions from a public image-editing benchmark (Section 4.1). Multiple editing models (SeedEdit-3.0, BAGEL, FLUX.Kontext) generated several edited outputs per input, which were manually annotated using pairwise preference comparisons. Annotators could mark pairs as "same" when edits were of comparable quality or no reliable preference could be established; these ambiguous pairs were excluded when evaluating verifier accuracy, yielding a cleaner pairwise benchmark. For downstream editing model evaluation, the paper uses GEdit-Bench-EN (Liu et al., 2025), a standardized benchmark with multi-dimensional automatic metrics evaluated by GPT-4.1. An additional public benchmark, EditRewardBench (Luo et al., 2025), is used for comparing reward model accuracy against concurrent work. Qualitative evaluation also includes the Emu Edit Test Set (shown in Appendix E, Figures 7 and 8).

• Base model(s). The Reasoning Reward Model (RRM) is built on Qwen-VL-2.5 (Bai et al., 2025) at two parameter scales: 3B and 7B. This model family was chosen as an open-source VLM with strong vision-language capabilities suitable for fine-tuning into an editing-specific verifier. For downstream image editing model optimization, the paper applies Edit-R1 to two strong open-source editing models: FLUX.Kontext (Batifol et al., 2025) and Qwen-Image-Edit (Wu et al., 2025). These represent state-of-the-art editing systems with different architectures, allowing the paper to demonstrate that Edit-R1's benefits generalize across model families.

• Metrics. For reward model evaluation, the primary metric is accuracy in predicting human-annotated pairwise preferences — given a source image, instruction, and two edited images, does the RRM correctly identify which image human annotators preferred? This is reported on both the internal benchmark and EditRewardBench. For downstream editing model evaluation on GEdit-Bench-EN, the paper reports three metrics, each assessed by GPT-4.1: Semantic Consistency (SC), which measures how well the edited image aligns with the given instruction; Perceptual Quality (PQ), which captures the visual fidelity of the edited image; and Overall Score (O), computed as the geometric mean of SC and PQ ($\sqrt{\text{SC} \times \text{PQ}}$). Additionally, per-category SC scores are reported across 11 editing categories (Background Change, Color Alteration, Material Modification, Motion Change, Portrait Beautification, Style Transfer, Subject Addition, Subject Removal, Subject Replacement, Text Modification, and Tone Transformation — defined in Appendix C). For human evaluation, the paper uses the Good-Same-Bad (GSB) protocol, computing the score as $(\text{Good} - \text{Bad}) / (\text{Good} + \text{Same} + \text{Bad})$, yielding a value between -100 (always worse) and +100 (always better).

• Baselines. For reward model evaluation (Table 2), baselines include: Seed-1.5-VL API (Guo et al., 2025) and Seed-1.6-VL API (Guo et al., 2025) — strong proprietary VLMs used both as holistic scorers (the "T+V" column without GCPO) and as reference points; Qwen-7B (VIESCORE) — a baseline using VIESCORE prompts (Ku et al., 2024) for SFT data generation rather than the paper's principle decomposition approach. For the public EditRewardBench (Table 4), the baseline is EditScore-7B (Luo et al., 2025), a concurrent editing reward model that uses holistic scoring without principle decomposition or CoT reasoning, with an additional variant using inference-time scaling. For downstream editing model evaluation (Table 3), baselines include: the original unoptimized editing models (FLUX.Kontext and Qwen-Edit without RL); RL with Seed-VLM-1.5 — FLUX.Kontext optimized via GRPO using Seed-1.5-VL as the reward signal instead of the RRM; and other contemporary editing models for contextual comparison: Step-Edit (Liu et al., 2025), UniPic2 (Wei et al., 2025), Bagel (Deng et al., 2025), and GPT-4o.

• Generation budget / compute accounting. For the reward model training, the SFT phase uses 200K curated samples (100K random + 100K hard) from which approximately 2 million quadruples are generated using multiple editing models. The GCPO phase uses 10K human-annotated preference pairs. For downstream editing model optimization, the paper uses Flow-GRPO (Liu et al., 2025) with a group size of $G = 24$ — meaning the editing model generates 24 edited images per training context, each scored by the RRM. The paper notes that "although GCPO requires rollouts from both preferred and non-preferred groups, the overall training cost remains manageable in practice due to the small rollout group size and efficient packed inference" (Section 4.1). The specific number of reasoning traces generated per image in GCPO ($N$ in Equation 2) is not explicitly stated, nor does the paper quantify the token generation overhead from the RRM's CoT reasoning traces compared to a scalar regressor.

• Cross-validation / statistical protocol. The paper does not report any explicit cross-validation or statistical significance testing. The internal benchmark uses 5,000 test samples, and the paper reports raw accuracy numbers without confidence intervals, standard deviations, or statistical tests between methods. For downstream editing model evaluation, the per-category SC scores in Table 3 are reported as single-point estimates without error bars. The human evaluation (Table 5) reports a single GSB score (+23.2) without confidence intervals, and the paper does not specify the number of human annotators, the number of samples evaluated, or any inter-annotator agreement metrics.

Main Quantitative Results

Reward Model Accuracy on Internal Benchmark

The central result for reward model quality is in Table 2, where the 7B RL-RRM (full two-stage pipeline with SFT + GCPO) achieves 82.2% accuracy on the internal pairwise preference benchmark. This represents the best reported accuracy, surpassing:

• Seed-1.5-VL API (79.3%), a strong proprietary VLM, by 2.9 percentage points.
• Seed-1.6-VL API (77.2% with Think+Verify), a more recent proprietary model, by 5.0 percentage points.
• Qwen-7B (VIESCORE) (68.3%), a baseline using alternative SFT data generation prompts, by 13.9 percentage points.
• Qwen-7B without GCPO (75.4% with Think+Verify), demonstrating that GCPO contributes an additional 6.8 percentage points beyond the best SFT-only configuration.

The table also reveals the marginal contribution of each pipeline component for Qwen-7B: "Think" only (CoT reasoning without external verification filtering) achieves 68.9%; "Verify" only (direct scoring without explicit CoT reasoning, but with verification-filtered training data) achieves 70.9%; "Think+Verify" (the full cold-start SFT pipeline) achieves 75.4%; and adding GCPO pushes this to 82.2%. The 6.5 percentage point gap between "Think" alone and "Think+Verify" (75.4% - 68.9%) quantifies the value of the external verification filtering step, while the 6.8 percentage point gap between "Think+Verify" and the full RL-RRM (82.2% - 75.4%) quantifies the value of GCPO-based human preference alignment.

For Qwen-3B, the same pattern holds: "Think+Verify" reaches 69.3% and GCPO improves this to 72.0%, demonstrating that GCPO provides benefits at both model scales, though the absolute improvement from GCPO is smaller for the 3B model (2.7 percentage points vs. 6.8 for 7B), suggesting that larger models benefit more from the reasoning refinement that GCPO provides.

The scaling trend is clear: moving from 3B to 7B improves the "Think+Verify" configuration from 69.3% to 75.4% (a 6.1 percentage point gain) and the full RL-RRM from 72.0% to 82.2% (a 10.2 percentage point gain). This non-linear scaling — where the absolute benefit of increased model size is larger after GCPO training — suggests that GCPO amplifies the advantages of increased model capacity rather than merely adding a constant improvement.

Reward Model Accuracy on Public Benchmark (EditRewardBench)

Table 4 provides external validation on the independently constructed EditRewardBench. The SFT-only RRM achieves 73.3% accuracy, already surpassing EditScore-7B's 65.9% by 7.4 percentage points. This gap is meaningful because EditScore-7B represents the concurrent state of the art in editing-specific reward models, and the EditRewardBench is independently constructed from the paper's internal pipeline — this gain indicates that the verifier paradigm provides benefits beyond potential benchmark bias.

After GCPO training, the RL-RRM reaches 78.2% accuracy, a 4.9 percentage point improvement over the SFT-only version, and a 12.3 percentage point advantage over EditScore-7B. Notably, the RL-RRM also outperforms EditScore-7B with inference scaling (72.7%), which uses additional test-time computation to improve its predictions. The 5.5 percentage point gap (78.2% vs. 72.7%) indicates that the verifier-based architecture with GCPO alignment provides more accurate judgments than a scorer-based model even when the latter is given inference-time compute advantages.

Since the GCPO stage uses only 10K human preference pairs — less than 1% of the SFT-scale training data — the 4.9 percentage point improvement from GCPO cannot be attributed to increased data volume. The paper explicitly argues that this gain is "mainly attributable to better human alignment rather than increased data volume" (Section 4.1), which is a strong claim that is supported by the magnitude of improvement relative to the tiny data addition.

Impact of Principled Data Curation (Internal Benchmark Ablation)

Table 2 provides direct evidence for the importance of the paper's data curation choices. The Qwen-7B (VIESCORE) baseline — which uses VIESCORE prompts rather than the paper's principle decomposition and external verification pipeline — achieves only 68.3% accuracy. The gap between this and the "Think+Verify" configuration (75.4%) represents a 7.1 percentage point improvement attributable to the combination of principle-based CoT reasoning generation and external verification filtering. While the paper does not ablate these two components independently within the same row (i.e., there is no "VIESCORE + Verify" or "Principles without Verify" for the 7B model in the table), the decomposition suggests that both the structured reasoning format and the quality-control filtering are necessary for the observed gains.

For Qwen-3B, the "Verify" only column (66.1%) shows that verification-filtered training data alone — without explicit CoT reasoning trace generation during training — provides only a modest 2.0 percentage point improvement over "Think" only (64.1%). The combination "Think+Verify" (69.3%) provides a 5.2 percentage point gain over "Think" only and a 3.2 percentage point gain over "Verify" only, indicating that the two components are synergistic: having the model learn to reason explicitly (Think) while training on factually accurate data (Verify) is substantially better than either alone.

Downstream Editing Model Performance

Table 3 presents the main results for downstream editing models on GEdit-Bench-EN, evaluated across 11 editing categories plus aggregate metrics. The headline results depend on model family:

FLUX.Kontext family: The base FLUX.Kontext model achieves an Overall Score (O) of 5.77, with SC of 6.27 and PQ of 7.25. Optimizing with the 7B RL-RRM (full Edit-R1 pipeline) produces the best results: SC improves to 6.86 (+9.4%), PQ remains essentially flat at 7.20 (-0.7%), and O improves to 6.24 (+8.1%). This surpasses all other FLUX.Kontext variants:

• FLUX.Kontext + RL w. SeedVLM-1.5: O = 6.03, SC = 6.74. The 7B RL-RRM outperforms this by 0.21 O and 0.12 SC, demonstrating that the verifier-based reward signal is more effective than even a strong proprietary VLM as a reward source.
• FLUX.Kontext + RL w. SFT-RRM (7B): O = 6.20, SC = 6.81. The RL-RRM (which adds GCPO training) provides an additional 0.04 O and 0.05 SC improvement.
• FLUX.Kontext + RL w. SFT-RRM (3B): O = 5.63, SC = 6.52. The 7B SFT-RRM provides a substantial 0.57 O improvement over the 3B SFT-RRM, reinforcing the scaling trend observed in reward model accuracy.
• FLUX.Kontext + RL w. RL-RRM (3B): O = 6.10, SC = 6.67. The 3B RL-RRM with GCPO outperforms the 3B SFT-RRM (6.10 vs. 5.63), again showing that GCPO benefits hold at both scales, but is outperformed by the 7B SFT-RRM (6.20) and 7B RL-RRM (6.24), indicating that model scale provides larger gains than GCPO alone at the smaller scale.

The per-category breakdown reveals that the 7B RL-RRM provides the strongest improvements in specific challenging categories. For Motion Change (category 4), the base FLUX.Kontext scores 3.54; the best variant (7B RL-RRM) reaches 4.22, a 19.2% relative improvement. For Material Modification (category 3), the improvement is from 6.90 to 7.33 for the 3B RL-RRM, though the 7B RL-RRM shows a slight regression to 6.82, suggesting some category-specific variance in optimization outcomes.

Qwen-Edit family: The base Qwen-Image-Edit is already substantially stronger than FLUX.Kontext, achieving O = 7.45, SC = 7.94, PQ = 7.78. Optimizing with the 7B RL-RRM yields modest aggregate improvements: O = 7.50 (+0.7%), SC = 7.99 (+0.6%), PQ = 7.76 (-0.3%). This small overall gain is expected because Qwen-Edit is already highly optimized and "already benefits from Best-of-N scaling" (Section 4.3). However, the per-category analysis reveals a much more significant pattern: the Motion Change category (4) shows a 15.2% relative gain, from 4.01 to 4.62. This is the largest single-category improvement in the entire table, and the paper explicitly highlights it as evidence that "the validity of Edit-R1 is highlighted in its ability to address the model's specific weaknesses" even on highly optimized models. Other categories show mixed results: Portrait Beautification (5) improves from 6.04 to 6.17 (+2.2%); Subject Replacement (9) is essentially flat (8.74 vs. 8.79); Text Modification (10) declines from 9.00 to 8.69 (-3.4%), which may indicate that the RRM's evaluation criteria for text editing differ from the GPT-4.1 evaluation used in GEdit-Bench-EN.

The paper's claim that Edit-R1 "delivers substantial gains to SOTA editors" (Section 4.3) should be understood as referring primarily to FLUX.Kontext, where the gains are substantial (+8.1% O), rather than Qwen-Edit, where aggregate gains are modest (+0.7% O) but category-specific gains are concentrated in challenging areas.

Training Dynamics and Reward Signal Quality

Figure 4 provides training dynamics for downstream editing model optimization, revealing the mechanism by which GCPO improves downstream performance. The figure is organized as four columns, each comparing two reward models, with the first row showing training reward and the second row showing evaluation reward.

First column (SFT-RRM 7B vs. Seed-1.5-VL): The SFT-RRM produces a reward signal that is "as stable and effective as the Seed-1.5-VL" (Section 4.3). Both training and evaluation reward curves follow similar trajectories, indicating that even without GCPO, the verifier-based approach provides supervision quality comparable to a strong proprietary VLM.

Second column (SFT-RRM 7B vs. SFT-RRM 3B): The 7B model exhibits stronger scalability, providing more reliable supervision and yielding higher evaluation rewards. The gap between the 7B and 3B curves widens over training, suggesting that larger verifiers not only start better but continue to provide more useful discrimination as the editing model improves.

Third and fourth columns (SFT-RRM vs. RL-RRM, at 3B and 7B): Here the critical finding emerges: RL-RRMs (trained with GCPO) provide lower training rewards but higher evaluation rewards compared to their SFT counterparts. The paper interprets this as evidence that "GCPO transforms the reward model into a stricter and more robust evaluator" — the lower training rewards mean the RL-RRM is harder to satisfy during optimization (it penalizes subtle failures that the SFT-RRM might miss), but the higher evaluation rewards mean its judgments better correlate with actual downstream editing quality as measured by the evaluation metric. In other words, GCPO reduces the gap between what scores well during training and what actually produces good edits, mitigating a form of reward hacking.

The evaluation reward curves for RL-RRM variants continue to increase throughout training without plateauing, while SFT-RRM evaluation rewards show earlier saturation. This suggests that the RL-RRM provides a more informative learning signal over longer training horizons, which could translate to continued improvements with additional training.

Human Evaluation on FLUX.Kontext

Table 5 reports human evaluation results using the GSB protocol. FLUX.Kontext optimized with the 7B RL-RRM achieves a GSB score of +23.2 against the original FLUX.Kontext baseline. This means that, across the evaluated samples, the Edit-R1-optimized model's outputs were preferred over the baseline's outputs by a margin of 23.2 percentage points (computed as the difference between the fraction of wins and the fraction of losses, normalized by total comparisons). This provides human-validated evidence that the automatic metric improvements translate to perceptible quality differences. However, the paper does not report the number of samples evaluated, the number of annotators, or any inter-annotator agreement statistics, which limits the strength of this evidence.

Qualitative Results

The paper provides extensive qualitative results in Figures 6-9 and Appendices E-F. These show the Edit-R1-optimized models successfully handling diverse editing categories: Color Alter (changing bus color to lime), Material Alter (puppy to clay, full scene to ceramic), Motion Change (plane taking off, cat sleeping, dog animated), Subject Addition (spoon on plate), Subject Removal (zebra removal), and Text Modification ("FREE" removal, changing "23" to "45", adding "skate"). In each case, the baseline model (FLUX.Kontext or Qwen-Edit without Edit-R1) either fails to perform the edit, produces artifacts, or modifies unintended regions, while the Edit-R1 version produces results that better align with the instruction.

The qualitative examples in Figure 10 and Appendix G are particularly instructive for understanding what GCPO fixes. In a case where the instruction is "change the shirt to red while preserving all other features," the SFT model incorrectly changes the hat color to red as well (attribute leakage/hallucination). The RRM correctly identifies this failure, and the RL-tuned model successfully preserves the blue hat. This case study provides concrete evidence that the RRM's principle-level verification catches specific, subtle failures that a holistic scorer would likely miss, and that the RL optimization can correct these failures.

Ablation Studies and Robustness Checks

• Verifier vs. scorer paradigm (Table 4): The comparison between Edit-RRM and EditScore-7B on the public EditRewardBench provides a clean test of the verifier vs. scorer paradigm. Edit-RRM (SFT only) achieves 73.3% vs. EditScore-7B's 65.9%, a 7.4 percentage point advantage. Since EditScore-7B represents the scorer paradigm applied to the same task (image editing reward modeling), this gap directly supports the paper's central claim that structured verification outperforms holistic scoring. The fact that EditScore-7B with inference scaling only reaches 72.7% — still below the SFT-only RRM — suggests that additional inference-time computation cannot fully compensate for the architectural advantage of principle-based verification.

• Think vs. Verify vs. Think+Verify (Table 2, Qwen rows): The ablation of cold-start SFT data components shows that "Think" only (68.9% for 7B) and "Verify" only (70.9%) each provide improvements over non-structured baselines like VIESCORE (68.3%), but the combination "Think+Verify" (75.4%) provides substantially larger gains than either alone. The synergy — where the combined improvement (6.5 points over "Think" only) exceeds what either component provides individually — indicates that having the model learn explicit CoT reasoning (Think) is most beneficial when the training data is also verified for factual accuracy (Verify), and vice versa.

• GCPO contribution at different model scales (Table 2, 3B vs. 7B): GCPO improves 3B accuracy from 69.3% to 72.0% (+2.7 points) and 7B accuracy from 75.4% to 82.2% (+6.8 points). The non-linear scaling — where GCPO's benefit is 2.5× larger at 7B than at 3B — suggests that GCPO's reasoning refinement is more effective when the base model already has strong reasoning capabilities from the SFT phase. This has practical implications: if computational budget is limited, investing in a larger base model before GCPO may yield higher returns than applying GCPO to a smaller model.

• Impact of RRM size on downstream editing performance (Table 3, FLUX.Kontext rows): Comparing the four FLUX.Kontext variants trained with different RRMs (SFT-RRM 3B, RL-RRM 3B, SFT-RRM 7B, RL-RRM 7B) reveals a consistent rank ordering: 7B RL-RRM (O=6.24) > 7B SFT-RRM (O=6.20) > 3B RL-RRM (O=6.10) > 3B SFT-RRM (O=5.63). Each step up in either model size or GCPO training provides a monotonic improvement, with model size providing larger absolute gains (3B SFT to 7B SFT: +0.57 O; 7B SFT to 7B RL: +0.04 O). This suggests that verifier accuracy — as measured by the internal benchmark — translates directly to downstream editing model quality, and that both scaling and alignment contribute independently.

• GCPO training dynamics (Figure 3): The weighted advantage metric (Figure 3c) being negative indicates that the model learns to generate longer reasoning traces for correct judgments. This is a diagnostic that is specific to generative verifiers — a scalar regressor would not show this pattern — and provides indirect evidence that GCPO is refining the reasoning process rather than just the final score calibration. Figure 3d shows stable training reward improvement and clear separation between 3B and 7B models, with the 7B model achieving higher rewards throughout training.

• Reward signal stability (Figure 4): The SFT-RRM (7B) produces training dynamics (first row, first column) that are comparable to Seed-1.5-VL, a proprietary VLM. This is a robustness check showing that the verifier-based approach, even without GCPO, does not introduce instability compared to established reward sources. The RL-RRM variants (third and fourth columns) maintain stable training trajectories despite being stricter evaluators, indicating that the GCPO training does not make the RRM's reward signal excessively noisy or sparse.

• Edit-Specific Weakness Correction (Table 3, Qwen-Edit Motion Change category): The 15.2% relative improvement on Qwen-Edit's Motion Change category (from 4.01 to 4.62) while the overall score only improves 0.7% is a specific robustness check: it demonstrates that the RRM's principle-based feedback is especially valuable for categories where the base model has systematic weaknesses, and that the optimization does not simply improve all categories uniformly but selectively addresses failure modes.

Critical Assessment

Does the paper demonstrate that the verifier paradigm is superior to the scorer paradigm for image editing reward models?

The evidence is strong but limited in scope. The comparison on EditRewardBench (Table 4) shows a clear 73.3% vs. 65.9% advantage over EditScore-7B, and the ablation in Table 2 (75.4% for "Think+Verify" vs. 68.3% for a VIESCORE-based scorer) supports the claim. However, the comparison is against a single scorer-based competitor (EditScore-7B) and one baseline prompt (VIESCORE). A more comprehensive validation would compare against a wider range of scorer architectures — for example, a regression-based reward model fine-tuned on the same training data, or other concurrent scoring-based editing reward models not cited in the paper. The claim that the verifier paradigm shift is universally superior to scoring for image editing is supported by the available evidence but is not exhaustively tested.

Additionally, the ablation between "Think" and "Verify" in Table 2 does not cleanly separate the verifier architecture from the data quality effect. The "Think+Verify" model receives both structured reasoning training (the model is trained to generate CoT traces) and higher-quality training data (the traces are filtered for factual accuracy). The improvement could be partly attributable to better training data quality rather than the verifier architecture itself. A cleaner ablation would train a scorer on the same verified, principle-labeled data but without the CoT reasoning objective — this would test whether the structured output format provides benefits beyond data quality improvements.

Does the paper demonstrate that GCPO effectively aligns the RRM with human preferences?

The evidence is convincing but limited in scale. The 78.2% vs. 73.3% improvement on EditRewardBench (Table 4) and the 82.2% vs. 75.4% improvement on the internal benchmark (Table 2) show consistent gains from GCPO across both benchmarks. The training dynamics in Figure 4 (RL-RRM provides lower training rewards but higher evaluation rewards) provide mechanistic evidence for why GCPO helps — it makes the verifier stricter and more discriminating. The qualitative example in Figure 10 demonstrates that GCPO corrects a specific hallucination (hat color leakage) that the SFT model missed.

However, GCPO is trained on only 10K human preference pairs, and the paper does not ablate the amount of preference data to determine whether the 4–7 percentage point gain is near the ceiling or whether additional human annotations would provide further improvements. The paper also does not compare GCPO against alternative methods for incorporating pairwise preferences into pointwise reward model training (e.g., converting pairwise preferences to scalar targets via Elo scores or Bradley-Terry model fitting, then training the RRM with regression). Without this comparison, it is unclear whether GCPO's cross-group win/loss ratio mechanism is genuinely more effective than simpler approaches, or whether any method of incorporating preference data would achieve similar gains.

The paper also does not report whether GCPO training changes the RRM's calibration (i.e., whether the absolute score values become more or less aligned with human judgments of absolute quality, as opposed to relative preferences). Since the RRM is used as an absolute scorer in the downstream GRPO training (the advantages are computed from absolute scores, not win/loss ratios), calibration matters. A model that perfectly ranks outputs within a group but systematically overestimates or underestimates absolute quality could still provide useful advantages (since advantages are group-normalized), but calibration errors could affect the stability of training if score distributions shift as the editing model improves.

Does the paper demonstrate that the RRM as a reward signal improves downstream editing models?

The evidence is generally strong but with important caveats about baseline strength and generalizability.

Strong evidence for FLUX.Kontext: The improvement from O=5.77 (base) to O=6.24 (7B RL-RRM) is substantial (+8.1%) and consistent across variants: the 7B RL-RRM outperforms the SeedVLM-1.5 baseline (6.03) and the 3B variants, and the performance monotonically improves with RRM scale and GCPO training. The human evaluation (GSB +23.2, Table 5) provides human-validated evidence. The per-category improvements (e.g., Motion Change from 3.54 to 4.22) show that the gains are not uniform but concentrated in challenging categories, consistent with the claim that the RRM provides fine-grained, principle-level feedback.

Weaker evidence for Qwen-Edit: The aggregate improvement from O=7.45 to O=7.50 (+0.7%) is small, and the paper's argument that this is "largely because the baseline model already benefits from Best-of-N scaling" (Section 4.3) is plausible but not experimentally verified. The paper does not show what improvement a simpler reward model (e.g., SeedVLM-1.5) would provide for Qwen-Edit — without this comparison, it is unclear whether the RRM's small improvement is due to the verifier architecture or simply because the model is near its performance ceiling given the training data and optimization algorithm. The 15.2% gain on Motion Change is notable but is a single category; it would be more convincing if multiple categories showed similar concentrated improvements.

Missing comparison to a "same-budget" non-verifier baseline: The paper does not train a non-verifier reward model (e.g., an ORM-style scalar regressor) on the same amount of training data and human preference pairs, then use it to optimize FLUX.Kontext via the same GRPO procedure. Without this comparison, it is unclear how much of the downstream improvement is attributable to the verifier architecture specifically versus simply having a decent reward model trained on a large amount of editing-specific data. The SeedVLM-1.5 baseline provides a partial comparison (it is a non-verifier and is used as a reward source), but it is a proprietary API with unknown training data and architecture, making it an imprecise comparison point.

Are the evaluation metrics aligned with the paper's claims?

A significant concern is the metric tautology risk in the downstream editing evaluation. The RRM is trained to predict human pairwise preferences on the internal benchmark (Table 2). The downstream editing models are optimized via GRPO to maximize the RRM's scores. The evaluation is performed using GPT-4.1-based metrics (GEdit-Bench-EN). If GPT-4.1's evaluation criteria correlate with the RRM's scoring criteria (both are based on principles like instruction fidelity, preservation, and quality), then improvements in the RRM's scores might translate to improvements in GPT-4.1 scores regardless of whether the edits are actually better by human standards. The human evaluation in Table 5 partially addresses this concern (showing that human judges also prefer the Edit-R1 outputs), but the paper does not report the number of human evaluation samples or any correlation between human judgments and GPT-4.1 scores, limiting the strength of this validation.

The paper also does not report reward hacking diagnostics for the downstream editing model optimization. In RLHF, a common failure mode is that the policy learns to produce outputs that score highly under the reward model but are actually low-quality — exploiting the reward model's blind spots. The training dynamics in Figure 4 show that evaluation rewards increase, but the paper never checks whether the RRM's scores on generated edits correlate with human judgments at the end of training (as opposed to the beginning). If reward hacking occurred, the RRM's scores would increase while human-judged quality would plateau or decrease. The human evaluation in Table 5 is a point comparison (final model vs. baseline) but does not track human-judged quality throughout training to detect reward hacking.

What is missing?

• No comparison to DPO-based optimization for downstream editing: The paper argues that DPO "inherently restricts policy exploration, risking suboptimal convergence" (Section 2.2), but never tests this claim empirically. A direct comparison of FLUX.Kontext + DPO (using the same preference data) vs. FLUX.Kontext + GRPO (using the RRM) would validate whether the added complexity of training a separate verifier is justified.

• No ablation of the number of reasoning traces in GCPO: The value of $N$ (the number of reasoning traces generated per image in GCPO training) is not specified in the paper, nor is there an ablation study showing how GCPO performance varies with $N$. Since generating reasoning traces is computationally expensive (each trace involves full CoT generation), this is an important practical consideration.

• No analysis of RRM inference cost vs. scalar reward model cost: The paper never quantifies the computational overhead of using an RRM (which generates full CoT reasoning traces) compared to a scalar regressor during downstream editing model training. In GRPO with $G=24$, the RRM must be run 24 times per training step, each time generating a reasoning trace that may be hundreds of tokens long. This cost should be reported and compared to a baseline using a scalar reward model.

• No per-category reward model accuracy breakdown: The paper reports aggregate accuracy on the pairwise preference benchmark (Table 2) but does not break this down by editing category (e.g., is the RRM more accurate on color changes than on motion changes?). This would help explain the downstream editing performance patterns — for instance, if the RRM is less accurate on text modification, this might explain why Qwen-Edit's Text Modification score declines from 9.00 to 8.69 after RL-RRM optimization.

• No statistical significance testing: Throughout the experimental section, all results are reported as point estimates without confidence intervals, standard deviations, or p-values. For the internal benchmark (5,000 samples), confidence intervals could be computed. For the per-category SC scores in Table 3, it is unclear whether differences of 0.1–0.3 points are statistically meaningful or within noise.

• Limited scale of human evaluation: The GSB score of +23.2 (Table 5) is reported without sample size, number of annotators, or inter-annotator agreement. A robust human evaluation would report these statistics and ideally compare human preferences against both GPT-4.1 scores and RRM scores to validate the entire evaluation pipeline.

• No test of generalization to unseen editing models: The RRM is trained on data generated by Flux-Kontext, Bagel, and SeedEdit 3.0. It is tested as a reward signal for optimizing FLUX.Kontext and Qwen-Edit. Qwen-Edit was not among the training data generators, so the results on Qwen-Edit test generalization to some extent. However, it would be stronger to evaluate the RRM as a reward model for an entirely held-out editing model not used in training data generation, to test whether the verifier's evaluation criteria generalize across model architectures and editing approaches.

Overall Assessment

The paper's experiments generally support its central claims within the tested scope, but the scope has important boundaries. The verifier paradigm is convincingly shown to outperform the scorer paradigm on the specific benchmarks and against the specific baselines tested, on the specific model family (Qwen-VL-2.5) used as the base for the RRM, and on the specific editing models (FLUX.Kontext, Qwen-Edit) optimized downstream. Whether these findings generalize to other base VLMs, other editing models, other benchmarks, and other preference datasets is untested. The paper's positioning as a paradigm shift is supported by the available evidence but requires replication across broader conditions to be fully convincing.

The GCPO algorithm is demonstrated to improve RRM accuracy and downstream editing performance, but the lack of comparison to simpler methods for incorporating pairwise preferences (e.g., Bradley-Terry regression) makes it difficult to assess whether the specific cross-group win/loss ratio mechanism is necessary or merely sufficient. The downstream editing improvements are solid for FLUX.Kontext but marginal for Qwen-Edit in aggregate, and the concentration of gains in specific categories (Motion Change) while other categories regress (Text Modification for Qwen-Edit, Material Modification for FLUX.Kontext 7B RL-RRM) suggests that the RRM's evaluation criteria may not be uniformly aligned with GPT-4.1's evaluation across all editing types — a limitation that is not discussed in the paper.

The most significant experimental gap is the absence of cost-benefit analysis: the paper never quantifies the computational overhead of the verifier paradigm (principle decomposition, CoT reasoning generation, external verification filtering, GCPO training with multiple reasoning traces per image) relative to simpler approaches, making it difficult for practitioners to assess whether the accuracy gains justify the added complexity and inference cost.

6. Limitations and Trade-offs

The Cost of Difficulty-Aware Data Curation Is Not Amortized into Headline Performance Numbers

The cold-start SFT data construction pipeline (Section 3.1.1) is a substantial engineering effort that the paper treats as an upfront investment independent of the model's inference-time performance. Building the training dataset involves: decomposing 200K editing instructions into principles via the Seed-1.5-VL API (Step 1); generating ~2 million edited candidates across three editing models (Step 2); running VLM pools to produce multiple CoT reasoning traces per quadruple, varying prompts, temperatures, and model variants (Step 3); and then running a separate external VLM verifier to re-evaluate every principle in every reasoning trace and select only the highest-accuracy trace (Step 4). This pipeline involves multiple expensive API calls to proprietary models (Seed-1.5-VL, Seed-1.6-VL, GPT-4o for the "Hard" subset filtering) and generates massive amounts of intermediate data. The paper does not report the FLOPs, API call count, dollar cost, or wall-clock time for this pipeline anywhere.

The consequence is practical: a team wanting to replicate this work for a different editing model or domain would need to budget for this entire data construction pipeline before training begins. The headline 82.2% accuracy and the 8.1% FLUX.Kontext O-score improvement cannot be achieved without this infrastructure. More importantly, the paper's implicit claim — that the verifier paradigm is the key architectural insight — cannot be cleanly separated from the possibility that the gains come substantially from the scale and quality of the training data produced by this expensive pipeline. A baseline that uses the same VLM APIs and similar data volume but with a simpler scorer architecture might close some of the gap. The paper does not provide this ablation (a scorer trained on the same verify-filtered data without CoT reasoning).

What evidence exists: No cost accounting appears in the paper. The SFT data construction is described qualitatively in Section 3.1.1 with no budget numbers. The internal benchmark accuracy gains from "Think+Verify" over "VIESCORE" (75.4% vs. 68.3%, Table 2) conflate the verifier architecture with the data construction pipeline quality — VIESCORE uses a different prompt and presumably a different (simpler) data construction process, so the 7.1 percentage point gap cannot be attributed solely to the verifier paradigm.

Mitigation status: The paper does not address this cost, frame it as a limitation, or suggest ways to reduce it. Future work on cheaper principle decomposition (e.g., using a smaller distilled model rather than the Seed-1.5-VL API) or amortizing data construction across multiple reward model versions would be natural extensions, but none are proposed.

The RRM Is Evaluated Only on Models and Data from Its Training Distribution, with No Test of Generalization to Unseen Editing Systems

The RRM is trained on edited images generated by Flux-Kontext, Bagel, and SeedEdit 3.0 (Section 3.1.1, Step 2). It is then deployed as a reward signal to optimize FLUX.Kontext (one of the training generators) and Qwen-Image-Edit (not explicitly listed as a training generator, but the paper does not confirm that Qwen-Edit outputs were entirely excluded from the 2M quadruples). The paper evaluates the RRM as a reward model on GEdit-Bench-EN (Table 3), whose edited outputs come from multiple models, but the benchmark itself may overlap with the training data distribution in terms of editing model families and editing types. The evaluation of reward model accuracy on EditRewardBench (Table 4) is stronger because this benchmark is "independently constructed from our internal pipeline" (Section 4.2), but the paper does not specify which editing models generated the outputs in EditRewardBench or whether those models were among the RRM's training generators.

The consequence is uncertainty about whether the RRM's evaluation criteria are genuinely general or are overfit to the characteristic editing patterns, artifacts, and failure modes of the specific models used during training. If a new editing model with a fundamentally different architecture — for example, one that produces edits with different types of visual artifacts, different levels of edit magnitude, or different preservation failure patterns — were evaluated by this RRM, the per-principle verification judgments might be less accurate. This matters for the paper's positioning because the RRM is presented as a general verifier for image editing, not as a verifier specifically tuned for the models it was trained on. The downstream application to Qwen-Edit is a partial test of generalization, but the aggregate performance improvement is very small (+0.7% O), and the paper does not report a separate reward model accuracy evaluation specifically on Qwen-Edit outputs to check whether the RRM's judgments remain calibrated.

What evidence exists: Table 3 shows that Qwen-Edit optimization with the 7B RL-RRM yields only marginal aggregate improvement (O: 7.45 → 7.50), which could indicate either that Qwen-Edit is near its ceiling or that the RRM's evaluation is less effective on Qwen-Edit's output distribution. The Motion Change category shows a notable 15.2% gain (4.01 → 4.62), suggesting some transfer, but other categories like Text Modification decline (9.00 → 8.69). Without a baseline showing what improvement a Qwen-Edit-specific reward model would provide, it is impossible to determine how much of the limited aggregate gain is due to distribution mismatch versus ceiling effects. The paper does not report the training data generation models' overlap with evaluation models.

Mitigation status: The paper does not acknowledge this as a limitation or discuss distribution shift between training and evaluation editing models. The Qwen-Edit result is presented as evidence of generalizability, but the small aggregate gain and category-specific regressions are not analyzed from a generalization perspective.

The Downstream Editing Model Optimization Shows Category-Specific Regressions That Are Not Diagnosed or Explained

Table 3 reports per-category SC scores for all FLUX.Kontext and Qwen-Edit variants. While the aggregate metrics (SC, PQ, O) improve for most Edit-R1 configurations, several categories show consistent regressions that the paper does not discuss or explain. For FLUX.Kontext optimized with the 7B RL-RRM (the best overall variant), Background Change (category 1) declines from 8.23 (RL w. SeedVLM-1.5) to 8.15, and Material Modification (category 3) declines from 7.10 (SFT-RRM 7B) to 6.82. For Qwen-Edit optimized with the 7B RL-RRM, Text Modification (category 10) declines from 9.00 to 8.69, and Style Transfer (category 6) declines from 7.61 to 7.76 (actually a small improvement, but let me correct — Style Transfer shows 7.61 → 7.76, which is an improvement; the regression is in Background Change from 8.85 to 8.75, and Subject Replacement from 8.74 to 8.79 which is flat).

The consequence is that the RRM-guided GRPO optimization does not uniformly improve editing quality — it trades off performance across categories, improving some (especially Motion Change, where gains are largest) while degrading others. For a deployed editing system, this is a significant practical concern: optimizing with Edit-R1 might make the model better at challenging spatial edits like motion changes but worse at seemingly simpler tasks like material modification or text editing. The paper's aggregation of results (Overall Score, which is the geometric mean of SC and PQ) masks these regressions because gains in high-weight categories can offset declines in others.

This trade-off pattern is not discussed as a limitation or analyzed for potential causes. It could arise from several mechanisms: (a) the RRM's evaluation criteria might be better aligned with human preferences for some editing categories than others, causing the GRPO optimization to over-optimize on categories where the RRM is a good judge and under-optimize or mis-optimize on categories where it is not; (b) the training data distribution might underrepresent certain editing categories, leading to poorer RRM calibration on those categories; (c) the GRPO optimization with a single scalar reward (even if generated from principle decomposition) might create perverse incentives where the model learns to produce edits that score well on average across principles but perform worse on specific principles that are harder to satisfy while also optimizing for others.

What evidence exists: The per-category regressions are visible in Table 3 but are not highlighted, discussed, or explained in the text. The paper's narrative focuses entirely on the positive results — the aggregate improvements for FLUX.Kontext (5.77 → 6.24 O) and the Motion Change gain for Qwen-Edit (4.01 → 4.62) — without mentioning that some categories move in the wrong direction. There is no per-category analysis of RRM accuracy to determine whether categories with regressions correspond to categories where the RRM's internal benchmark accuracy is lower.

Mitigation status: The paper does not acknowledge this trade-off, does not provide per-category reward model accuracy breakdowns, and does not discuss whether per-category reward model fine-tuning or category-aware optimization could mitigate these regressions. Given that the overall gains are modest for Qwen-Edit (+0.05 O) and the category-specific regressions are of comparable magnitude, a practitioner might reasonably ask whether the optimization is worth the risk of degrading performance on specific editing types that are important for their application.

GCPO's Computational Overhead and the Missing Ablation Against Simpler Pairwise-to-Pointwise Conversion Methods

GCPO (Section 3.1.2) requires generating N reasoning traces (the exact value of N is never specified in the paper) for both the winner and loser images in each of the 10K human preference pairs, then performing N × N cross-group comparisons to compute win/loss ratios, then running a full GRPO optimization step. For each training step, the RRM must generate 2 × N full Chain-of-Thought reasoning traces (each potentially hundreds of tokens long) before any gradient computation occurs. The paper states that "the overall training cost remains manageable in practice due to the small rollout group size and efficient packed inference" (Section 4.1), but never quantifies this cost or compares it to alternatives.

The consequence is twofold. First, practitioners cannot estimate the computational budget required to replicate GCPO training, since N is not reported and the inference cost per reasoning trace depends on trace length (which Figure 3c indicates increases during GCPO training, as the model learns to produce longer reasoning for correct judgments). Second, and more fundamentally, the paper does not demonstrate that GCPO's cross-group win/loss ratio mechanism is necessary rather than merely sufficient. A much simpler alternative would be: for each preference pair, generate one score per image (no reasoning traces, or a single reasoning trace), compute whether the winner scores higher than the loser (binary accuracy reward), and train the RRM to maximize this binary accuracy using standard policy gradient. Alternatively, fit a Bradley-Terry model to convert pairwise preferences into scalar quality estimates for individual images, then train the RRM with regression against these estimates. Both alternatives would be computationally cheaper (no N × N comparisons, no multiple traces per image) and conceptually simpler. Without comparing against these baselines, the paper cannot claim that GCPO's specific mechanism provides benefits beyond simpler methods for incorporating preference data.

What evidence exists: The paper provides no ablation of GCPO against alternative pairwise-to-pointwise conversion methods. The GCPO contribution is validated only against not using GCPO at all (SFT-RRM vs. RL-RRM in Table 2, Table 4). The improvement from GCPO is clear (e.g., 75.4% → 82.2% on the internal benchmark for 7B), but it is unknown whether a simpler method using the same 10K preference pairs would achieve similar or only slightly lower gains at a fraction of the computational cost. The paper also never specifies N, the number of reasoning traces, making it impossible to estimate the cost multiplier of GCPO relative to single-trace inference.

Mitigation status: The paper does not compare GCPO to any alternative method for incorporating pairwise preferences, does not report N, and does not discuss the cost-accuracy trade-off of GCPO's design choices. The claim that GCPO is a novel and necessary contribution rests on the assumption that standard RLHF algorithms (DPO, standard GRPO) "are ill-suited for this task" (Section 1), but this claim is never tested empirically — no DPO or standard GRPO baseline for RRM training is reported.

The Hardest Editing Categories Show Near-Zero Improvement from Test-Time Optimization, and the Paper Does Not Characterize This Capability Boundary

Across all editing model variants in Table 3, certain categories remain stubbornly low regardless of the RRM variant used or the optimization applied. For FLUX.Kontext, the Motion Change category (category 4) starts at 3.54 for the base model. The best variant (7B RL-RRM) reaches 4.22 — an improvement, but still far below the model's performance on easier categories like Subject Removal (7.54 → 7.99) or Tone Transformation (7.85 → 7.82). Portrait Beautification (category 5) shows a similar pattern: base FLUX.Kontext scores 3.54, the best variant reaches 4.22. These categories represent editing tasks that fundamentally challenge the base model's capabilities — changing an object's pose, facial expression, or motion state while preserving identity and background coherence.

The consequence is that the RRM-guided GRPO optimization, much like the test-time compute methods in the reference example paper, amplifies existing capability but does not create it from nothing. If the base editing model cannot produce a correct edit for a given category at any reasonable probability, no amount of reward model feedback will help — the policy gradient has no positive examples to reinforce. The paper never explicitly characterizes this capability boundary or quantifies how much improvement is possible as a function of base model quality. This matters for practitioners because it sets expectations: Edit-R1 can make a good editing model better, especially on categories where it already has some competence, but it cannot rescue a model from fundamental architectural limitations on challenging editing types.

More subtly, this capability boundary may interact with the scaling trends observed in the RRM. If the RRM's principle decomposition and verification are most accurate on edit types where the base editing model already performs reasonably well (because the training data contains more success examples to learn from), then the RRM's reward signal is weakest precisely where the editing model needs the most help — on difficult categories. The paper does not analyze the correlation between RRM accuracy per category and editing model improvement per category.

What evidence exists: Table 3 shows that categories with the lowest base model scores (Motion Change, Portrait Beautification) show the largest relative improvements (19.2% for FLUX.Kontext Motion Change, 15.2% for Qwen-Edit Motion Change), but also remain far below the model's average performance even after optimization. The paper does not break down RRM accuracy by editing category, so it is impossible to determine whether the RRM is equally reliable at evaluating edits in these difficult categories versus easier ones. The training dynamics in Figure 4 show evaluation rewards increasing throughout training without plateauing, but this does not indicate whether the rewards are increasing uniformly across categories or primarily on already-solvable edits.

Mitigation status: The paper does not discuss capability boundaries, does not analyze per-category reward model accuracy, and does not characterize the relationship between base model performance and achievable improvement. This is a significant gap for practitioners trying to determine whether Edit-R1 is worth deploying for their specific editing use case and model combination.

Human Evaluation Is Underreported, Making the Claim that Automatic Metric Gains Translate to Perceptible Quality Improvements Difficult to Verify

The paper includes exactly one human evaluation result: FLUX.Kontext optimized with the 7B RL-RRM achieves a GSB score of +23.2 against the baseline (Table 5). No other human evaluation is reported — not for the RRM's accuracy against human judgments (the pairwise preference benchmark in Table 2 is the internal benchmark, but the paper does not report human inter-annotator agreement or the number of annotators), not for Qwen-Edit optimization, and not for any ablation comparisons (e.g., RL-RRM vs. SFT-RRM, or 7B vs. 3B).

The consequence is that the paper's central claim — that the verifier paradigm and GCPO training lead to better image editing as perceived by humans — rests almost entirely on automatic metrics (GPT-4.1 scores on GEdit-Bench-EN). The paper argues, implicitly, that GPT-4.1 evaluation correlates with human judgment, but this correlation is never measured or cited. The risk of metric tautology is real: the RRM is trained to predict human pairwise preferences on a certain distribution of edits, the editing model is optimized to maximize the RRM's scores, and the evaluation uses GPT-4.1 (another VLM) to score the edits. If GPT-4.1's evaluation criteria share biases with the VLM pools used to generate the RRM's training data or with the VLM used for external verification (SeedVLM-1.5), then improvements in GPT-4.1 scores might reflect overfitting to a shared set of VLM-specific evaluation heuristics rather than genuine perceptual improvement.

The single GSB score of +23.2 is reported without sample size, without the number of annotators, without inter-annotator agreement (e.g., Fleiss' kappa), and without confidence intervals. The paper does not specify whether the GSB evaluation was conducted on a held-out test set distinct from the GEdit-Bench-EN evaluation set, or whether it used the same editing instructions and source images. Without these details, the +23.2 figure is effectively uninterpretable — it could represent a robust preference across hundreds of samples with high agreement, or a weak preference across dozens of samples with noisy annotations.

What evidence exists: Table 5 is a single-row, single-number table with no supporting statistics. The paper's qualitative examples (Figures 6–9) show compelling visual improvements for selected edits, but qualitative cherry-picking does not substitute for systematic human evaluation. The paper does not report whether the human evaluation covered a representative sample of editing categories or was biased toward categories where Edit-R1 excels.

Mitigation status: The paper does not acknowledge the thinness of its human evaluation or the metric tautology concern. No suggestions for more comprehensive human evaluation are made in the conclusion or future work sections. For a paper whose central contribution is improving image editing through better reward modeling — where human perception is the ultimate ground truth — this is a significant evidential gap.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper changes the conversation around reward modeling for visual generation by establishing that a verifier is fundamentally different from — and better than — a scorer, and that this difference matters for downstream RLHF performance. Before Edit-R1, the field's default assumption was that improved reward models meant larger VLMs, better training data, or fancier prompting strategies — all within the scorer paradigm where the model outputs a single number. The paper's central reframing is that the structure of the evaluation matters more than its scale: a 7B model that verifies explicit principles through CoT reasoning outperforms proprietary VLMs of unknown (likely much larger) scale that produce holistic scores. The headline number supporting this is the 82.2% internal benchmark accuracy for the 7B RL-RRM versus 79.3% for Seed-1.5-VL and 77.2% for Seed-1.6-VL (Table 2) — the open-source verifier beats the proprietary scorers by a margin that is not attributable to model scale.

The magnitude of this shift is substantial but bounded. It is not a paradigm shift in the Kuhnian sense — the paper does not introduce a new scientific framework that redefines the field's questions. It is better understood as a reframing with demonstrated practical consequences: the paper shows that explicit decomposition of the evaluation task into sub-tasks (principles), combined with verifiable reasoning traces and alignment through the novel GCPO algorithm, produces both better reward models and better downstream editing models. The reframing is from "how do we get a VLM to output the right score?" to "how do we get a VLM to perform the right verification checks, and then train it to reason correctly about those checks?" This shifts attention from score calibration (adjusting the magnitude of outputs) to reasoning fidelity (ensuring the model's internal evaluation logic is factually consistent with the image content and aligned with human preferences).

The paper also reconciles a tension in the reward model design space that was visible but unresolved in prior work. Table 1 catalogues the field's fragmentation: some models are pointwise (usable as RLHF reward sources but lacking reasoning), some support CoT (interpretable but designed for non-visual tasks), some use principles (structured but without RL fine-tuning of the reward model itself), and some use RL (but are pairwise and cannot provide absolute scores). Edit-R1 is the first model to simultaneously satisfy all four desiderata for image editing. This resolution matters because it demonstrates that these capabilities are composable — you can have a pointwise, principle-decomposing, CoT-reasoning, RL-aligned reward model — and that their combination produces gains beyond what any subset achieves. The ablation in Table 2 provides evidence for this: "Think" alone (CoT without verification) reaches 68.9%, "Verify" alone (verification-filtered training without CoT reasoning during inference) reaches 70.9%, and "Think+Verify" (the full combination) reaches 75.4% — a 6.5 point gain over "Think" alone and 4.5 points over "Verify" alone. These components are synergistic, not merely additive.

A consequential shift the paper initiates is redirecting RLHF research effort from optimization algorithms to reward model architecture. Prior work in visual RLHF has invested heavily in developing better optimization algorithms — DPO variants (DreamFuse), REFL-based approaches (ByteEdit, OneReward), and GRPO adaptations (Flow-GRPO, DanceGRPO). The paper's implicit argument is that this focus is partly misallocated: GRPO already works for editing models (the paper uses it without modification), and the bottleneck is the quality and structure of the reward signal itself. The evidence is that switching from a strong proprietary VLM (Seed-1.5-VL) to the 7B RL-RRM as the reward source improves FLUX.Kontext's Overall Score from 6.03 to 6.24 (Table 3), while the optimization algorithm remains identical (Flow-GRPO with G=24, β=0.04). This suggests that marginal research investment in better verifiers will have higher returns than marginal investment in better optimizers, at least in the current regime where even SOTA proprietary VLMs produce suboptimal reward signals.

The paper also introduces a new diagnostic lens for reward model quality: rather than evaluating reward models solely by their ability to predict pairwise preferences (the standard metric), the verifier paradigm makes it natural to evaluate per-principle verification accuracy — does the model correctly judge each specific sub-task of the editing instruction? This is the logic behind the external verification step in the cold-start pipeline (Section 3.1.1, Step 4), where reasoning traces are filtered by their agreement with a gold-standard per-principle verifier. The paper does not use this as an evaluation metric for the final model, but it is an obvious extension: a benchmark that evaluates reward models on per-principle accuracy (not just aggregate preference prediction) would provide a richer diagnostic of where and why reward models fail. This shifts the evaluation philosophy from "does the model get the final answer right?" to "does the model get the intermediate reasoning right?," which is a more granular and actionable signal for model improvement.

Finally, the paper narrows the plausibility space for certain alternative approaches. The concurrent EditScore advocates for holistic scoring as the path forward for editing reward models. The 73.3% vs. 65.9% gap on EditRewardBench (Table 4) between Edit-RRM (SFT only) and EditScore-7B — and the 78.2% vs. 72.7% gap including GCPO and inference scaling, respectively — makes it harder to argue that scaling scorer-based approaches will close the gap with verifier-based approaches. The paper does not prove that scoring can never catch up, but it establishes a substantial performance deficit that future scorer-based work must address, and it provides a mechanistic hypothesis for the gap: scorers conflate multiple evaluation dimensions into a single number, making the mapping from image content to score less learnable and more prone to hallucinated feedback.

Follow-Up Research This Work Enables

Directly comparing GCPO against simpler pairwise-to-pointwise conversion methods, measuring accuracy per unit of human annotation cost. The paper demonstrates that GCPO improves RRM accuracy (75.4% to 82.2% on the internal benchmark, 73.3% to 78.2% on EditRewardBench) using only 10K human preference pairs. However, it never tests whether these gains require GCPO's specific cross-group win/loss ratio mechanism. A strong ablation study would train the same SFT-RRM using: (a) Bradley-Terry model fitting to convert pairwise preferences into scalar Elo scores, then regression fine-tuning against those scores; (b) a simplified version of GCPO with N=1 (single reasoning trace per image, binary accuracy reward); (c) DPO-style direct optimization on the preference pairs (treating the RRM's reasoning traces as the policy's responses and the preference pairs as the ranking signal). The key metric would be internal benchmark accuracy per 1,000 human annotations, tracking whether GCPO's more complex mechanism provides better data efficiency or merely equal performance at higher computational cost. A negative result — where Bradley-Terry regression matches GCPO performance — would be valuable because it would suggest that the GCPO mechanism is not necessary and that simpler methods suffice, simplifying replication and deployment. A positive result — where GCPO outperforms at low annotation budgets but the gap shrinks with more data — would characterize the regime where GCPO's complexity is justified.

Building a public benchmark for per-principle verification accuracy in image editing reward models. The paper's cold-start pipeline introduces the concept of evaluating reasoning traces by their principle-level factual accuracy (Step 4), but this evaluation is only used during training data construction, not as a final model benchmark. A natural extension would construct a dataset of (source image, edit instruction, edited image, principles, per-principle ground-truth judgments) where the per-principle judgments are verified by multiple expert human annotators with high inter-annotator agreement. This benchmark would enable answering questions the current evaluation cannot: Does the RRM's reasoning trace correctly identify that a background preservation principle is violated, even when the final score is accurate? Are there systematic categories of principles (e.g., spatial positioning, material properties, fine-grained detail preservation) where the RRM consistently hallucinates? Does GCPO improve per-principle accuracy uniformly or only on principles where the SFT model was already strong? The paper's verification infrastructure (Appendix A.3's system prompt, which instructs a VLM to produce gold-standard 0/1 judgments per principle using images as ground truth) provides a starting point for bootstrapping such a dataset, but final validation would require human verification of a subset to calibrate the VLM's per-principle accuracy.

Testing whether the verifier paradigm generalizes across editing model architectures when the RRM is trained exclusively on a held-out set of models. The paper trains the RRM on edited images from Flux-Kontext, Bagel, and SeedEdit 3.0, then deploys it to optimize FLUX.Kontext (an in-distribution model) and Qwen-Edit (partially out-of-distribution, with limited aggregate improvement). A clean generalization experiment would: (1) train the RRM exclusively on data from model families A and B (e.g., FLUX.Kontext and Bagel), (2) freeze the RRM, (3) use it as the reward signal to optimize a new editing model from family C (e.g., a model with a fundamentally different architecture like a masked generative transformer or a GAN-based editor, held out from training data generation entirely), and (4) measure both the RRM's accuracy on family C's outputs (on a human-annotated preference benchmark) and the downstream editing improvement after GRPO optimization. The hypothesis is that the verifier paradigm should generalize better than holistic scoring because principle-based evaluation is architecture-agnostic (checking "is the car now red?" does not depend on how the edit was produced), but there may be systematic generalization failures if family C produces edits with characteristic artifacts that the RRM has never encountered and misinterprets. Characterizing this generalization boundary would determine whether the RRM can serve as a universal editing reward model or needs to be fine-tuned for each new editing model architecture.

Characterizing the relationship between RRM scale, principle granularity, and downstream editing improvement across difficulty tiers. The paper shows a scaling trend from 3B to 7B in both RRM accuracy (69.3% → 75.4% for SFT, Table 2) and downstream FLUX.Kontext performance (O: 5.63 → 6.20 for SFT-RRM, 6.10 → 6.24 for RL-RRM, Table 3). But it does not explore the interaction between model scale and evaluation granularity. A systematic study would train RRMs at multiple scales (1B, 3B, 7B, 13B) with varying numbers of principles per instruction (e.g., 3, 5, 10, 20), then measure both reward model accuracy and downstream editing improvement as a function of (scale × granularity). The hypothesis is that smaller models might benefit from coarser principles (fewer, simpler checks that are easier to verify reliably), while larger models can productively use finer-grained decomposition (more principles catching subtle preservation failures or nuanced instruction requirements). The paper's per-category downstream results provide suggestive evidence: the 7B RL-RRM improves FLUX.Kontext's Motion Change performance (4.22 vs. 3.54 base) but Material Modification regresses slightly (6.82 vs. 7.10 for the 7B SFT-RRM baseline). A finer-grained analysis might reveal that the 7B model's principle decomposition for material modifications is too complex for reliable verification at current model scale, causing noisy reward signals that hurt optimization. If verified, this would suggest that optimal principle granularity is scale-dependent, with immediate practical implications for RRM design at different computational budgets.

Measuring and mitigating reward hacking in verifier-guided editing model optimization. The paper does not report any reward hacking diagnostics — metrics that would detect whether the editing model is learning to exploit the RRM's evaluation weaknesses rather than genuinely improving edit quality. A critical follow-up would track, throughout GRPO training: (a) the correlation between the RRM's scores and GPT-4.1 scores (or human judgments) on a held-out evaluation set — if this correlation decreases over training while RRM scores increase, it indicates the editing model is finding adversarial examples that score well under the RRM but poorly under other evaluators; (b) the RRM's per-principle score distribution — if certain principles consistently receive high scores while others are ignored (e.g., the model learns to satisfy "instruction following" principles while sacrificing "feature preservation" principles because the RRM weights them differently), this indicates the editing model is exploiting the RRM's aggregation function; (c) the KL divergence from the reference policy — if this grows large, the editing model may be drifting into regions of output space where the RRM's evaluation is poorly calibrated. The paper's Figure 4 (evaluation reward) shows evaluation rewards increasing throughout training, which is encouraging but not sufficient — it does not distinguish between genuine quality improvement and reward hacking that happens to correlate with the evaluation metric. A negative result (evidence of reward hacking) would be valuable because it would motivate future work on verifier robustness (e.g., adversarial training where the RRM is periodically updated on editing model outputs, or ensemble verification where multiple RRMs with different principle decompositions provide complementary signals).

Adapting the verifier paradigm to tasks where clean principle decomposition is harder — open-ended generation, multi-modal reasoning, or subjective aesthetic evaluation. The paper's principle decomposition pipeline (Section 3.1.1, Step 1) relies on the fact that image editing instructions can be naturally factorized into "what should change," "what should stay the same," and "does it look good?" But this factorization may not transfer to other visual generation tasks. For text-to-image generation, principles might need to capture compositional relationships ("is the red ball to the left of the blue cube?"), attribute binding ("does the cat have stripes and the dog have spots?"), or stylistic adherence ("is the image in the style of Van Gogh?"). For video editing, principles might need to capture temporal consistency ("does the object's motion trajectory match the instruction?") or frame-to-frame coherence. A systematic study would apply the verifier paradigm (principle decomposition + CoT verification + GCPO alignment) to these adjacent tasks, measuring whether the cold-start data construction pipeline transfers without modification or requires task-specific adaptations to the principle generation prompts and verification criteria. The paper's observation that GCPO improves the RRM's ability to catch subtle failures (like the hat-color hallucination in Figure 10) suggests that verifier-based reward models might be particularly valuable for compositional tasks where holistic scorers are known to fail systematically (the "binding problem" in T2I generation where models incorrectly associate attributes with objects). A positive result on compositional T2I benchmarks (e.g., T2I-CompBench, GenEval) would significantly expand the claimed scope of the paradigm shift.

Practical Applications and Downstream Use Cases

Cost-efficient post-training of open-source image editing models for deployment-specific editing requirements. An organization deploying an open-source editing model (e.g., FLUX.Kontext) for a specific use case — say, an e-commerce platform needing to change product colors, backgrounds, and materials while preserving product identity — can use Edit-R1's pipeline to fine-tune the model on their domain-specific editing distribution. The cold-start SFT phase can be adapted by decomposing the organization's common editing instructions into principles using the same Seed-1.5-VL API, generating edited candidates with the base model, and constructing domain-specific quadruple data. Critically, only 10K human preference pairs are needed for the GCPO phase — a small fraction of the total training data — meaning the organization can incorporate their own annotators' preferences about what constitutes a good product edit (e.g., "the logo must remain sharp," "the fabric texture should not change") without commissioning a large-scale annotation effort. The paper's downstream results provide a rough estimate of expected improvement: an 8.1% Overall Score improvement over the base FLUX.Kontext (5.77 to 6.24) with the 7B RL-RRM, concentrated in challenging categories like Motion Change and Material Modification. For an e-commerce use case where even small improvements in edit reliability translate to reduced manual retouching costs, this represents a concrete ROI argument. The key practical considerations would be: (a) verifying that the RRM's accuracy transfers to the domain-specific editing distribution (the paper does not test this, so an initial small-scale validation would be necessary), and (b) amortizing the cost of the SFT data construction pipeline across multiple fine-tuning iterations as the base editing model improves.

Automated quality filtering and benchmarking for large-scale editing model evaluation. The 82.2% internal benchmark accuracy (Table 2) and 78.2% EditRewardBench accuracy (Table 4) suggest that the RL-RRM can serve as a reliable automated evaluator for editing model outputs — reliably ranking edited images in agreement with human preferences. For platform operators or model developers who regularly evaluate multiple editing model versions across thousands of editing instructions, the RRM provides a principled alternative to: (a) expensive human evaluation (which does not scale to the volume of comparisons needed during rapid model iteration), or (b) GPT-4.1-based automated metrics (which may have unknown biases and calibration issues). The RRM's principle-level verification traces make the evaluation auditable: if a new model version underperforms on a specific editing category, the developer can inspect per-principle judgments to diagnose whether the failure is in instruction following, preservation, or quality. The paper's comparison to GPT-4.1 (Table 3) shows that the RRM-guided optimization produces results that also improve GPT-4.1 scores, suggesting alignment between the RRM's criteria and the GPT-4.1 evaluation — but the paper does not directly measure this alignment. A practitioner would need to calibrate the RRM against their specific evaluation criteria by measuring correlation with their own annotators' preferences on a small validation set before relying on the RRM for large-scale automated evaluation.

Self-improvement pipelines where editing models generate their own training data, filtered and scored by the RRM. The paper's two-stage training pipeline (SFT for the RRM, GRPO for the editing model) can be closed into an iterative loop: (1) use the current editing model to generate edited candidates for a large set of source images and instructions, (2) score them with the frozen RRM, (3) select high-scoring edits as pseudo-ground-truth for a supervised fine-tuning round, (4) re-optimize the editing model with GRPO using the RRM as reward, (5) repeat. This is analogous to the ReST$^{EM}$ self-improvement loop in language models, but with the verifier paradigm addressing the critical bottleneck that made prior image editing self-improvement attempts fragile (as the paper notes, the ReST$^{EM}$ experiment in Appendix K caused the revision model to degrade). The RRM's principle-based verification makes the filtering step more reliable than using a holistic scorer: rather than selecting edits based on a single potentially-hallucinated score, the pipeline can filter based on per-principle performance (e.g., only accept edits that satisfy all "instruction following" and "quality" principles, even if some "preservation" principles fail, then use the accepted edits as training data for the specific editing categories where the model is already competent). The paper's results suggest this could be particularly effective for improving model performance on categories where it already has non-trivial competence but makes frequent errors (e.g., FLUX.Kontext's Motion Change improving from 3.54 to 4.22, Table 3). The key risk, which the paper does not address, is distribution collapse: if the RRM has systematic blind spots (e.g., it consistently fails to detect certain types of preservation failures), the self-improvement loop will amplify those blind spots, producing edits that score well under the RRM but are increasingly flawed by human standards. Regular human spot-checks on the filtered training data would be essential.

When to Prefer This Method

The paper positions the verifier paradigm (principle decomposition + CoT verification + GCPO alignment) against the scorer paradigm (holistic VLM scoring, as represented by Seed-VLM APIs and EditScore), with the downstream choice between GRPO-based optimization (compatible with the non-differentiable RRM) and REFL-based optimization (requiring differentiable scalar reward models). The following decision rules emerge from the paper's results and design constraints:

• Prefer the verifier paradigm and GCPO training when the editing domain involves complex, multi-faceted instructions where preservation of unedited elements is as important as instruction following (the standard image editing setting), and you have access to strong VLMs (for training data construction) and a small budget of human preference annotations (10K pairs used in the paper). The paper's evidence supports this with a 7.4 percentage point gain over EditScore-7B on EditRewardBench (Table 4) and an 8.1% downstream improvement on FLUX.Kontext (Table 3). The verifier approach is also preferable when auditability matters — the CoT reasoning trace allows diagnosing why a particular score was assigned, which is valuable for debugging editing model failures or explaining automated evaluation decisions to stakeholders.

• Prefer the scorer paradigm and REFL-based optimization when the editing domain has simple, single-step instructions where principle decomposition provides limited benefit (e.g., "make the image brighter," "apply a sepia filter"), the reward model must be differentiable for end-to-end training with the editing model, or the computational budget for reward model inference during training is severely constrained (since RRM inference generates full CoT reasoning traces, potentially hundreds of tokens per evaluation, versus a scalar regressor's single forward pass). The paper does not test the verifier paradigm on simple edits, so this boundary is speculative, but the logic is that the overhead of principle decomposition and CoT reasoning is less justified when the evaluation task is straightforward.

• Prefer GRPO over REFL for the downstream optimization when the reward model is non-differentiable (as the RRM is) or when online exploration is important (the editing model learns from its own generations rather than a static dataset). The paper's argument for GRPO over REFL is structural (GRPO is compatible with non-differentiable rewards; REFL is not) and empirical (DPO restricts exploration, Section 2.2), though the paper does not provide a direct DPO vs. GRPO comparison for editing model optimization.

• Prefer scaling the RRM over scaling the GCPO annotation budget when additional computational resources are available. The paper shows that moving from 3B to 7B provides larger downstream gains (FLUX.Kontext O: 5.63 → 6.20 for SFT-RRM) than adding GCPO at the smaller scale (3B SFT-RRM O=5.63 → 3B RL-RRM O=6.10, a 0.47 gain vs. the 0.57 gain from scaling). This suggests that, given a fixed total budget, investing in a larger base model for the RRM may yield higher returns than investing in more human preference annotations for GCPO — though the paper does not test this trade-off explicitly (e.g., by comparing a 7B SFT-RRM against a 3B RL-RRM trained with a proportionally larger preference dataset).