SPARK: Synergistic Policy And Reward Co-Evolving Framework

🎯 Pitch

A single model can bootstrap its own reward function from discarded RL training rollouts, eliminating the need for separate reward models or human preference data. On Qwen2.5-VL-7B, this approach yields a 9.7% average gain on reasoning benchmarks and even boosts its own reward-modeling accuracy by 12.1%, all without external supervision.

1. Executive Summary

This paper introduces the Synergistic Policy And Reward Co-Evolving FrameworK (SPARK), an on-policy RL method that recycles rollouts and correctness signals from RL with Verifiable Rewards (RLVR) to simultaneously train a single model as both a policy and a generative reward model — eliminating the need for separate reward models and costly human preference data. SPARK reuses RLVR-discarded rollouts to construct auxiliary training objectives (pointwise correctness judgment, pairwise comparison, and self-reflection conditioned on incorrect responses) that teach the model to evaluate and improve its own outputs, creating a positive feedback loop where improved reward accuracy yields better policy gradients, which in turn produce higher-quality rollouts that further refine the reward capability. Applied to Qwen2.5-VL-7B and evaluated across 17 benchmarks spanning reasoning, reward modeling, and general multimodal understanding, SPARK-VL-7B achieves a 9.7% average gain on 7 reasoning benchmarks, 12.1% on 2 reward benchmarks, and 1.5% on 8 general benchmarks over the baseline, establishing that policy and reward can co-evolve synergistically within a single model without external supervision — though the approach's test-time self-reflection scaling degrades performance on untrained base models, confirming that the integrated judgment capability must be learned through the co-evolving process rather than being an emergent property of scale alone.

2. Context and Motivation

The Core Problem: Current RL Pipelines for LLMs Waste Supervision and Operate in Isolation

The paper addresses a fundamental inefficiency in how post-pretraining reinforcement learning is conducted for Large Language Models (LLMs) and Large Vision-Language Models (LVLMs). The current landscape, as the authors describe in Section 1, is bifurcated into two standard paradigms:

1. RL with Verifiable Rewards (RLVR): For objective, automatically-checkable tasks like math problem solving or code generation, a verifier (e.g., exact answer matching, unit test passing) assigns a binary reward to model-generated outputs. Algorithms like GRPO (Shao et al., 2024) compute advantages from these rewards and update the policy. This approach is powerful — it powered the reasoning breakthroughs in DeepSeek-R1 (Guo et al., 2025) — but it discards a tremendous amount of information after each update. Specifically, for each training question, the model generates $n$ candidate rollouts (complete reasoning traces plus final answers), the verifier scores them as correct or incorrect, the advantages are computed, the policy is updated, and then all of those rollouts and correctness labels are thrown away. The paper's key observation in Section 1 is:

"these valuable rollouts are typically discarded after this single use"

This is wasteful in a very concrete sense. Each rollout contains the model's reasoning trace — its step-by-step attempt at solving the problem — plus a ground-truth binary label from the verifier. This is exactly the kind of data one would use to train a reward model that learns to predict output correctness. Yet in standard RLVR, it evaporates after the gradient step.

2. RL from Human Feedback (RLHF): For subjective tasks where automatic verification is impossible (helpfulness, safety, stylistic quality), a separate reward model is trained on human preference judgments or teacher-model annotations. This reward model then scores policy outputs during RL optimization. The problems here are manifold and well-documented in the alignment literature:

Cost: Collecting high-quality human preference data is expensive and slow. Using teacher models (LLM-as-a-judge) reduces cost but introduces teacher bias and limits the ceiling on reward quality to the teacher's capabilities.
Reward–policy mismatch: The reward model is trained on data from one model distribution (often the base model or an earlier policy checkpoint), but the policy evolves during training. The reward model's assessments become increasingly miscalibrated as the policy moves into regions of output space where the reward model has never seen labeled examples. This induces the well-known pathology of reward hacking (Skalse et al., 2022; Gao et al., 2023), where the policy learns to produce outputs that score highly under the reward model but are not actually better — they just exploit blind spots in the reward's training distribution.
Brittle generalization: The external reward model, being a separate network, may fail to generalize to novel query types, leading to unreliable reward signals.
Infrastructure overhead: Maintaining a separate reward model doubles GPU memory requirements, introduces inter-process communication latency between the policy and reward models during training, and adds serving costs for the reward model at inference time if it is used for test-time verification.

The Gap: No Framework Unifies Policy and Reward in a Mutually Reinforcing Cycle

Beyond these individual limitations, the paper identifies a deeper structural gap. Policy models and reward/judge models are developed in isolation, with fundamentally different training regimes and data sources. The policy learns to generate; the reward model learns to evaluate. Their interaction is unidirectional — the reward model provides feedback to the policy, but the policy's improving capabilities never feed back to improve the reward model. As the policy gets stronger and produces more sophisticated outputs, the static reward model (trained on earlier, weaker policy outputs) becomes an increasingly poor judge.

The authors frame this as:

"a key limitation of existing paradigms is that policy and reward models are usually developed in isolation, which restricts their interaction and reduces the potential for mutual improvement" (Section 2)

This isolation is not a necessary constraint — it is an artifact of how these systems have historically been engineered. Conceptually, the ability to solve a problem (reasoning) and the ability to judge whether a solution is correct (evaluation) are complementary cognitive skills that could, in principle, reinforce each other. A stronger reasoner generates better training examples for evaluation training; a better evaluator provides more accurate feedback for reasoning improvement. But existing pipelines provide no mechanism for this virtuous cycle to operate.

Why This Matters: Practical and Theoretical Significance

The practical implications are substantial:

Cost and efficiency. The dual pipeline (policy model + external reward model) doubles the computational and human-annotation costs of post-training. For organizations deploying RL-based alignment, the reward model training stage — often requiring large-scale human preference collection or expensive teacher model API calls — can rival or exceed the cost of the policy training itself. Eliminating the need for a separate reward model, as SPARK proposes, represents a significant reduction in the total cost of building aligned LLMs.

Scalability. As models scale to hundreds of billions of parameters, maintaining a comparably-sized reward model becomes increasingly burdensome. SPARK's single-model approach means that improvements in backbone architecture or training infrastructure only need to be implemented once, not duplicated across two separate model pipelines.

Real-time deployment. At inference time, many systems use reward models or judge models for test-time compute scaling — verifying outputs, selecting among candidates, or triggering revisions. An external judge adds latency (inter-model communication) and GPU memory overhead. SPARK's internalized judge, which runs as a second forward pass through the same model with a different prompt, eliminates the need for a second model in GPU memory, enabling test-time scaling on resource-constrained deployments.

Theoretical significance. The paper challenges the prevailing assumption that policy and reward must be separate systems. By demonstrating that a single model can simultaneously serve as both a strong reasoner and an effective reward model — and, crucially, that training both capabilities jointly yields better results than training either in isolation — SPARK suggests that reasoning and evaluation are not competing objectives but complementary skills that share underlying representations. This has implications for how we think about model architecture and training objectives in the broader context of AI alignment.

Prior Approaches and Where They Fall Short

The paper situates itself relative to several lines of prior work, each of which addresses pieces of the problem but leaves the core inefficiency unresolved.

RLVR-based methods (GRPO and its variants). Since DeepSeek-R1 (Guo et al., 2025), GRPO with verifiable rewards has become the standard approach for reasoning-focused post-training (Section 2, Related Works). Extensions have applied this paradigm to perception (Zheng et al., 2025; Su et al., 2025b; Liu et al., 2025a), agent tasks (Jin et al., 2025; Liu et al., 2025b), and general multimodal reasoning. These methods all share the same structural property: the verifier signal is used exclusively for policy optimization, and the generated rollouts are discarded. SPARK's contribution here is not a new RL algorithm but a new way to use the data that RLVR already generates — the rollouts and correctness labels — to simultaneously train the model's evaluation capability.

RLHF-based methods and external reward models. Standard RLHF (Ouyang et al., 2022; Bai et al., 2022) trains an independent reward model on human preferences, then uses it for PPO-based optimization. More recent variants (Cai et al., 2024a; Zhu et al., 2023; Zang et al., 2025) refine reward model architectures and training procedures. Direct Preference Optimization (DPO; Rafailov et al., 2023) eliminates the explicit reward model by reparameterizing the policy itself as an implicit reward, but this approach does not generate a model that can explicitly judge outputs — it optimizes the policy directly from preference data without producing a reusable evaluation capability. All RLHF variants still require preference data (human or synthetic) and maintain a policy–reward separation either architecturally or conceptually.

The paper's critique of RLHF, as summarized in Table 7, is stark: RM-based RL requires extra preference data, extra reward model training, and roughly 2× GPU cost for reward inference during training. SPARK requires none of these, using only the verifiable reward signal that RLVR already computes.

Self-reward and self-reflection methods. Prior work has explored teaching models to evaluate and improve their own outputs, but these approaches (discussed in Section 2 and Appendix A.3) maintain dependence on external supervision:

STaR (Zelikman et al., 2022) iteratively generates chain-of-thought rationales and filters them with ground-truth answers, but the self-improvement loop only improves generation — it does not produce an internalized evaluation capability.
S2R (Ma et al., 2025) uses pre-annotated self-verification and self-correction data for both SFT and RL training, meaning the supervision still originates from external sources.
COOPER (Hong et al., 2025) leverages an external assistant to generate preference data, which is then used to train a reward model — maintaining the separate reward model architecture and the dependence on external annotation.

The common thread in these prior self-reward/self-reflection works is that they either (a) rely on externally provided evaluation data or (b) use that data to train a separate reward model. SPARK claims to be the first to unify policy and reward capabilities within a single model by optimizing the GRPO framework, with all evaluation training data generated on-policy from the verifiable reward signal. The authors state:

"our method is the first to unify policy and reward capabilities within a single model by optimizing the GRPO framework. This co-evolving design breaks the conventional paradigm of separately trained reward models" (Appendix A.3)

How This Paper Positions Itself

The paper's positioning can be understood as addressing the inefficiency at the intersection of RLVR and RLHF. Rather than proposing a new RL algorithm or a new reward model architecture, SPARK asks: given that RLVR already generates valuable rollouts with correctness labels, why not use them to teach the model to evaluate itself?

The mechanism is what the authors call co-evolution — a positive feedback loop described in the abstract and elaborated in Section 3:

The policy generates $n$ rollouts for a given question under GRPO.
Verifiable rewards assign binary correctness to each rollout.
Standard GRPO step: Compute advantages from these rewards and update the policy (as in normal RLVR).
SPARK's innovation: Rather than discarding the rollouts, construct auxiliary training data from them — pointwise judgment ("Is this answer correct?"), pairwise comparison ("Which of these two answers is better?"), and self-reflection ("This answer was wrong; correct it") — and train the same model on these objectives.
The auxiliary training improves the model's ability to judge correctness.
In the next iteration of GRPO, the improved judgment capability (even if not explicitly used for advantage computation — the advantage still comes from the verifiable reward) influences the policy's output distribution through the shared parameters, producing better rollouts.
Better rollouts provide higher-quality training data for the judgment objectives, further improving evaluation accuracy.
The cycle continues.

This framing positions SPARK not as a competitor to RLVR or RLHF but as an augmentation of RLVR that extracts additional value from data that is otherwise wasted. The paper explicitly builds on GRPO, not as a replacement, but as a foundation. The co-evolving training is an auxiliary objective added to the standard GRPO loss (Equation 4 in Section 3.1), not a replacement for it.

The test-time implications further differentiate SPARK. Because the model learns both to reason and to judge, it can perform self-reflection at inference time — generating an answer, judging whether it is correct, and if not, revising it (Section 3.3). This test-time scaling capability is enabled by the co-evolving training, and the paper's ablation (Table 5) demonstrates that applying the same self-reflection procedure to an untrained base model degrades performance, confirming that the judgment ability must be learned through the SPARK process rather than being an emergent property of pre-trained models.

In summary, SPARK positions itself as addressing a specific, high-leverage inefficiency: the discarding of supervised data in RLVR pipelines. By recycling this data into auxiliary training objectives for evaluation and self-correction, it simultaneously eliminates the need for a separate reward model (solving the cost and mismatch problems of RLHF) while creating a mutually reinforcing cycle between generation and evaluation capabilities.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper presents a training framework — a specific recipe for how to run reinforcement learning on language models so that a single model learns to both solve problems and evaluate its own solutions. The core system is not a new model architecture or a new RL algorithm, but rather a careful reorganization of the data and training objectives within an existing RL paradigm (GRPO with verifiable rewards) to extract additional learning signal from rollouts that are normally discarded.

The problem it solves is the waste of supervised data in RL with Verifiable Rewards (RLVR) and the cost and fragility of maintaining separate reward models for RL from Human Feedback (RLHF). The shape of the solution is a co-evolving feedback loop: during standard GRPO training on problems with verifiable correct answers, the model generates candidate solutions which are scored by a rule-based verifier. Normally, these rollouts are used once to compute policy gradients and then thrown away. SPARK instead recycles them — using the same verifier-assigned correctness labels to construct auxiliary training data that teaches the model to judge correctness (pointwise and pairwise) and to self-correct (reflection). Because these auxiliary objectives are trained on the same model that is being optimized as a policy, the model's improved judging ability feeds back into producing better generations in subsequent RL iterations, which in turn provide higher-quality training data for the judging objectives. The result is a single model that serves as both an effective policy and an effective reward model, eliminating the need for separate reward model training and human preference data.

3.2 Big-Picture Architecture (Diagram in Words)

The SPARK system has five major structural components, all operating within a single model:

Unified Model (π_θ) — the single model being trained, which serves simultaneously as the policy (generating answers), the judge (evaluating correctness), and the reflector (correcting errors). It is initialized from a pre-trained LLM or LVLM checkpoint (e.g., Qwen2.5-VL-7B).
Verifiable Reward Function (R) — an external, rule-based oracle that compares model outputs against ground-truth answers and returns a binary 0/1 score. This is used only for problems with objectively verifiable answers (math, code). It is not a neural network — it is an exact matching function or programmatic verifier.
GRPO Optimization Engine — the standard GRPO algorithm that generates groups of $n$ candidate answers (rollouts) for each training question, computes standardized advantages from the verifiable rewards, and updates the policy via a clipped surrogate objective with KL regularization against a frozen reference model.
On-Policy Data Generator — a module that takes the rollouts and their verifiable reward scores from the GRPO step and reformats them into three types of auxiliary training data: pointwise judgment samples, pairwise comparison samples, and reflection samples. This module produces the training data for the auxiliary objectives that teach judging and self-correction.
Test-Time Self-Reflection Pipeline — at inference time, the model generates an answer, invokes its internal judge (via a judgment prompt) to assess correctness, and either accepts the answer (if judged correct) or performs iterative refinement (if judged incorrect) until a self-judged-correct answer is produced or a maximum number of iterations is reached.

Information flow during training: A training question enters → the unified model generates $n$ candidate rollouts (reasoning traces + final answers) → the verifiable reward function assigns binary scores → the GRPO engine computes advantages and updates the policy → the on-policy data generator reformats the rollouts and correctness labels into judgment/comparison/reflection training samples → the unified model is further trained on these auxiliary objectives → the next GRPO iteration uses the now-improved model to generate new rollouts → the cycle repeats.

Information flow during inference: A test question enters → the unified model generates an initial answer → the same model judges its own answer via a judgment prompt → if judged correct, the answer is returned; if judged incorrect, the model reflects on the errors and generates a revised answer → the cycle repeats until self-judged correct or a maximum of 2 iterations → the final answer is compared against ground truth for evaluation.

3.3 Roadmap for the Deep Dive

The GRPO training loop with verifiable reward (Section 3.1): First, I will walk through the standard GRPO optimization procedure that SPARK builds upon — how answer groups are generated, how verifiable rewards are computed, how advantages are derived, and the overall training objective with KL regularization. This is the foundation on which all of SPARK's innovations are layered, and understanding the "discarding" point in this pipeline makes SPARK's recycling mechanism intuitive.
On-policy reward and reflection data generation (Section 3.2): Next, I will explain how SPARK repurposes the normally-discarded GRPO rollouts into three types of auxiliary training data — pointwise judgment, pairwise comparison, and self-reflection — each serving a distinct purpose in teaching the model to evaluate and improve its own outputs. I will detail the construction of each data type from the rollouts and verifiable rewards, and how these samples are combined into a single auxiliary training set.
Integration of auxiliary objectives with the base GRPO loss: I will explain how the auxiliary judgment/reflection data is used during training — specifically, that the same unified model is optimized on both the standard GRPO objective and the auxiliary objectives, creating the co-evolving feedback loop. I will clarify the training data composition (the mix of standard GRPO data and auxiliary data) and the practical implementation.
Test-time scaling with self-reflection (Section 3.3): Finally, I will explain how the trained model's integrated policy-and-judge capabilities are leveraged at inference time for iterative self-reflection — the prompt-based mechanism for invoking the internal judge, the decision rule for accepting vs. revising answers, and why this capability only works after co-evolving training and not in base models.

This order follows the natural lifecycle of the SPARK system: how rollouts are generated → how they are recycled into training data → how the auxiliary training integrates with policy optimization → how the resulting capabilities are deployed.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a training framework paper whose core idea is that the binary correctness signals generated by verifiable rewards during GRPO optimization can be recycled as supervision for teaching the same model to judge and self-correct its own outputs, creating a mutually reinforcing co-evolution between generation and evaluation capabilities — all without external reward models or human preference data.

The GRPO Training Loop with Verifiable Reward (What SPARK Builds Upon)

SPARK is built on top of the GRPO (Group Relative Policy Optimization) algorithm, which the paper adopts from Shao et al. (2024). To understand SPARK's innovation, we must first understand the standard GRPO training loop that it augments.

Step 1: Sampling an answer group. For each training sample, the model generates not a single answer but a group of $n$ candidate answers. The paper denotes a training sample as $d = (q, a, I)$ for vision-language tasks (where $q$ is the question, $a$ is the ground-truth answer, and $I$ is the input image) or $d = (q, a)$ for text-only tasks. The model produces:

$G = \left\{ \left( o^i_{\text{cot}}, o^i_{\text{ans}} \right) \right\}_{i=1}^{n}$

where $G$ is the answer group of size $n$ , $o^i_{\text{cot}}$ is the $i$ -th reasoning trace (chain of thought), and $o^i_{\text{ans}}$ is the corresponding final answer extracted from the trace.

The paper uses a specific prompting convention to facilitate this decomposition: the model is instructed to enclose its final answer within \box{} using LaTeX notation, as illustrated in Appendix Figure 5. The prompt suffix appended to each mathematical evaluation question is:

"Please first conduct reasoning, and then answer the question. Repeat the final answer using a '\boxed{}'."

This allows the system to mechanically extract the final answer $o^i_{\text{ans}}$ by parsing the \boxed{} content, separating it from the reasoning trace $o^i_{\text{cot}}$ . The paper explicitly notes (Section 3.1) that they avoid rigid structural formats like think<answer> to prevent over-constraining the model's reasoning style, opting instead for this lighter LaTeX-based delimiter.

Step 2: Verifiable reward. Each final answer is evaluated by a rule-based verifier that compares it against the ground truth:

$R(q, o) = \begin{cases} 1, & \text{if } o = a, \\ 0, & \text{otherwise.} \end{cases}$

where $R$ is the verifiable reward function, $q$ is the input question, $o$ is the model's extracted final answer, and $a$ is the ground-truth answer. For the $i$ -th sample in group $G$ , the reward is denoted $r_i = R(q, o^i_{\text{ans}})$ .

What this computes: a binary 0/1 score per candidate answer based on exact match (or programmatic equivalence for mathematical expressions). It is an oracle — it always returns the true correctness because it has access to the ground-truth answer.

Why this form: this is the simplest possible reward for tasks with objectively verifiable answers. It requires no learned model, no human annotation, and no calibration. The binary nature means the reward signal is sparse but perfectly accurate. This is the standard formulation in RLVR and is not SPARK's contribution — SPARK's innovation is in how it reuses these $r_i$ values beyond the advantage computation.

Crucially, this is the point in standard RLVR where rollouts would be discarded. After computing $r_i$ for each candidate, standard GRPO immediately proceeds to advantage computation and policy update, and the rollouts $(o^i_{\text{cot}}, o^i_{\text{ans}})$ and their correctness labels $r_i$ are never used again. SPARK intervenes after this step to recycle them into auxiliary training data (described in the next section).

Step 3: Advantage computation. GRPO normalizes the rewards within each answer group to compute standardized advantages, following a group-relative normalization that avoids needing a learned value function (a key architectural difference from PPO):

$\bar{r} = \frac{1}{n} \sum_{j=1}^{n} r_j$

$s = \sqrt{\frac{1}{n} \sum_{j=1}^{n} \left( r_j - \bar{r} \right)^2 + \epsilon}$

$A_i = \frac{r_i - \bar{r}}{s}$

where $\bar{r}$ is the mean reward across the $n$ candidates in the group, $s$ is the standard deviation with a small constant $\epsilon > 0$ for numerical stability (preventing division by zero when all candidates have the same reward), and $A_i$ is the normalized advantage for the $i$ -th candidate.

What this computes: for each candidate $i$ , how much better (positive $A_i$ ) or worse (negative $A_i$ ) its reward is compared to the group average, measured in standard deviation units.

Why this form: GRPO's group-relative normalization is a deliberate design choice that eliminates the need for a separate value function (critic network) required by PPO. By normalizing within each group of $n$ candidate answers, the advantages are always zero-mean and roughly unit-variance, providing a stable gradient signal regardless of the absolute difficulty of the question. Questions where all candidates are correct (all $r_i = 1$ ) yield $A_i = 0$ for all candidates — no update occurs because the model already solves the problem perfectly under its current sampling distribution. Questions where all candidates are wrong (all $r_i = 0$ ) also yield $A_i = 0$ — no update occurs because there is no within-group variation to learn from. Only questions with mixed correctness across the $n$ candidates produce non-zero advantages and thus drive learning. This implicit difficulty filtering is a key property of GRPO that SPARK inherits.

Step 4: Overall training objective. The GRPO objective maximizes expected reward while regularizing the learned policy toward a frozen reference model:

$\mathbb{E}_{o \sim \pi_\theta(\cdot \mid q)} \left[ R(q, o) \right] - \lambda \, \text{KL} \left( \pi_\theta(\cdot \mid q) \, \| \, \pi_{\text{ref}}(\cdot \mid q) \right)$

where $\pi_\theta$ is the current policy (the model being trained), $\pi_{\text{ref}}$ is a frozen copy of the initial model (the reference policy), $\lambda$ is a hyperparameter controlling the strength of the KL penalty, and $\text{KL}(\cdot \| \cdot)$ is the Kullback-Leibler divergence between the two distributions.

What this computes: the expected verifiable reward under the current policy, minus a penalty proportional to how far the policy has diverged from the reference model. The gradient of this objective (approximated via the GRPO clipped surrogate loss) updates the model parameters $\theta$ .

Why this form: the KL penalty prevents the policy from collapsing to a degenerate distribution that maximizes reward but loses the general capabilities encoded in the pre-trained model — a phenomenon known as "reward hacking" or "policy collapse." The reference model acts as an anchor, and $\lambda$ controls the trade-off between optimization toward the verifiable reward and preservation of the original model's behavior. SPARK does not modify this objective; the auxiliary training objectives (described next) are added in addition to this GRPO objective, not as a replacement for it.

On-Policy Reward and Reflection Data Generation (SPARK's Core Innovation)

This is where SPARK diverges from standard GRPO. Rather than computing advantages and discarding the rollouts, SPARK recycles the answer group $G$ and the verifiable rewards $\{r_i\}_{i=1}^n$ to construct three distinct types of auxiliary training data. These data are generated on-policy — they are constructed from the current model's own outputs at each training iteration, ensuring the training distribution for the auxiliary objectives tracks the evolving model.

The paper categorizes the auxiliary data into three forms (Section 3.2), each designed to teach a different facet of evaluation and self-improvement:

Pointwise judgment data. For each candidate answer $o^i_{\text{ans}}$ paired with its verifiable reward $r_i = R(q, o^i_{\text{ans}})$ , the system constructs a binary judgment sample:

$D_{\text{pointwise}} = \left\{ \left( q, \, o^i_{\text{ans}}, \, R(q, o^i_{\text{ans}}) \right) \right\}$

where the model is asked a question of the form: "You are a specialist in answer evaluation. For the given question and answer, perform logical reasoning to assess the answer's correctness. Then state your judgment as 'Yes' or 'No'." (exact prompt illustrated in Appendix Figure 7).

What this produces: a set of supervised binary classification training examples where the input is (question, candidate answer) and the target label is the verifiable reward (correct/incorrect). For a group of size $n$ , this yields $n$ pointwise samples per training question.

What it teaches: this objective trains the model to perform absolute correctness evaluation — looking at a single answer and determining whether it is right or wrong. This is analogous to what a standard outcome reward model (ORM) does, but the "reward model" is the same model being trained as the policy, sharing all parameters.

Pairwise comparison data. From the same answer group $G$ , the system constructs preference-style samples that compare two candidate answers:

$D_{\text{pairwise}} = \left\{ \left( q, \, o^i_{\text{ans}}, \, o^j_{\text{ans}}, \, R(q, o^i_{\text{ans}}), \, R(q, o^j_{\text{ans}}) \right) \right\}$

where two candidate answers $o^i_{\text{ans}}$ and $o^j_{\text{ans}}$ are drawn from $G$ , and the model must select which one is better. The prompt template (Appendix Figure 7) frames it as: "You're given a question and two possible answers. (A) and (B) may differ in correctness. Please analyze both answers carefully and decide which one is correct."

What this produces: a set of comparative evaluation training examples where the input is (question, answer A, answer B) and the target is the verifiably-correct answer. The construction implicitly ensures that at least one of the two answers differs in correctness (otherwise the comparison is uninformative), since both are drawn from the same answer group which contains a mix of correct and incorrect candidates for the GRPO advantage to be non-zero.

What it teaches: this objective trains the model to perform relative quality assessment — distinguishing between a better and worse answer even when both might be partially reasonable. This is the skill that underlies reward model training in RLHF (where pairs of human-preferred and human-dispreferred responses are compared), but here the comparison ground truth comes from the verifiable reward rather than human preference. The paper notes (Section 3.2) that in both pointwise and pairwise settings, $o_{\text{ans}}$ can be replaced with reasoning traces $o_{\text{cot}}$ to shift supervision toward intermediate steps, an extension explored in the ablation study (Table 6) which finds that combining both answer-based and CoT-based data yields the best performance.

Reflection data. For incorrect candidate answers, the system constructs self-correction samples:

$D_{\text{reflect}} = \left\{ \left( q, \, o^i_{\text{ans}}, \, R(q, o^i_{\text{ans}}) \right) \right\}$

where the model is presented with the question, an incorrect answer, and instructed to both verify correctness and (if incorrect) provide the correct answer. The prompt (Appendix Figure 7) reads: "You are given a question and a proposed answer. Judge whether the answer is correct. If it is, briefly confirm it. If it is incorrect, provide correct reasoning and the correct answer." An alternative variant also accepts reasoning traces: "You are given a question and a reasoning process (CoT). Your task is to judge whether both the reasoning and the final answer are correct. If correct, summarize the logic and repeat the answer. If not, provide a correct reasoning process and answer."

What this produces: a set of examples that train the model to not just detect errors but to fix them — the core capability needed for test-time self-reflection.

What it teaches: this objective explicitly stimulates the model's self-reflection and self-correction ability. Unlike pointwise judgment (which only requires a yes/no assessment) or pairwise comparison (which only requires relative preference), reflection training requires the model to generate a complete, correct alternative when it identifies an error. This links the judgment capability back to the generation capability — the model must not only know that an answer is wrong, but also know how to produce the right one.

Combined auxiliary dataset. The three data types are combined:

$D_{\text{on-policy}} = D_{\text{pointwise}} \cup D_{\text{pairwise}} \cup D_{\text{reflect}}$

This combined dataset is used to further train the unified model on the auxiliary objectives, in addition to the standard GRPO policy gradient updates. The paper does not specify exact mixing ratios between the three data types or between the standard GRPO data and the auxiliary data — this is left as an implementation detail that may vary by experimental configuration.

Training data scale. The paper uses relatively small training sets compared to typical LLM post-training: 19K randomly sampled instances from ViRL-39K (Wang et al., 2025) for SPARK-VL-7B, 24K difficulty-filtered instances from the same dataset for SPARK-VL-32B, and the Simple-RL-Zero-25K dataset (Zeng et al., 2025) for the text-only SPARK-7B (Appendix A.1.3). All data are in the form of $(q, a, I)$ triples with no manually annotated reward data, reflection traces, or judgment-oriented CoT trajectories — the auxiliary data is generated entirely on-policy during training.

Why on-policy generation matters: a fundamental challenge in RLHF is that the reward model is trained on data from one policy distribution but used to score outputs from a different (evolving) policy distribution, causing the reward signal to become miscalibrated. SPARK avoids this by generating auxiliary training data from the current policy at each iteration. As the policy improves and produces different distributions of correct/incorrect answers, the auxiliary data automatically tracks this distribution. This means the model's internal judge is always calibrated to the model's current output distribution — a property the authors call "on-policy and stable" (Section 1, advantage 2).

Integration of Auxiliary Objectives with the GRPO Loss

The paper is somewhat implicit about the exact training procedure for the auxiliary objectives, but the logic can be reconstructed from Section 3.1 and the ablation design (Table 1, "Policy&Reward" vs. "SPARK"):

What the Policy&Reward baseline does: The GRPO-trained model receives auxiliary training on judgment data but without the reflection component and without the iterative co-evolution. This is the "Policy&Reward" row in Table 1, which achieves an average of 55.1% across all benchmarks — better than Policy-Only (53.3%) and Reward-Only (52.7%), but substantially below SPARK (57.1%).

What SPARK adds beyond Policy&Reward: Two key differences:

Reflection data: SPARK includes $D_{\text{reflect}}$ in the auxiliary training data, which Policy&Reward does not. This teaches the model to not just judge correctness but to correct errors, creating a tighter coupling between evaluation and generation capabilities.
Iterative co-evolution: The auxiliary training and GRPO policy optimization proceed in an iterative cycle. After the auxiliary training on $D_{\text{on-policy}}$ improves the model's judgment capability, the next GRPO iteration uses this improved model (with shared parameters) to generate new rollouts. These rollouts benefit from the improved evaluation capability embedded in the shared parameters, even though the GRPO advantage computation still uses the external verifiable reward (not the model's internal judge). The improved rollouts then provide higher-quality auxiliary training data, which further improves judgment, ad infinitum.

Practical training procedure (reconstructed):

For each training iteration, sample a batch of questions with ground-truth answers.
For each question, generate an answer group $G$ of size $n$ from the current model $\pi_\theta$ .
Compute verifiable rewards $\{r_i\}$ and GRPO advantages $\{A_i\}$ .
Update $\pi_\theta$ using the GRPO policy gradient (standard RLVR step).
From the same rollout batch, construct $D_{\text{on-policy}}$ using the procedure described above.
Further train $\pi_\theta$ on $D_{\text{on-policy}}$ using standard supervised learning (cross-entropy loss on the judgment/reflection targets).
Proceed to the next iteration with the updated model.

The paper does not provide explicit hyperparameters for the auxiliary training (learning rate, batch size, number of auxiliary steps per GRPO step, or mixing ratios between the three data types). This is a notable omission that limits reproducibility.

Why the co-evolution works (conceptual mechanism): The shared parameters of the unified model create an information channel between the policy and the judge. When the model learns to better judge correctness (from the auxiliary objectives), the representations that support this judgment — understanding of logical structure, detection of reasoning errors, calibration of uncertainty — are also available to the generation pathway. When the model next generates rollouts under GRPO, these improved representations produce higher-quality reasoning traces (even though the advantage signal still comes from the verifiable reward). The higher-quality rollouts then provide cleaner auxiliary training data (correct answers are genuinely well-reasoned, incorrect answers contain identifiable mistakes rather than random noise), which further improves the judgment representations, and the cycle continues.

This is fundamentally different from simply training a separate reward model on the same data and using it to score policy outputs. In that case, the reward model's improved evaluation capability does not feed back into the policy's generation capability except through the scalar reward signal. In SPARK, the feedback operates through the shared parameter space — a much richer channel that can transfer representational knowledge, not just scalar scores.

Test-Time Scaling with Self-Reflection

Once SPARK training is completed, the unified model possesses three integrated capabilities: reasoning (generating answers), judging (evaluating correctness), and reflection (correcting errors). Section 3.3 describes how these capabilities are deployed together at inference time through a test-time scaling (TTS) procedure that iteratively refines answers until the model's internal judge deems them correct.

The TTS procedure (formal description).

Step 1 — Initial generation: Given a question $q$ and (optionally) an image $I$ , the model generates a candidate answer:

$o_t = \pi_\theta(q, I)$

where $t$ denotes the current refinement step (starting at $t = 0$ for the initial generation), and $o_t = (c_t, a_t)$ consists of a reasoning chain $c_t$ and a final prediction $a_t$ extracted from the \box{} content.

Step 2 — Self-judgment: The model assesses its own output by invoking a judgment prompt that instructs it to evaluate the correctness of its answer:

$r_t = \pi_\theta \left( q, I, \text{judge}(c_t, a_t) \right)$

where $\text{judge}(c_t, a_t)$ is a prompt wrapper that asks the model to verify whether $(c_t, a_t)$ is correct, and $r_t \in \{0, 1\}$ is the model's binary self-assessment (0 = judged incorrect, 1 = judged correct). The judgment prompt, as illustrated in Figure 3 of the paper, asks the model to produce a structured judgment with reasoning, followed by a final verdict in \box{} (e.g., \boxed{Yes} or \boxed{No}). The exact judgment prompt used is:

"You are a specialist in answer evaluation. For the given question and answer, perform logical reasoning to assess the answer's correctness. Then state your judgment as 'Yes' or 'No'. Finally, repeat your judgment inside \box{}."

Step 3 — Decision and refinement: Based on the self-judgment, the model either accepts the answer or performs iterative refinement:

$o_{t+1} = \begin{cases} o_t, & \text{if } r_t = 1, \\ \pi_\theta \left( q, I, \text{reflect}(c_t, a_t) \right), & \text{if } r_t = 0. \end{cases}$

where $\text{reflect}(c_t, a_t)$ is a prompt wrapper that asks the model to critique its prior reasoning and generate a revised solution. The reflection prompt (Figure 3) reads:

"You are given a question and a proposed answer. Judge whether the answer is correct. If it is, briefly confirm it. If it is incorrect, provide correct reasoning and the correct answer."

The process terminates when the model produces an answer it judges as correct ( $r_t = 1$ ), or when a maximum number of iterations is reached. The paper uses a maximum of 2 refinement rounds in the ablation study (Table 5), based on the practical observation that additional rounds beyond this show diminishing returns and can lead to correct answers being incorrectly revised to incorrect ones (a reversion problem discussed in the main analysis paper).

Final evaluation: The actual correctness for benchmark reporting is computed by comparing the final accepted prediction $a_t$ with the ground-truth answer, not by trusting the model's self-judgment. The self-judgment is used only to decide whether to stop or refine; the metric is always ground-truth accuracy.

Why this works only after SPARK training (Ablation, Table 5): The paper provides a critical ablation demonstrating that the TTS procedure is not a general-purpose capability of language models — it only works when the model has been trained with SPARK's co-evolving objectives. When the same judge-reflection TTS is applied to the base Qwen2.5-VL-7B (without any GRPO or SPARK training), performance drops from 49.1% to 39.9% on the average across 7 math benchmarks. This degradation occurs because the base model's self-judgments are inaccurate and its reflection attempts are ineffective — the model frequently misclassifies correct answers as incorrect and "revises" them to wrong answers, or iterates without actually improving.

When TTS is applied to the Policy-Only GRPO baseline (trained with GRPO but no auxiliary judgment/reflection objectives), the improvement is marginal: from 55.2% to 56.6% (a +1.4% gain). This model has improved reasoning from GRPO but lacks trained self-evaluation capability.

When TTS is applied to SPARK-VL-7B (trained with full co-evolving objectives including reflection data), the model achieves 58.8% — a substantial gain over the non-TTS SPARK baseline, and dramatically better than the base model with TTS. This confirms that the self-judgment and self-reflection capabilities are learned through the SPARK co-evolving process and are not emergent from scale or from GRPO training alone.

Practical considerations: The TTS procedure doubles the inference cost when a refinement is triggered (the model performs a second forward pass for the reflection step). The paper does not quantify the average refinement rate across benchmarks, but the performance gain (+3.6% over SPARK without TTS, from 55.2% to 58.8% as extrapolated from Table 5) suggests that the additional cost is justified by meaningful accuracy improvements on the subset of questions where the model correctly identifies its own errors and successfully corrects them. A more detailed cost-benefit analysis of TTS (what fraction of questions trigger refinement, what is the success rate of refinements, what is the false-positive rate where correct answers get incorrectly revised) is not provided but would be valuable for practical deployment decisions.

Ablation: Answer-Based vs. CoT-Based Auxiliary Data

The paper's ablation in Table 6 examines whether the auxiliary training data should be constructed from final answers only, from chains of thought (CoT), or from both. This is an important design choice because it determines what information the model learns to evaluate.

Answer-only auxiliary data (SPARK + Ans, 55.3% average): The judgment, comparison, and reflection training samples present only the final extracted answers ( $o_{\text{ans}}$ ) without the intermediate reasoning traces. This trains the model to evaluate the correctness of final conclusions without seeing the reasoning that led to them — analogous to how an outcome reward model (ORM) operates.

CoT-only auxiliary data (SPARK + CoT, 56.2% average): The training samples present the full reasoning traces ( $o_{\text{cot}}$ ) including intermediate steps, requiring the model to evaluate the reasoning process as well as the conclusion. This is analogous to process reward model (PRM) training (Lightman et al., 2023), where the model learns to judge step-by-step reasoning quality.

Combined answer and CoT (SPARK + Ans&CoT, 57.1% average): Both sources are used, providing richer and complementary training signals. The paper reports this as the best-performing configuration, with the combined approach outperforming answer-only by +1.8% and CoT-only by +0.9% on average across all benchmarks.

Why the combination works best: Answer-based data teaches the model to evaluate final correctness, which is the ultimate criterion that matters for benchmark accuracy. But final answers can be correct for wrong reasons (false positives in evaluation) or incorrect despite partially valid reasoning (false negatives). CoT-based data teaches the model to evaluate the process of reasoning, which provides a richer supervisory signal — the model learns to identify specific reasoning errors (e.g., arithmetic mistakes, logical leaps, misreading of problem constraints) rather than just the binary outcome. The combination ensures the model can both detect flawed reasoning (via CoT training) and make accurate final correctness judgments (via answer training), with the two capabilities reinforcing each other.

Cost Analysis: SPARK vs. RM-Based RL

The paper's Table 7 provides a concise comparison of SPARK against traditional reward-model-based RL pipelines:

Requirement	RM-based RL	SPARK
Extra Data (Preference)	Required	Not required
Extra RM Training	Required (separate model)	Not required (internal judge)
GPU Cost	~2× (policy + reward model)	1× (single model)
Reward Signal	RM inference (neural network)	Rule-based signal (verifier)
Efficiency	Slower (two models, inter-process communication)	Faster (single model, no external calls)

What "~2× GPU cost" means concretely: In RM-based RL, the policy model and reward model are typically loaded on separate GPUs or GPU partitions. During training, for each generated rollout, the policy model's output is sent to the reward model for scoring, which requires inter-process communication. The reward model's forward pass consumes GPU memory and compute comparable to the policy model (assuming similar architecture). SPARK eliminates this entirely — the rule-based verifier runs on CPU (simple string matching or programmatic evaluation), and the auxiliary training data is generated from the verifier outputs rather than requiring a separate neural network forward pass.

Why the verifiable reward remains external: SPARK does not replace the verifiable reward with the model's internal judge during GRPO training. The GRPO advantage computation (Equation 3) still uses the external, ground-truth-based verifiable reward $R(q, o)$ . This is a deliberate design choice: using the model's own (potentially imperfect) judge for advantage computation would create a circular training loop where errors in the judge propagate into policy updates — the classic reward hacking problem. The external verifier provides a ground-truth anchor that ensures the policy is always optimized toward actual correctness, while the internal judge developed through auxiliary training is used only for test-time self-reflection (where its imperfections only affect when to stop refining, not the fundamental policy optimization). This separation of concerns — ground-truth reward for optimization, learned judge for self-reflection — is a subtle but important design property that the paper does not explicitly articulate but that is evident in the system architecture.

Training Data Preparation

The paper (Appendix A.1.3) emphasizes that SPARK requires minimal curated training data compared to RLHF pipelines:

Input data format: Only VQA triples are needed — question $q$ , ground-truth answer $a$ , and optionally image $I$ . No manually annotated reward data, reflection traces, preference pairs, or judgment rationales are required.

Data sources used:

SPARK-VL-7B: 19K instances randomly sampled from ViRL-39K (Wang et al., 2025), a dataset of vision-language reasoning problems with verifiable answers.
SPARK-VL-32B: 24K difficulty-filtered instances from the same ViRL-39K dataset. The difficulty filtering (selecting harder problems) is motivated by the observation that larger models benefit more from training on challenging instances where the baseline success rate is moderate — easy problems provide little learning signal, and impossible problems provide none.
SPARK-7B (text-only): Simple-RL-Zero-25K dataset (Zeng et al., 2025), a collection of text-based mathematical reasoning problems with verifiable answers.

Why this data efficiency is significant: Traditional RLHF pipelines require two distinct data collection efforts: (1) SFT data for initial instruction tuning and (2) preference data (human annotations or teacher model outputs) for reward model training. SPARK collapses both requirements into a single modestly-sized dataset of problems with ground-truth answers — the type of data that is already widely available for math and code domains, and which can be synthetically generated at scale for many reasoning tasks. The on-policy data generation mechanism then amplifies this seed data by generating diverse auxiliary training samples from the model's own outputs. The paper's claim of being "data-efficient" (Section 1, advantage 1) rests on this elimination of preference annotation requirements.

Summary of Key Design Choices and Their Justifications

1. Shared parameters between policy and judge, rather than separate models: Enables the co-evolving feedback loop where improved evaluation representations benefit generation and vice versa. Avoids the GPU memory overhead, training cost, and reward-policy mismatch of separate reward models. The paper's results (Table 1: Policy&Reward > Policy-Only > Reward-Only) empirically demonstrate that joint training of both capabilities outperforms either in isolation.

2. External verifiable reward for GRPO optimization, internal judge only for auxiliary training and TTS: Prevents circular self-reinforcement of errors. The ground-truth verifier anchors the policy optimization to actual correctness, while the learned judge is used only for self-reflection — a lower-stakes application where errors cause inefficiency (unnecessary refinements or premature stopping) rather than systematic policy misdirection.

3. On-policy data generation for auxiliary training: Ensures the judgment training data tracks the evolving policy distribution, preventing the calibration drift that plagues offline reward models. Each iteration's auxiliary data reflects the current model's error patterns and reasoning style.

4. Three complementary auxiliary objectives (pointwise, pairwise, reflection): Each teaches a distinct evaluation skill — absolute correctness assessment (pointwise), relative quality discrimination (pairwise), and error correction (reflection). The combination produces a judge that can not only detect errors but also fix them, enabling the test-time self-reflection pipeline.

5. Prompt-based invocation of internal judge and reflector, rather than separate architectural components: The model's judging and reflecting capabilities are accessed through specialized prompts (Appendix Figures 5, 7) rather than separate output heads or decoder modules. This makes the system architecturally simple — the same model with the same parameters handles all three modes (generation, judgment, reflection) through prompt conditioning. However, this also means the quality of the judge/reflector depends entirely on how well the model has internalized these capabilities during training. The ablation in Table 5 (base model + TTS degrades performance) confirms that prompt conditioning alone does not elicit these capabilities from untrained models.

6. LaTeX \box{} delimiter for answer extraction rather than structural XML tags: A lightweight formatting convention that separates final answers from reasoning traces without constraining the model's natural reasoning style. The paper explicitly notes avoiding rigid formats like think<answer> to prevent over-constraining.

7. Modest training data scale (19K–25K instances): Demonstrates that SPARK's on-policy data amplification can achieve substantial improvements with relatively little seed data, making the approach accessible to researchers and organizations without massive annotation budgets. The comparison to RLHF pipelines that require large-scale human preference data underscores this efficiency advantage.

4. Key Insights and Innovations

Innovation 1: The Wasted Rollout as a Free Source of Reward Modeling Supervision

The paper's most fundamental conceptual move is not a new algorithm, but a reframing of what counts as "waste" in the RLVR pipeline and what that waste is worth. In the standard GRPO workflow — which the field has adopted rapidly since DeepSeek-R1 (Guo et al., 2025) — each training question prompts the model to generate a group of n candidate answers. The verifier scores them, the advantages are computed, the policy is updated, and then the entire batch of rollouts (reasoning traces plus final answers plus their verifier-assigned binary correctness labels) is discarded. This is treated as an unavoidable cost of the policy gradient computation: the rollouts are the computation you must do to get the advantage signal, and once you have it, they have no further value.

SPARK challenges this directly. The paper's opening statement in Section 1 — "these valuable rollouts are typically discarded after this single use" — is not merely an observation about inefficiency. It is a diagnostic move that recategorizes data that the field had implicitly classified as "intermediate computation" (analogous to activations in a forward pass) as "labeled training data" (analogous to the preference pairs used to train reward models in RLHF). The claim is that the RLVR rollout stream is not just a cost center for policy optimization; it is a free, on-policy, continuously refreshing stream of supervised data for reward modeling, generated as a side effect of operations you are already running.

The significance of this reframing is best appreciated by comparing the data requirements of traditional RLHF against what SPARK needs. RLHF requires an explicit, costly data collection step: human annotators or teacher models must compare pairs of model outputs and indicate preferences. This data is then used to train a separate reward model — an entirely distinct training pipeline with its own hyperparameters, validation procedures, and infrastructure requirements. The field had accepted this as necessary for any task where automated verification is unavailable: if you cannot write a rule-based verifier, you must pay humans (or large teacher models) to provide the reward signal.

SPARK shows that this dichotomy is a false one — or at least, that its boundary is more porous than previously recognized. The verifiable reward signal, which the field viewed exclusively as a policy optimization tool, can also serve as a teacher for reward modeling. The binary correctness labels generated by the verifier are exactly the type of supervised signal one would collect to train an outcome reward model. By restructuring the training loop to capture this signal and feed it back into the model through carefully designed auxiliary objectives (pointwise judgment, pairwise comparison, self-reflection), SPARK essentially runs a reward model training loop "for free" on the side of the policy optimization.

This is a fundamental conceptual contribution, not an incremental refinement of GRPO. The paper is not proposing a better advantage estimator or a new KL penalty schedule — it is proposing that the data architecture of RLVR should be restructured to view the verifier not just as a reward function for the policy but as a labeling function for an auxiliary supervision task. The downstream implications are significant: if this framing holds, then any RLVR training run can simultaneously produce a reward model at essentially zero additional data cost, eliminating the primary bottleneck (preference data collection) that has made RLHF expensive and inaccessible for many practitioners.

The evidence that this reframing is empirically consequential comes from the consistent stratification in Table 1's GRPO ablation rows. The "Policy-Only" GRPO variant (standard RLVR, no auxiliary data) achieves 53.3% average accuracy across all benchmarks. Adding the auxiliary training on recycled rollouts ("Policy&Reward") raises this to 55.1%. Adding the full co-evolving cycle with reflection data (SPARK) raises it further to 57.1%. Each additional extraction of value from the "wasted" rollout stream yields measurable, monotonic improvements — demonstrating not just that the data has value, but that the value compounds when the extraction is done in a way that creates a feedback loop.

Innovation 2: Co-Evolution as a First-Class Training Dynamic — Not Just Joint Training

The term "co-evolution" appears throughout the paper (title, abstract, Section 3 heading, multiple claims), and it is not merely decorative. The paper is making a specific architectural claim: that training a model to simultaneously be a policy and a reward model produces a qualitative dynamic — a positive feedback loop — that is categorically different from simply training both capabilities in parallel on independent objectives.

To see why this is distinct from standard multi-task learning, consider the "Policy&Reward" baseline in Table 1. This variant trains the model on both the GRPO policy objective and the auxiliary judgment/comparison objectives, but does so without the iterative recycling mechanism that characterizes full SPARK. The auxiliary data is generated once (or from early checkpoints) and used statically. Policy&Reward achieves 55.1% — better than Policy-Only (53.3%), confirming that training both capabilities helps. But SPARK achieves 57.1%, a further jump of 2.0 percentage points that the paper attributes specifically to the co-evolving dynamic.

What does "co-evolving" mean concretely in this architecture, and why is it more than joint training? The mechanism (described in Section 3's technical breakdown) is that the auxiliary training on judgment/reflection objectives improves the shared model parameters, which means the next GRPO iteration generates rollouts from a model that has better internal representations of correctness. These improved rollouts provide cleaner auxiliary training data — correct answers are more robustly reasoned, incorrect answers contain more identifiable and learnable error patterns — which further improves the judgment capability, which further improves the next generation's rollouts, in a cycle.

The intellectual contribution here is the recognition that policy and reward are not merely compatible objectives that can coexist in a single model; they are synergistic in a specific, mechanistic sense. The shared parameter space acts as an information channel. Improvements in evaluation capability (better understanding of what makes an answer correct, what common errors look like, how to verify logical steps) transfer to generation capability through the shared representations, even though the GRPO advantage computation continues to use the external verifiable reward rather than the internal judge. The generation doesn't get better because the internal judge is scoring it — it gets better because the representations that support accurate judgment also support accurate reasoning.

This is a conceptual advance over prior work on self-reward and self-reflection. Systems like STaR (Zelikman et al., 2022) improve generation through iterative filtering, but do not produce an internal evaluation capability. Systems like S2R (Ma et al., 2025) train evaluation and generation, but from externally annotated data without a feedback mechanism. Systems like COOPER (Hong et al., 2025) maintain a separate reward model, so the policy's improvements never feed back into the evaluator. SPARK is the first to demonstrate that, when both capabilities share parameters and are trained iteratively on the policy's own outputs, a measurable positive feedback loop emerges that outperforms static joint training.

The evidence for this feedback loop being real — not just an assertion — comes from two data points. First, the Policy&Reward vs. SPARK gap in Table 1 (+2.0%) is the marginal benefit of the iterative recycling over static multi-task training. Second, the difficulty-dependent data scaling for larger models (Appendix A.1.3: SPARK-VL-32B uses difficulty-filtered data) suggests that the co-evolving mechanism benefits from challenging training instances where the model's initial pass rate is moderate — precisely the regime where the feedback loop has room to operate (easy problems provide no correction signal, impossible problems provide no initial success to build on).

This innovation is fundamental rather than incremental. The concept of mutual reinforcement between generation and evaluation through shared parameters is not a minor tweak to an existing recipe; it is a new way of thinking about what can be achieved when the policy-reward separation is removed. It suggests that future post-training pipelines should be designed with the explicit goal of creating these feedback channels, rather than treating policy optimization and reward modeling as separable stages.

Innovation 3: Test-Time Self-Reflection as a Learned Capability, Not an Emergent Property

The paper's ablation in Table 5 delivers a finding that is as important for what it rules out as for what it demonstrates: when the test-time self-reflection (TTS) procedure — generate an answer, judge it, and revise if incorrect — is applied to the base Qwen2.5-VL-7B model without any GRPO or SPARK training, performance crashes from 49.1% to 39.9% across the seven math benchmarks. The base model's self-judgments are so inaccurate, and its attempted revisions so ineffective, that the iterative procedure is actively harmful: the model frequently "corrects" already-correct answers into incorrect ones.

This negative result is not merely an unsurprising sanity check. It is a direct empirical counter to a hypothesis that has circulated in the LLM literature: that sufficiently large pre-trained models possess latent self-evaluation capabilities that can be elicited through appropriate prompting alone. Under this hypothesis, the reason prior work found self-correction ineffective (Huang et al., 2023) was insufficient prompting engineering or model scale, not a fundamental lack of the underlying capability. SPARK's Table 5 provides strong evidence against this view: a capable 7B-parameter vision-language model (Qwen2.5-VL-7B), when prompted with carefully designed judgment and reflection templates, not only fails to improve its outputs through self-reflection but actively degrades them. The capability is not latent; it must be learned.

The positive counterpart is equally significant: after SPARK training, the same TTS procedure yields substantial gains (from the non-TTS SPARK baseline to 58.8% with TTS). The Policy-Only GRPO baseline (trained only on the standard RLVR objective, no auxiliary judgment/reflection data) sees only a marginal TTS gain (+1.4%), confirming that GRPO training alone — which demonstrably improves reasoning — does not incidentally produce reliable self-evaluation. The self-judgment capability requires explicit training through the auxiliary objectives that SPARK introduces.

This finding changes the conceptual landscape around test-time compute scaling and self-improvement. Prior to this work, it was unclear whether self-reflection was (a) an emergent capability that would appear at sufficient model scale, (b) a capability that could be prompted out of instruction-tuned models, or (c) a capability that required specific training. SPARK provides evidence for (c), at least for the 7B–32B scale range studied, and in doing so clarifies what kind of research investment is needed: not better prompts, not larger models, but training procedures — like SPARK's co-evolving auxiliary objectives — that explicitly teach models to evaluate and correct their own outputs.

This is a conceptual advance with practical consequences. The field has been exploring test-time scaling strategies (best-of-N sampling, beam search, iterative refinement, tree search) largely under the assumption that the base model's outputs can be effectively scored or selected by either external verifiers or the model's own implicit confidence. SPARK shows that if you want the model itself to drive the test-time scaling process — judging its own outputs and deciding when to refine — this requires dedicated training, and that the training can be obtained as a byproduct of the same RLVR process that improves the policy. This closes the loop between training-time RL and test-time scaling in a way that prior work had not articulated: the same auxiliary objectives that improve training efficiency (by recycling wasted rollouts) also produce the self-evaluation capability needed for effective test-time scaling.

The innovation here is not the test-time scaling procedure itself (iterative refinement is well-known) but the demonstration that it is a learned capability rather than an emergent property, and that the learning can be achieved through a training framework that requires no additional data beyond what RLVR already generates.

Innovation 4: Internalizing the Reward Model Eliminates the Reward–Policy Mismatch Problem by Construction

The paper's comparison to RM-based RL (Table 7) makes a practical cost argument: SPARK eliminates the need for separate reward model training and the associated GPU overhead. But beneath this practical claim is a deeper conceptual insight about the nature of reward–policy mismatch and how it can be addressed.

Reward–policy mismatch — the divergence between the distribution on which a reward model was trained and the distribution of outputs produced by the evolving policy — is one of the central pathologies of RLHF. It causes the reward model's scores to become miscalibrated, leading to reward hacking (the policy exploits blind spots in the reward model's training) and ultimately to policy collapse. The standard mitigation is to periodically retrain the reward model on new preference data collected from the current policy — which is expensive and creates a complex, multi-phase training schedule.

SPARK addresses this problem architecturally rather than procedurally. Because the auxiliary judgment data is generated on-policy — constructed from the current model's own rollouts at each training iteration — the "reward model" (the model's internal judge) is always trained on data drawn from the current policy distribution. There is no distribution shift between the data used to train the judge and the outputs the judge needs to evaluate, because they are the same data. The judge doesn't need to be periodically retrained on fresh data, because it is continuously retrained on fresh data as an inherent part of the training loop.

This is a fundamentally different approach to the reward–policy mismatch problem than what the RLHF literature has explored. Rather than trying to make the reward model generalize better to out-of-distribution policy outputs (through larger training sets, better architectures, or uncertainty calibration), SPARK eliminates the out-of-distribution problem entirely by ensuring the reward model is always trained in-distribution. The evaluator and the evaluated are, literally, the same system at the same point in training.

The conceptual significance extends beyond the specific SPARK implementation. It suggests a design principle for future RL systems: evaluation capability should be trained on the same distribution it will be used to evaluate, and the simplest way to guarantee this is to make the evaluator and the generator share training iterations. This principle runs counter to the dominant paradigm of training reward models as standalone artifacts, frozen and deployed across many policy training runs. A frozen reward model is, by definition, out of distribution for any policy that has been updated since the reward model was trained. SPARK's architecture avoids this entirely.

The evidence that this architectural property translates to practical benefits is indirect but consistent. The paper's reward benchmark results (Table 1: SPARK-VL-7B achieves +12.1% average improvement over the baseline on RewardBench2 and VL-RewardBench) demonstrate that the internal judge generalizes to held-out evaluation tasks — it is not merely overfitting to the training distribution. And the paper's observation that these reward benchmark gains occur despite all auxiliary training data being mathematics-specific (Section 4.2: "all reward-related data generated during training are mathematics-specific, these two reward benchmarks span diverse domains") suggests that the on-policy, continuously updated training produces a judge that is robust rather than brittle — exactly the property that reward–policy mismatch undermines in traditional RLHF.

This innovation is fundamental rather than incremental. While the specific mechanism (on-policy auxiliary data generation from GRPO rollouts) is novel to SPARK, the underlying insight — that internalizing the evaluator and tying its training distribution to the policy's output distribution eliminates the mismatch problem by construction — is a conceptual advance that could influence how future RL systems are architected, even those that do not adopt SPARK's exact training recipe. It reframes the reward–policy mismatch from a training problem (how do we make reward models more robust to distribution shift?) to an architecture problem (how do we structure the system so that distribution shift never occurs?).

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use training data drawn from ViRL-39K (Wang et al., 2025) — a dataset of vision-language reasoning problems with verifiable answers — with 19K randomly sampled instances for SPARK-VL-7B, 24K difficulty-filtered instances for SPARK-VL-32B, and the Simple-RL-Zero-25K dataset (Zeng et al., 2025) for the text-only SPARK-7B. Evaluation spans 17 benchmarks across three categories: mathematical reasoning, reward-related evaluation, and general multimodal understanding. For mathematical reasoning, the multimodal benchmarks include MathVista (Lu et al., 2023), MathVision (Wang et al., 2024), WeMath (Qiao et al., 2024), MathVerse (Zhang et al., 2024), DynaMath (Zou et al., 2024), LogicVista (Xiao et al., 2024), and MMK12 (Meng et al., 2025), while text-only benchmarks include AIME24, AIME25, AMC23, GSM8k (Cobbe et al., 2021), Math500 (Lightman et al., 2023), and MMLU-STEM (Hendrycks et al., 2020). For reward-related evaluation, the paper uses RewardBench2 (RB2; Malik et al., 2025) and VL-RewardBench (VL-RB; Li et al., 2025), including their mathematical subsets for fine-grained analysis. For general capabilities, eight multimodal benchmarks are used: MMBench (Liu et al., 2023b), MMStar (Chen et al., 2024), MMMU (Yue et al., 2024), MMVet (Yu et al., 2023), ScienceQA (Lu et al., 2022), POPE (Li et al., 2023), SeedBench (Ying et al., 2025), and RealWorldQA.
Base model(s). All experiments are built on the Qwen model family: Qwen2.5-VL-7B and Qwen2.5-VL-32B (Bai et al., 2025) for vision-language experiments, and Qwen2.5-7B (Yang et al., 2025a) for text-only experiments. These models are chosen to span both modalities and multiple scales (7B and 32B parameters), with the paper arguing that this allows comprehensive evaluation of SPARK's effectiveness across different architectures and capacities. The 7B scale is the primary focus, with the 32B experiments serving as a scalability test.
Metrics. The primary metric across all benchmarks is accuracy — the fraction of test instances for which the model's final answer matches the ground truth. For mathematical benchmarks, answers are extracted from the model's output by parsing the \box{} LaTeX delimiter and compared against ground-truth answers using exact match or programmatic equivalence. For reward benchmarks, accuracy is measured by the fraction of preference pairs or judgment queries where the model correctly identifies the better or correct answer, following the official evaluation protocols of RewardBench2 and VL-RewardBench. For general benchmarks, accuracy follows each benchmark's standard metric. Aggregate metrics are reported as simple averages across the relevant benchmark sets (e.g., Avg-M for the 7 math benchmarks, Avg-R for the 2 reward benchmarks, Avg-All for the combined set). Per-benchmark accuracy improvements are reported in percentage points relative to the Qwen2.5-VL-7B or Qwen2.5-7B baseline.
Baselines. The paper compares against several categories of baselines. (1) Base models without RL: Qwen2.5-VL-7B-Instruct and Qwen2.5-VL-32B-Instruct (Bai et al., 2025), Qwen2.5-7B-Instruct (Yang et al., 2025a). (2) RL-based multimodal baselines: VL-Rethinker-7B and VL-Rethinker-32B (Wang et al., 2025), MM-Eureka-7B and MM-Eureka-32B (Meng et al., 2025), OpenVLThinker-7B (Deng et al., 2025), Vision-R1-7B and Vision-R1-32B (Huang et al., 2025), R1-OneVision-7B (Yang et al., 2025b). (3) RL-based text-only baselines: Qwen2.5-Math-7B-Instruct (Yang et al., 2024), Simple-RL-Zero-7B (Zeng et al., 2025), Eurus-2-7B-PRIME (Cui et al., 2025), Open-Reasoner-Zero-7B (Hu et al., 2025). (4) GRPO ablation baselines: Within the paper's own experimental framework, three GRPO variants are tested to isolate SPARK's contributions — Policy-Only (trained on standard GRPO with verifiable rewards, no auxiliary judgment/reflection data), Reward-Only (trained on GRPO but optimized for reward/judgment capability rather than policy reasoning), and Policy&Reward (jointly trained on both policy and reward objectives but without the iterative co-evolving cycle and without reflection data).
Generation budget / compute accounting. The paper does not report generation budgets in terms of number of rollouts per question for training — the GRPO group size n is not specified. For inference, all results are reported as single-generation accuracy (greedy or single-sample), except for the test-time scaling ablation (Table 5) where the budget is measured in number of self-reflection rounds (capped at 2 refinement iterations). The cost analysis in Table 7 compares GPU cost multiplicatively — RM-based RL requires approximately 2× the GPU resources of SPARK because it must maintain a separate reward model — but no quantitative FLOPs or GPU-hour measurements are reported. This means the paper's efficiency claims are based on architectural comparisons (one model vs. two) rather than empirical runtime or FLOPs measurements.
Cross-validation / statistical protocol. The paper does not report any cross-validation, statistical significance testing, confidence intervals, or error bars. Results are reported as single-point accuracy numbers without variance estimates. For the test-time scaling ablation (Table 5), the paper specifies that self-reflection is limited to a maximum of 2 rounds but does not report the distribution of reflection counts across the test set or the fraction of questions where reflection was triggered. The lack of statistical rigor — particularly given test set sizes that vary considerably across benchmarks (some with only hundreds of questions) — is a notable methodological limitation.

Main Quantitative Results

Results on SPARK-VL-7B: Mathematical and Reward Benchmarks

The headline result for the 7B vision-language model is reported in Table 1. SPARK-VL-7B achieves a 9.7% average improvement on 7 mathematical reasoning benchmarks (from the Qwen2.5-VL-7B baseline average of 49.1% to 58.8%), a 12.1% average improvement on 2 reward benchmarks (from the baseline average of 42.0% to 54.1%), and an overall average gain of 10.6% (from 46.5% to 57.1%). These are the numbers highlighted in the abstract and Figure 1(c).

Breaking down the mathematical benchmarks individually (Table 1, comparing Qwen2.5-VL-7B vs. SPARK-VL-7B): MathVista improves from 68.2% to 75.9% (+7.7), MathVision from 25.1% to 31.1% (+6.0), WeMath from 62.1% to 70.3% (+8.2), MathVerse from 49.2% to 53.0% (+3.8), DynaMath from 53.3% to 58.7% (+5.4), LogicVista from 40.4% to 50.7% (+10.3), and MMK12 from 45.1% to 71.8% (+26.7). The MMK12 gain of 26.7 percentage points is notably large — more than double the gain on any other benchmark — which warrants scrutiny. MMK12 is a recently introduced benchmark (Meng et al., 2025) and the paper does not discuss whether the ViRL-39K training data has any overlap or distributional similarity with it that might partially explain this outsized improvement.

On the reward benchmarks (Table 1): RewardBench2 overall improves from 45.8% to 49.3% (+3.5), with the math subset (RB2-Math) showing a minimal gain from 49.1% to 49.3% (+0.2 — essentially flat). In contrast, VL-RewardBench jumps from 38.8% to 62.7% (+23.9), with the math subset (VL-RB-Math) improving from 35.5% to 65.1% (+29.6). The dramatic asymmetry between the RewardBench2 and VL-RewardBench gains — +3.5% vs. +23.9% — is striking and the paper does not fully explain it, though Section 4.2 notes that the generalization of judgment ability "beyond mathematics" is evidence of the approach's robustness. An alternative interpretation is that VL-RewardBench's evaluation protocol or difficulty characteristics are more aligned with the type of judgment training SPARK provides, while RewardBench2 tests a different dimension of reward model quality.

A critical observation from the GRPO ablation rows in Table 1: the progression from Policy-Only (53.3% average) to Policy&Reward (55.1%) to SPARK (57.1%) demonstrates that each component — adding auxiliary judgment training (+1.8%), then adding the co-evolving cycle with reflection data (+2.0% further) — yields a measurable improvement. Reward-Only training (52.7%) underperforms Policy-Only, confirming the paper's claim that training exclusively on reward/judgment signals degrades reasoning capability compared to training on the policy objective.

Comparing against external baselines in Table 1: SPARK-VL-7B (Avg-M 58.8%) outperforms VL-Rethinker-7B (55.6%), MM-Eureka-7B (55.2%), Vision-R1-7B (51.0%), OpenVLThinker-7B (with incomplete data but reported at 60.6% on MMK12 vs. SPARK's 71.8%), and R1-OneVision-7B (with incomplete data). The gap over the strongest external baseline (VL-Rethinker-7B) is approximately 3.2 percentage points on average math accuracy. This is a meaningful but not enormous margin, and the paper does not discuss whether the differences in training data, hyperparameters, or compute budget between these methods might account for part of the gap.

Results on SPARK-7B (Text-Only LLM)

Table 2 reports the results for the text-only Qwen2.5-7B backbone. SPARK-7B achieves a 5.4% average gain on 6 math benchmarks (from 51.2% to 56.6%), a 12.0% gain on reward benchmarks (from 45.3% to 57.3%, with RewardBench2 improving from 49.5% to 58.8% [+9.3] and RB2-Math from 41.0% to 55.7% [+14.7]), and a 7.1% overall gain (from 49.7% to 56.8%).

The math benchmark breakdown (Table 2): AIME24 improves from 6.7% to 16.7% (+10.0), AIME25 stays flat at 6.7% — essentially no improvement on this particularly challenging competition benchmark, AMC23 improves from 50.0% to 62.5% (+12.5), GSM8k from 91.9% to 93.2% (+1.3, near ceiling), Math-500 from 76.2% to 79.4% (+3.2), and MMLU-STEM from 75.8% to 81.1% (+5.3). The AIME25 flatline is notable: it suggests that SPARK provides essentially zero benefit on problems where the base model's pass@1 rate is extremely low (6.7%), consistent with the broader finding in the RL scaling literature that RL-based post-training cannot create capability from nothing — it can only amplify existing capability where the base model already has some non-trivial probability of success.

Compared to external LLM baselines (Table 2): SPARK-7B (Avg-All 56.8%) substantially outperforms Simple-RL-Zero (48.8%), which notably drops to 31.0% on RewardBench2 and 38.7% on RB2-Math — a decline the paper attributes to the trade-off between reasoning and judgment ability in standard RLVR. Eurus-2-7B-PRIME (PRIME variant) achieves stronger math results (26.7% on AIME24 vs. SPARK's 16.7%, 79.2% on Math-500 vs. SPARK's 79.4%) but the comparison is incomplete across all benchmarks. Open-Reasoner-Zero-7B shows a similar pattern — stronger on some math metrics (79.2% on Math-500) but substantially weaker on reward benchmarks (31.4% on RewardBench2 and 37.3% on RB2-Math).

The GRPO ablation pattern in Table 2 mirrors the VL findings: Policy&Reward (53.2% Avg-M, 46.0% Avg-R) outperforms Policy-Only (51.8% Avg-M, 44.4% Avg-R) and both substantially outperform Reward-Only (51.6% Avg-M, 42.0% Avg-R). The Reward-Only degradation on both math and reward benchmarks reinforces the paper's claim that training exclusively on judgment signals is harmful — the model overfits to reward signals, "which in turn weakens its reasoning ability and prevents it from excelling in either skill" (Section 4.2). SPARK-7B's additional 3.4-percentage-point gain in Avg-M over Policy&Reward (56.6% vs. 53.2%) and 11.3-point gain in Avg-R (57.3% vs. 46.0%) demonstrates the value of the co-evolving cycle and reflection data.

Results on SPARK-VL-32B (Scalability Test)

Table 3 reports the scalability experiment with Qwen2.5-VL-32B. SPARK-VL-32B achieves a 7.8% average improvement on math benchmarks (from the baseline 57.2% to 65.0%) and a 3.0% improvement on reward benchmarks (from 58.0% to 61.0%), for an overall gain of 6.0% (from 57.5% to 63.5%).

Compared to the 7B results, the percentage gains are smaller in absolute terms but from a higher baseline — the 32B model starts at 57.2% math accuracy (vs. 49.1% for the 7B), meaning there is less headroom. The MMK12 benchmark again shows the largest individual improvement (+24.5%, from 52.9% to 77.4%), consistent with the 7B results and further suggesting that either SPARK's training is particularly well-suited to this benchmark or that there is some distributional alignment between the training data and MMK12.

Comparing against the Policy&Reward GRPO baseline for the 32B scale (Table 3): SPARK-VL-32B (65.0% Avg-M) outperforms Policy&Reward (63.4%) by 1.6 percentage points, and SPARK's reward benchmark average (61.0%) outperforms Policy&Reward's (55.9%) by 5.1 points. The gap is smaller than at the 7B scale (+2.0% for math, +1.4% for reward at 7B), which might indicate that the marginal benefit of the co-evolving mechanism diminishes as base model capability increases — though the different training data sizes (24K for 32B vs. 19K for 7B) and the difficulty-filtered data for the 32B model confound this comparison.

Compared to external 32B baselines: SPARK-VL-32B (65.0% Avg-M) outperforms VL-Rethinker-32B (62.9%), MM-Eureka-32B (61.0%), and Vision-R1-32B (59.1%). The margin over VL-Rethinker-32B is 2.1 percentage points, consistent with the 7B comparison where the margin was 3.2 points.

Results on General Multimodal Benchmarks

Table 4 reports results on 8 general-purpose multimodal benchmarks, testing whether SPARK's improvements generalize beyond mathematics and reward evaluation to broader vision-language understanding tasks. SPARK-VL-7B achieves an average gain of 1.5% over the Qwen2.5-VL-7B baseline (from 74.3% to 75.8%). Individual benchmarks: MMBench improves from 82.2% to 84.4% (+2.2), MMStar from 64.1% to 67.3% (+3.2), MMMU from 58.0% to 58.7% (+0.7), MMVet from 69.7% to 71.5% (+1.8), ScienceQA from 89.0% to 90.8% (+1.8), POPE from 85.9% to 88.2% (+2.3), SeedBench from 77.0% to 77.2% (+0.2), and RealWorldQA from 68.4% to 68.5% (+0.1).

The gains are positive and consistent — no benchmark shows degradation — but they are substantially smaller than the math and reward benchmark improvements (1.5% vs. 9.7% and 12.1%). This is expected given that SPARK's training data is entirely mathematics-focused, with auxiliary judgment data generated from math problem rollouts. The fact that improvement transfers at all to diverse domains (knowledge QA, visual perception, hallucination detection, scientific reasoning) is nontrivial, but the modest magnitude of the gains suggests that the learned evaluation and self-reflection capabilities are partially domain-specific, with the strongest transfer occurring to tasks that share structural features with mathematical reasoning (e.g., MMBench's comparative reasoning tasks, MMStar's multi-step visual reasoning).

Compared to other RL-based methods on general benchmarks (Table 4): SPARK-VL-7B (75.8%) outperforms VL-Rethinker-7B (74.4%) by 1.4 points and MM-Eureka-7B (74.1%) by 1.7 points. The paper frames this as evidence that SPARK "extends its reasoning and reflection abilities beyond the mathematical domain, demonstrating stronger generalization to diverse tasks" (Section 4.3). While true directionally, the margins are small enough — particularly on SeedBench (+0.2) and RealWorldQA (+0.1) — that they could potentially be explained by variance or differences in training data scale and composition between methods, factors the paper does not control for.

Judgment Accuracy Analysis (Self-Evaluation Quality)

The paper conducts a focused evaluation (Figure 4) to measure the model's self-judgment accuracy — how well the trained model can determine whether its own answers are correct. Using data from seven mathematical datasets, the model generates an answer and then assesses its own correctness. Precision, recall, and F1 scores are reported for both Qwen2.5-VL-7B and SPARK-VL-7B.

The results (Figure 4) show that SPARK-VL-7B consistently outperforms the baseline across all three metrics on all seven datasets. The F1 score improvements range from +0.25 (DynaMath) to +0.79 (MMK12), with the largest improvements occurring on MMK12, MathVerse, and WeMath — the same benchmarks where SPARK shows the largest accuracy improvements in Table 1. The paper does not report the absolute precision and recall values (only the radar chart comparisons), making it difficult to assess whether the model's self-judgments are reliable enough for practical deployment (e.g., what fraction of the model's "I'm correct" judgments are actually correct, and what fraction of its errors does it catch). This is a significant omission for a paper that proposes self-reflection as a deployment strategy.

Test-Time Scaling Results

Table 5 reports the ablation on test-time scaling (TTS). The key findings:

Base model + TTS degrades performance. Applying the judge-reflection loop to the untrained Qwen2.5-VL-7B drops average accuracy from 49.1% to 39.9% — a 9.2-percentage-point decline. Individual benchmark drops are severe: MathVision falls from 25.1% to 18.4% (-6.7), WeMath collapses from 62.1% to 24.1% (-38.0), MathVerse drops from 49.2% to 29.9% (-19.3), and MMK12 drops from 45.1% to 42.8% (-2.3). The paper explains this as the base model's "weak judgment and reflection skills causing frequent misjudgments, especially when the number of reasoning rounds increases" (Section 4.5). This is the strongest evidence that self-reflection is not an emergent capability of pre-trained models.
Policy-Only GRPO + TTS yields marginal gains. The Policy-Only baseline (trained with standard GRPO, no auxiliary judgment/reflection objectives) improves from 55.2% to 56.6% (+1.4) with TTS. Individual benchmarks: MathVista improves from 72.0% to 73.1% (+1.1), MathVision is essentially flat (28.5% to 28.7%), WeMath slightly declines (67.9% to 67.5%), DynaMath improves from 54.9% to 57.7% (+2.8), LogicVista improves from 44.9% to 48.9% (+4.0), MMK12 improves from 66.9% to 68.9% (+2.0). This small average gain confirms that GRPO training alone — which improves reasoning — does not incidentally produce reliable self-evaluation.
SPARK-VL-7B with TTS achieves the best results. The full SPARK model achieves 58.8% average with TTS (the exact TTS result for SPARK is not separately tabulated but can be inferred as the SPARK-VL-7B row in Table 5, where 58.8% is reported as the average across the 7 benchmarks). Compared to the Policy-Only + TTS result (56.6%), SPARK's TTS gain is approximately +2.2 additional percentage points beyond what GRPO alone provides with TTS.

Ablation Studies and Robustness Checks

Answer-based vs. CoT-based auxiliary data generation (Table 6): The paper compares three configurations for constructing the auxiliary training data: using only final answers (SPARK + Ans, 55.3% Avg-All), using only chains of thought (SPARK + CoT, 56.2%), and combining both (SPARK + Ans&CoT, 57.1%). The combined approach outperforms answer-only by +1.8% and CoT-only by +0.9%. On math benchmarks specifically, the combined approach achieves 58.8% Avg-M vs. 56.9% for both answer-only and CoT-only — a +1.9% improvement. On reward benchmarks, the CoT-only variant (54.9% Avg-R) outperforms the answer-only variant (52.7%) by +2.2%, but the combined variant (54.1%) is slightly worse than CoT-only on average, driven by a lower RB2-Math score (39.2% for combined vs. 44.0% for CoT-only vs. 41.8% for answer-only). This anomalous dip on RB2-Math for the combined configuration is not discussed in the paper but suggests that the interaction between answer-based and CoT-based auxiliary training may have complex effects on judgment capability that vary by evaluation domain.

Test-time scaling effectiveness across training configurations (Table 5): As discussed above, the TTS ablation demonstrates that self-reflection is effective only after SPARK's co-evolving training, with the base model actually harmed by TTS (-9.2%) and the Policy-Only GRPO baseline showing only marginal improvement (+1.4%). This is a critical ablation because it rules out the hypothesis that test-time self-reflection is a general capability of language models that can be elicited through prompting alone.

Policy-Only vs. Reward-Only vs. Joint Training (Tables 1 and 2): The GRPO ablation rows across all three model scales consistently demonstrate that (a) training on policy alone improves reasoning but not judgment, (b) training on reward alone degrades both reasoning and judgment relative to the base model, and (c) joint training (Policy&Reward) outperforms either single objective. At the 7B VL scale (Table 1): Policy-Only achieves 53.3% Avg-All, Reward-Only achieves 52.7%, and Policy&Reward achieves 55.1%. At the 7B LLM scale (Table 2): Policy-Only achieves 49.9%, Reward-Only achieves 49.2%, and Policy&Reward achieves 51.4%. The Reward-Only degradation is a consistent finding that supports the paper's claim about the complementarity of policy and reward training: judgment ability without reasoning ability is brittle and counterproductive.

Cost analysis (Table 7 and Section 4.5 narrative): The paper provides a qualitative comparison of SPARK against RM-based RL, claiming that SPARK eliminates the need for extra preference data, extra reward model training, and the associated 2× GPU cost. This is a legitimate architectural claim — SPARK does not require a separate reward model — but the paper provides no quantitative measurements of actual training time, GPU-hours, wall-clock time, or FLOPs to substantiate the "2×" figure or the "faster" efficiency claim. The comparison is conceptual, not empirical, and a rigorous cost analysis would need to measure the actual training throughput of SPARK vs. an equivalent GRPO + external reward model pipeline with comparable hyperparameters and hardware.

Scalability test (Table 3): The 32B experiment demonstrates that SPARK's improvements persist at larger scale (+7.8% Avg-M, +3.0% Avg-R), though the absolute percentage gains are smaller than at the 7B scale (+9.7% Avg-M, +12.1% Avg-R). This could reflect diminishing returns from co-evolution as base model capability increases, or it could reflect the different training data configuration (24K difficulty-filtered instances for 32B vs. 19K randomly sampled for 7B). The paper does not discuss which factor is more likely. A controlled ablation varying data size and difficulty filtering at both scales would be needed to disentangle these effects.

Domain generalization of judgment ability (Tables 1, 2, and 4): The paper observes that SPARK's judgment improvements transfer to reward benchmarks spanning diverse domains despite all auxiliary training data being mathematics-specific (Section 4.2). The evidence for this claim is the +12.1% improvement on the reward benchmarks (Table 1), with VL-RewardBench showing +23.9% improvement. However, the transfer to general multimodal benchmarks is substantially weaker (+1.5% average in Table 4), suggesting that the generalization is partial — the learned evaluation capability transfers better to structured evaluation tasks (reward benchmarks) than to open-ended understanding tasks (general benchmarks). This is not a weakness of the approach per se, but it bounds the claim of generalization: the model learns to be a better evaluator in general, but the downstream benefits for generation diminish for tasks that are structurally dissimilar from the training domain.

Critical Assessment

Claim: "SPARK achieves significant performance gains on multiple LLM and LVLM models"

This claim is well-supported across the three model configurations tested: SPARK-VL-7B (+10.6% overall, Table 1), SPARK-7B (+7.1% overall, Table 2), and SPARK-VL-32B (+6.0% overall, Table 3). The gains are positive and consistent across all three scales and both modalities, with no reported benchmark showing degradation (with the possible exception of AIME25 for SPARK-7B, which is flat at 6.7%). However, the claim's strength is limited by the narrow model family tested — all experiments use Qwen-series models. Whether SPARK's effectiveness transfers to other model families (LLaMA, Mistral, DeepSeek, etc.) with different pre-training distributions, tokenizers, and architectural choices is untested. The paper's assertion that SPARK demonstrates "robustness across different model scales and architectures" (Section 1) is technically true within the Qwen ecosystem but overstates the architectural diversity tested — Qwen2.5-VL and Qwen2.5 share substantial architectural heritage.

Claim: "SPARK eliminates the need for a separate reward model and costly human preference data"

This claim is true by construction of the SPARK framework: the auxiliary training data for judgment and reflection is generated on-policy from verifiable rewards, with no human preference annotations or teacher model outputs required. The cost comparison in Table 7 correctly identifies the components that SPARK eliminates (extra preference data, extra RM training). However, the claim is operational rather than empirical — the paper does not actually compare SPARK against an RM-based RL pipeline on the same tasks with the same base model to quantify the cost savings or to verify that the performance is comparable. The ablation rows in Tables 1 and 2 compare against GRPO variants (Policy-Only, Reward-Only, Policy&Reward), not against an external PPO + reward model pipeline. A direct comparison — SPARK vs. standard RLHF on the same benchmarks with the same base model and comparable total compute — would be needed to substantiate the claim that SPARK achieves comparable or better performance while eliminating reward model costs.

The paper also does not account for the computational cost of the auxiliary training itself. While SPARK eliminates the separate reward model forward pass during policy optimization, it adds auxiliary training steps (pointwise, pairwise, and reflection objectives) that require additional forward and backward passes through the model. The net computational saving relative to RM-based RL is therefore not as clear-cut as the "1× vs. 2× GPU" comparison implies, since SPARK adds computation that RM-based RL does not require. A thorough FLOPs or GPU-hours comparison would need to account for: GRPO rollouts (same in both), auxiliary data construction (SPARK-specific), auxiliary training steps (SPARK-specific), and reward model training + inference (RM-based RL-specific).

Claim: "SPARK creates a positive co-evolving feedback loop: improved reward accuracy yields better policy gradients, which in turn produce higher-quality rollouts that further refine the reward model"

This claim — the central conceptual contribution — is supported but with important caveats about the nature of the evidence. The paper's evidence for co-evolution consists of the performance gap between Policy&Reward (static joint training, no iterative recycling) and SPARK (iterative co-evolving training with reflection data): +2.0% Avg-All for VL-7B (Table 1, 55.1% vs. 57.1%) and +5.4% Avg-All for LLM-7B (Table 2, 51.4% vs. 56.8%). These gaps are consistent with a co-evolution effect, but they are confounded by the fact that SPARK includes reflection training data while the Policy&Reward baseline explicitly does not. The paper does not include a "Policy&Reward + Reflection" ablation that would isolate the contribution of the iterative recycling from the contribution of reflection data — without this, the gap attributed to co-evolution could partially reflect the value of reflection training rather than the feedback loop per se.

Furthermore, the paper does not provide any direct evidence of the feedback loop dynamics. There are no learning curves showing that the model's judgment accuracy improves over training iterations, no measurements of rollout quality (e.g., average reward) across iterations, and no demonstration that judgment accuracy at iteration k predicts reasoning improvement at iteration k+1 or vice versa. The co-evolution claim is an inference from the final performance difference between SPARK and Policy&Reward, not a directly observed phenomenon. This is a significant gap between the paper's conceptual framing and its empirical evidence.

Claim: "SPARK enables test-time scaling via self-reflection without external reward models and their associated costs"

This claim is the most thoroughly ablated in the paper (Table 5) and is genuinely well-supported. The demonstration that base model + TTS degrades performance (-9.2%), Policy-Only + TTS yields marginal gains (+1.4%), and SPARK + TTS yields substantial gains establishes a clear causal chain: self-reflection capability requires explicit training through SPARK's auxiliary objectives. The ablation cleanly rules out the alternative hypotheses (emergent capability from scale, incidental acquisition through GRPO reasoning training). This is the strongest empirical contribution in the paper.

However, the test-time scaling results have important limitations. The paper does not report what fraction of test questions trigger reflection (i.e., how many answers the model judges as incorrect), what the false-positive rate of self-judgments is (fraction of correct answers mistakenly judged as incorrect and subsequently "revised" to wrong answers), or what the success rate of reflections is (fraction of genuinely incorrect answers that are successfully corrected by reflection). These metrics are essential for understanding whether the TTS gains come from catching genuine errors or from other mechanisms, and whether the approach is reliable enough for deployment where the ground-truth answer is unknown.

Claim: "SPARK achieves 9.7% average gains on 7 reasoning benchmarks, 12.1% on 2 reward benchmarks, and 1.5% on 8 general benchmarks"

These specific numbers (from Table 1 and Table 4) are accurately reported and internally consistent. However, the aggregation across heterogeneous benchmarks with different scales, difficulties, and scoring protocols masks substantial variance: the 26.7% gain on MMK12 is an extreme outlier that inflates the average, while other benchmarks show much more modest improvements (3.8% on MathVerse, 5.4% on DynaMath). The paper does not report a weighted average, a median, or any measure of dispersion. The 1.5% gain on general benchmarks is reported as the primary generalization result, but individual benchmark gains range from 0.1% (RealWorldQA) to 3.2% (MMStar), with SeedBench at 0.2% — at these margins, even small variations in evaluation protocol or data splits could change the sign of the result. Without confidence intervals or statistical tests, it is impossible to determine which of these per-benchmark improvements are reliably above noise.

Missing experiments and comparisons

Several experiments would have substantially strengthened the paper's contributions:

Comparison against a reward-model-based RL baseline trained on the same data. The paper claims cost and performance advantages over RM-based RL but never tests against such a baseline. A simple experiment — train a separate ORM on the GRPO rollout data (using the verifiable rewards as labels), then use this ORM for policy optimization via PPO — would directly test whether SPARK's internalized judge outperforms a separate reward model trained on the same information.
Ablation of reflection data vs. co-evolving iterations. Policy&Reward + Reflection data (without iterative recycling) vs. SPARK (with iterative recycling) would isolate the contribution of the feedback loop from the contribution of reflection training. Currently these are confounded.
Training curves showing the co-evolution dynamic. Measurements of judgment accuracy and generation quality at each training iteration, showing that improvements in one track improvements in the other, would provide direct evidence for the central conceptual claim.
Experiments on non-Qwen model families. At minimum, a single experiment with a LLaMA or Mistral backbone would test whether the approach transfers across architectures.
Statistical significance testing or confidence intervals. All results are reported as single-point estimates without any quantification of uncertainty, making it difficult to assess the reliability of the reported improvements, particularly for the smaller benchmarks and the smaller gains on general benchmarks.
Quantitative cost analysis. Actual GPU-hours, wall-clock time, and memory usage measurements for SPARK training compared to equivalent GRPO-only and hypothetical RM-based RL training would substantiate the efficiency claims.
Detailed breakdown of TTS dynamics. What fraction of questions trigger reflection? What is the precision and recall of self-judgments? What is the success rate of reflections? Under what conditions does reflection harm performance?

In summary, the paper demonstrates that adding auxiliary judgment and reflection training to GRPO — using on-policy data recycled from the normally-discarded rollout stream — improves both reasoning and evaluation performance across multiple benchmarks and model scales, with the improvements being most pronounced on mathematical reasoning and reward evaluation, modest on general multimodal tasks, and dependent on the auxiliary training for effective test-time self-reflection. The evidence for the specific claim of a "co-evolving feedback loop" is suggestive but indirect, and several important ablations and comparisons that would strengthen the central narrative are absent.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Efficiency Claims

The assumption or constraint. The paper's headline efficiency claim — that SPARK requires only "1× GPU cost" versus "~2×" for RM-based RL (Table 7) — compares architectural requirements (one model vs. two) without measuring the actual computational cost of SPARK's auxiliary training loop. SPARK adds several computationally significant operations to the standard GRPO pipeline at each training iteration: constructing three types of auxiliary training data from the rollout group (pointwise, pairwise, and reflection samples, Section 3.2), formatting these samples with specialized prompt templates (Appendix Figure 7), and running additional forward and backward passes for the auxiliary objectives on top of the GRPO policy gradient step. None of this auxiliary computation exists in the standard RLVR pipeline that SPARK is compared against.

The consequence. The "1× vs. 2× GPU" framing in Table 7 is misleading because it compares SPARK's single-model architecture against RM-based RL's dual-model architecture while ignoring that SPARK adds computation that RM-based RL does not require. A fair comparison would account for: (a) the cost of generating $n$ rollouts per question (same in both), (b) the cost of auxiliary data construction and formatting (SPARK-specific), (c) the cost of auxiliary training steps — forward and backward passes for pointwise, pairwise, and reflection objectives (SPARK-specific), and (d) the cost of reward model training and inference (RM-based RL-specific). Without a FLOPs-matched or GPU-hours-matched comparison, practitioners cannot determine whether SPARK is genuinely more compute-efficient or simply trades one form of computation (reward model inference) for another (auxiliary objective training). The paper's efficiency claims are entirely qualitative, and a practitioner choosing between SPARK and an external reward model has no quantitative basis for the decision.

What evidence exists in the paper. The paper provides no measurements of training time, GPU-hours, wall-clock time, FLOPs, or memory usage for any configuration. Table 7 is a conceptual comparison, not an empirical one. Section 4.5's cost analysis narrative states that RM-based RL "repeatedly calls the RM for reward inference, which doubles GPU usage and slows training" and that SPARK "directly employs lightweight rule-based verifiable rewards to generate feedback on-policy, allowing a single unified model to optimize both policy and reward," but "lightweight" is not quantified. The ablation tables (Tables 1, 2, 3) compare final accuracy, not training cost.

Mitigation status. The paper does not acknowledge this as a limitation and makes no attempt to measure or model the computational cost of the auxiliary training. The efficiency claim is presented as an established fact rather than a hypothesis requiring empirical validation. A future study would need to run controlled comparisons measuring total FLOPs or GPU-hours for SPARK vs. an equivalent GRPO + external ORM pipeline at matched final performance to determine the actual efficiency tradeoff.

Self-Reflection Degrades Performance on Untrained Models — The Capability Is Not Emergent

The assumption or constraint. The paper's test-time scaling (TTS) procedure — generate an answer, self-judge correctness, and revise if judged incorrect — assumes the model possesses reliable self-evaluation capability. The paper demonstrates conclusively that this assumption fails for base models and for models trained only on standard GRPO (Table 5). This means SPARK's TTS benefits are contingent on the specific auxiliary training that SPARK provides, and the TTS procedure cannot be applied as a general-purpose inference-time strategy to arbitrary models.

More subtly, the paper does not characterize the reliability of SPARK's self-judgments even after training. The judgment accuracy analysis (Figure 4) shows that SPARK-VL-7B improves over the baseline in self-judgment F1 score, but the absolute precision and recall values are not reported — only the relative improvement on a radar chart. A practitioner needs to know: if the model judges an answer correct, what is the probability it is actually correct (precision)? If the answer is actually incorrect, what is the probability the model catches it (recall)? Without these numbers, the practical reliability of the TTS procedure is unknown.

The consequence. Two failure modes arise from imperfect self-judgments at test time:

False negatives (correct answer judged as incorrect): The model "revises" a correct answer into an incorrect one, degrading accuracy relative to single-pass generation. The paper's Table 5 shows this occurring at scale for the base model (average drops from 49.1% to 39.9%), and even SPARK may suffer from this on some fraction of questions — the paper does not report the false-negative rate.
False positives (incorrect answer judged as correct): The model accepts a wrong answer and terminates refinement early, missing the opportunity to correct it. This limits the ceiling on TTS improvement — the procedure only helps on questions where the model both (a) generates an incorrect initial answer and (b) correctly identifies it as incorrect.

The paper does not report either rate, the distribution of reflection counts across the test set, or the per-question dynamics (initial correctness → judgment → post-reflection correctness). A practitioner deploying TTS in a setting where ground-truth answers are unavailable (the normal inference scenario) has no way to estimate how often the model's "I'm done" judgment can be trusted.

What evidence exists in the paper. Table 5 provides aggregate TTS accuracy for three model configurations, but no breakdown of the TTS dynamics (trigger rate, judgment accuracy, reflection success rate). Figure 4 provides relative self-judgment F1 improvements but no absolute precision/recall values. The case study in Figure 3 shows a successful reflection (incorrect initial answer → model correctly judges it as wrong → model produces correct revision), but this is a single illustrative example, not a systematic characterization.

Mitigation status. The paper implicitly acknowledges the reliability concern by limiting TTS to a maximum of 2 refinement rounds (Section 4.5 narrative: "especially when the number of reasoning rounds increases" as a factor in base model degradation), but does not explain how this limit was chosen or whether additional rounds would help or hurt for SPARK specifically. The limitation of self-judgment reliability is not discussed as a systematic constraint on the approach.

The Co-Evolution Claim Lacks Direct Empirical Validation

The assumption or constraint. The paper's central conceptual contribution is the "co-evolving feedback loop": improved reward accuracy yields better policy gradients, which produce higher-quality rollouts, which further refine the reward model, in a positive cycle (Section 1, advantage 3; Section 3). This claim is used to distinguish SPARK from static multi-task learning (the "Policy&Reward" baseline in Tables 1 and 2) and to motivate the iterative recycling of auxiliary training data.

The consequence. If the co-evolution claim is true, SPARK's design — iterative recycling of on-policy rollouts into auxiliary training — is essential and the Policy&Reward baseline should underperform because it lacks the feedback loop. If the claim is false or overstated, then the performance gap between SPARK and Policy&Reward might be attributable to other factors (e.g., the inclusion of reflection data, which Policy&Reward lacks, or differences in hyperparameters between the two configurations), and a simpler training procedure (one-time auxiliary data generation followed by joint training) might achieve comparable results.

The paper's evidence for co-evolution consists entirely of endpoint comparisons — the final accuracy difference between Policy&Reward and SPARK. There is no process evidence: no learning curves showing judgment accuracy improving across iterations, no measurements of rollout quality (e.g., average verifiable reward) evolving over training, and no demonstration that iteration $k$ 's judgment accuracy predicts iteration $k+1$ 's generation improvement. The causal structure of the feedback loop is asserted, not demonstrated.

What evidence exists in the paper. The Policy&Reward vs. SPARK gap is +2.0% Avg-All for VL-7B (Table 1: 55.1% vs. 57.1%) and +5.4% Avg-All for LLM-7B (Table 2: 51.4% vs. 56.8%). These gaps are attributed to co-evolution, but they are confounded: Policy&Reward does not include reflection data ( $D_{\text{reflect}}$ from Section 3.2), while SPARK does. The paper does not include a "Policy&Reward + Reflection" ablation (static joint training with reflection data but without iterative recycling) that would isolate the marginal contribution of the feedback loop from the marginal contribution of reflection training. Without this ablation, the gap attributed to co-evolution could be partially or entirely explained by the value of reflection data.

There are no training curves, no per-iteration metrics, and no analysis of how the model's judgment or generation quality changes over the course of training. The "co-evolving" dynamic is an inference from final performance, not a measured phenomenon.

Mitigation status. The paper does not acknowledge this as a limitation. The co-evolution language is used throughout as a description of what SPARK does, not as a hypothesis requiring validation. A future study would need to track per-iteration metrics (judgment accuracy, average reward, rollout quality), vary the frequency of auxiliary data regeneration, and include the missing "Policy&Reward + Reflection" ablation to establish whether the iterative feedback loop provides benefits beyond those of static multi-task learning with the same objectives.

Single Model Family and Single Task Domain Limit Claims of Generality

The assumption or constraint. All experiments use Qwen-series models (Qwen2.5-VL-7B, Qwen2.5-VL-32B, Qwen2.5-7B) as backbones (Appendix A.1.1), and all training data consists of problems with verifiable answers — predominantly mathematical reasoning (ViRL-39K for VL models, Simple-RL-Zero-25K for the LLM). The paper claims SPARK demonstrates "robustness across different model scales and architectures" (Section 1) and that judgment ability "generalizes beyond mathematics" (Section 4.2).

The consequence. The claim of architectural generality is unsupported. Qwen2.5-VL and Qwen2.5 share substantial architectural heritage — they are from the same model family developed by the same team. Whether SPARK's effectiveness transfers to models with meaningfully different architectures (LLaMA, Mistral, DeepSeek, Gemma) — which differ in tokenization, attention mechanisms, pre-training data mixtures, and instruction-tuning recipes — is untested. A practitioner using a non-Qwen base model has no evidence that SPARK will work for them.

The claim of domain generality is partially supported (the +1.5% gain on general multimodal benchmarks in Table 4) but the magnitude is an order of magnitude smaller than the math-domain gains (+9.7% in Table 1). This suggests that while some transfer occurs, the learned self-evaluation and self-reflection capabilities are substantially domain-specific. The paper's framing emphasizes the positive transfer while not discussing the large gap in transfer magnitude between math-adjacent tasks (reward benchmarks: +12.1%) and general tasks (general benchmarks: +1.5%). A practitioner deploying SPARK for a non-mathematical application cannot expect gains of the magnitude reported in the headline numbers.

What evidence exists in the paper. All model backbones are Qwen (Appendix Table 8). All training data is mathematical (ViRL-39K and Simple-RL-Zero-25K, described in Appendix A.1.3). The 32B experiments (Table 3) demonstrate scalability within the Qwen family but are not a cross-architecture test. The general benchmark results (Table 4) demonstrate some domain transfer but with substantially smaller gains than in-domain tasks. The paper observes that RewardBench2 and VL-RewardBench "span diverse domains" and that SPARK's improvements on these benchmarks indicate generalization (Section 4.2), but these are still reward evaluation tasks — structurally similar to the auxiliary training objectives — not open-ended generation or reasoning in non-mathematical domains.

Mitigation status. The paper does not acknowledge the limitation to Qwen architectures or the domain-specificity of the primary gains. The claim of "robustness across different model scales and architectures" (Section 1) overstates the evidence: scales are tested within a single architecture family, and no cross-architecture experiment is conducted. Future work would need to replicate SPARK on at least one non-Qwen backbone and on training data from a substantially different domain (e.g., code generation, scientific reasoning, or factual QA) to establish the claimed generality.

The Reward Benchmark Evaluation Protocol Mismatches the Training Signal

The assumption or constraint. The paper evaluates SPARK's reward modeling capability using RewardBench2 and VL-RewardBench (Tables 1, 2, 3), which test a model's ability to compare two candidate responses and identify which is better — a pairwise preference task. SPARK's training includes pairwise comparison data ( $D_{\text{pairwise}}$ in Section 3.2), but the ground-truth signal for these pairwise training samples is always derived from the verifiable reward: the "better" answer is the one that matches the ground-truth answer. This is a very specific type of preference — correctness preference — which is a subset of the types of preferences that general-purpose reward models are expected to handle (helpfulness, harmlessness, stylistic quality, factual accuracy in domains without verifiable answers, etc.).

The consequence. The reward benchmark results are being used to claim that SPARK produces a general-purpose reward model ("strong judgment ability on broader tasks," Section 4.2). But VL-RewardBench and RewardBench2 evaluate reward models on tasks where the "better" response is not always determined by objective correctness alone — they include dimensions of completeness, clarity, relevance, and helpfulness that SPARK's training never sees. The paper's judgment prompting for VL-RewardBench evaluation (Appendix Figure 6) shows that the model is asked to evaluate based on accuracy, completeness, clarity, and relevance — yet SPARK was only trained to identify verifiably correct vs. incorrect answers. This creates an evaluation mismatch: the benchmark is testing capabilities that SPARK was not explicitly trained for, and it is unclear whether SPARK's improvements on these benchmarks reflect genuine reward modeling generalization or artifact of the evaluation protocol (e.g., correctness often correlates with the other dimensions, so a model that identifies the correct answer may incidentally score well on completeness and relevance).

The asymmetry between RewardBench2 (+3.5% overall) and VL-RewardBench (+23.9%) gains for SPARK-VL-7B (Table 1) further suggests that something about VL-RewardBench's construction makes it more amenable to correctness-based evaluation, while RewardBench2 — which the reward modeling community treats as a more rigorous test of general reward model quality — shows much smaller improvements. The near-zero gain on RewardBench2's math subset (+0.2%) is particularly striking given that this is exactly the domain where SPARK's training is concentrated.

What evidence exists in the paper. Table 1 shows the asymmetric gains: +3.5% on RewardBench2 overall, +0.2% on RB2-Math, +23.9% on VL-RewardBench, +29.6% on VL-RB-Math. The paper notes the domain gap ("all reward-related data generated during training are mathematics-specific, these two reward benchmarks span diverse domains") and interprets the positive results as evidence of generalization. But it does not discuss why the generalization is dramatically stronger on VL-RewardBench than on RewardBench2, nor whether the evaluation criteria used in these benchmarks (which include non-correctness dimensions) are actually being learned by SPARK or simply correlated with correctness in the benchmark instances.

Mitigation status. The paper does not acknowledge this evaluation mismatch as a limitation. The reward benchmark results are presented as evidence of general reward modeling capability without analyzing the gap between what SPARK is trained to do (identify verifiably correct answers) and what the benchmarks measure (identify better responses according to multi-dimensional criteria). A proper evaluation would need to either (a) use reward benchmarks that exclusively test correctness-based preference (isolating the capability SPARK actually trains) or (b) test whether SPARK's judgments align with human preferences on dimensions other than correctness, which would require a targeted evaluation not present in the paper.

No Comparison Against a Standard RM-Based RL Baseline

The assumption or constraint. The paper's thesis is that SPARK eliminates the need for separate reward models and their associated costs while achieving strong performance. The cost comparison in Table 7 and the narrative throughout the paper position SPARK as an alternative to RM-based RL pipelines. Yet the paper never actually compares SPARK against such a pipeline.

The consequence. A practitioner choosing between SPARK and a traditional RLHF setup has no evidence about how their final performance would compare. The paper compares SPARK against several GRPO variants (Policy-Only, Reward-Only, Policy&Reward) and against other published methods that use various RL recipes (VL-Rethinker, MM-Eureka, Simple-RL-Zero, etc.), but none of these represent the canonical RM-based RL alternative that SPARK claims to replace: a pipeline that (1) trains a separate reward model on preference data, then (2) uses that reward model for PPO-based policy optimization. Without this comparison, the paper's central value proposition — that SPARK achieves comparable performance to RM-based RL at lower cost — remains a hypothesis, not a demonstrated result.

The paper has all the components needed to construct this baseline. The GRPO rollouts with verifiable reward labels ( $D_{\text{pointwise}}$ and $D_{\text{pairwise}}$ from Section 3.2) could be used to train a separate outcome reward model (a standard ORM training procedure), which could then guide policy optimization via PPO. A comparison of SPARK against this baseline would directly test whether internalizing the judge in the policy model is better, worse, or equivalent to maintaining a separate reward model trained on the same information.

What evidence exists in the paper. No comparison against an external reward model baseline is provided. The baseline methods in Tables 1, 2, and 3 are either base instruction-tuned models, other published RL-based methods with different training recipes and data, or GRPO variants that do not include a separate reward model. Table 7 makes comparative claims about RM-based RL without any empirical RM-based RL results.

Mitigation status. The paper does not acknowledge this as a missing comparison. The claim that SPARK eliminates the need for separate reward models is supported by the architecture description (SPARK does not have one) and the cost analysis (Table 7), but not by a demonstration that performance is preserved relative to an RM-based pipeline. A future study would need to train a separate reward model on the same on-policy data that SPARK uses for auxiliary training, use it for PPO-based policy optimization on the same base model and training data, and compare the resulting policy and reward model quality against SPARK's unified model.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper reframes a waste product—the discarded rollouts from RL with Verifiable Rewards (RLVR)—as a free, on-policy, continuously refreshing stream of supervised data for reward modeling. The shift is conceptual rather than algorithmic: the field had implicitly categorized GRPO rollouts as intermediate computation (analogous to activations in a forward pass, to be discarded after the gradient step), and SPARK recategorizes them as labeled training data (analogous to the preference pairs used to train reward models in RLHF). This reframing matters because it challenges the assumption that reward modeling requires a separate data collection effort, separate model architecture, and separate training pipeline. If the rollout stream from any RLVR training run can simultaneously produce a competent reward model at essentially zero additional data cost, then the primary bottleneck that has made RLHF expensive—preference data collection—is bypassed for any task where verifiable rewards exist.

The magnitude of this shift should be calibrated carefully. This is not a paradigm shift on the order of "RLHF is obsolete" or "reward models are unnecessary." SPARK's approach is contingent on the availability of verifiable rewards—it works for math, code, and other domains with ground-truth answer checking, but does not directly address the subjective alignment tasks (helpfulness, safety, stylistic quality) that motivated RLHF in the first place. Rather, SPARK demonstrates that for the growing fraction of post-training that uses verifiable rewards (a fraction that has expanded dramatically since DeepSeek-R1), the separate reward model pipeline is an unnecessary inheritance from the RLHF era rather than a technical requirement. This is a reframing with practical consequences: it suggests that organizations investing in RLVR-based reasoning improvements can obtain reward modeling capability as a byproduct, eliminating a significant line item from their post-training budget.

The paper also resolves a latent contradiction in the self-correction literature. Prior work had reached opposite conclusions about whether models can self-correct—some studies found it helpful (Madaan et al., 2023), others found it ineffective or harmful (Huang et al., 2023). SPARK's Table 5 provides a clear resolution: self-correction is not an emergent property of pre-trained models, nor is it incidentally acquired through reasoning-focused RL training. The base Qwen2.5-VL-7B model, when asked to judge and revise its own outputs, degrades performance by 9.2 percentage points (from 49.1% to 39.9%). The Policy-Only GRPO baseline, despite improved reasoning, gains only 1.4 points from self-reflection. Only after SPARK's explicit auxiliary training on judgment and reflection objectives does self-reflection become reliably beneficial. This three-way comparison cleanly resolves the prior ambiguity: self-correction works only when models are explicitly trained for it, not when it is prompted from general capabilities.

This finding redirects research attention in two ways. First, it makes test-time self-reflection a training problem rather than a prompting problem—the question is not "what prompt elicits self-correction from a capable model?" but "how do we train models to evaluate and correct themselves?" This shifts investment from prompt engineering toward training objective design. Second, it makes RLVR data recycling an attractive research direction, because the paper demonstrates that the same auxiliary training that enables self-reflection also improves reasoning (the Policy&Reward row in Table 1 outperforms Policy-Only), meaning the investment in auxiliary objectives pays off in both training-time and inference-time improvements.

The identification of reward–policy mismatch as an architectural problem rather than a training problem is a subtler but potentially influential conceptual move. The RLHF literature has treated reward–policy mismatch (the divergence between the distribution on which a reward model was trained and the distribution of outputs from the evolving policy) as a training challenge—addressed through better reward model architectures, uncertainty calibration, or periodic retraining on fresh preference data. SPARK eliminates the mismatch by construction: because the auxiliary judgment data is generated on-policy from the current model's own rollouts, the "reward model" (the model's internal judge) is always trained on data drawn from the current policy distribution. Distribution shift never occurs because the evaluator and the evaluated are the same system at the same point in training. This reframes the problem from "how do we make reward models robust to distribution shift?" to "how do we structure the training loop so that distribution shift never arises?"—an architectural solution to what had been treated as a statistical robustness problem.

Finally, the paper's results on reward benchmarks (Table 1: SPARK-VL-7B achieves +12.1% average over the baseline, with VL-RewardBench jumping +23.9%) suggest that models trained to reason about correctness in verifiable domains develop evaluation capabilities that partially transfer to broader judgment tasks. The transfer is imperfect (the +1.5% gain on general multimodal benchmarks in Table 4 is an order of magnitude smaller than the math-domain gains) but directionally consistent—no benchmark shows degradation. This implies that correctness judgment is a scaffoldable skill: training on objective correctness in narrow domains produces representations that support broader evaluation capabilities, even without explicit training on those broader capabilities. This finding positions RLVR not just as a policy optimization tool but as a potential foundation for general-purpose evaluator training, a reframing that could influence how future post-training pipelines allocate their training budget across verifiable and non-verifiable tasks.

Follow-Up Research This Work Enables

Direct comparison of SPARK against a reward-model-based RL baseline trained on identical data. The paper's central value proposition is that SPARK eliminates the need for a separate reward model while achieving strong performance, but this claim is never tested empirically. The paper has all the components needed: the GRPO rollouts with verifiable reward labels ( $D_{\text{pointwise}}$ and $D_{\text{pairwise}}$ from Section 3.2) could be used to train a separate outcome reward model via standard supervised learning, which could then guide policy optimization via PPO on the same base model and training data. This comparison would directly answer the question: does internalizing the evaluator in the policy model produce better, worse, or equivalent results compared to a separate reward model trained on the same information? The experiment should measure three outcomes: (1) final policy accuracy on math benchmarks, (2) reward model accuracy on RewardBench2 and VL-RewardBench, and (3) total GPU-hours to reach matched performance. If SPARK matches or exceeds the separate reward model pipeline at lower total compute, the cost-saving claim is validated. If the separate reward model outperforms SPARK's internal judge, the architectural tradeoff (simplicity vs. evaluation quality) becomes the central question. The paper's own "Reward-Only" GRPO ablation in Table 1 (which trains the model on reward signals without policy co-training and sees degradation to 52.7% Avg-All vs. Policy&Reward's 55.1%) suggests that a standalone reward model might outperform a policy model that is only trained on reward objectives, but doesn't answer whether a separate reward model guiding PPO would outperform SPARK's integrated approach.

Isolating the co-evolving feedback loop from reflection data with a controlled ablation. The paper attributes the performance gap between Policy&Reward (static joint training without reflection data) and SPARK (iterative recycling with reflection data) to the "co-evolving feedback loop," but these two configurations differ in two ways: SPARK includes reflection data and SPARK iteratively regenerates auxiliary data from the current policy. A Policy&Reward + Reflection configuration—joint training on pointwise, pairwise, and reflection objectives, but with auxiliary data generated once from initial or early checkpoints rather than iteratively recycled—would isolate the marginal contribution of the feedback loop. If this configuration matches SPARK's performance, then the co-evolution claim is unsupported and the gains come from adding reflection data to the training mix, not from the iterative recycling. If it underperforms SPARK, the feedback loop is genuinely important and the mechanism deserves further study. This ablation is simple to run, requires no new infrastructure, and would substantially clarify which aspects of SPARK's design are necessary versus incidental. The paper's omission of this ablation is the single most important gap in its empirical validation of the central conceptual claim.

Measuring the co-evolution dynamics directly with per-iteration metrics. The paper asserts a positive feedback loop but provides only endpoint comparisons, not process evidence. A follow-up study should track, at each training iteration: (a) the model's judgment accuracy on a held-out set of correctness evaluation queries (to measure how evaluation capability improves), (b) the average verifiable reward across GRPO rollouts (to measure how generation quality improves), and (c) the cross-correlation between iteration $k$ 's judgment accuracy and iteration $k+1$ 's generation quality. If the feedback loop is real, improvements in judgment should precede improvements in generation (or vice versa) with a measurable lag, and the two metrics should exhibit mutual reinforcement rather than independent improvement. The study should also vary the frequency of auxiliary data regeneration—regenerating every iteration (full SPARK), every $m$ iterations, and only once (Policy&Reward)—to characterize how quickly the auxiliary training data becomes stale and how much the feedback loop depends on fresh on-policy data. This would transform the co-evolution claim from a post-hoc interpretation of endpoint results into a directly observed and characterized phenomenon.

Testing SPARK on non-Qwen model families and non-math domains to establish generality. The paper's architectural generality claim ("robustness across different model scales and architectures," Section 1) is tested only within the Qwen ecosystem. A replication of SPARK on a LLaMA-3 or Mistral backbone would test whether the co-evolving mechanism depends on properties specific to Qwen models (tokenizer behavior, pre-training data distribution, instruction-tuning recipe) or generalizes to meaningfully different architectures. The domain generality claim is partially tested by the general multimodal benchmarks in Table 4 (average +1.5%), but the transfer is weak. A replication on a code generation training set (e.g., programming problems with unit-test-based verifiable rewards) and evaluation on both code benchmarks (HumanEval, MBPP) and general reasoning benchmarks would test whether the auxiliary training benefits are specific to mathematical reasoning or transfer across verifiable domains. If the approach works on code but the general benchmark transfer remains weak, that would suggest the learned evaluation capability is tied to the structural features of the training domain rather than being a general-purpose judgment skill. If it fails on code entirely, that would suggest SPARK's effectiveness depends on properties of mathematical reasoning data that may not hold in other domains.

Characterizing the reliability and failure modes of test-time self-reflection for deployment. The paper's TTS ablation (Table 5) demonstrates that self-reflection works after SPARK training but provides no characterization of when it works and when it fails. A deployment-focused follow-up should measure: (a) the self-judgment precision (fraction of answers the model judges as correct that are actually correct), (b) the self-judgment recall (fraction of genuinely incorrect answers that the model catches), (c) the reflection success rate (fraction of caught errors that are successfully corrected), (d) the false-correction rate (fraction of correct answers mistakenly judged as incorrect and "revised" to wrong answers), and (e) the distribution of reflection rounds across the test set. These metrics would allow a practitioner to estimate the expected benefit of TTS for their specific accuracy-reliability tradeoff. The study should also correlate these metrics with problem difficulty (using the difficulty binning approach from the main analysis paper): does self-judgment reliability degrade on harder problems, where the model is more likely to be uncertain? Does reflection succeed primarily on easy problems (where the model can recognize and fix simple errors) or on medium problems (where the initial attempt is in the right ballpark but contains a specific mistake)? The paper's case study in Figure 3 shows a successful reflection, but a systematic characterization of failure modes—when does the model confidently assert a wrong answer is correct? When does it "correct" a correct answer into an incorrect one?—is essential for any practical deployment where ground-truth answers are unavailable.

Extending SPARK to subjective alignment tasks without verifiable rewards. SPARK's core mechanism relies on verifiable rewards to generate the auxiliary training data. A natural extension is to incorporate a small amount of human preference data or teacher-model judgments to train the auxiliary objectives for tasks where verifiable rewards are unavailable (helpfulness, safety, stylistic quality). The hypothesis would be that the co-evolving mechanism—where improved evaluation feeds back into improved generation—generalizes beyond the verifiable reward setting, and that the on-policy data generation approach (using the model's own rollouts scored by an external judge) would similarly reduce reward–policy mismatch compared to a static reward model. A concrete experiment: start with a small set of human-annotated preference pairs, train an initial reward model, use it to score policy rollouts in a SPARK-style loop (where the reward model provides the labels for auxiliary judgment/reflection training on the policy model), and compare against standard RLHF with the same preference data. This would test whether the architectural insight—internalizing the evaluator and tying its training distribution to the policy's output distribution—provides benefits even when the evaluation signal originates from human preferences rather than verifiable ground truth. The key question is whether the positive feedback loop depends on the quality of the evaluation signal (verifiable rewards are perfect; human preferences are noisy) or only on the architecture of the training loop.

Practical Applications and Downstream Use Cases

Cost-efficient post-training pipelines for organizations with verifiable-domain applications. For teams building LLMs or LVLMs for mathematical reasoning, code generation, or scientific QA—domains where ground-truth verification is available—SPARK offers a concrete cost reduction strategy. Instead of running separate pipelines for GRPO-based reasoning improvement and reward model training, a single SPARK training run produces both a stronger policy and a competent internal evaluator. The paper's data (Table 7) claims roughly 2× GPU cost reduction by eliminating the separate reward model, and while this specific number is not empirically validated, the architectural elimination of the second model and its associated training loop is a genuine simplification. A team currently spending 50% of their post-training compute on reward model training could redirect that budget toward more policy training iterations, larger batch sizes, or longer training runs—potentially amplifying the reasoning gains beyond what SPARK demonstrates. The paper's finding that only 19K training instances suffice for SPARK-VL-7B (Appendix A.1.3) means the seed data requirement is modest, further lowering the barrier to adoption.

Test-time self-reflection for deployed models without external judge infrastructure. In deployment scenarios where GPU memory is constrained (edge devices, mobile inference, small-server deployments), running a separate reward model or judge model for test-time verification may be infeasible—loading a second model doubles memory requirements and introduces inter-model communication latency. SPARK's internalized judge enables test-time self-reflection with a single model: the model generates an answer, invokes its internal judge via a prompt (a second forward pass through the same model), and revises if needed. The paper's TTS results (Table 5) show that SPARK-VL-7B with TTS achieves 58.8% average accuracy vs. 55.2% for the Policy-Only GRPO baseline without TTS—a +3.6 percentage point gain at the cost of at most 2 additional forward passes per question. For applications where accuracy matters more than latency (batch evaluation, offline data processing, high-stakes decision support), this represents a meaningful accuracy improvement with no additional model storage or serving infrastructure. The caveat is that the paper does not characterize TTS reliability (false-positive rate, false-negative rate), so practitioners would need to evaluate whether the self-judgment precision is adequate for their specific accuracy requirements before deploying without ground-truth oversight.

Data generation for self-improvement loops with built-in quality filtering. In self-improvement pipelines (STaR, ReST $^{\text{EM}}$ , rejection sampling fine-tuning), models generate training data for themselves and the quality of this data determines the effectiveness of subsequent training rounds. SPARK's internal judge provides a learned quality filter that can assess generated solutions without requiring ground-truth answers. For self-improvement on tasks where verification is eventually possible but expensive (e.g., code generation where unit tests exist but running them at scale is costly; mathematical proofs where automated verification is possible but slow), the internal judge could serve as a fast pre-filter: generate $n$ candidate solutions, use the internal judge to rank them or filter out those judged incorrect, and only run the expensive verifier on the top candidates. The paper's judgment accuracy analysis (Figure 4) shows that SPARK's self-judgments are substantially more accurate than the baseline model's, though the absolute precision and recall are not reported—a practitioner would need to calibrate this for their specific domain. The key advantage is that the judge is the same model generating the solutions, so no additional serving infrastructure is needed for quality filtering, and the judge is always on-policy with respect to the current generator.

Unified policy-and-evaluator deployment for interactive tutoring or assistive systems. In applications where a model must both solve problems and explain to a user why a solution is correct or incorrect (interactive math tutoring, code review assistants, scientific reasoning support), SPARK's unified architecture is particularly valuable. Rather than running a separate model to evaluate user submissions or to verify the system's own reasoning, the same model that generates solutions can also judge them and produce explanatory feedback. The paper's reflection training (Section 3.2) explicitly teaches the model to not just identify errors but to produce correct alternatives, which is precisely the capability needed for a tutoring system that says "that's not quite right—here's why, and here's the correct approach." The case study in Figure 3 demonstrates this integrated capability: the model identifies its own error ("The given answer is 54 cm², which is incorrect"), provides reasoning for the judgment, and produces a corrected solution. The paper's general benchmark results (Table 4, average +1.5%) suggest that some of this evaluation capability transfers to non-math domains, though the transfer is weak enough that domain-specific training would likely be needed for tutoring applications outside mathematics.

When to Prefer This Method

The paper explicitly positions SPARK against reward-model-based RL (Table 7 and the cost analysis narrative in Section 4.5), articulating a clear tradeoff: SPARK eliminates the need for separate reward model training and human preference data, but is currently demonstrated only for verifiable-reward tasks. The decision rule implied by the paper's results is:

Prefer SPARK over RM-based RL when:

Your task domain has ground-truth verifiable answers (math, code, logic puzzles, factual QA with answer keys), because SPARK uses these to generate auxiliary training data for free.
You lack access to human preference data or teacher-model annotations for reward model training, because SPARK requires none.
Your deployment environment has GPU memory constraints that make running a separate reward model infeasible, because SPARK's internal judge runs as a second forward pass through the same model with no additional model loading.
You need both generation and evaluation capabilities from a single model (e.g., interactive tutoring, self-improvement pipelines), because SPARK trains both simultaneously rather than requiring separate models for each.
Your base model is from the Qwen family and at a scale of 7B–32B parameters, since these are the only configurations empirically validated in the paper.

Prefer RM-based RL over SPARK when:

Your task domain lacks verifiable rewards and requires alignment on subjective dimensions (helpfulness, safety, stylistic quality), since SPARK's auxiliary training depends on objective correctness labels and its generalization to subjective judgment is untested.
Your deployment scenario can accommodate a separate reward model and you prioritize maximizing evaluation accuracy over architectural simplicity, because the paper does not compare SPARK's internal judge against a dedicated reward model trained on the same data—the specialized model might achieve higher judgment accuracy.
You are using a non-Qwen model family, since SPARK's effectiveness has not been demonstrated on other architectures.
You need strong generalization of evaluation capability to domains far from the training distribution, because the paper's general benchmark gains (+1.5% average in Table 4) are substantially weaker than in-domain gains (+9.7% on math, +12.1% on reward benchmarks), suggesting limited transfer.

Prefer standard RLVR (GRPO without auxiliary training) over SPARK when:

Your sole objective is to maximize reasoning accuracy on verifiable tasks and you do not need evaluation or self-reflection capabilities, because the auxiliary training adds computational overhead that may not be justified if only generation quality matters. The paper does not quantify this overhead, making it impossible to determine whether the +2.0% gain from Policy&Reward over Policy-Only (Table 1) is cost-effective.
You are resource-constrained and cannot afford the additional forward/backward passes for auxiliary training, regardless of the eventual benefit—standard GRPO has a simpler training loop with fewer hyperparameters to tune.