CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning

ArXiv: 2509.22647

🎯 Pitch

Training image captioning models with a vision-free LLM as a verifiable judge—rewarding captions based on how well they let a separate model answer questions about the image—outperforms standard supervised fine-tuning and even matches a 72B-parameter competitor, all while resisting the reward hacking that crippled every alternative scoring method tried.

---

1. Executive Summary

This paper introduces Captioning Reinforcement Learning (CapRL), a novel training framework that applies Reinforcement Learning with Verifiable Rewards (RLVR) to the subjective image captioning task by redefining caption quality through its utility — specifically, whether a vision-free LLM can accurately answer Multiple-Choice Questions about the image based solely on the generated caption (a decoupled two-stage VQA pipeline). Using Qwen2.5-VL-3B as the policy model trained with GRPO on a curated 75k-image QA dataset, CapRL-3B achieves performance comparable to Qwen2.5-VL-72B in the Prism Framework for caption quality evaluation while outperforming the baseline by an average margin of 8.4%, and pretraining on the CapRL-5M dataset annotated by CapRL-3B yields substantial gains across 12 benchmarks. The results establish that objective, verifiable rewards from decoupled VQA can effectively drive dense and accurate caption generation, but only when the reward signal resists the exploitation — such as verbosity bias or brevity collapse — that plagues LVLM-as-a-judge and unified reward model alternatives.

2. Context and Motivation

The Core Problem: We Can't Scale Image Captioning Training Without Ground-Truth Labels

The fundamental tension this paper addresses is one that has haunted image captioning since its inception: how do you train a model to produce high-quality image descriptions without requiring a matching high-quality description for every training image? This is not merely an academic concern — it's the central bottleneck preventing image captioning from scaling the way language modeling has scaled.

The problem arises from a structural mismatch between the task and the dominant training paradigm. Image captioning is inherently open-ended: a single image can be accurately described in countless valid ways, each emphasizing different objects, attributes, spatial relationships, or stylistic choices. There is no single "correct" caption. Yet the standard training approach — Supervised Fine-Tuning (SFT) — inherently assumes there is one. SFT trains models to reproduce a specific target text for each input image, typically by maximizing the likelihood of that exact token sequence.

The paper identifies three cascading consequences of this mismatch (Section 1):

1. Data acquisition is expensive and non-scalable. High-quality dense captions require either human annotators (slow, costly, hard to scale to millions of images) or proprietary models like GPT-4V (which incur API costs and raise reproducibility concerns). The paper states this directly:

"SFT requires large datasets annotated by humans or proprietary models, which are expensive and not scalable."

This creates a ceiling: you can only train on as many captions as you can afford to produce or purchase. For the community building open-source LVLMs, this is a persistent barrier.

2. Models memorize rather than generalize. When an SFT-trained captioning model is exposed only to single ground-truth descriptions per image during training, it learns to reproduce those specific phrasings rather than developing a robust understanding of the concepts that make any caption good. The paper is explicit about this mechanism:

"Since SFT models are trained to match a single ground-truth description for each image, they tend to memorize specific answers rather than learning the underlying concepts. As a result, the SFT models become less general and struggle to generate the diverse range of valid captions possible for a single image."

This is a specific instantiation of a broader phenomenon the paper cites from Chu et al. (2025): "SFT Memorizes, RL Generalizes." The SFT paradigm's reliance on exact token-level imitation creates brittle models that have not learned the deeper objective of informativeness — they've only learned to satisfy the teacher distribution.

3. The field needs scalable training for LVLM pretraining. Captions are not just a standalone task — they are a critical ingredient in the pretraining pipeline for Large Vision-Language Models. Work like CLIP (Radford et al., 2021) and modern LVLMs (Liu et al., 2023b) rely on massive image-text datasets for modality alignment. If the captions used in pretraining are sparse, inaccurate, or monotonous, the resulting LVLM inherits those limitations in its cross-modal understanding. The paper positions better captioning as a lever that amplifies all downstream multimodal capabilities, not just caption generation itself.

The Paradigm Shift That Hasn't Reached Captioning: RLVR

The paper places itself at the intersection of two trends that have developed largely in parallel:

Trend 1: RLVR has transformed LLM training for objective tasks. The Reinforcement Learning with Verifiable Rewards (RLVR) paradigm (Lambert et al., 2024) has exploded in the past two years as the way to post-train LLMs for tasks with unambiguous correctness signals. DeepSeek-R1 (Guo et al., 2025) used binary math-answer verification as a reward to incentivize chain-of-thought reasoning. DeepSeekMath (Shao et al., 2024) used the same for mathematical problem-solving. Code generation models (Team et al., 2025) use unit test pass/fail signals. The key insight unifying these successes: when you can objectively verify correctness, you can let the model explore diverse solution paths and reinforce whatever works, rather than forcing it to imitate a single reference solution.

The paper contrasts this sharply with SFT:

"Unlike SFT, which teaches a model to mimic a single ground-truth response, RLVR encourages the model to generate more diverse and robust outputs that meet the verifiable criteria."

Trend 2: Image captioning has remained stuck in SFT or complex multi-model pipelines. Modern captioning models — the paper cites FuseCap (Rotstein et al., 2024), FastVLM (Vasu et al., 2025), and the ShareGPT4V family (Chen et al., 2024b) — are all trained with SFT on LVLM-generated or human-annotated captions. Other approaches use expensive multi-expert ensembles (Li et al., 2024b; Sun et al., 2024) that are costly at inference time. None of them apply RLVR.

The gap is clear and puzzling: RLVR has revolutionized post-training for reasoning, math, and code, but has not been successfully applied to open-ended vision-language tasks like captioning. The paper identifies this as the specific gap it aims to fill:

"We contribute the first study of applying Reinforcement Learning with Verifiable Rewards for the open-ended and subjective image captioning task."

Why Hasn't RLVR Been Applied to Captioning Before? The Reward Design Problem

The paper's central motivation is that applying RLVR to captioning is fundamentally harder than applying it to math or code, but the difficulty is not in the RL optimization — it's in the reward design. The paper makes this explicit in Section 1:

"A primary challenge, however, is designing an objective reward function for the inherently subjective nature of what constitutes a 'good' caption."

To understand why this is hard, consider what makes reward design straightforward in math and code RLVR:

• Math: The reward is binary — does the final answer match the ground truth? The answer "42" is either correct or it isn't. The verification is exact string matching after extraction from the model's output.
• Code: The reward is whether the generated code passes a set of unit tests. Again, binary and objective.

Captioning has none of these properties. There is no single correct answer to match against. "A red fox holding a popsicle on a sunny day" and "On the left, there is a tall red fox wearing a light yellow short-sleeved shirt and a dark green tie" are both valid descriptions of different aspects of the same image, and neither is inherently "more correct." The paper emphasizes:

"A good caption can be subjective, with multiple valid descriptions possible for the same image."

This subjectivity breaks the standard RLVR recipe. You cannot simply define a binary correctness function the way you can for math problems. The entire challenge of the paper is constructing an objective reward signal for a subjective task, and this is where prior attempts have failed.

Where Prior Reward Design Attempts Fall Short

The paper is unusually thorough in characterizing the failure modes of existing approaches to subjective reward design, making it clear why none of them work and what specific pathological behaviors they induce. Figure 1(a) and the accompanying analysis in Section 4.2 catalog three families of failure:

Failure Mode 1: LVLM-as-a-Judge — Verbosity Bias and Reward Hacking

A natural instinct when faced with a subjective evaluation problem is to use a larger, more capable LVLM as a judge. The idea is simple: show the judge model the image and the generated caption, ask it to rate the caption quality (e.g., on a 1–5 scale), and use that rating as the RL reward. The paper tests this using Qwen2.5-VL-3B as the judge model during GRPO training.

The problem, as the paper documents, is that the judge model has intrinsic biases that the policy model learns to exploit. Specifically, Qwen2.5-VL-3B-as-a-judge exhibits a verbosity preference — it assigns higher scores to longer captions, regardless of whether the extra length adds genuine visual information. The policy model, optimizing for reward, quickly discovers this:

"Using Qwen2.5-VL-3B as the judge, the bias is in the opposite direction: it prefers overly verbose captions. This makes the policy model prone to exploiting the bias by generating long passages of content irrelevant to the image."

The qualitative examples in the appendix (Figures 10 and 11) are striking. In Figure 11, the model trained with Qwen2.5-VL-as-a-Judge produces a caption that begins with reasonable chart data but then degenerates into a bizarre meta-commentary: "This level of detail also enables other AI systems to process and analyze the information accurately... The description is both factual and insightful, providing a robust basis for any further inquiries..." The model has learned to pad its output with self-praise and irrelevant exposition because the judge model rewards length, not accuracy. The paper labels this as reward hacking: the model discovers a shortcut to high reward that circumvents the intended objective of producing informative captions.

The training curve for this approach (Figure 1(c), left panel) shows the reward increasing — because the judge keeps giving high scores to long outputs — but this does not translate to better caption quality as measured by downstream benchmarks. The optimization is misaligned.

Failure Mode 2: Unified Reward Models — Brevity Collapse

An alternative approach is to use a dedicated reward model trained to evaluate multimodal outputs. The paper tests UnifiedReward-2.0-qwen-3b (Wang et al., 2025), a model specifically trained for multimodal understanding and generation evaluation. The expectation is that a purpose-built reward model would be more calibrated than a general LVLM.

The paper finds the opposite. UnifiedReward-2.0-qwen-3b exhibits a brevity bias — it prefers short captions that only mention the most salient objects, likely because its training data included captions from text-to-image datasets that are typically brief. The consequence is catastrophic:

"The model's captioning ability actually deteriorates during GRPO training... the average caption length during training continuously decreases and eventually collapses to producing only ':description'."

This is what the paper calls brevity collapse: the policy model optimizes for the reward so aggressively that it learns that a near-empty caption (just the word "description" preceded by a colon) maximizes the reward-to-length ratio. The caption becomes useless for conveying visual information, but the reward model is satisfied. This is a complete failure of the reward signal — the optimization works perfectly in terms of maximizing reward, but the reward itself is measuring the wrong thing.

Failure Mode 3: Reference-Based Metrics (ROUGE, BLEU) — Insufficient for Complex Captions

The paper briefly acknowledges reference-based evaluation metrics like ROUGE (Lin, 2004) and BLEU (Papineni et al., 2002), which have been used in prior RL-for-captioning attempts (Gurung & Lapata, 2025; Yu et al., 2025). These metrics compare generated captions to human-written references by measuring n-gram overlap. The paper dismisses them because:

"Using reference answer as rewards like ROUGE and BLEU is constrained when evaluating complex and long-form captions."

This is a known limitation: n-gram overlap penalizes valid paraphrases, fails to capture semantic equivalence, and cannot assess factual accuracy beyond surface-level word matching. For the dense, detailed captions the paper aims to produce — captions that might include specific numerical values from charts, precise spatial relationships between objects, or fine-grained attribute descriptions — ROUGE and BLEU are essentially blind to whether the information is correct. A caption could achieve high ROUGE by getting the structure right while hallucinating all the numbers, which is a common failure mode the paper documents for baseline models (e.g., Figure 7, where Qwen2.5-VL-3B fabricates athlete names and scores).

The Deeper Issue: All Subjective Rewards Are Gameable

The paper's analysis of these failure modes points to a deeper pattern. All three approaches — LVLM-as-a-judge, unified reward models, reference-based metrics — attempt to provide a subjective reward for a subjective task. They evaluate caption quality by having some model or metric make a holistic judgment about the caption's goodness. The paper's key insight is that this is fundamentally unstable: any model with biases (and all models have biases) creates an exploitable signal that the policy model will discover and amplify through RL optimization.

The paper captures this in Figure 1(a), where both the "Unified Reward Model" and "General LVLM" paths lead to the same destination — "Reward Hacking" — just through different routes (brevity for the former, verbosity for the latter). The lesson is not that some reward models are better than others, but that the entire category of subjective holistic evaluation is vulnerable to exploitation when used as an RL training signal.

How CapRL Positions Itself: Objective Reward via Decoupled Utility

Against this backdrop of failed approaches, the paper proposes a radically different philosophy: don't judge the caption directly — measure what the caption enables. The key conceptual move is to define caption quality through its utility rather than its intrinsic properties:

"We introduce a novel perspective, where a caption's quality is proportional to its utility. When the image caption is detailed and accurate, a text-based LLM that can't directly 'see' the image can still answer Visual Question Answering (VQA) questions about the image."

This transforms the problem from subjective evaluation to objective measurement. Instead of asking "Is this a good caption?" (a question with no ground truth), the paper asks "Can this caption enable an LLM to correctly answer questions about the image?" (a question with a binary, verifiable answer for each question). The elegance of this construction is that it converts the entire caption evaluation problem into a series of multiple-choice question-answering tasks where correctness is determined by exact string matching against known ground-truth answers.

The paper's decoupled two-stage pipeline (Figure 1(b), Figure 3) enforces this separation:

1. Stage 1 (Captioning): An LVLM (the policy model) generates a caption for an image.
2. Stage 2 (VQA evaluation): A completely separate, vision-free LLM receives only the caption text — not the image — and answers Multiple-Choice Questions about the image. Its accuracy becomes the reward.

This decoupling is deliberate and important. The LLM in Stage 2 cannot "see" the image, so its ability to answer correctly depends entirely on whether the caption contains the necessary information. If the caption omits a key object, gets a number wrong, or hallucinates a detail, the LLM will answer incorrectly, and the reward will be low. If the caption is comprehensive and accurate, the LLM can extract the relevant facts and answer correctly, earning a high reward.

The paper's framing of this as a verifiable reward is precise: the reward is computed by an objective process (VQA accuracy) that does not involve any model making a subjective judgment about caption quality. The LLM answering the questions has no idea it's being used to evaluate a caption — it's simply performing the task it was trained for (multiple-choice QA given a text context). The paper can therefore claim:

"This question-answering process can be formulated as: $a_m = M_L(c_i, q_m)$ ... the accuracy of the LLM's responses can therefore serve as a reliable indicator of caption quality."

The QA Curation Challenge

The paper also identifies a critical requirement that makes this approach non-trivial: the QA pairs used for reward computation must be strictly visually-grounded. If a question can be answered correctly without seeing the image — for example, "What is the capital of France?" paired with an image of the Eiffel Tower — then the LLM can answer correctly regardless of caption quality, creating a spurious reward signal.

The paper addresses this with its QA curation pipeline (Section 3.1, Figure 3(b)), which is motivated by the need to guarantee that:

"all questions are strictly visually-grounded and answerable exclusively through analysis of the image content."

The three-stage process — image collection from diverse sources, automated QA generation with Qwen2.5-VL-72B, and a stringent filtering step (Equation 4) that verifies each question requires the image to answer — is necessary infrastructure for the reward signal to be reliable. The filtering step uses Qwen2.5-VL-3B as a cheaper verifier: for each candidate QA pair, it checks whether the model can answer correctly when shown the image (it should) and whether it can answer correctly without the image (it should not). Only pairs satisfying both conditions are retained.

Connection to Broader Trends

The paper positions itself within two converging research narratives:

1. The shift from SFT to RL for post-training. The paper cites the growing body of evidence (Chu et al., 2025) that SFT encourages memorization while RL encourages generalization. This is not just a claim about captioning — it's a claim about the nature of these training paradigms. SFT optimizes for token-level imitation, which is fundamentally about reproducing surface patterns. RL, when equipped with a meaningful reward, optimizes for outcomes, which can be achieved through diverse surface realizations. This is the theoretical motivation for why RLVR should produce captions that are "more creative and more general" than SFT captions.

2. The growing importance of caption quality for multimodal pretraining. The paper notes that captions are "a core component in the pre-training stage of Large Vision-Language Models" (Section 1), citing work like CLIP and LLaVA-style training. The implication is that improvements in captioning models have a multiplier effect: better captions → better pretraining data → better downstream LVLMs. This frames CapRL not just as a captioning method but as infrastructure for the broader multimodal AI ecosystem.

Summary of the Gap and the Paper's Response

To synthesize: the paper identifies a specific, high-impact gap — RLVR has not been successfully applied to subjective vision-language tasks because no one has designed a reliable objective reward signal for caption quality — and proposes a specific solution — use decoupled VQA accuracy as a verifiable proxy for caption utility, with rigorous QA curation to ensure the signal is clean. The failure modes of alternative approaches (verbosity bias, brevity collapse, n-gram blindness) are documented not as incidental observations but as evidence that the problem of reward design is the critical bottleneck, and that CapRL's utility-based philosophy is the first approach to navigate it successfully.

3. Technical Approach

3.1 Reader orientation

The CapRL system is a training pipeline that teaches a vision-language model (LVLM) to produce dense, accurate image captions using reinforcement learning — specifically, the model generates candidate captions for an image, and a separate text-only LLM attempts to answer Multiple-Choice Questions about that image by reading only the caption, with the LLM's answer accuracy becoming the reward signal that updates the LVLM. The problem it solves is the absence of any reliable, objective reward function for the inherently subjective task of image captioning — previous attempts using reward models or LVLM-as-judge approaches produce trivial exploitable signals (verbosity bias, brevity collapse) — and the "shape" of the solution is a decoupled two-stage architecture where caption quality is never judged directly but instead measured through its downstream utility: whether a caption contains enough accurate visual information to serve as a substitute for the image itself in a VQA task.

3.2 Big-picture architecture

The CapRL framework consists of five interconnected components that operate in a closed training loop:

1. The Policy Model (MV): A vision-language model (initialized as Qwen2.5-VL-3B) that receives an image and an instruction ("Describe this image in detail") and generates a candidate caption. This is the model being trained — the one whose captioning capability CapRL aims to improve.

2. The Curated VQA Dataset: A pre-built collection of approximately 75k images, each paired with multiple Multiple-Choice Questions whose answers can only be determined by examining the image (not from external knowledge or question wording). This dataset is fixed before training begins and serves as the ground-truth source for reward computation.

3. The Answering LLM (ML): A separate, vision-free language model (default: Qwen2.5-3B-Instruct) that receives a caption and a Multiple-Choice Question as text input and produces an answer. Critically, this model never sees the image — its ability to answer correctly depends entirely on whether the caption contains the relevant information.

4. The Reward Computation Module: For each candidate caption, this module samples N Multiple-Choice Questions associated with the source image, presents each to the Answering LLM (with shuffled answer options), checks whether the LLM's answer matches the ground truth via exact matching, and computes the average accuracy as the caption's reward score.

5. The GRPO Optimizer: The Group Relative Policy Optimization algorithm that takes a group of candidate captions (all generated for the same image), computes advantages from their reward scores (using the group mean and variance), adds a KL-divergence penalty against a reference model to maintain training stability, and updates the Policy Model via policy gradient optimization.

The information flow is: Image + Instruction → Policy Model → Group of Candidate Captions → (paired with) Curated MCQs → Answering LLM → Binary Correctness Scores → Averaging → Reward → GRPO Advantage Computation → Policy Gradient Update. The Answering LLM and the VQA Dataset are frozen throughout training — only the Policy Model's parameters are updated.

3.3 Roadmap for the deep dive

• First, the GRPO training framework — how the policy model generates captions, how groups of captions are compared, and how the KL-constrained policy gradient update works — because this is the optimization backbone that the reward signal plugs into, and understanding it clarifies what the reward function needs to provide.
• Second, the reward design — the core innovation: how decoupled VQA accuracy is computed, why the multiple-choice format is essential, why answer options are shuffled, and why multiple sampling rounds are needed — because this is what distinguishes CapRL from all prior approaches and is the mechanism that makes the subjective task objective.
• Third, the QA curation pipeline — how the 75k-image VQA dataset is constructed, generated, and filtered to ensure questions are strictly visually-grounded — because the reliability of the entire reward signal depends on the quality of this data, and the paper's filtering equation (Equation 4) encodes the specific leakage-prevention criterion.
• Fourth, the CapRL-5M dataset construction — how the trained CapRL-3B model is used as a captioner to annotate 5 million images, including image sourcing, safety filtering, and the rationale for using the 3B model — because this demonstrates the practical scalability of the approach and connects the training method to its downstream pretraining applications.

3.4 Detailed, sentence-based technical breakdown

This is primarily a methodology + empirical validation paper whose core idea is that an objective, verifiable reward signal for the subjective image captioning task can be constructed by measuring the downstream utility of captions — specifically, their ability to enable a separate, vision-free language model to correctly answer Multiple-Choice Questions about the source image — and that training an LVLM with this reward via GRPO produces captions that are substantially more informative than those from SFT-trained baselines or from models trained with subjective judge-based rewards.

---

GRPO Training Framework

The paper adopts Group Relative Policy Optimization (GRPO) as the reinforcement learning algorithm, following the same framework used in DeepSeek-R1 (Guo et al., 2025). GRPO is a policy-gradient method designed for language model training that eliminates the need for a separate value function (critic) by using the group of candidate outputs generated for the same input to estimate the baseline for advantage computation.

Training loop structure. For each training step, the process unfolds as follows. First, an image $I$ is sampled from the curated VQA dataset along with a captioning instruction (the paper uses "Describe this image in detail" as the default prompt, though exact prompt text is not exhaustively specified). Second, the policy model — denoted $M_V$ (V for vision) and initialized from Qwen2.5-VL-3B — generates a group of $G$ candidate captions $\{c_1, c_2, \ldots, c_G\}$ for this same image, using stochastic sampling (the paper does not explicitly specify the sampling temperature for caption generation, but GRPO typically uses non-zero temperature to ensure diversity within the group). Third, each caption $c_i$ is evaluated through the reward computation pipeline (described in the next subsection) to produce a scalar reward $R_{c_i}$. Fourth, the mean and standard deviation of rewards across the group are computed:

$\mu_R = \frac{1}{G} \sum_{i=1}^{G} R_{c_i}, \quad \sigma_R = \sqrt{\frac{1}{G} \sum_{i=1}^{G} (R_{c_i} - \mu_R)^2}$

where $\mu_R$ is the group mean reward and $\sigma_R$ is the group standard deviation, both computed over the $G$ candidate captions generated for the same input image.

What this computes: the average and spread of caption quality within the group. Because all captions are generated for the same image with different sampling noise, the mean represents the expected quality of the model's current policy on this image, and the standard deviation captures how much quality varies across samples. The subscript $R$ indicates these are reward statistics (as opposed to policy statistics).

Why compute group statistics: the key insight of GRPO is that comparing each caption's reward to the group mean — rather than to an absolute threshold — provides a relative advantage signal that is invariant to the inherent difficulty of the image. If an image is inherently hard (e.g., a complex infographic), all captions will receive low rewards, but the best caption in the group still deserves positive reinforcement relative to the others. Conversely, if an image is easy, all rewards will be high, but the worst caption should still receive negative reinforcement. The group normalization via $\mu_R$ and $\sigma_R$ automatically adapts the advantage scale to per-image difficulty without requiring a separately trained value function.

Advantage computation. For each caption $c_i$, the advantage $A_i$ is computed as the standardized reward:

$A_i = \frac{R_{c_i} - \mu_R}{\sigma_R}$

where $R_{c_i}$ is the reward for caption $c_i$, $\mu_R$ is the group mean, and $\sigma_R$ is the group standard deviation (with potential small epsilon added for numerical stability, though not explicitly stated in the paper).

What this computes: how many standard deviations above or below the group average a particular caption's reward falls. A positive advantage means the caption was better than the group average; a negative advantage means it was worse. The advantage is a zero-centered, unit-variance signal that feeds into the policy gradient update.

Why standardization: the policy gradient update magnitude is proportional to the advantage. If the raw reward range varies across images (e.g., hard images might have rewards between 0 and 0.2, easy images between 0.8 and 1.0), standardization ensures that the gradient step size is comparable across images and that the optimizer does not over-fit to easy images simply because they produce larger raw reward differences. This is standard practice in GRPO and is consistent with the DeepSeek-R1 implementation.

KL-divergence penalty. To prevent the policy model from diverging too far from its initial behavior (which can cause language degradation or catastrophic forgetting), GRPO incorporates a Kullback-Leibler (KL) divergence penalty between the current policy and a frozen reference model (typically the initial checkpoint before RL training). The paper states:

"To ensure training stability, and consistent with the original GRPO framework, we incorporate a KL-divergence penalty."

The KL penalty is added to the reward during the policy gradient update, effectively penalizing the model for producing token distributions that deviate significantly from the reference model's distributions. The exact weighting coefficient for the KL penalty is not specified in the paper (common values in the GRPO literature range from 0.01 to 0.1). The reference model is initialized from the same Qwen2.5-VL-3B checkpoint and remains frozen throughout training.

Why KL-divergence is necessary: without this penalty, the policy model can rapidly drift into degenerate regions of output space — for example, memorizing the exact phrasing that maximizes VQA accuracy for specific images but losing general language coherence. The KL penalty acts as a regularizer that keeps the policy "close" to the pretrained language distribution, ensuring that the captions remain natural language rather than degenerating into keyword lists or other exploitative formats. This is particularly important for CapRL because the reward signal comes from a separate LLM that might have different linguistic preferences than the LVLM — without the KL penalty, the LVLM could learn to produce captions that are optimized for the answering LLM's reading patterns rather than for general human readability.

Policy gradient update. The policy model's parameters are updated using standard policy gradient optimization. The objective is to maximize the expected reward (including the KL penalty) with respect to the policy parameters. The paper does not provide the explicit policy gradient loss equation, but the standard GRPO formulation maximizes:

$\mathcal{J} = \mathbb{E}_{c \sim \pi_\theta} \left[ A(c) - \beta \cdot D_{KL}(\pi_\theta \| \pi_{\text{ref}}) \right]$

where $\pi_\theta$ is the current policy, $\pi_{\text{ref}}$ is the frozen reference model, $A(c)$ is the standardized advantage for caption $c$, and $\beta$ is the KL penalty coefficient (value not specified in the paper).

What this computes: the expected advantage-weighted log-probability of the generated tokens, minus a penalty proportional to how much the current policy's output distribution differs from the reference policy's distribution. The expectation is over captions sampled from the current policy, and the advantage $A(c)$ determines both the direction and magnitude of the update for each token in the caption — tokens from high-advantage captions get their log-probabilities increased, tokens from low-advantage captions get theirs decreased.

Why this form: the advantage-weighted policy gradient is the standard REINFORCE-style estimator with a learned baseline (here, the group mean). The subtraction of $\mu_R$ in the advantage computation serves as the baseline, reducing variance compared to using raw rewards. The division by $\sigma_R$ provides adaptive step-size scaling. The KL penalty term ensures that the policy does not deviate into regions where the reward signal may be unreliable (because the answering LLM was never trained on such captions) or where language quality degrades.

Default training configuration. The paper does not provide an exhaustive GRPO hyperparameter table, but specifies: the policy model is initialized from Qwen2.5-VL-3B, the answering LLM $M_L$ defaults to Qwen2.5-3B-Instruct, and the group size $G$ (number of candidate captions per image) is not explicitly stated but typically ranges from 4 to 16 in GRPO implementations. The training uses the curated VQA dataset of approximately 75k images with their associated MCQs.

Critical distinction from DeepSeek-R1's GRPO usage. The paper explicitly notes that CapRL differs from reasoning-focused GRPO in an important way:

"Since we compute the caption reward directly from the original caption, there is no need to perform intermediate reasoning steps as in DeepSeek-R1, which first carries out a thinking process before formatting an answer. As a result, our method avoids the need for any format-specific rewards and retains a clean, flexible reward computation process that fully respects the free-form nature of the policy model's output."

In DeepSeek-R1, the model must produce a structured output (e.g., think tags followed by a final answer box), and format-compliance rewards are needed to enforce this structure. CapRL has no such requirement — the caption is the entire output, and the reward computation operates on the caption text as-is. This is a simplification that reduces the number of hyperparameters and eliminates format-reward tuning, making the RL setup cleaner.

---

Reward Design

The reward function is the central innovation of CapRL. The paper's key insight is to convert subjective caption evaluation into objective multiple-choice question answering accuracy, creating a reward signal that is binary, verifiable, and resistant to the reward hacking that plagues judge-based approaches.

The reward computation pipeline. Given a candidate caption $c_i$ generated by the policy model for image $I$, the reward is computed through a multi-step process:

Step 1: Retrieve associated questions. Image $I$ has $M$ associated Multiple-Choice Questions in the curated VQA dataset, denoted $\{q_1, q_2, \ldots, q_M\}$. Each question $q_m$ has a ground-truth answer $GT_m$ and a set of answer options (typically four options labeled A through D, with an occasional "E) Cannot answer based on the caption" option, as shown in the prompt in Figure 15).

Step 2: Answer a question using the caption. For a specific question $q_m$, the answering LLM $M_L$ receives the caption text and the question text (with answer options) as input and produces an answer:

$a_m = M_L(c_i, q_m)$

where $M_L$ is the frozen answering LLM (default: Qwen2.5-3B-Instruct), $c_i$ is the candidate caption text, $q_m$ is the m-th question (including all answer options), and $a_m$ is the LLM's selected answer (e.g., "A", "B", "C", "D").

What this computes: a single inference call to the answering LLM, producing a discrete answer choice. The LLM never sees the image — it only reads the caption and the question. This is the decoupling that makes the reward objective: if the caption is missing information about, say, the color of a specific object, the LLM will be forced to guess or answer incorrectly because it has no other source of that information.

Why the LLM is frozen: the answering LLM is never updated during CapRL training. This is crucial because updating it would change the reward function over time, making the optimization a moving target. A frozen LLM provides a stationary reward landscape, which is necessary for stable RL training. The paper's choice of Qwen2.5-3B-Instruct is motivated by both efficiency (3B parameters is relatively lightweight for the answering step) and stability (instruction-tuned models tend to follow the MCQ format reliably).

Step 3: Compute binary correctness. The LLM's answer is compared to the ground truth using exact matching:

$r(a_m) = \begin{cases} 1, & \text{if } a_m = GT_m \\ 0, & \text{otherwise} \end{cases}$

where $r(a_m)$ is the reward for this single question-answer attempt, and $GT_m$ is the ground-truth answer to question $q_m$.

What this computes: a binary signal — either the LLM answered correctly (1) or incorrectly (0). There is no partial credit, no graded evaluation. This is what makes the reward "verifiable" in the RLVR sense: correctness is determined by exact string matching against a known answer, with no model judgment involved.

Why binary reward matters: the binary nature of the reward prevents the rich-get-richer dynamics that drive reward hacking in continuous reward models. A reward model that assigns scores on a 1–5 scale can be exploited by the policy model generating outputs that trigger the reward model's biases (e.g., longer outputs get higher scores regardless of content). A binary reward, by contrast, has no gradation to exploit — a caption either enables the LLM to answer a specific question correctly or it doesn't. The only way to increase reward is to actually include the information needed to answer questions correctly.

Step 4: Option shuffling and multiple sampling. The paper identifies an important subtlety: LLMs can exhibit biases toward specific answer option labels (e.g., a tendency to prefer option "A" or "C" regardless of content). To eliminate this bias from the reward signal, the paper applies two techniques:

Option shuffling: each time a question is presented to the LLM, the order of the answer options is randomly permuted. For example, if the original question has options "A) Red, B) Blue, C) Green, D) Yellow", the shuffled version might present them as "A) Green, B) Red, C) Yellow, D) Blue" with the ground truth adjusted accordingly. This means the LLM cannot exploit any positional preference — it must read the option content to select the correct answer.

Multiple sampling rounds: rather than evaluating the caption on a single question once, the paper samples $N$ questions (with replacement) from the set of $M$ questions associated with the image, and for each sampled question, presents it to the LLM with shuffled options:

$R_{c_i} = \frac{1}{N} \sum_{k=1}^{N} r\left(M_L\left(c_i, \text{Shuffle}(q_{m_k})\right)\right), \quad m_k \sim \{1, \ldots, M\}$

where $R_{c_i}$ is the final reward for caption $c_i$, $N$ is the number of sampling rounds, $m_k$ is the index of the k-th randomly sampled question, $\text{Shuffle}(q_{m_k})$ denotes the question with its answer options randomly permuted, and $r(\cdot)$ is the binary correctness function from Step 3.

What this computes: the average accuracy of the LLM across $N$ independently sampled and shuffled question-answering attempts. If the caption contains comprehensive information, the LLM will answer most questions correctly, and the average will be close to 1. If the caption is sparse or inaccurate, the average will be closer to 0.

Why multiple sampling: relying on a single question to evaluate a caption is noisy — a caption might happen to include information about one aspect of the image but not another, and if the single sampled question happens to ask about the missing aspect, the reward would be 0 even though the caption is partially good. By averaging over $N$ questions, the reward becomes a more robust estimate of overall caption informativeness. The paper's ablation (Table 5) shows that $N = 4$ provides the best performance — lower values (particularly $N = 1$) produce unreliable rewards due to option bias and question sampling noise, while higher values ($N = 8$) provide diminishing returns because the additional samples are redundant.

Why option shuffling combined with averaging: even with multiple questions, if the options are always presented in the same order, a biased LLM might achieve inflated accuracy on certain questions by consistently picking position A regardless of content. Shuffling each time and averaging over samples decorrelates the reward from positional bias, because the correct answer will appear in different positions across samples, and a position-biased LLM would be wrong on most of them. This makes the reward a more honest measure of whether the caption truly contains the needed information.

The reward computation process is entirely decoupled from the policy model's training step. The policy model generates the caption; the answering LLM evaluates it. Neither model is updated during the other's operation. This decoupling is what the paper means by "decoupled two-stage pipeline" — the first stage (captioning) and second stage (VQA evaluation) involve completely separate models with no shared parameters or gradients.

Training efficiency consideration. The paper notes that using Qwen2.5-3B-Instruct as the answering LLM "makes the overall training highly efficient." This is because:

1. The 3B model is relatively small, so each reward computation (forward pass through the LLM) is computationally cheap.
2. The LLM only needs to generate a single token (the answer letter) or at most a few tokens — it does not need to produce long reasoning chains or explanations.
3. The reward computation can potentially be parallelized across questions within the group (though the paper does not specify whether this is done).

Why this reward design resists hacking. The paper's analysis of why this approach avoids the failure modes in Figure 1(a) is implicit but clear from the mechanism. The reward is:

• Content-dependent, not style-dependent: the LLM only cares about whether the information needed to answer the question is present in the caption text. It does not evaluate writing style, length, or eloquence. A verbose caption that adds irrelevant self-praise (as seen in Figure 11) does not help answer factual questions about chart data — the LLM will still get those wrong if the numbers are wrong.
• Verifiable and binary: there is no gradient of "sort of good" to climb via exploitation. To get a reward of 1 on a question about the color of a frisbee, the caption must contain the phrase "red frisbee" (or equivalent). A caption that says "the frisbee is beautifully colored" scores 0. The policy model cannot hack this by adding generic positive language — it must include specific, accurate details.
• Multi-dimensional: each image has multiple questions covering different aspects of the visual content. A caption that focuses exclusively on one object and ignores everything else will score well on questions about that object but poorly on questions about other objects. The averaging across questions encourages comprehensive coverage — the policy model is rewarded for including all relevant information, not just a subset.

Ablation evidence (Table 5). The paper empirically validates the sampling rounds design choice. Training with $N = 1$ (single question, no shuffling) yields an average score of 47.3% across five Prism benchmarks. Increasing to $N = 4$ improves to 48.4%. Further increasing to $N = 8$ results in 48.3%, essentially saturated. The paper explains the $N = 1$ degradation: "Due to inherent option biases in the LLM, the measured accuracy fails to serve as a reliable proxy for reward, thereby misdirecting the optimization of the policy model." This ablation demonstrates that the shuffling and averaging are not merely cosmetic — they are necessary for the reward signal to be reliable enough to guide optimization.

---

QA Curation Pipeline

The reward signal's reliability depends entirely on the quality of the VQA dataset used to compute it. If questions can be answered without the image (from external knowledge, common sense, or cues in the question text), then the LLM can achieve high accuracy even with an empty caption, and the reward signal becomes meaningless. The paper constructs this dataset through a three-stage pipeline that ensures questions are strictly visually-grounded.

Stage 1: Image Collection. The paper sources diverse images to maximize the variety of visual content the policy model encounters during training:

"We begin by sourcing diverse images from the web and existing open-source datasets, including natural scenes, charts, and documents, to maximize variety."

The inclusion of charts and documents (in addition to natural images) is deliberate: these image types require precise extraction of text, numbers, and structural information, which places high demands on caption accuracy. A caption that says "a chart showing some data" is useless for answering "What percentage of Republicans aged 65+ see Trump as fighting for their beliefs?" — the caption must include the specific number. This domain diversity is what enables CapRL to produce captions that are not just natural-language descriptions but precise information extractions, as shown in the infographic examples in Figures 6 and 13.

The paper does not specify the exact sources or total number of images collected before QA generation, but the final filtered dataset contains approximately 75k images with associated QA pairs.

Stage 2: QA Generation. For each collected image, the paper uses Qwen2.5-VL-72B (a much larger and more capable model than the policy model being trained) to automatically generate multiple-choice questions and their answers:

"For each image, we then use Qwen2.5-VL-72B to automatically generate multiple question-answer pairs."

The specific prompt used for QA generation is provided in Figure 18 of the appendix. The prompt instructs the model to:

1. Generate exactly five multiple-choice questions about the image content.
2. Make questions challenging and focused on image content.
3. Follow a strict formatting template: question text, four answer options (A through D), and an explicit "Answer:" line with the correct option.
4. Avoid generating irrelevant sentences.

The paper generates five questions per image initially, but does not enforce that all five survive the filtering stage — as noted in Appendix E, "We do not deliberately control the number of QA pairs per image, prioritizing instead the overall dataset size and diversity." After filtering, some images may retain multiple questions while others may retain only one.

Why use a 72B model for generation: the QA generator needs to be significantly more capable than the policy model to produce questions that are (a) answerable from the image, (b) genuinely require visual analysis, and (c) have unambiguous correct answers. If the generator produces ambiguous or error-containing questions, the reward signal becomes noisy. Using a model 24 times larger than the policy model (72B vs. 3B) provides a strong teacher signal.

Stage 3: QA Filtering. This is the most critical stage and the paper's key contribution to data quality assurance. The goal is to remove any question that can be answered correctly without actually examining the image — what the paper calls "information leakage" or "leaking data." The filtering criterion is formalized in Equation 4:

$Q = \{(q, a) \in D \mid M_{V_f}(q, I) = a \land M_{V_f}(q) \neq a\}$

where $D$ is the initial set of generated question-answer pairs, $(q, a)$ is a candidate QA pair, $I$ is the corresponding image, $M_{V_f}$ is the LVLM used for filtering (set to Qwen2.5-VL-3B for cost efficiency, not the 72B generator), $M_{V_f}(q, I)$ is the answer produced when the filter model sees both the question and the image, and $M_{V_f}(q)$ is the answer produced when the filter model sees only the question text without the image.

What this equation computes: the set $Q$ of retained QA pairs that satisfy two conditions simultaneously. The first condition ($M_{V_f}(q, I) = a$) verifies that the filter model can answer the question correctly when given the image — if even a model with vision access gets it wrong, the question might be ambiguous or incorrectly annotated, and it should be discarded. The second condition ($M_{V_f}(q) \neq a$) verifies that the filter model cannot answer correctly without the image — if the model can answer from the question text alone (e.g., by recognizing a well-known fact or pattern-matching the question wording), then the question leaks information and must be discarded.

Why both conditions: the first condition alone would retain questions that are answerable from common sense (e.g., "What color is the sky?" paired with an outdoor image — the model answers "blue" correctly both with and without the image, so the condition $M_{V_f}(q, I) = a$ passes but the question is not visually-grounded). The second condition alone would discard genuinely visual questions if the model guesses correctly by chance. Both conditions together ensure that the retained questions require visual analysis and have verifiably correct answers.

The $\land$ (logical AND) enforces that both must hold for a QA pair to be retained. If either condition fails, the pair is discarded.

Why use a 3B model for filtering instead of the 72B generator: the filtering step requires running inference on every candidate QA pair, potentially multiple times (the paper notes that due to temperature uncertainty, "we sample responses four times for each question, shuffling the answer options each time"). Using the 72B model for this would be computationally prohibitive. The 3B model is a cheaper proxy that can still reliably determine whether a question is answerable from text alone. The paper explicitly states this efficiency motivation:

"To manage computational costs effectively, the QA filtering step is performed using the Qwen2.5-VL-3B model as MVf."

The filtering protocol in detail (from Appendix E). The paper provides additional details on how the filtering is implemented:

1. For each question, the filter model $M_{V_f}$ generates answers under both conditions (with image and without image) four separate times, with answer options shuffled each time.
2. The accuracy across the four attempts is computed for both the image-conditioned and non-image-conditioned settings.
3. A threshold is applied to select questions where image-conditioned accuracy is high and non-image-conditioned accuracy is low. The exact threshold values are not specified numerically, but the principle is that "questions with high image-based accuracy but low question-only accuracy" are retained.

The paper also notes an important calibration point about the filtering strictness:

"It is worth noting that, because our filtering criteria are quite strict, some discarded QA pairs contain only mild or even negligible leakage. This also explains why, as shown in Figure 6 [sic — likely refers to Table 6], training with the leaked data does not cause training collapse but merely led to degraded performance."

This is a deliberate design choice: the paper errs on the side of over-filtering (discarding some borderline-usable questions) to ensure that retained questions are definitively clean. The ablation in Table 6 validates this, showing that training with filtered (clean) data achieves 48.5% average Prism score versus 47.4% for unfiltered (leaking) data — a 1.1% degradation from leakage, confirming that while collapse does not occur, signal quality matters.

Final dataset composition. After filtering, the curated dataset contains approximately 75k images with their associated QA pairs. The number of QA pairs per image varies (some images retain multiple questions, some retain only one), and the paper's ablation in Table 4 shows that even a single QA per image ("CapRL-1QA-20k") is highly effective, achieving 48.0% average Prism score versus 48.5% for three QA per image ("CapRL-3QA-20k") — a difference of only 0.5 percentage points. This demonstrates that "sparse QA supervision is sufficient for CapRL."

The MCQ format choice. The paper deliberately constructs all questions as multiple-choice rather than open-ended. This is not incidental — it is a design choice that enables the verifiable reward computation. Multiple-choice answers can be evaluated via exact string matching (comparing the LLM's output letter or text to the ground-truth answer). Open-ended answers would require either a separate judge model to evaluate correctness (reintroducing the subjectivity problem) or reference-based metrics like ROUGE (which, as discussed in Section 2, are insufficient for complex captions). The multiple-choice format makes the reward computation fully automatic, binary, and objective — exactly the properties needed for RLVR.

Handling uncertainty: "Cannot answer based on the caption." The prompt provided to the answering LLM (Figure 15 in the appendix) includes an additional option: "E) Cannot answer based on the caption." This is shown in the prompt examples, where a question about a flag's color paired with a caption that mentions a "British flag" but not its colors yields "The answer is E." This option gives the answering LLM a way to signal that the caption lacks the necessary information, rather than forcing a guess. However, the paper does not explicitly state how this "E" option is handled in the reward computation — whether it counts as incorrect (reward 0) or whether it is treated specially. From the binary reward formulation in Equation 2 ($r(a_m) = 1$ only if $a_m = GT_m$), we can infer that "E" would count as incorrect (reward 0) since it would not match the ground-truth answer (which is one of A–D). This is consistent with the design philosophy: if the caption is missing information, the model should be penalized, incentivizing more comprehensive captions.

---

CapRL-5M Dataset Construction

After training the CapRL-3B model using the GRPO framework with the curated QA dataset, the paper leverages this trained model as a high-quality captioner to construct a large-scale pretraining dataset. This demonstrates the scalability of the approach: once the model is trained, it can annotate arbitrary numbers of images at low cost.

Image sourcing. The paper draws images from three categories:

1. Open-source datasets: ShareGPT4V-1M (Chen et al., 2024b) and DenseFusion-1M (Li et al., 2024b). These datasets have already undergone extensive filtering and clustering for quality, so the paper "directly incorporated all images from them." This provides approximately 2 million images (1M from each dataset).

2. Web-crawled images: The paper collects an additional large set of images from the web, "spanning natural photographs, documents, charts, and user interfaces" to enhance diversity. The initial quantity is not specified, but after filtering, 3 million high-quality images are retained.

3. Combined: The final pool consists of 5 million images (2M from existing datasets + 3M from filtered web crawl).

Image filtering pipeline. The web-crawled images require substantial processing to ensure quality and safety. The paper describes a three-stage pipeline (Appendix D):

1. Semantic deduplication: Following the SemDeDup approach (Abbas et al., 2023), the paper constructs semantic clusters of images and removes images that are redundant (too similar to others in the cluster). This reduces dataset bloat and ensures diversity. Additionally, low-resolution and "overly simple" images are discarded, and content involving violence, pornography, or other safety concerns is filtered.

2. Benchmark leakage prevention: To avoid contaminating evaluation, the paper identifies images from commonly used evaluation benchmarks and forms clusters with them. Any web-crawled images that are "overly similar to benchmark samples" are removed. This is a critical step for fair evaluation — if CapRL-5M contained images similar to those in the downstream benchmarks, the pretraining improvements could be attributed to memorization rather than genuine caption quality.

3. Human safety verification: Human annotators perform sample-based screening of the remaining images. "Once the proportion of unsafe images falls to a negligible level, we stop filtering." This provides a final safety net against edge cases missed by automated filters.

Caption model selection. For annotating 5 million images, computational cost is a primary concern. The paper explicitly justifies using the 3B model:

"In typical multimodal pretraining scenarios, the pretraining dataset often requires a massive number of image-text pairs, making annotation costs substantial. Considering practical applications, we decide to train a highly lightweight yet powerful captioner to keep annotation costs more acceptable."

The choice of Qwen2.5-VL-3B (rather than a larger model) is motivated by throughput and cost: a 3B model can process images much faster than a 7B or 72B model, making 5M annotations feasible within reasonable compute budgets. The paper's results (Table 1, Table 3) then demonstrate that despite its small size, the CapRL-3B model produces captions of sufficient quality that pretraining on them yields substantial benefits — the CapRL training has effectively extracted more capability from the 3B architecture than SFT training does.

Annotation process. The trained CapRL-3B model processes each of the 5 million images with a standard captioning prompt (likely "Describe this image in detail" or similar) to generate a dense caption. No iterative refinement or VQA evaluation occurs during annotation — the model is used in pure inference mode. The resulting (image, caption) pairs constitute CapRL-5M.

Subset for controlled experiments. For the pretraining experiments in Section 4.1, the paper creates a 1M subset (CapRL-1M) by randomly sampling from the full 5M dataset. This enables fair comparison with ShareGPT4V-1M and DenseFusion-1M, which are also 1M-scale datasets.

Why this demonstrates scalability. The CapRL training pipeline (GRPO with decoupled VQA reward) requires the curated QA dataset, which is expensive to construct (75k images with filtered MCQs). However, once the RL-trained model is obtained, it can annotate any image — no QA pairs are needed at annotation time. This decouples the expensive RL training (which produces a strong captioner) from the cheap inference-time annotation (which produces the large dataset). The paper leverages this property to create a 5M dataset that would be prohibitively expensive to create via SFT (since SFT would require dense captions for all 5M images, likely from proprietary models or human annotators).

Training details for CapRL-3B (inference from context). The paper does not provide exhaustive hyperparameters for the GRPO training phase beyond what is discussed above, but key configuration points are:

• Policy model initial checkpoint: Qwen2.5-VL-3B
• Answering LLM: Qwen2.5-3B-Instruct
• Training dataset size: approximately 75k images with associated filtered QA pairs
• Sampling rounds $N$ for reward computation: 4 (based on Table 5 ablation showing this as optimal)
• The model is trained until convergence (the paper does not specify exact step counts or learning rates for GRPO)

Cross-validation note. Unlike some RLVR setups that require careful train/test splitting to avoid reward overfitting, CapRL's reward signal comes from the QA dataset, which is distinct from the downstream evaluation benchmarks. The paper does not need to split the QA dataset for validation because the reward is a training signal, not an evaluation metric — evaluation is done on separate benchmarks (the 12 pretraining benchmarks in Table 1 and the Prism Framework in Table 3).

4. Key Insights and Innovations

Innovation 1: Redefining Caption Quality Through Utility Rather Than Intrinsic Properties

The paper's most fundamental conceptual contribution is not a new architecture or training trick, but a reframing of what it means for a caption to be "good." Before CapRL, the dominant mental model in captioning research — whether using SFT with human references or RL with judge models — treated caption quality as an intrinsic property of the text itself: a good caption has certain stylistic, structural, or factual qualities that can be recognized by looking at the caption (and perhaps the image). This is the assumption behind every approach that asks a model to "evaluate the caption quality" — whether that model is a trained reward model (UnifiedReward), a general LVLM acting as a judge (Qwen2.5-VL-as-Judge), or a reference-based metric (ROUGE, BLEU). All of them share the premise that caption quality is something you can see in the caption.

CapRL rejects this premise entirely. Its core reframing — captured in the line "a caption's quality is proportional to its utility" (Section 1) — defines quality extrinsically, through the downstream task the caption enables. A caption is good not because it looks comprehensive or reads fluently, but because it serves as an effective substitute for the image in a visual question-answering task. This is a pragmatic, functionalist definition that bypasses the entire problem of subjective evaluation: you don't need to judge whether the caption is good; you need to measure whether it works.

What makes this reframing intellectually distinctive is that it converts an entire category of failure modes into category errors. Every attempt to use holistic subjective evaluation for RL rewards — the paper documents three distinct families (verbosity-biased LVLM judges, brevity-biased reward models, and n-gram-matched references) — failed because they were asking the wrong question. They asked "Is this caption good?" and got answers that were gameable because "good" is not a property that can be measured objectively from the caption text alone. CapRL asks "Can an LLM answer these questions using this caption?" and gets answers that are binary, verifiable, and — crucially — contingent on the information content of the caption rather than its surface form.

The significance of this reframing extends beyond captioning. It suggests a general principle for applying RLVR to subjective generation tasks: identify a downstream task that requires the generated output to contain specific, verifiable information, and use performance on that task as the reward. The paper itself hints at this generality in the introduction when it motivates the approach through the lens of "utility," but the principle could apply to summarization (can a reader answer comprehension questions?), instruction-following (can the output be executed?), or translation (can a monolingual reader recover the original meaning?). This is not a claim the paper makes explicitly, but it is the natural conceptual extension of the utility-based framing.

Evidence anchoring: The contrast between CapRL and the LVLM-as-Judge baselines in Table 3 makes this reframing concrete. Qwen2.5VL-as-Judge-3B — which uses the intrinsic-evaluation approach — achieves 42.5% average Prism score, while CapRL-3B achieves 48.3%, matching the 72B model. The failure is not that the judge model is poorly implemented; it's that the intrinsic-evaluation paradigm itself creates an exploitable signal, as demonstrated by the qualitative examples in Figures 10 and 11 where the judge-trained model produces lengthy self-praise ("This description is both factual and insightful...") that satisfies the judge's verbosity bias but contains no additional image information.

---

Innovation 2: Decoupled VQA as a Verifiable Reward That Resists Hacking by Construction

The second major conceptual contribution is the specific mechanism that operationalizes the utility-based reframing: using a completely separate, vision-free LLM as the evaluator in a decoupled two-stage pipeline. This is not merely a clever implementation detail — it embodies a design principle that directly addresses why prior reward approaches were gameable.

In prior approaches, the evaluator had access to the image. Whether it was a reward model, an LVLM judge, or a reference-based metric, the evaluation process involved comparing the generated caption against either the image itself or a reference caption that was derived from the image. This created an information asymmetry problem: the policy model (which also sees the image) and the evaluator (which sees the image) share context that the policy model can exploit. For example, if an LVLM judge has a known bias toward verbose outputs, the policy model can pad its captions with irrelevant text that sounds authoritative in the context of the image — the judge, seeing both the image and the verbose caption, is satisfied because the caption seems comprehensive even though it adds no new information beyond what a shorter caption would convey.

CapRL eliminates this asymmetry by depriving the evaluator of visual access entirely. The answering LLM in Stage 2 receives only the caption text and the question — never the image. This means that the policy model cannot exploit any shared context between itself and the evaluator, because there is no shared context beyond the caption. The only way to increase reward is to put the specific information the LLM needs into the caption text itself. There is no shortcut, no style-based exploitation, no way to trick the evaluator into thinking information is present when it isn't — the LLM either finds the answer in the text or it doesn't.

Why this is a fundamental shift rather than incremental. Prior work on RL for captioning attempted to fix the reward model — better training data, better calibration, larger judge models. The implicit assumption was that the reward model was the problem and that a better reward model would solve it. CapRL's insight is that the architecture of evaluation matters more than the quality of the evaluator. By structurally preventing the evaluator from accessing the image, CapRL makes the reward contingent on a property that cannot be faked: the presence of extractable information in the caption. This is a qualitatively different kind of robustness than what better reward model training could achieve — it's robustness by design, not by optimization.

The paper demonstrates this empirically through the failure of even purpose-built reward models. UnifiedReward-2.0-qwen-3b (Wang et al., 2025) is a model specifically trained for multimodal evaluation, yet when used as the GRPO reward, it causes brevity collapse — the policy model learns to output near-empty captions because the reward model, trained on text-to-image dataset captions, prefers short descriptions. This is not a failure of reward model quality; it's a failure of the evaluation architecture. No amount of additional training data would fix the fundamental problem that a holistic evaluator with image access has exploitable biases. CapRL's decoupled design sidesteps this entire class of failures by changing the rules of the game.

The option-shuffling detail as a microcosm of the design philosophy. Even within the decoupled VQA framework, the paper identifies a subtler form of potential exploitation: LLMs can have positional biases toward specific answer option labels (e.g., preferring "A" or "C"). Without shuffling, a policy model could potentially learn to produce captions that trigger the answering LLM's positional bias — for instance, consistently describing objects in an order that correlates with the answer key. The paper's solution — random shuffling of options combined with averaging over multiple samples (Table 5) — applies the same design principle at a finer granularity: eliminate correlations that could be exploited, even if the exploitation seems unlikely. This attention to adversarial robustness in reward design distinguishes CapRL from RLVR applications in math and code, where the reward function is naturally adversarial-robust because there is a single correct answer that cannot be gamed by output style.

Evidence anchoring (Table 5 and Figure 1(c)). The necessity of the decoupled architecture is shown by the training curves in Figure 1(c). The CapRL curve (left panel) shows stable reward improvement that translates to genuine caption quality gains. The UnifiedReward curve shows rapid reward increase followed by collapse — the model learned to hack the reward, but the reward itself was measuring the wrong thing. The Qwen2.5-VL-as-Judge curve shows increasing reward without corresponding quality improvement — the judge was satisfied, but the captions were not more informative. Only the decoupled approach produces aligned optimization, where maximizing reward and maximizing caption utility are the same objective.

---

Innovation 3: Sparse QA Supervision Is Sufficient — A New Scaling Relationship for RLVR in Perception Tasks

The third conceptual contribution is an empirical discovery with theoretical implications: that highly sparse question-answer supervision — even a single QA pair per image — is sufficient to drive substantial improvements in captioning capability through RLVR. This finding, presented in Table 4 and Figure 5 (Right), is not obvious from the RLVR literature and has significant implications for the scalability of this approach.

In the standard RLVR paradigm (math, code), the reward is computed from the same instance that generated the output: the model solves a math problem, and the reward is whether the answer matches the ground truth. Each training instance provides exactly one reward signal. One might naturally assume that for captioning, where the "correctness" is not a single answer but a multidimensional assessment of information coverage, you would need many questions per image — perhaps dozens — to provide a sufficiently dense reward signal that captures all the dimensions along which a caption can be good or bad.

CapRL shows the opposite. Table 4 compares models trained with 3 QA pairs per image (CapRL-3QA-20k, 48.5% average Prism score), 2 QA pairs (48.5%), and 1 QA pair (48.0%). The degradation from 3 QA to 1 QA is only 0.5 percentage points — a negligible cost for a 3× reduction in annotation effort. Even more striking, Figure 5 (Right) shows a clear but gentle scaling curve: performance improves from 63.1% to 70.6% on ChartQA as QA training data scales from 2.5k to 20k images, with no sign of saturation at 20k. This means the model benefits more from seeing diverse images with sparse QA supervision than from dense QA coverage on fewer images.

Why this is conceptually significant. This finding reveals an implicit structure in the relationship between captions and questions that the paper does not fully articulate but that the results imply: a well-structured caption that covers one aspect of an image well tends to also cover other aspects well. If the policy model learns — through sparse QA rewards — to produce captions that are detailed, accurate, and well-organized for the specific aspects that happen to be questioned, it generalizes this behavior to aspects that were never directly rewarded. This is a form of reward generalization that is not guaranteed by the RLVR framework and represents an emergent property of the captioning task as structured by the pretrained model's capabilities. The model already "knows" from pretraining what a comprehensive description looks like; the sparse QA rewards provide just enough signal to steer its generation toward that latent capability.

This has direct practical implications for scaling. The paper's QA curation pipeline — generating questions with a 72B model and filtering with a 3B model — is the most computationally expensive part of the CapRL framework. The finding that 1 QA per image is nearly as effective as 3 means that the annotation cost can be reduced by 3× for the same image diversity, or equivalently, that 3× more images can be covered for the same annotation budget. Given the scaling trend in Figure 5 (Right) showing continued improvement with more data, this points toward a strategy of maximizing image diversity with minimal per-image QA annotation — a counterintuitive but empirically supported allocation of annotation resources.

The contrast with dense reward in RLHF is instructive. In RLHF, the reward model is trained on preference pairs and provides a dense, continuous signal for every token of the output. This density is what makes RLHF prone to reward hacking — the model can find subtle output modifications that increase the reward score without improving actual quality. CapRL's sparse, binary, question-based reward is the opposite: it provides signal only at the level of factual correctness on specific queries, completely ignoring stylistic dimensions. The discovery that this sparse signal is sufficient suggests that for perception-to-language tasks, factual coverage is the dominant dimension of quality, and optimizing it implicitly improves other dimensions. This is a hypothesis the paper does not test directly but that its results strongly support.

Evidence anchoring (Table 4, Figure 5). Table 4: CapRL-1QA-20k achieves 48.0% average across five Prism benchmarks versus 48.5% for CapRL-3QA-20k — a 0.5% gap that is smaller than the gap between any of these and the baseline (40.6%). Figure 5 (Right): ChartQA performance scales from 65.6% (baseline, 0 QA data) to 79.9% (20k QA images), with a monotonic improvement curve that shows no sign of plateauing, suggesting that further scaling of image diversity would yield continued gains.

---

Innovation 4: Verifier Over-Optimization as the Central Failure Mode in Subjective Reward Design — Diagnosed Through Systematic Comparison

While the paper's primary contribution is constructive (CapRL itself), it also makes a significant diagnostic contribution to the RLVR literature by providing the first systematic empirical characterization of why subjective reward signals fail when used as RL training objectives for open-ended generation tasks. This is not merely a list of failure cases — it's a structured analysis that reveals a common underlying mechanism: any holistic evaluator with intrinsic biases will create an exploitable optimization landscape that the policy model will discover and amplify.

The paper compares three fundamentally different reward paradigms under identical experimental conditions (same base model, same GRPO algorithm, same training data scale):

1. Dedicated reward model (UnifiedReward-2.0-qwen-3b): trained specifically for multimodal evaluation, represents the "best available" approach to learned reward functions.
2. General LVLM as judge (Qwen2.5-VL-3B): represents the "LLM-as-a-judge" paradigm that has become popular in the post-GPT-4 era.
3. CapRL's decoupled VQA: represents the utility-based, objective-verification paradigm.

The results (Table 3) are stark and structured. The dedicated reward model degrades captioning performance relative to the untrained baseline (38.4% vs. 39.9% average Prism score), meaning RL training with this reward actually made the model worse. The LVLM-as-judge approach improves modestly (42.5%) but is still substantially below CapRL (48.3%). The training curves in Figure 1(c) reveal the dynamics: both subjective approaches show increasing reward during training, but only CapRL's reward increase translates to genuine quality improvement.

What makes this analysis intellectually distinctive is its demonstration that reward hacking is not a bug to be fixed but an inevitable consequence of the subjective-evaluation architecture. The paper does not frame the failure cases as implementation problems (e.g., "if we had a better reward model, this would work") but as architectural problems (e.g., "any reward model that makes holistic judgments about caption quality will have biases that can be exploited"). This is a stronger claim than prior work has made. Previous RLVR papers have documented reward hacking as an empirical nuisance; CapRL argues it is a structural property of subjective reward functions applied to open-ended generation.

The brevity collapse observed with UnifiedReward is particularly instructive because it demonstrates that the problem is not reward model accuracy but reward model alignment. UnifiedReward-2.0-qwen-3b is presumably well-calibrated on its training distribution — it accurately predicts which captions humans prefer in its training data. But those training preferences encode a brevity bias (from text-to-image captions), and the GRPO optimizer finds the global optimum of that biased objective: a near-empty caption that maximizes reward-per-token. This is not a failure of optimization; it's a perfect optimization of the wrong objective. The paper's framing makes clear that no amount of reward model improvement within the subjective-evaluation paradigm can solve this, because the problem is the paradigm itself — holistic evaluation always encodes biases, and RL will always find them.

This diagnostic framework has implications for the broader RLVR field. As RLVR expands beyond math and code to more subjective domains (summarization, dialogue, creative writing), the CapRL analysis provides a template for evaluating reward designs: does the reward emerge from an objective, verifiable downstream task, or does it involve a model making a holistic judgment? If the latter, what biases might that model encode, and how might the policy model exploit them? The paper's demonstration that even purpose-built reward models fail this test raises the bar for what constitutes an acceptable RLVR reward function in subjective domains.

Evidence anchoring (Table 3, Figure 1). Table 3: UnifiedRW-as-Judge-3B achieves 38.4% average Prism score (below the 39.9% baseline), Qwen2.5VL-as-Judge-3B achieves 42.5%, CapRL-3B achieves 48.3%. Figure 1(c): training curves show all three approaches with increasing reward, but only CapRL's reward curve corresponds to meaningful quality improvement. The qualitative examples in Figures 9–11 show the specific exploitation patterns: brevity collapse for UnifiedReward ("The image appears to be a bar chart..."), verbosity padding with self-praise for the LVLM judge ("This description is both factual and insightful...").

5. Experimental Analysis

Evaluation Methodology

• Dataset (Pretraining Setting). The pretraining evaluation (Section 4.1) uses 12 standard benchmarks: InfoVQA, DocVQA, ChartQA, RealWorldQA, MathVista, SEED2-Plus, MME-RW, MMBench, MMStar, MMVet, AI2D, and GQA. These are drawn from the Open-LLaVA-NeXT evaluation suite (Chen & Xing, 2024) and cover document understanding, chart interpretation, general VQA, and mathematical reasoning. The paper does not specify exact split sizes but follows standard protocols for each benchmark.

• Dataset (Prism Setting). The caption quality evaluation (Section 4.2) uses the Prism Framework (Qiao et al., 2024), which decouples VQA into two stages: a captioner generates descriptions, and a separate LLM answers questions based solely on those captions. The evaluation spans 12 benchmarks: ChartQA, ChartQA-Pro, InfoVQA, MMMU-Pro, MathVerse, CharXiv, WeMath, MathVision, MMStar, SEED, MMMU, and the average across all 12. The exact number of test instances per benchmark is not specified, but these are standard benchmarks with publicly available test sets.

• Base model(s). For the core CapRL training, the policy model is initialized from Qwen2.5-VL-3B (Bai et al., 2025). The answering LLM in the reward computation is Qwen2.5-3B-Instruct. For the pretraining experiments (Table 1), three architecture combinations are tested: Qwen2.5-3B + Qwen2.5-ViT, Qwen2.5-7B + Qwen2.5-ViT, and InternLM2.5-7B + CLIP-ViT-L. This spans different parameter scales (3B, 7B), LLM backbones (Qwen2.5, InternLM2.5), and visual encoders (Qwen2.5-ViT, CLIP-ViT-L). The QA generation for the training dataset uses Qwen2.5-VL-72B, and the QA filtering uses Qwen2.5-VL-3B.

• Metrics. In the pretraining setting, each benchmark has its own accuracy metric (exact match for QA, correctness for multiple-choice, etc.), and the paper reports per-benchmark scores plus an unweighted average across all 12. In the Prism setting, the metric is VQA accuracy as measured by the decoupled two-stage framework: the captioner produces a caption, a fine-tuned Qwen2.5-3B-Instruct answers questions based solely on that caption, and accuracy is computed via exact match against ground-truth answers. Both settings use percentage accuracy (0–100).

• Baselines (Pretraining). In Table 1 and Table 2, the baselines are: (1) Vanilla: skips the Further Pretraining stage entirely (only Initial Alignment + SFT); (2) ShareGPT4V-1M: uses the ShareGPT4V dataset (Chen et al., 2024b) for Further Pretraining; (3) DenseFusion-1M: uses the DenseFusion dataset (Li et al., 2024b) for Further Pretraining. In Table 2, two additional baselines are constructed by using CapRL-3B to re-annotate the existing ShareGPT4V and DenseFusion images, isolating caption quality from image diversity: CapRL-ShareGPT4V-1M and CapRL-DenseFusion-1M.

• Baselines (Prism). In Table 3, the baselines are: (1) Qwen2.5-VL-3B, Qwen2.5-VL-7B, Qwen2.5-VL-72B: the untrained base models at three scales, representing the SFT pretrained baselines; (2) UnifiedRW-as-Judge-3B: GRPO-trained using UnifiedReward-2.0-qwen-3b (Wang et al., 2025) as the reward model; (3) Qwen2.5VL-as-Judge-3B: GRPO-trained using Qwen2.5-VL-3B itself as the holistic caption quality judge. These two baselines represent the subjective reward paradigms that CapRL aims to replace.

• Generation budget / compute accounting. For the GRPO training phase, the "budget" is measured implicitly through the number of training images (75k for the main CapRL model, with ablations at 2.5k, 5k, 10k, and 20k for scaling experiments in Figure 5). For the pretraining experiments, the budget is measured in dataset size (1M vs. 5M image-caption pairs). For the scaling comparison in Figure 4, the budget is measured in the number of pretraining samples (varying from 0 to 1M). There is no FLOPs-based compute accounting analogous to the Chinchilla-style analysis in the reference example; the paper measures data efficiency rather than computational efficiency.

• Cross-validation / statistical protocol. The pretraining experiments use standard train/test splits from each benchmark with no custom cross-validation. For the Prism evaluation, the answering LLM in Stage 2 is fixed to a fine-tuned Qwen2.5-3B-Instruct, ensuring that benchmark performance differences directly reflect caption quality differences from Stage 1. The paper does not report confidence intervals, error bars, or statistical significance tests. For the GRPO training, the QA dataset used for reward computation is entirely separate from the evaluation benchmarks, avoiding contamination. The paper employs two-fold cross-validation only implicitly through the standard benchmark evaluation protocols, not for strategy selection as in the reference example.

---

Main Quantitative Results

Pretraining Setting: CapRL-Annotated Data Outperforms Existing Caption Datasets

The headline result from Table 1 is that pretraining on CapRL-1M (a 1M subset of the CapRL-5M dataset annotated by CapRL-3B) yields substantial and near-universal improvements over the strongest existing open-source caption datasets across all three model architecture configurations. Under the Qwen2.5-3B + Qwen2.5-ViT setting, CapRL-1M achieves an average 12-benchmark score of 59.7%, compared to 57.1% for DenseFusion-1M, 56.7% for ShareGPT4V-1M, and 55.5% for the Vanilla baseline (no Further Pretraining). This represents a +2.6 percentage point improvement over the best alternative dataset (DenseFusion-1M) and a +4.2 point improvement over Vanilla.

The gains are not uniform across benchmarks — they are concentrated in domains that require fine-grained visual extraction. On InfoVQA (infographic VQA), CapRL-1M achieves 56.2% versus 49.4% for DenseFusion-1M — a +6.8 point gap that is the largest single-benchmark improvement. On DocVQA, the gap is +2.7 points (87.3% vs. 84.6%). On ChartQA, it is +3.6 points (78.0% vs. 74.4%). These three benchmarks — InfoVQA, DocVQA, ChartQA — all involve charts, documents, and infographics where precise extraction of text, numbers, and structural information is critical. This pattern directly validates CapRL's design: the decoupled VQA reward incentivizes captions that contain specific, extractable facts, which is exactly what document- and chart-oriented downstream tasks require.

On natural image benchmarks, the gains are smaller but consistent. On MMStar, CapRL-1M achieves 47.0% versus 45.6% for DenseFusion-1M (+1.4 points). On MMBench, it achieves 70.5% versus 69.0% (+1.5 points). On MMVet, CapRL-1M achieves 50.0% versus 40.2% for DenseFusion-1M — a +9.8 point gap that is the second-largest single-benchmark improvement. MMVet requires complex visual reasoning and detailed attribute recognition, suggesting that CapRL's captions are not just factually accurate but also contain the kind of fine-grained object descriptions that support compositional reasoning.

Cross-architecture consistency (Table 1). The same pattern holds under all three model configurations:

• Qwen2.5-7B + Qwen2.5-ViT: CapRL-1M achieves 62.2% average vs. 60.2% for DenseFusion-1M (+2.0 points). InfoVQA gap: +6.4 points (59.9% vs. 53.5%). ChartQA gap: +3.9 points (80.6% vs. 76.7%).
• InternLM2.5-7B + CLIP-ViT-L: CapRL-1M achieves 59.6% average vs. 57.4% for DenseFusion-1M (+2.2 points). InfoVQA gap: +4.0 points (43.3% vs. 39.3%). ChartQA gap: +5.0 points (75.8% vs. 70.8%).

The gains are not only consistent but larger in absolute terms for the Qwen-based architectures than for the InternLM-based one, likely reflecting the fact that CapRL-3B was trained from Qwen2.5-VL-3B and its captions may be better aligned with the Qwen visual encoder's feature space. However, even with the completely different CLIP-ViT-L encoder and InternLM2.5 backbone, CapRL-1M still substantially outperforms DenseFusion-1M, indicating that the caption quality improvement is not merely an artifact of distribution matching.

Scaling from 1M to 5M (Table 1). When scaling the pretraining dataset from CapRL-1M to CapRL-5M, performance improves further across all three configurations:

• Qwen2.5-3B: 59.7% → 62.0% (+2.3 points)
• Qwen2.5-7B: 62.2% → 63.8% (+1.6 points)
• InternLM2.5-7B: 59.6% → 62.2% (+2.6 points)

The fact that 5× more data continues to yield improvements (rather than saturating at 1M) suggests that CapRL-annotated captions have not exhausted their utility even at the 5M scale. The improvement from 1M to 5M is roughly comparable in magnitude to the improvement from DenseFusion-1M to CapRL-1M, indicating that both data quality and data quantity contribute independently to downstream performance.

Isolating caption quality from image diversity (Table 2). A critical confounding factor in Table 1 is that CapRL-5M uses different source images than ShareGPT4V-1M and DenseFusion-1M. The paper addresses this directly in Table 2 by using CapRL-3B to annotate the exact same images from ShareGPT4V-1M and DenseFusion-1M, creating CapRL-ShareGPT4V-1M and CapRL-DenseFusion-1M. Under the Qwen2.5-3B + Qwen2.5-ViT setting:

• ShareGPT4V-1M (original captions): 56.7% average. CapRL-ShareGPT4V-1M (same images, CapRL captions): 58.7% (+2.0 points).
• DenseFusion-1M (original captions): 57.1% average. CapRL-DenseFusion-1M (same images, CapRL captions): 59.9% (+2.8 points).

These gains, achieved with identical image sets, isolate the effect of caption quality. The +2.8 point improvement over DenseFusion-1M with identical images is actually larger than the +2.6 point improvement in Table 1 (where image sets differed), suggesting that DenseFusion's image selection may be somewhat better than CapRL's (perhaps offsetting some caption quality difference) but that CapRL's caption quality advantage is robust. The paper explicitly draws this conclusion: "the substantial advantage of the CapRL dataset over the baselines largely stems from the superior quality of its captions, rather than from differences in image diversity."

Scaling trend comparison (Figure 4). The paper compares the scaling curves of CapRL and DenseFusion as pretraining data increases from 0 to 1M samples on four representative benchmarks (ChartQA, SEED2Plus, InfoVQA, AI2D). The key finding is that CapRL consistently outperforms DenseFusion at every data scale, and the gap widens as data increases. For InfoVQA at 1M samples: CapRL reaches approximately 56% versus DenseFusion at approximately 49% — a 7-point gap. At 200k samples, the gap is only about 3 points (CapRL ~46% vs. DenseFusion ~43%). This widening gap with scale is evidence that CapRL captions provide more learnable signal — the model continues to extract value from them as data volume increases, whereas DenseFusion captions saturate earlier. For ChartQA, the pattern is similar: at 1M, CapRL reaches approximately 78% vs. DenseFusion at approximately 74% (4-point gap), while at 200k the gap is approximately 2 points.

Prism Setting: CapRL-3B Matches the 72B Model in Caption Informativeness

The headline result from Table 3 is that CapRL-3B achieves an average 12-benchmark Prism score of 48.3%, matching Qwen2.5-VL-72B (also 48.3%) and substantially exceeding Qwen2.5-VL-3B (39.9%), Qwen2.5-VL-7B (44.9%), and all subjective-reward GRPO baselines. This is the paper's most striking claim: a 3B model, trained with CapRL, produces captions that are as informative as those from a model 24× its size.

The per-benchmark breakdown reveals where CapRL's advantage is concentrated:

• ChartQA: CapRL-3B achieves 80.5% versus 65.6% for the untrained Qwen2.5-VL-3B (+14.9 points), and even exceeds Qwen2.5-VL-72B (80.2%). This is the largest absolute improvement on any benchmark.
• ChartQA-Pro: CapRL-3B achieves 39.9% versus 27.1% for baseline (+12.8 points) and exceeds 72B (38.0%).
• InfoVQA: CapRL-3B achieves 64.8% versus 40.2% for baseline (+24.6 points) and exceeds 72B (60.8%) by 4.0 points. This is the largest relative improvement and the benchmark where CapRL most decisively outperforms even the 72B model.
• CharXiv: CapRL-3B achieves 32.4% versus 21.8% for baseline (+10.6 points) and exceeds 72B (30.7%).
• MMStar: CapRL-3B achieves 55.0% versus 46.4% for baseline (+8.6 points), matching 72B (55.0%).
• SEED: CapRL-3B achieves 70.6% versus 64.1% for baseline (+6.5 points), exceeding 72B (69.3%).

The pattern is clear: CapRL's gains are largest on document, chart, and infographic understanding benchmarks (InfoVQA, ChartQA, ChartQA-Pro, CharXiv) where precise extraction of structured information matters most, and smaller but still substantial on natural image benchmarks (MMStar, SEED). On MMMU and MMMU-Pro (multimodal understanding with college-level questions), CapRL-3B achieves 38.1% and 30.7% respectively — improvements over baseline but not matching the 72B model's 39.4% and 34.1%. This is consistent with the design: MMMU questions often require reasoning that goes beyond what can be captured in a caption, so even a perfect caption may not contain the answer.

Comparison with subjective reward baselines (Table 3). The two subjective-reward GRPO baselines demonstrate the failure modes discussed in Section 2:

• UnifiedRW-as-Judge-3B: Achieves 38.4% average — actually worse than the untrained Qwen2.5-VL-3B baseline (39.9%). This is the brevity collapse phenomenon: the model learned to output near-empty captions that maximize the biased reward signal but contain minimal visual information. On InfoVQA, it achieves only 33.6% (vs. 40.2% baseline and 64.8% CapRL). On ChartQA, 54.9% (vs. 65.6% baseline). This is the only model in the table where GRPO training reduced caption quality.

• Qwen2.5VL-as-Judge-3B: Achieves 42.5% average — an improvement over baseline (39.9%) and UnifiedReward (38.4%), but still far below CapRL (48.3%). On InfoVQA, it reaches 49.3% vs. CapRL's 64.8% — a 15.5-point gap. On ChartQA, 71.4% vs. 80.5% — a 9.1-point gap. The qualitative examples in Figures 10 and 11 explain this gap: the judge-trained model produces verbose but information-sparse output, padding captions with self-praise and generic language rather than specific extractable facts.

The key takeaway from these comparisons is not merely that CapRL works better, but that the two subjective-reward approaches fail in qualitatively different and complementary ways — brevity collapse vs. verbosity padding — yet both fail to produce captions that enable downstream VQA. This validates the paper's core claim that subjective holistic evaluation is structurally unreliable as an RL reward, regardless of the specific evaluator used.

Domain-specific training and generalization (Figure 5, Left). The paper trains CapRL on two domain-specific subsets: CapRL-DocChart-20k (20k document/chart images only) and CapRL-Natural-20k (20k natural images only). The key finding is that domain-specific training generalizes to out-of-domain benchmarks. CapRL-DocChart-20k improves ChartQA from 65.6% (baseline) to 77.8% (+12.2 points) — expected, since it was trained on similar images — but also improves MMStar from 46.4% to 50.0% (+3.6 points) and SEED from 64.1% to 68.9% (+4.8 points) — both natural image benchmarks. Conversely, CapRL-Natural-20k improves ChartQA from 65.6% to 76.3% (+10.7 points) despite never seeing chart images during training. This cross-domain generalization is evidence that CapRL teaches the model general principles of informative captioning (be specific, include numbers, describe visual elements precisely) rather than domain-specific memorization. The model trained only on natural images learns to be detailed and accurate in ways that transfer to charts, and vice versa.

Scaling with QA data quantity (Figure 5, Right). The paper evaluates CapRL models trained with 2.5k, 5k, 10k, and 20k QA images. ChartQA performance scales from 65.6% (baseline, 0 QA data) to approximately 79.9% at 20k images, with a monotonic improvement curve showing no sign of saturation. SEED scales from 64.1% to approximately 70.6% at 20k. The curves have not flattened at 20k, suggesting that further data scaling would yield continued gains. This is practically significant because 20k images is a relatively small dataset — the QA curation pipeline can feasibly scale to much larger sizes, and the paper's results suggest this would be worthwhile.

---

Ablation Studies and Robustness Checks

• Image sources for QA training data (Figure 5, Left): Training on document/chart images only (CapRL-DocChart-20k) improves both in-domain benchmarks (ChartQA: +12.2 points to 77.8%) and out-of-domain natural image benchmarks (MMStar: +3.6 points to 50.0%; SEED: +4.8 points to 68.9%). Conversely, training on natural images only (CapRL-Natural-20k) improves ChartQA (+10.7 points to 76.3%) despite never seeing charts during training. This demonstrates strong cross-domain generalization of the captioning skill learned through CapRL.

• Number of QA pairs per image (Table 4): Using 3 QA pairs per image achieves 48.5% average Prism score; using 2 QA pairs achieves 48.5%; using only 1 QA pair achieves 48.0% — a degradation of only 0.5 percentage points. The near-equivalence of 1 QA and 3 QA is the paper's key efficiency finding: sparse supervision is sufficient, which dramatically reduces QA annotation costs for scaling.

• Number of sampling rounds N for reward computation (Table 5): N=1 (single question, single evaluation) achieves 47.3% average. N=2 achieves 47.6%. N=4 achieves 48.4% — the optimal value. N=8 achieves 48.3% — essentially saturated. The paper explains the N=1 degradation as due to "inherent option biases in the LLM" that cause the measured accuracy to misrepresent true caption quality. The saturation at N=4 indicates that four independent evaluations per caption (each with shuffled options) are sufficient to average out positional biases and provide a reliable reward signal.

• QA data leakage filtering (Table 6): Training on unfiltered QA data (containing questions answerable without the image, "Leaking20k") achieves 47.4% average versus 48.5% for filtered data ("Refined20k") — a 1.1-point penalty for leakage. The paper notes that this degradation exists but does not cause training collapse, explaining: "because our filtering criteria are quite strict, some discarded QA pairs contain only mild or even negligible leakage." This positions the filtering as a quality improvement rather than a necessity for stability.

• Scaling of pretraining data quantity (Figure 4): When pretraining data is scaled from 0 to 1M samples on four benchmarks (ChartQA, SEED2Plus, InfoVQA, AI2D), CapRL-annotated captions consistently outperform DenseFusion-annotated captions at every data point, and the gap widens with increasing data. On InfoVQA, the gap grows from approximately 3 points at 200k samples to 7 points at 1M samples. This suggests that CapRL captions provide more learnable signal per sample and that their advantage compounds with scale.

• Robustness to model architecture (Table 1): The CapRL advantage holds across three distinct architecture families (Qwen2.5-3B + Qwen2.5-ViT, Qwen2.5-7B + Qwen2.5-ViT, InternLM2.5-7B + CLIP-ViT-L), with average improvements over DenseFusion-1M of +2.6, +2.0, and +2.2 points respectively. This demonstrates that the caption quality benefit is not tied to a specific visual encoder or language model backbone.

---

Critical Assessment

Claim 1: "CapRL significantly enhances multiple settings... CapRL achieves performance comparable to Qwen2.5-VL-72B while exceeding the baseline by an average margin of 8.4%."

This claim is anchored to Table 3: CapRL-3B achieves 48.3% average Prism score, exactly matching Qwen2.5-VL-72B (48.3%), and exceeding Qwen2.5-VL-3B (39.9%) by 8.4 percentage points. The evidence is direct and strong. However, the paper omits a critical comparison: it does not report the Prism score of Qwen2.5-VL-72B with CapRL training. If CapRL-3B can match the 72B model, what would CapRL-72B achieve? The absence of this experiment leaves open the question of whether CapRL's gains are bounded by base model scale or whether they compound with it. Given that the 72B model's raw capability is substantially higher (as shown by its superior performance on reasoning-heavy benchmarks like MMMU, where it achieves 39.4% vs. CapRL-3B's 38.1%), CapRL-72B might significantly exceed CapRL-3B. The claim of "performance comparable to Qwen2.5-VL-72B" is true on average but masks benchmark-level variation: on MMMU-Pro (34.1% vs. 30.7%) and MathVerse (39.4% vs. 38.1%), the 72B model maintains a lead, while on InfoVQA (60.8% vs. 64.8%) CapRL-3B actually exceeds the 72B model. The claim holds best for factual extraction benchmarks and weakest for reasoning benchmarks.

A subtler caveat: the "72B model comparable" claim is benchmark-dependent. The Prism framework evaluates caption quality through the lens of VQA accuracy — the same metric CapRL optimizes for during training. This creates a potential evaluation-metametric alignment: CapRL is being evaluated on exactly the kind of task it was trained to excel at (enabling VQA from captions), while the 72B model was trained with a general SFT objective that may not prioritize this specific capability. A caption that is excellent for human reading or for CLIP-style contrastive learning might score differently. The paper does not evaluate on non-VQA caption quality metrics (e.g., human preference studies, caption diversity measures, or traditional metrics like CIDEr or SPICE), which limits the breadth of the "comparable to 72B" claim.

Claim 2: "Pretraining on the CapRL-5M caption dataset annotated by CapRL-3B results in substantial gains across 12 benchmarks."

This claim is robustly supported by Table 1 across all three model configurations and both data scales (1M and 5M). The improvements are consistent, non-trivial (2–4 percentage point average gains over strong baselines), and concentrated in domains where CapRL's design predicts advantage (document/chart understanding). The image-source ablation in Table 2 cleanly isolates caption quality as the causal factor. The scaling analysis in Figure 4 shows the advantage persisting and widening with data scale, addressing the concern that the gains might be a small-data artifact.

However, there is a circularity concern the paper does not fully address. CapRL-3B was trained from Qwen2.5-VL-3B using the GRPO reward framework. The pretraining experiments then use CapRL-3B's captions to train new models that also use Qwen2.5-based architectures. Could the gains partly reflect that CapRL-3B's captions are in-distribution for Qwen-based models (since the base model shares the same pretraining)? The paper partially addresses this by showing gains with the InternLM2.5-7B + CLIP-ViT-L architecture (a completely different model family), but the InternLM gains (+2.2 points over DenseFusion-1M) are smaller than the Qwen-3B gains (+2.6 points). This leaves residual uncertainty about how much of the gain is caption quality versus distribution matching.

Additionally, the pretraining experiments follow a specific three-stage training recipe (Initial Alignment with BLIP-558K → Further Pretraining with the target dataset → SFT with Open-LLaVA-NeXT-1M). The paper does not ablate this recipe — for instance, what if CapRL-5M were used in the SFT stage instead of Open-LLaVA-NeXT-1M? What if the Further Pretraining stage were skipped entirely and CapRL-5M were used only for Initial Alignment? The results are conditional on this specific training curriculum, and the paper does not explore how robust they are to curriculum changes.

Claim 3: "CapRL effectively trains models to produce more general and accurate image descriptions, moving beyond the limitations of traditional SFT-based image captioning models."

This claim has two parts: "more general" and "more accurate." The accuracy claim is well-supported by the Prism results (Table 3) and the pretraining results (Table 1), both of which measure downstream task performance that depends on caption accuracy. The generality claim — that CapRL captions enable models to "generate a broader range of creative and general descriptions" — is not directly tested by any experiment in the paper.

The paper evaluates captions through their utility (VQA accuracy), not through their diversity or creativity. A model that produces the same highly-informative caption for every image containing a fox would score perfectly on VQA accuracy but would not be "creative" or "diverse." The paper does not report any diversity metrics (e.g., n-gram diversity, self-BLEU, embedding dispersion) or any evaluation of whether CapRL-trained models produce multiple valid captions for the same image when sampled repeatedly. This is a gap between the claim and the evidence: the paper has demonstrated that CapRL produces useful captions, but it has not demonstrated that they are diverse or creative in the sense implied by the motivation (Section 1: "generate a broader range of creative and general descriptions").

The qualitative examples in Figures 2, 6, 8, 12–14 show that CapRL captions are more detailed and structured than baseline captions, but these are single examples that cannot establish diversity. The paper's motivation contrasts SFT ("models tend to memorize specific answers") with RL ("encourages the model to explore and discover optimal solutions"), but no experiment quantifies this exploration or diversity advantage. This is a significant missing piece of evidence for one of the paper's central motivational claims.

Missing experiments and open questions:

1. No comparison with SFT on the same QA reward data. The paper presents CapRL as an alternative to SFT, but does not compare against an SFT baseline trained on the same images with the best available captions. If one trained Qwen2.5-VL-3B via SFT on the CapRL-5M dataset itself (using CapRL-3B's captions as targets), would the resulting model match or exceed the GRPO-trained CapRL-3B? This would test whether the RL optimization is necessary or whether the caption quality in CapRL-5M is the primary driver.

2. No human evaluation. All results are measured through automated benchmarks (VQA accuracy, Prism scoring). While the paper's core argument is that VQA accuracy is a valid proxy for caption quality, a small-scale human study (even on 100 images) comparing CapRL captions to baseline captions on dimensions like accuracy, completeness, and readability would substantially strengthen the claim that the optimization objective aligns with human preferences.

3. No evaluation of caption faithfulness to images. The paper evaluates captions through their downstream VQA utility, but does not measure whether CapRL captions hallucinate less (or more) than baseline captions. The VQA accuracy metric penalizes missing information (the LLM gets questions wrong because the caption lacks details) but may not penalize hallucinated information (the LLM might ignore hallucinations that aren't relevant to the specific questions asked). The qualitative examples in Figure 2 suggest reduced hallucinations, but no quantitative hallucination evaluation is provided.

4. No latency or inference cost analysis. The paper claims CapRL-3B is efficient for annotating 5M images, but provides no throughput numbers, GPU-hours consumed, or comparison with alternative annotation approaches (e.g., using Qwen2.5-VL-72B directly). This matters for the practical claim that CapRL enables "very low annotation cost."

5. No sensitivity analysis of GRPO hyperparameters. The paper does not report the effects of varying the KL penalty coefficient, the group size G, the learning rate, or the number of training steps. This makes it difficult to assess whether the reported results are robust or the product of careful hyperparameter tuning that may not transfer.

Conditional nature of the claims. The paper's claims hold most strongly under specific conditions that should be made explicit:

• The CapRL advantage is largest for document, chart, and infographic images where precise factual extraction matters. On reasoning-heavy benchmarks (MMMU, MathVerse), the advantage is smaller or nonexistent.
• The advantage requires a clean QA dataset with strict leakage filtering — Table 6 shows degradation with unfiltered data.
• The approach assumes the availability of a capable QA generation model (Qwen2.5-VL-72B) and a capable answering LLM (Qwen2.5-3B-Instruct). The paper does not test sensitivity to these model choices — what if a weaker QA generator is used? What if the answering LLM is switched to a different model family?
• The pretraining gains are demonstrated with a specific three-stage training recipe and may not generalize to other pretraining paradigms (e.g., single-stage training, different SFT datasets).

What would strengthen the paper:

The most impactful missing experiment is training CapRL on larger base models (7B, 72B) to test whether the approach scales with model size or hits diminishing returns. A second high-value experiment is a direct SFT baseline using CapRL-annotated captions as targets, which would isolate the contribution of the RL optimization from the contribution of the high-quality training data. Finally, diversity metrics (measuring whether CapRL models produce varied captions when sampled multiple times for the same image) would directly address the "creativity and generality" claims that currently lack empirical support.

6. Limitations and Trade-offs

The QA Curation Pipeline Requires a Strong LVLM Teacher and Has Unaccounted Costs

The CapRL framework depends on a high-quality VQA dataset for reward computation, and the paper constructs this dataset using Qwen2.5-VL-72B — a model 24 times larger than the policy model being trained — to generate Multiple-Choice Questions for each training image. The paper acknowledges this dependency implicitly by specifying the model used ("we use Qwen2.5-VL-72B to automatically generate multiple question-answer pairs," Section 3.1), but it does not treat the availability or cost of this teacher model as a limitation.

The consequence is that CapRL's reward signal quality is bounded by the QA generator's capability. If the 72B model produces ambiguous questions, misses important visual details when generating QA pairs, or introduces systematic biases in question selection, the reward signal will be correspondingly noisy or incomplete. A practitioner without access to a 72B-scale LVLM — or working in a domain where even the best available model cannot reliably generate high-quality VQA pairs — cannot replicate the CapRL pipeline as described. The paper provides no sensitivity analysis on QA generator quality: would CapRL still work with a 7B generator? A 3B generator? Does performance degrade gracefully or collapse? The filtering step (Equation 4) can remove obviously bad questions, but it cannot create good questions that the generator failed to produce in the first place — if the generator systematically ignores certain types of visual content (e.g., spatial relationships, small text, unusual objects), those aspects will never be rewarded during training, and the policy model will not learn to caption them.

Furthermore, the cost of QA generation and filtering is not accounted for in any of the paper's efficiency claims. Generating five questions per image with a 72B model across 75k images, then running the 3B filter model four times per candidate QA pair — this represents a substantial one-time compute investment that precedes any RL training. The paper characterizes CapRL as enabling "very low annotation cost" (Section 3.2) for the downstream 5M dataset, but this claim refers only to the inference cost of running the trained CapRL-3B captioner, not the cost of obtaining the QA dataset needed to train it. A complete accounting would amortize the QA curation cost over the number of captions ultimately generated, but the paper provides no such analysis.

Evidence in the paper: The QA generation and filtering parameters are specified in Section 3.1 and Appendix E, but no ablation studies test alternative QA generators or measure the sensitivity of final caption quality to QA generator scale or domain expertise. Table 6 shows that filtered QA data outperforms unfiltered data by 1.1 percentage points, confirming that QA quality matters, but this does not address the question of how QA generator capability affects outcomes. The paper's generalization experiment (Figure 5, Left) — showing that training on chart-only images improves natural image captioning — provides indirect evidence that the QA data captures generalizable captioning principles, but it does not isolate the effect of QA quality from other factors.

Mitigation status: Not addressed. The paper treats the 72B QA generator as a fixed component of the pipeline and does not discuss how practitioners with different resource constraints might adapt it. Future work on cheaper QA generation methods or on training without a strong teacher model is not mentioned.

---

All Results Are on a Single Base Model Family and the MATH-Analogous Benchmark Suite, with No Evidence of Cross-Architecture Transfer for the RL Training Itself

Every CapRL model in the paper is initialized from Qwen2.5-VL-3B and trained with the GRPO framework using Qwen2.5-3B-Instruct as the answering LLM. The paper never trains CapRL starting from a different base model family (e.g., InternVL, LLaVA, Phi-Vision) or using a different answering LLM for reward computation. This is not a minor omission — it means the paper provides no evidence that CapRL's reward signal transfers across model architectures or even across scales within the same family (the 7B and 72B Qwen variants are evaluated only as untrained baselines in Table 3, never as CapRL-trained models).

The consequence is that we cannot distinguish whether CapRL's success is due to the general principle of decoupled VQA reward or due to some favorable interaction between Qwen2.5-VL-3B's pretraining, Qwen2.5-3B-Instruct's answering patterns, and the specific QA data distribution. For example, the answering LLM (Qwen2.5-3B-Instruct) shares the same pretraining corpus and tokenizer as the policy model's language backbone. It is possible that the answering LLM has learned to extract information from captions in ways that are particularly well-aligned with how the policy model naturally expresses visual information, creating a smoother reward landscape than would exist with a mismatched answering LLM. If a practitioner attempted to use CapRL with a LLaMA-based vision model and a Qwen-based answering LLM (or vice versa), the reward signal might be noisier or less effective.

The pretraining experiments in Table 1 partially mitigate this concern by showing that CapRL-annotated captions benefit models with different architectures (InternLM2.5-7B + CLIP-ViT-L), but these experiments test the downstream utility of the captions, not the trainability of the captioner itself. A practitioner wanting to apply CapRL to train their own custom LVLM — rather than using CapRL-3B's pre-generated captions — has no evidence that the approach works beyond the Qwen2.5-VL-3B + Qwen2.5-3B-Instruct combination.

Evidence in the paper: Section 4.1 trains CapRL only from Qwen2.5-VL-3B. Section 4.2 evaluates only CapRL-3B. The pretraining experiments (Table 1) use CapRL-3B's captions but do not train new CapRL models from other base architectures. The answering LLM ablation is limited to testing different sampling rounds (Table 5) with the same Qwen2.5-3B-Instruct model — the paper never swaps the answering LLM for a different model family or tests whether CapRL is robust to this choice. The paper's claim that CapRL "offers a scalable framework for applying RLVR to the open-ended image captioning task" (Figure 3 caption) assumes cross-architecture generality that is not demonstrated.

Mitigation status: Not addressed. The authors present CapRL as a general framework but test it on a single model ecosystem. This is a significant gap for practitioners using non-Qwen model families, who would need to conduct their own validation before adopting the approach.

---

CapRL-3B's Gains Are Concentrated in Factual Extraction Domains; Reasoning-Heavy Benchmarks Show Minimal or No Improvement

The paper's strongest results — the +14.9 point improvement on ChartQA, +24.6 points on InfoVQA, +12.8 points on ChartQA-Pro (Table 3) — all occur on benchmarks that require extracting and reporting specific factual information from images: numbers, labels, text content, structured data. On benchmarks that require reasoning beyond factual extraction, the gains are substantially smaller or nonexistent.

In Table 3, CapRL-3B achieves 30.7% on MMMU-Pro, only 2.1 points above the untrained Qwen2.5-VL-3B baseline (28.6%) and still well below Qwen2.5-VL-72B (34.1%). On MathVerse, CapRL-3B achieves 36.4% versus 32.8% baseline (+3.6 points), but the 72B model reaches 39.9%. On WeMath, CapRL-3B achieves 23.4% versus 22.6% baseline (+0.8 points) — essentially no improvement — while the 72B model reaches 24.5%. On MMMU, CapRL-3B achieves 38.1% versus 35.1% baseline (+3.0 points), compared to 72B at 39.4%.

The consequence is that CapRL's reward design — which optimizes for information coverage as measured by VQA accuracy — primarily teaches the model to extract and report, not to reason. This is a direct structural consequence of the reward function: the answering LLM evaluates whether the caption contains the answer to a factual question, not whether the caption demonstrates understanding or draws inferences. A caption that states "The chart shows 51% for very well and 36% for fairly well" will score perfectly on a question asking "What percentage said very well?" regardless of whether the model understood what the chart means or why those numbers matter. CapRL incentivizes comprehensive transcription of visual content into text, which is valuable for document and chart understanding but does not address — and may not improve — higher-level visual reasoning.

This limitation is especially important for practitioners considering CapRL as a general post-training method for LVLMs. If the downstream application requires reasoning about visual content (e.g., "Why might older Republicans be more likely to view Trump as fighting for their beliefs?") rather than merely extracting facts from it, CapRL-trained models may offer little advantage over SFT baselines, and the 72B model's inherent reasoning capability remains unattainable through this form of RL training.

The paper's framing as moving "beyond the limitations of traditional SFT-based image captioning models" (Abstract) does not adequately distinguish between captioning for factual coverage and captioning that supports reasoning — the latter may require capabilities that CapRL's reward structure does not optimize for.

Evidence in the paper: Table 3, comparing CapRL-3B with Qwen2.5-VL-72B across the full 12-benchmark suite. The benchmarks where CapRL-3B matches or exceeds 72B (ChartQA: 80.5% vs. 80.2%; InfoVQA: 64.8% vs. 60.8%; CharXiv: 32.4% vs. 30.7%; MMStar: 55.0% vs. 55.0%; SEED: 70.6% vs. 69.3%) are all factual extraction or recognition benchmarks. The benchmarks where CapRL-3B trails 72B (MMMU-Pro: 30.7% vs. 34.1%; MathVerse: 36.4% vs. 39.9%; WeMath: 23.4% vs. 24.5%; MMMU: 38.1% vs. 39.4%) are reasoning-heavy. This pattern is consistent and large enough that it cannot be attributed to noise.

Mitigation status: Not acknowledged. The paper presents CapRL's results as a unified success story without discussing the domain-dependence of the gains. Section 5 (Conclusion) states that CapRL "effectively encourages models to generate dense and precise image descriptions" — which is true — but does not address the boundary between description and reasoning. The paper does not suggest modifications to the reward design that might improve reasoning capabilities or acknowledge this as a fundamental limitation of the utility-based reward paradigm.

---

The Paper Does Not Compare CapRL Against an SFT Baseline Trained on the Same High-Quality Data, Leaving the RL vs. SFT Contribution Unquantified

The paper's core motivation is that RLVR can overcome SFT's limitations for image captioning — specifically, that "SFT models are trained to match a single ground-truth description for each image, they tend to memorize specific answers" while "RLVR encourages the model to generate more diverse and robust outputs" (Section 1). To support this claim, the paper compares CapRL-3B against the untrained Qwen2.5-VL-3B baseline and against GRPO-trained models using subjective rewards, but it never compares CapRL-3B against an SFT-trained model that had access to the same information.

The consequence is that we cannot determine how much of CapRL's advantage comes from the RL optimization and how much comes from simply having access to a high-quality training signal (the 75k QA pairs and their associated images). A plausible alternative hypothesis is: if you took the same 75k images with their QA pairs, generated high-quality captions for them (perhaps using the 72B model that generated the QA), and trained Qwen2.5-VL-3B via standard SFT to reproduce those captions, the resulting model might match or approach CapRL-3B's performance. The paper provides no evidence against this hypothesis.

This is especially important because the paper's primary practical contribution — CapRL-5M — is a dataset of captions, not a training method. Practitioners who download CapRL-5M and use it for SFT pretraining (as done in Table 1) benefit from CapRL's annotation quality without needing to implement the GRPO training pipeline. If SFT on CapRL-annotated captions achieves similar results to CapRL's RL training, then the RL framework is unnecessary for the paper's main demonstrated use case (pretraining data generation), and the paper's theoretical motivation (RL generalizes better than SFT) remains untested.

The paper does evaluate one form of this: the pretraining experiments in Table 1 train models via SFT on CapRL-annotated captions and show they outperform models trained on other caption datasets. But this tests the downstream value of CapRL captions for pretraining, not whether CapRL's RL training is better than SFT for training the captioner itself. The missing experiment is: (1) take the 75k images used for CapRL's GRPO training, (2) generate captions for them using the best available method (e.g., Qwen2.5-VL-72B with a detailed prompt), (3) SFT-train Qwen2.5-VL-3B on these captions, (4) evaluate the resulting model in the Prism framework and compare with CapRL-3B. This would directly measure the value added by RL over SFT given equivalent information access.

Evidence in the paper: No such SFT baseline exists. The closest comparison is in Table 2, where "CapRL-ShareGPT4V-1M" and "CapRL-DenseFusion-1M" are created by having CapRL-3B re-annotate existing images and then using those annotations for SFT pretraining — but this evaluates the captions' quality for pretraining downstream models, not the training method that produced the captioner. The paper's claim that "this work marks a significant step away from the restrictive, data-hungry SFT paradigm for RLVR in open-ended tasks" (Section 5) is therefore not directly supported by comparative evidence — it is a claim about the training paradigm that the experiments do not isolate.

Mitigation status: Not addressed. The paper does not acknowledge this as a missing comparison or discuss what such a comparison might reveal. The Chu et al. (2025) citation ("SFT Memorizes, RL Generalizes") provides theoretical motivation but does not substitute for empirical evidence within the CapRL framework itself.

---

CapRL Optimizes for a Specific Evaluation Paradigm That It Also Uses for Assessment, Creating a Potential Circularity in the Headline Claim

The paper's primary quantitative claim — that CapRL-3B achieves "performance comparable to Qwen2.5-VL-72B" (Abstract) — is measured using the Prism Framework, which evaluates caption quality through the exact same mechanism that CapRL uses as its training reward: decoupled VQA where a vision-free LLM answers Multiple-Choice Questions based solely on the generated caption. This is not merely a methodological convenience — it means the evaluation metric and the training objective are functionally identical.

The consequence is a potential reward-metametric alignment problem: CapRL is explicitly trained to maximize VQA accuracy from captions, and it is evaluated on VQA accuracy from captions. A model optimized for this specific metric would be expected to excel at it, potentially at the expense of other dimensions of caption quality that the Prism framework does not measure — such as natural language fluency, conciseness, avoidance of hallucination when the VQA questions happen not to probe a hallucinated detail, or captions being useful for tasks other than VQA (e.g., text-to-image generation, cross-modal retrieval, human reading experience, or structure-aware tasks like spatial reasoning from captions).

The paper's comparison against Qwen2.5-VL-72B in Table 3 implicitly assumes that Prism score is a comprehensive measure of caption quality, such that matching the 72B model's Prism score means matching its captioning capability. But the 72B model was trained with a general SFT objective that did not specifically optimize for decoupled VQA performance. It is plausible — and the paper provides no evidence to the contrary — that the 72B model produces captions that are superior to CapRL-3B's on dimensions not captured by Prism (e.g., more natural phrasing, better global coherence, fewer subtle hallucinations that don't affect the specific VQA questions asked), and that CapRL-3B has overfit to the specific evaluation paradigm, producing captions that are highly optimized for answering MCQs but less useful for other purposes.

This concern is amplified by the paper's choice not to report any non-VQA caption quality metrics: no human evaluation, no traditional captioning metrics (CIDEr, SPICE), no diversity measures, no hallucination rates, no retrieval benchmarks where captions serve as queries. The entire evaluation apparatus is built on the same principle as the training objective, making it impossible to detect whether CapRL has found a genuine improvement in captioning or a narrow specialization to VQA-based evaluation.

Evidence in the paper: The evaluation methodology in Section 4.2 explicitly states: "We leverage the Prism framework primarily because it can evaluate caption quality in an objective and stable manner. In our setup, we fix Stage 2 with a fine-tuned Qwen2.5-3B-Instruct as the answering LLM, ensuring that benchmark performance directly reflects the quality of captions produced by the captioner." The pretraining experiments in Section 4.1 evaluate downstream LVLM performance on standard benchmarks, which partially addresses this concern by showing that CapRL captions improve models on tasks beyond caption-VQA (e.g., MMStar, MMBench, AI2D) — but these are still VQA-style benchmarks and do not measure caption quality dimensions that fall outside the VQA paradigm.

Mitigation status: Partially addressed through the pretraining experiments (Table 1), which show that CapRL captions improve downstream LVLM performance across 12 diverse benchmarks, not just Prism-style evaluations. This provides independent evidence that CapRL captions are genuinely higher quality rather than merely Prism-optimized. However, the paper does not acknowledge this potential circularity or discuss what dimensions of caption quality Prism might miss. The claim of "performance comparable to Qwen2.5-VL-72B" is presented without qualification, when it should more precisely be stated as "comparable to Qwen2.5-VL-72B on the Prism benchmark suite, which measures VQA utility specifically."

---

The Difficulty Estimation Bottleneck Analogy: No Mechanism Exists to Detect When an Image Is Beyond the Captioner's Capability

CapRL's reward signal measures whether a caption contains the information needed to answer specific questions. This works well when the policy model is capable of extracting the relevant information from the image — the reward provides a gradient that pushes the model toward more complete extraction. But for images where the base model fundamentally lacks the perceptual or reasoning capability to extract certain information, the reward provides no useful signal: all captions will score poorly regardless of how they are generated, and the GRPO optimizer has no gradient to follow.

This is directly analogous to the "hardest problems" limitation identified in the reference example's Section 6 ("Hard Problems Remain Essentially Unsolved"), where test-time compute could not help on difficulty bin 5 because the base model's pass@1 was near zero. In CapRL's case, the paper provides no difficulty estimation, no stratification by image complexity, and no analysis of which types of visual content benefit from CapRL training versus which are beyond the 3B model's reach. The paper reports only aggregate benchmark scores.

The consequence is that a practitioner cannot predict whether CapRL will help for their specific image distribution. For natural images with clear, salient objects (the Zootopia scene in Figure 3), CapRL may provide large gains. For images requiring specialized domain knowledge (medical images, complex mechanical diagrams, abstract art), the 3B base model may be incapable of producing captions that answer VQA questions regardless of optimization, and CapRL training would waste compute without improving performance. The paper's generalization experiment (Figure 5, Left) shows that training on one domain transfers to another, but this is evidence of cross-domain generalization of learned captioning skills, not of cross-difficulty generalization — both chart images and natural images are within the 3B model's capability range.

The paper also provides no diagnostic tools for this situation. Unlike the reference example, which explicitly bins problems by difficulty and shows that test-time compute fails on the hardest bin, CapRL provides no per-image difficulty scores, no confidence estimates, and no analysis of failure cases grouped by image type. The qualitative examples in the appendix (Figures 6–14) are cherry-picked successes — we see none of the cases where CapRL-3B likely still produces inadequate captions despite RL training.

Evidence in the paper: Table 3 shows that CapRL-3B matches Qwen2.5-VL-72B on factual extraction benchmarks (ChartQA, InfoVQA) but trails on reasoning-heavy benchmarks (MMMU, MMMU-Pro). This provides indirect evidence of a capability boundary: for reasoning tasks that require more than factual extraction, CapRL-3B cannot match the 72B model's performance. However, the paper does not explicitly analyze this as a difficulty-dependent limitation or characterize which specific question types or image categories fall on each side of the boundary. The scaling experiment in Figure 5 (Right) shows continued improvement as QA data scales from 2.5k to 20k, but this measures the effect of data quantity, not the effect of image difficulty on the achievable performance ceiling.

Mitigation status: Not addressed. The paper does not discuss the concept of difficulty-dependent effectiveness, does not bin images or benchmarks by estimated difficulty, and does not analyze failure cases. This is a significant gap for practitioners who need to know whether CapRL will work on their specific image domain or whether they should invest in a larger base model instead.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological reframing rather than an architectural breakthrough — it does not introduce a new model design, training algorithm, or loss function. Instead, it changes how the field should think about reward design for subjective vision-language tasks. The shift is from intrinsic quality assessment (asking "Is this caption good?") to extrinsic utility measurement (asking "Does this caption enable an independent system to perform a downstream task?"). This reframing is significant because it converts an entire category of RLVR applications — those involving open-ended, multi-valid-output generation — from theoretically out-of-reach to practically implementable, at least for domains where a downstream verification task can be constructed.

The magnitude is not a paradigm shift on the scale of RLVR itself (which replaced SFT for reasoning tasks) but rather a critical design principle that unlocks RLVR for a new task class. Before CapRL, the conventional wisdom — supported by the failure cases the paper systematically documents — was that RLVR could only work for tasks with naturally verifiable rewards (math answers, code execution, game outcomes). Subjective tasks like captioning, summarization, and dialogue were considered the domain of RLHF with learned reward models. CapRL demonstrates that this boundary is not fundamental: if you can construct an objective proxy task that depends on the quality of the generated output, you can create a verifiable reward without any model making a holistic quality judgment.

This directly reconciles the conflict between two competing narratives in the post-training literature:

• Narrative 1 (RLVR triumphalism): "RLVR is the future of post-training — SFT merely memorizes, RL generalizes" (Chu et al., 2025; DeepSeek-R1; Kimi K1.5). This narrative has been supported by math and code results but has struggled to extend to open-ended generation.
• Narrative 2 (subjective reward skepticism): "Reward models are gameable, LLM-as-a-judge is unreliable, and RL for subjective tasks will always collapse to reward hacking." This narrative is supported by the UnifiedReward and Qwen2.5-VL-as-Judge baselines in the paper itself, which demonstrate verbosity bias and brevity collapse.

CapRL shows that both narratives can be true simultaneously: RLVR does generalize beyond math and code, but only when the reward is constructed from an objective downstream task rather than from holistic evaluation. The failure of subjective reward approaches is not evidence against RLVR — it is evidence that the reward must be architecturally prevented from being gameable. This reconciliation is valuable because it clarifies where to invest effort: not in building better reward models (which the paper shows will always have exploitable biases), but in designing clever proxy tasks that measure output quality through objective downstream performance.

Research directions that become more attractive:

• Proxy task design for other subjective generation domains. The decoupled VQA reward is one instance of a broader pattern: find a task that requires extracting information from the generated output, where correctness can be verified objectively. For summarization, could the reward be whether a reader can answer factual questions about the original document using only the summary? For instruction-following, could it be whether the generated plan executes successfully in a simulator? For translation, could it be whether a monolingual reader can answer questions about the source text using only the translation? CapRL provides both a template and a proof-of-concept that this approach works for at least one subjective task.

• Combining utility-based rewards with diversity objectives. The paper's motivation emphasizes that RLVR should produce "more diverse and robust outputs" than SFT, but CapRL's reward optimizes only for VQA accuracy — it provides no signal about diversity, creativity, or style. A natural extension is to add a diversity bonus to the reward (e.g., entropy regularization across the group of captions, or a term that penalizes captions that are too similar to each other) to explicitly encourage the exploration the paper theorizes about but does not measure.

Research directions that become less attractive:

• Better holistic reward models for captioning. The paper's comparison of UnifiedReward (a purpose-built reward model) against CapRL's decoupled VQA reward is a strong negative result for the reward-modeling approach. UnifiedReward-2.0-qwen-3b is not a toy baseline — it is a state-of-the-art multimodal reward model — and it degrades caption quality when used as a GRPO reward. The paper's analysis suggests this is not because the reward model is inaccurate, but because any holistic evaluator has biases that RL will exploit. This implies that effort spent on incrementally improving reward model architectures or training data is unlikely to solve the fundamental alignment problem.

• LLM-as-a-judge for RL training of open-ended generators. The Qwen2.5-VL-as-Judge baseline produces captions that are verbose but information-sparse — the model learns to pad outputs with self-praise rather than adding genuine visual detail. While LLM-as-a-judge may remain useful for evaluation (in zero-shot settings where the judge is not used as a training signal), its use as an RL reward for open-ended generation appears fundamentally unreliable based on this paper's evidence.

The paper's most enduring contribution may be its diagnostic framework rather than CapRL itself. By testing three qualitatively different reward paradigms (dedicated reward model, general LVLM judge, decoupled objective proxy) under identical experimental conditions, the paper provides a template for evaluating reward designs in any subjective domain. The key diagnostic question is: does the reward evaluator have access to information beyond the generated output? If yes, the policy model can exploit correlations between output features and evaluator biases without improving output quality. CapRL's design principle — deprive the evaluator of any information not present in the output itself — is the concrete takeaway that transfers across tasks.

---

Follow-Up Research This Work Enables

1. Training CapRL from non-Qwen base models and with non-Qwen answering LLMs to test cross-architecture generality. The paper's results are entirely within the Qwen ecosystem: the policy model is Qwen2.5-VL-3B, the answering LLM is Qwen2.5-3B-Instruct, and the QA generator is Qwen2.5-VL-72B. A critical stress test is to replicate CapRL training starting from a different base model family (e.g., InternVL2-2B, LLaVA-OneVision-0.5B) and using a different answering LLM (e.g., LLaMA-3.1-8B-Instruct). The specific question is whether the reward signal remains effective when the policy model's language backbone and the answering LLM do not share pretraining corpora, tokenizers, or instruction-tuning styles. If CapRL works robustly across architectures, it confirms that decoupled VQA reward is a general principle. If it fails — if the answering LLM struggles to extract information from captions generated by a differently-trained model — it reveals a hidden dependency on distribution matching between the captioner and the evaluator, which would constrain CapRL's applicability and suggest that the answering LLM may need to be fine-tuned on captions from the specific policy model family.

2. Measuring whether CapRL actually produces more diverse captions than SFT, using controlled same-data comparison. The paper's motivation claims that "RLVR encourages the model to generate more diverse and robust outputs" and that SFT models "tend to memorize specific answers" — but no diversity or memorization metrics are reported. A targeted experiment would: (1) take the 75k images used for CapRL's GRPO training, (2) use Qwen2.5-VL-72B to generate a single high-quality reference caption per image, (3) SFT-train Qwen2.5-VL-3B on these reference captions, (4) sample 100 captions from both the SFT-trained model and CapRL-3B for each of 100 held-out test images, and (5) measure pairwise self-BLEU within each model (capturing how similar repeated samples are), n-gram diversity, and embedding dispersion. If CapRL-3B's captions are significantly more diverse than the SFT model's, it validates the paper's theoretical motivation. If they are comparably diverse (or less diverse — CapRL might converge on a single optimal information-dense captioning style), it would reframe CapRL as a method for improving accuracy rather than diversity, and the RL vs. SFT generalization claim would need to be qualified.

3. Training CapRL at 7B and 72B scales to characterize how the RL gain interacts with base model capability. The paper's headline result — CapRL-3B matches Qwen2.5-VL-72B on Prism — raises the obvious question of what CapRL-72B would achieve. Training CapRL-7B and CapRL-72B would answer several questions simultaneously: (a) Does the CapRL gain compound with model scale (i.e., does the 72B model benefit as much from RL training as the 3B model, yielding a CapRL-72B that dramatically exceeds its base version), or do larger models already produce near-optimal captions for VQA purposes, making the RL gain shrink? (b) On reasoning-heavy benchmarks (MMMU, MMMU-Pro, MathVerse) where CapRL-3B trails the 72B baseline, does CapRL-72B close the gap or even exceed the baseline? If CapRL-72B matches or exceeds baseline-72B on reasoning benchmarks, it would demonstrate that the training method genuinely enhances reasoning capability, not just factual extraction. If it doesn't, it would confirm that CapRL's gains are bounded to the factual-coverage dimension and that reasoning improvements require different reward structures. This experiment is the single highest-impact follow-up for understanding CapRL's scaling properties.

4. Exploring alternative proxy tasks beyond VQA for caption evaluation, and testing whether they produce captioners with different strengths. The paper's choice of Multiple-Choice VQA as the proxy task is one of many possible ways to measure caption utility. Alternative proxy tasks would produce captioners optimized for different aspects of caption quality, and comparing them would reveal whether the utility-based reward paradigm is robust to task choice. Examples: (a) Visual grounding: given a caption and an image with bounding boxes, the answering system must identify which region corresponds to each noun phrase. This rewards spatial precision in captions. (b) Caption-to-image retrieval: the caption is used as a query against a database of images; retrieval accuracy is the reward. This rewards captions that capture the most discriminative visual features. (c) Instruction-following: the caption is used as an instruction for an image editing model; the reward is CLIP similarity between the edited image and a target. Each proxy task would produce a captioner with a different emphasis, and the paper's framework makes it straightforward to swap the task while keeping the GRPO infrastructure identical. Comparing the resulting captioners on a diverse set of downstream evaluations (not just the task they were optimized for) would characterize how proxy task choice shapes caption properties.

5. Difficulty-stratified analysis of CapRL's effectiveness to identify when base model capability saturates. The paper reports only aggregate benchmark scores, which obscure whether CapRL's gains are uniform across image types or concentrated in specific regimes. A follow-up analysis would: (1) use the Qwen2.5-VL-72B QA generator to produce questions for a diverse set of images, (2) bin images by the base Qwen2.5-VL-3B's pass@1 VQA accuracy (analogous to the difficulty binning in the reference example), (3) evaluate CapRL-3B's improvement within each bin. The hypothesis — motivated by the paper's observation that reasoning-heavy benchmarks show minimal gain — is that CapRL helps most on images where the base model can already partially extract relevant information (medium difficulty) but provides diminishing returns on images where the base model fundamentally cannot perceive the necessary details (very hard) or already extracts everything (very easy). This would give practitioners a principled way to predict whether CapRL will help for their specific image distribution and would connect the paper's results to the broader literature on compute-optimal allocation and difficulty-dependent scaling.

6. Combining CapRL's utility reward with a lightweight diversity or style reward to test whether multi-objective RL overcomes the single-proxy limitation. CapRL optimizes exclusively for VQA accuracy, producing captions that are information-dense but potentially monotonous in style. A natural extension is to add a second reward term that encourages diversity — for example, a penalty based on the similarity between captions in the group (measured by embedding cosine similarity or n-gram overlap), or a reward for producing captions that differ from the model's own previous outputs on similar images. The GRPO framework naturally supports multiple reward components, and the KL penalty already provides some regularization. The experiment would sweep the relative weight of the utility reward vs. the diversity reward and measure both VQA accuracy (to ensure utility is preserved) and diversity metrics (self-BLEU, embedding variance) on a held-out image set. This directly addresses the paper's unsubstantiated claim that RLVR produces "more creative and more general" captions by providing an explicit mechanism to encourage creativity.

---

Practical Applications and Downstream Use Cases

1. Low-cost annotation of pretraining data for vision-language models. The paper's most immediately actionable result is that CapRL-3B — a 3B-parameter model — can generate captions whose quality, as measured by downstream pretraining utility, substantially exceeds existing manually-curated caption datasets (ShareGPT4V-1M, DenseFusion-1M) and approaches that of a 72B model (Table 1, Table 3). The concrete workflow: (1) train CapRL-3B once using the GRPO pipeline on a curated 75k-image QA dataset (one-time cost), (2) run inference with CapRL-3B on millions of new images to generate captions (low per-image cost due to the 3B model size), (3) use these captions as pretraining data for larger LVLMs. The paper demonstrates this at 5M scale (CapRL-5M) with +2.3 to +2.6 point average gains over 12 benchmarks when scaling from 1M to 5M samples (Table 1), and the scaling trend in Figure 4 shows no saturation at 1M, suggesting that even larger datasets would yield continued improvements. For organizations building LVLMs from scratch or fine-tuning on domain-specific images, this provides a path to high-quality pretraining data without relying on proprietary models (GPT-4V) or expensive human annotation.

2. Dense captioning for document and chart understanding applications. CapRL-3B's largest gains over baselines are on InfoVQA (+24.6 points over the untrained 3B model, exceeding even the 72B model by 4.0 points in Table 3) and ChartQA (+14.9 points). These benchmarks require extracting precise numerical values, text labels, and structural information from infographics, charts, and documents — exactly the capability that CapRL's VQA-based reward incentivizes. A practical deployment would use CapRL-3B (or a fine-tuned variant) as a captioning front-end for document VQA systems: given a document image, CapRL produces a dense structured caption, and a lightweight text-only QA model (which is much cheaper than a full LVLM) answers user queries by reading only the caption. This decoupled architecture (CapRL captioner + text QA model) could serve queries at substantially lower latency and cost than running a large LVLM on every user request, because the expensive visual processing happens once per document (during captioning) while the cheap text-based QA handles all subsequent questions. The paper's results on InfoVQA and DocVQA suggest this approach would be particularly effective for financial documents, scientific papers, and administrative forms where users ask multiple questions about the same document.

3. Bootstrapping domain-specific vision-language capabilities with minimal labeled data. The paper's finding that sparse QA supervision is sufficient (Table 4: 1 QA per image achieves 48.0% vs. 48.5% for 3 QA per image) and that domain-specific training generalizes across domains (Figure 5, Left: training on charts improves natural image captioning) has a direct practical implication: CapRL can be applied to new domains with very little per-domain annotation effort. A practitioner working in a specialized domain (medical imaging, satellite imagery, industrial inspection) would: (1) collect a few thousand in-domain images, (2) use a large general LVLM (or, ideally, a domain-expert model if available) to generate one or two QA pairs per image, (3) filter for visual grounding using Equation 4, (4) train CapRL on this small in-domain QA dataset, (5) use the resulting model to annotate a much larger corpus of in-domain images. The paper's scaling results (Figure 5, Right) suggest that even 2.5k training images yield significant improvements over the baseline, making this feasible for domains where large annotated datasets do not exist. The key insight is that the QA pairs need only be good enough to provide a reward signal — they don't need to cover every possible question about every image, because the model generalizes captioning skills learned from sparse rewards.

4. Improving open-source LVLM pretraining datasets to reduce dependence on proprietary models. The paper notes that current state-of-the-art captioning pipelines rely on "expensive, non-scalable data annotated by humans or proprietary models" (Abstract). CapRL-3B is fully open-source (trained from Qwen2.5-VL-3B, which is publicly available) and the paper releases the CapRL-5M dataset. This directly addresses the reproducibility and cost barriers that have made high-quality caption data a bottleneck for the open-source multimodal community. An open-source LVLM project could replace GPT-4V-generated captions (used in ShareGPT4V, ALLaVA, and similar datasets) with CapRL-3B-generated captions, eliminating API costs and enabling unlimited scaling — the only cost is GPU inference time for the 3B model, which is trivially parallelizable. The paper demonstrates that CapRL captions outperform ShareGPT4V captions on downstream pretraining (Table 1: CapRL-1M at 59.7% average vs. ShareGPT4V-1M at 56.7% under Qwen2.5-3B), so this substitution is not merely cost-saving but quality-improving as well.

---

When to Prefer This Method

The paper does not explicitly articulate a decision rule comparing CapRL against named alternatives (e.g., "use CapRL when X, use SFT when Y"), but the results imply clear boundary conditions that can be extracted into a practical guide. A conditional recommendation derived from the evidence is:

• Prefer CapRL for training captioners when the downstream use case involves factual information extraction from structured or semi-structured visual content (charts, documents, infographics, UI screenshots, scientific figures). The paper's largest gains are in these domains (InfoVQA: +24.6 points; ChartQA: +14.9 points; CharXiv: +10.6 points), and the mechanism — VQA-based reward incentivizing precise transcription of visual details — is directly aligned with the task requirements.

• Prefer SFT (or the base model without additional training) when the primary evaluation criterion involves visual reasoning, inference, or abstraction beyond factual extraction. The paper shows that CapRL-3B trails Qwen2.5-VL-72B on MMMU (38.1% vs. 39.4%), MMMU-Pro (30.7% vs. 34.1%), and MathVerse (36.4% vs. 39.9%), and provides essentially zero improvement on WeMath (23.4% vs. 22.6% baseline). For applications where users ask "why" or "what if" questions about images rather than "what" or "how many," CapRL's reward structure does not target the relevant capability.

• Prefer CapRL when a QA dataset can be constructed with acceptable cost and a sufficiently capable QA generator model is available. The paper uses Qwen2.5-VL-72B for QA generation; practitioners without access to a model of comparable capability in their domain (e.g., specialized medical or scientific imagery where even large general LVLMs may not generate high-quality questions) should verify QA quality before committing to the CapRL pipeline. The ablation in Table 6 shows that low-quality QA data degrades performance, and the paper provides no evidence on the minimum viable QA generator capability.

• Prefer CapRL over LVLM-as-a-Judge or reward model approaches for any RL-based captioning training. The paper's direct comparison (Table 3) shows that both subjective reward approaches underperform CapRL — UnifiedReward causes brevity collapse (38.4% average, below the untrained baseline) and Qwen2.5-VL-as-Judge produces verbose but information-sparse captions (42.5%). There is no regime in the paper's results where a subjective reward approach is preferable to CapRL for training captioners via RL. This is the strongest decision rule the paper supports: if you are using RL to train a captioner, use a utility-based verifiable reward rather than a holistic judge.