InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model

ArXiv: 2501.12368

🎯 Pitch

A single open-source 7B reward model matches or beats proprietary giants like GPT-4o on multi-modal preference benchmarks, then simultaneously improves chatbot performance through RL training, boosts accuracy via test-time response selection, and automatically detects bad training data—all without needing separate specialized tools.

---

1. Executive Summary

This paper introduces IXC-2.5-Reward, a simple yet effective discriminative multi-modal reward model that aligns Large Vision Language Models (LVLMs) with human preferences by predicting scalar reward scores from image, video, and text inputs. Built upon the InternLM-XComposer-2.5 SFT model with an added score head and trained on a newly constructed multi-modal preference dataset spanning instruction following, text-rich documents, mathematical reasoning, and video understanding, IXC-2.5-Reward achieves state-of-the-art results on VL-RewardBench (70.0% macro accuracy, outperforming all proprietary generative RMs including GPT-4o at 62.4%) while also demonstrating competitive performance on text-only reward benchmarks (88.6% on Reward-Bench, 68.8% on RM-Bench). The authors further demonstrate three key applications: providing supervisory signals for PPO-based RL training that yields meaningful improvements in instruction following and multi-modal open-ended dialogue across WildVision, MIA, MM-MT, and MM-Vet benchmarks; enabling test-time scaling via best-of-N sampling for additional performance gains beyond RL training alone; and filtering outlier or noisy samples from instruction tuning datasets through the observed strong correlation between low reward scores and problematic examples, establishing that a single open-source 7B reward model can serve as a unified foundation for RLHF, inference-time optimization, and data quality control in multi-modal settings.

2. Context and Motivation

The Core Problem: Multi-Modal Reward Models Are Practically Non-Existent in the Open-Source Ecosystem

The central gap this paper addresses is straightforward but consequential: open-source, general-purpose reward models for Large Vision Language Models (LVLMs) essentially do not exist. While the LLM community has developed a rich ecosystem of reward models — discriminative classifiers like InternLM2-Reward (Cai et al., 2024) and Skywork-Reward (Liu et al., 2024a), generative evaluators like Nemotron-4-340B-Reward (Wang et al., 2024f), and implicit DPO-based models — the multi-modal counterpart has lagged dramatically. The paper states this explicitly in Section 1:

"publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear."

This is not merely an academic observation. The absence of multi-modal RMs creates a cascading set of practical constraints on what open-source LVLM developers can do. Without a reward model, three critical capabilities — each well-established in the LLM domain — remain out of reach for LVLMs:

1. On-policy reinforcement learning with human feedback (RLHF). The dominant paradigm for aligning LLMs with user preferences involves training a reward model on human preference data, then using that RM to provide dense supervisory signals for PPO-based optimization (Ouyang et al., 2022; Bai et al., 2022b). Without a multi-modal RM, LVLM developers cannot run on-policy RL algorithms like PPO, RLOO (Ahmadian et al., 2024), or GRPO (Shao et al., 2024). Instead, they are forced to rely on off-policy alternatives — most commonly Direct Preference Optimization (DPO) (Rafailov et al., 2024) and its variants — which the paper explicitly notes "potentially compromise performance compared to the on-policy RL solutions like PPO" (Section 2). This is not a minor implementation detail. Off-policy methods optimize against a fixed dataset of preferences collected from a previous model version, while on-policy methods continuously generate new responses from the current policy and receive fresh reward signals. The gap between these approaches can be substantial, especially as the policy model diverges from the data-collection model during training.

2. Test-time scaling via best-of-N or search. The paper we analyzed in detail earlier — Snell et al. (2024) on compute-optimal test-time scaling — demonstrated that a reward model (specifically, a Process Reward Model) enables systematic improvements through best-of-N sampling, beam search, and lookahead search at inference time. That work showed that a smaller model with compute-optimal test-time strategies can outperform a ~14× larger model on problems within its capability range. But the entire framework depends on having a reliable verifier. For LVLMs, the absence of multi-modal RMs means this entire paradigm of inference-time optimization — selecting the best output from multiple candidates, performing rejection sampling, or guiding iterative refinement — is unavailable. The paper explicitly identifies this as motivation:

"The growing community interest in RLHF and test-time scaling highlights the need for multi-modal RMs" (Section 1).

3. Automated data quality control. Large-scale vision-language datasets are noisy. Image-question mismatches, hallucinated answers, empty responses, and irrelevant pairings are common in widely-used instruction tuning datasets like ALLaVA (Chen et al., 2024a) and LLaVA-Video-178K (Zhang et al., 2024e). Existing filtering approaches typically use CLIP-based similarity scores (Radford et al., 2021), which operate only on image-text alignment and fail to detect more subtle issues like hallucinated content in responses, poor instruction following, or problems requiring long-context understanding (Zhang et al., 2025a). A multi-modal RM that scores the full (image/video + question + answer) tuple offers a more comprehensive quality signal, but this has been unavailable.

Why This Gap Exists: The Data Scarcity Bottleneck

The paper identifies a structural reason for this gap that goes deeper than "nobody has built one yet." Training a reward model requires pairwise preference data — triples of (prompt, chosen response, rejected response) that reflect human judgments about response quality. For language-only LLMs, such data has become abundant: datasets like UltraFeedback (Cui et al., 2024), HelpSteer2 (Wang et al., 2024f), and the Tulu-3 preference collection (Lambert et al., 2024a) provide hundreds of thousands of high-quality preference pairs across diverse domains (chat, safety, reasoning, instruction following).

For multi-modal settings, the situation is fundamentally different:

• Existing multi-modal preference data is scarce and narrow. The paper notes in Section 2 that "current preference data is predominantly text-based and skewed toward specific domains (e.g., safety)." The limited multi-modal preference data that does exist covers only a fraction of the domains needed for a general-purpose RM.

• Existing multi-modal RMs are domain-restricted or methodologically limited. The paper catalogs three categories of prior approaches, each with significant shortcomings:

  1. Domain-specific discriminative RMs. Models like RoVRM (Wang et al., 2024a) focus narrowly on hallucination reduction. While useful for that specific problem, they don't generalize to evaluating instruction following quality, reasoning correctness, or conversational helpfulness — precisely the dimensions that matter for real-world chat applications.

  2. Generative RMs via LVLM prompting. Several works (Yu et al., 2024c; Ouali et al., 2025; Xiyao et al., 2024) repurpose existing LVLMs as evaluators by prompting them with evaluation criteria and parsing their textual output. This approach — sometimes called "LLM-as-a-judge" — has three limitations. First, it is expensive: running a full LVLM forward pass to produce a textual critique is far more computationally costly than a discriminative model predicting a single scalar score. Second, it is brittle: the quality of the evaluation depends on prompt engineering, and the model may exhibit systematic biases (e.g., length bias, position bias) that are hard to control. Third, as the VL-RewardBench results in Table 3 demonstrate, the best proprietary generative RMs (GPT-4o at 62.4%, Gemini-1.5-Pro at 62.5%) significantly underperform the discriminative IXC-2.5-Reward (70.0%) on multi-modal preference judgments, suggesting that generative evaluation is inherently less reliable for this task.

  3. Reliance on proprietary models for preference labeling. Several prior LVLM alignment efforts (Zhang et al., 2024a; Zhao et al., 2023) use GPT-4 or GPT-4V as a "generative RM" to construct DPO preference pairs — generating responses, asking the proprietary model to judge which is better, and training on those judgments. The paper correctly identifies this as "expensive and unsustainable for large-scale applications" (Section 2). At scale, routing every preference judgment through a paid API is economically infeasible and introduces dependencies on models whose behavior may change unpredictably.

• No process supervision capability. The paper notes that existing multi-modal RM benchmarks (specifically VL-RewardBench; Li et al., 2024b) "focus solely on generative RMs and lack the evaluation of process supervision" (Section 2). This means not only are there no process reward models (PRMs) for multi-modal reasoning — there isn't even a benchmark to evaluate one. PRMs, which score individual reasoning steps rather than just final answers, were shown by Lightman et al. (2023) and the Snell et al. (2024) paper to be critical for effective test-time search. The absence of multi-modal PRMs means that sophisticated inference-time optimization techniques (beam search over reasoning chains, lookahead search) are entirely unavailable for visual reasoning tasks.

The Broader Implications: A Bottleneck on LVLM Progress

The unavailability of multi-modal RMs is not an isolated inconvenience — it represents a systematic bottleneck on the entire LVLM development pipeline. The paper frames this through three interconnected impacts:

Training bottleneck. Without RMs, LVLM developers cannot run on-policy RL, which has been the most effective method for improving instruction following and conversational quality in LLMs. The paper's own results (Table 6) demonstrate what becomes possible when this bottleneck is removed: IXC-2.5-Chat, trained with PPO using IXC-2.5-Reward, shows dramatic improvements on instruction-following benchmarks — WildVision jumps from 37.5 (SFT model) to 74.6, and MIA improves from 80.4 to 84.0. These are gains of a magnitude that off-policy DPO has not demonstrated for LVLMs, suggesting that the absence of on-policy RL was indeed holding back performance.

Inference bottleneck. Test-time scaling — selecting the best output from multiple candidates — is one of the simplest and most reliable ways to improve generation quality without modifying the model. The paper's Best-of-N results (Table 8) show that even a modest N=4 sampling with IXC-2.5-Reward selection improves WildVision from 74.6 to 77.7. Without a multi-modal RM, this capability is unavailable, forcing users to accept whatever single output the LVLM produces.

Data bottleneck. The observation that low reward scores correlate with genuinely problematic samples (Figure 2) suggests that multi-modal RMs can serve as automated data quality filters — a capability that becomes increasingly important as the community scales up multi-modal datasets. Manual inspection of every (image, question, answer) triple is infeasible at scale, and CLIP-based filtering only catches image-text mismatches, not response quality issues.

How This Paper Positions Itself

The paper's positioning is straightforward: it argues that rather than transferring uni-modal (text-only) reward models to the vision domain or relying on expensive proprietary models as evaluators, the community needs a purpose-built, open-source, discriminative multi-modal reward model trained on diverse preference data spanning multiple modalities and domains. The key design choices that distinguish IXC-2.5-Reward from prior efforts:

• Discriminative rather than generative architecture. By predicting a scalar reward score through a dedicated score head rather than generating textual critiques, IXC-2.5-Reward is both more computationally efficient at inference time and — as the VL-RewardBench results demonstrate — more accurate at preference judgments. The paper belongs to the discriminative RM and Outcome RM (ORM) family, as explicitly stated in Section 2.

• Built upon a strong, aligned LVLM backbone. Rather than training a reward model from scratch, the paper leverages the pretrained InternLM-XComposer-2.5 model, which has already aligned visual and textual modalities through its SFT training. This means the RM training can focus exclusively on learning preference judgments without needing to re-learn modality alignment — the vision encoder and projector are frozen during RM training (Section 3).

• Comprehensive domain coverage. The paper constructs a new multi-modal preference dataset that spans text, image, and video inputs across instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding (Tables 1, 2). This diversity is what enables IXC-2.5-Reward to serve as a general-purpose RM rather than a domain-specific one.

• Demonstrated utility across the full pipeline. Rather than merely reporting benchmark scores, the paper validates IXC-2.5-Reward across three distinct applications (RL training, test-time scaling, data cleaning), demonstrating that a single reward model can serve as infrastructure for the entire LVLM development lifecycle. This is a strategic positioning: the paper is arguing not just "we built a good RM," but "a good RM unlocks capabilities that the LVLM community has been missing."

The paper explicitly connects to the broader trajectory of the field in Section 2:

"Although open-source RMs for LVLMs have lagged behind their LLM counterparts, the growing community interest highlights the need for multi-modal RMs, which motivates our work."

This framing positions IXC-2.5-Reward not as an incremental improvement over existing multi-modal RMs (there are few to improve upon), but as filling a fundamental infrastructure gap that has been holding back the entire open-source LVLM ecosystem. The parallel to language-only RMs is explicit throughout: just as InternLM2-Reward, Skywork-Reward, and others enabled on-policy RL and test-time scaling for LLMs, IXC-2.5-Reward aims to enable those same capabilities for LVLMs.

3. Technical Approach

3.1 Reader Orientation

IXC-2.5-Reward is a discriminative multi-modal reward model — a system that takes an image, video, or text prompt plus a candidate response, and produces a single scalar score indicating how well that response aligns with human preferences. The core problem it solves is providing a reliable, efficient, and general-purpose evaluator for Large Vision Language Models that works across diverse domains (chat, reasoning, text-rich documents, video understanding) and modalities, enabling the three downstream applications that have been standard in language-only settings but effectively unavailable for vision-language models: on-policy RL training with PPO, test-time best-of-N selection, and automated data quality filtering.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, arranged in a pipeline from data to deployment:

1. Multi-Modal Preference Data Pipeline — collects prompts from diverse domains (general chat, text-rich documents, math reasoning, video understanding), generates candidate responses from the base LVLM, and labels which responses are preferred using GPT-4o judgments or verifier functions. This produces the (prompt, chosen_response, rejected_response) triples needed for RM training.

2. IXC-2.5-Reward Model Architecture — the core discriminative model, built by taking the pretrained InternLM-XComposer-2.5 SFT model, freezing its vision encoder and modality-alignment projector, and replacing its final language-modeling head with a scalar score head. Given a (prompt, response) pair, the model processes it through the frozen visual pipeline and trainable LLM, averages the hidden states of all tokens, and projects to a single scalar reward score.

3. PPO Training Pipeline (Consumer #1) — uses IXC-2.5-Reward as the frozen reward provider while training a policy model (IXC-2.5-Chat) via Proximal Policy Optimization. A separate critic model, initialized from IXC-2.5-Reward, estimates value functions to reduce gradient variance. The policy generates responses on-policy, the RM scores them, and the PPO objective updates the policy to maximize expected reward while staying close to a reference model.

4. Test-Time Scaling and Data Cleaning (Consumers #2 and #3) — at inference time, the policy model generates N candidate responses with different random seeds, IXC-2.5-Reward scores each, and the highest-scoring response is selected (Best-of-N). For data cleaning, IXC-2.5-Reward scores existing instruction-tuning samples, and those with anomalously low scores are flagged as likely problematic (hallucinations, mismatched image-text pairs, empty answers).

Information flows as follows: diverse prompts → response generation → preference labeling → RM training on preference pairs → frozen RM provides scalar rewards → PPO updates policy model → trained policy generates candidates → RM selects best candidate (test-time) or flags low-quality samples (data cleaning).

3.3 Roadmap for the Deep Dive

• First, the data construction pipeline — since the quality and diversity of preference data fundamentally determines what the RM can learn, and because the paper's core contribution is partly a dataset contribution filling the multi-modal preference gap.
• Second, the model architecture and training — how the scalar reward head works, what gets frozen vs. trained, the loss function, and the critical length-constraint design choice.
• Third, the PPO training procedure — how the frozen RM provides rewards, how the advantage estimation and policy gradient work, and the interaction between the policy, reference, RM, and critic models.
• Fourth, the test-time scaling mechanism — the Best-of-N sampling procedure and why it produces gains beyond RL training alone.
• Fifth, the data cleaning application — the observed correlation between low reward scores and problematic samples, and what this enables for dataset curation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and infrastructure paper whose core idea is that building a general-purpose discriminative multi-modal reward model — by constructing diverse preference data and training a scalar prediction head on a strong LVLM backbone — unlocks on-policy RL, test-time scaling, and automated data filtering for the open-source LVLM ecosystem, capabilities that were previously unavailable.

---

3.4.1 Multi-Modal Preference Data Construction

The reward model's effectiveness is bounded by the quality and diversity of its training data. Since publicly available multi-modal preference data is scarce and narrow (Section 2), the paper constructs a new dataset that complements existing open-source collections. The data construction follows a systematic pipeline: select prompts across diverse domains and modalities → generate candidate responses from the base LVLM → apply preference judgments using either GPT-4o or verifier functions → filter by length constraints.

Prompt sources (Tables 1 and 2). The paper draws prompts from two categories. Existing open-source datasets (Table 1) provide text-only preference data covering instruction following (Tulu-3-IF-augmented-on-policy-8b, UltraFeedback), safety (HHH alignment, PKU-Safe, SHP, Anthropic-HHRLHF), and image-based chat and general understanding (WildVision-Battle, LLaVA-Critic, VL-Feedback, RLAIF-V, MIA-DPO). These datasets already contain preference annotations — they provide (prompt, chosen, rejected) triples directly — so they require no additional labeling.

Newly collected data (Table 2) fills domain gaps where preference annotations do not exist. The paper identifies prompts from supervised fine-tuning (SFT) datasets across four under-represented categories:

• Image instruction following and general VQA: MM-IFDPO-23k for multi-modal instruction following, plus KVQA, A-OKVQA, and PMC-VQA for knowledge-aware visual question answering.
• Text-rich document understanding: AI2D (scientific diagrams), IconQA (abstract diagram reasoning), TextbookQA (textbook question answering), ChartQA (chart reasoning), DVQA (data visualization QA), and ScienceQA (science reasoning with explanations).
• Mathematical reasoning: GeoQA (geometry), CLEVR-Math (compositional visual math), Super-CLEVR (complex visual reasoning), and TabMWP (tabular math word problems).
• Video understanding: TrafficQA (video reasoning about traffic events), FunQA (surprising video comprehension), and MiraData (long-duration video understanding).

For each prompt in these new domains, the paper generates the chosen response y_c from the SFT model InternLM-XComposer-2.5 (IXC-2.5). This is the model's standard supervised output, treated as the positive example.

Rejected response generation. To obtain rejected responses y_r — examples of what not to produce — the paper generates multiple candidate outputs from IXC-2.5 for each prompt (the exact number of candidates is not specified) and then applies distinct selection criteria depending on the domain. For general understanding and text-rich data, GPT-4o serves as the preference judge: the paper provides GPT-4o with a pairwise evaluation prompt (the specific prompt template is not disclosed) comparing each candidate response against the SFT ground-truth answer, and the candidate evaluated as worse becomes the rejected response. For math reasoning and instruction following data, the paper uses verifier functions (following Lambert et al., 2024a) — programmatic checks that compare generated responses against ground-truth solutions. For example, a math verifier would extract the final answer from the generated response and compare it to the known correct answer; responses with incorrect answers become rejected examples. The paper states:

"For math reasoning and instruction following data, we build verifier functions that compare generated responses against ground-truth solutions to label the chosen and rejected data."

Length constraints. A critical filtering step removes data pairs where the chosen response is significantly longer than the rejected response. The motivation is to prevent the reward model from learning a spurious correlation between response length and quality — a known vulnerability of LLM-based evaluation (Dubois et al., 2024) where longer responses tend to receive higher scores regardless of content quality. The paper notes that this length bias "has also significant implications for LVLMs," specifically that open-ended VQA benchmarks using GPT-4o as a judge "are susceptible to inflated scores from overly long responses." The effect of this constraint is demonstrated in an ablation (Table 7): removing length constraints during RM training causes the downstream PPO-trained chat model to produce substantially longer responses (average 361 tokens vs. 274 tokens on WildVision) and achieve inflated benchmark scores because the GPT-4o judge model itself exhibits length bias. The paper retains the length constraint in the main experiments to optimize for genuine user experience rather than benchmark scores.

What this pipeline produces. The final dataset consists of (prompt x, chosen response y_w, rejected response y_l) triples spanning text, image, and video inputs across instruction following, safety, general knowledge, text-rich document understanding, mathematical reasoning, and video understanding domains. This diversity is what enables IXC-2.5-Reward to serve as a general-purpose RM rather than a domain-specific one.

---

3.4.2 Model Architecture and Reward Model Training

IXC-2.5-Reward is a discriminative reward model built by modifying the InternLM-XComposer-2.5 (IXC-2.5) supervised fine-tuned model. The core architectural change is replacing the language modeling head with a scalar score predictor, while the vision encoder and modality-alignment components are frozen.

Base model components (frozen). The pretrained IXC-2.5 model consists of three main components: a vision encoder that processes image and video inputs into visual feature representations, an MLP projector that maps these visual features into the token embedding space of the language model (achieving modality alignment), and a Large Language Model (InternLM-2-7B) that processes the combined visual and textual token sequence. The vision encoder and MLP projector are frozen during RM training because "the IXC-2.5-Reward is merely required to train preference data to predict the reward score and avoid using other pre-training data for modality alignment." In other words, the modality alignment learned during IXC-2.5's SFT training is already sufficient, and the RM training should focus exclusively on learning preference judgments. Freezing these components also prevents catastrophic forgetting of visual understanding capabilities and reduces training cost.

Score head (trainable). The final linear layer of IXC-2.5 — which normally projects hidden states to vocabulary logits for next-token prediction — is replaced with a score head f. Given an input consisting of the prompt x and response y concatenated as a single token sequence, the model processes the entire sequence through the vision encoder (if visual inputs are present), projector, and LLM. The score head f then takes the averaged hidden state features of all tokens in the sequence and transforms them into a single scalar value r(x, y):

$r(x, y) = f\left(\frac{1}{T}\sum_{t=1}^{T} h_t\right)$

This is an implicit description — the paper does not provide the exact equation but states:

"the score head f transforms the averaged hidden state features of all tokens into a binary scalar r(x, y). This scalar value r(x, y) serves as the predicted reward score for the inputs."

What it computes: The hidden state h_t at each token position t is a vector representing the model's contextualized understanding at that position after processing the entire (prompt, response) sequence. Averaging across all T tokens produces a single fixed-size vector that summarizes the model's overall assessment of the full input. The score head f — a learned linear transformation (and possibly additional layers, though the paper doesn't specify the exact architecture beyond it being a replacement for the final linear layer) — maps this summary vector to a single scalar. The output r(x, y) is the predicted reward: higher values indicate responses that better align with human preferences.

Why this form: Averaging across all tokens rather than using only the final token's hidden state (as is common in some sequence classification approaches) ensures that the reward prediction incorporates information from the entire response, not just its conclusion. This is important because reward-relevant signals — factual errors, poor formatting, incomplete reasoning — can appear anywhere in the response. Using only the final token would make the reward model vulnerable to responses that end well despite containing errors earlier. The scalar output (rather than a multi-dimensional reward vector or textual critique) makes the model efficient to use in RL training and test-time scaling, where it needs to score many candidate responses quickly.

Trainable parameters. Only the LLM (InternLM-2-7B) and the score head are trained. The vision encoder and projector remain frozen at their IXC-2.5-initialized values. This is a significant design choice: it means the RM training is essentially a text-only fine-tuning task (since the visual processing pipeline is fixed), which is computationally efficient and avoids the need for large-scale multi-modal pre-training data during RM training.

Other components of IXC-2.5 remain unchanged, including the dynamic image partitioning mechanism for high-resolution inputs — the model can still process images at multiple resolutions by splitting them into sub-images and encoding each separately, a capability inherited from the base model.

Loss function. The reward model is trained using the standard Bradley-Terry preference modeling loss, which is the dominant approach for training discriminative reward models (used by InternLM2-Reward, Skywork-Reward, and others). The loss function is:

$\mathcal{L}_{RM} = -\mathbb{E}\left[\log\left(\sigma\left(r(x, y_w) - r(x, y_l)\right)\right)\right]$

where $r(x, y_w)$ is the reward score assigned to the prompt x with the chosen (preferred) response y_w, $r(x, y_l)$ is the reward score for the same prompt with the rejected response y_l, and $\sigma(\cdot)$ is the sigmoid function that maps any real value to the (0, 1) interval.

What it computes: The term inside the sigmoid, $r(x, y_w) - r(x, y_l)$, is the difference in reward scores between the chosen and rejected responses. If the model correctly assigns a higher score to the chosen response, this difference is positive, $\sigma$ of a positive value is close to 1, $\log(1) = 0$, and the loss is near zero — the model is doing well. If the model incorrectly assigns a higher score to the rejected response, the difference is negative, $\sigma$ of a negative value is close to 0, $\log(0) \to -\infty$, and the loss is large — the model is penalized heavily. The expectation $\mathbb{E}$ averages this loss over all preference pairs in the training data, producing a single scalar that measures how well the model's reward scores align with the observed preferences.

Why this form: This loss is derived from the Bradley-Terry model of paired comparisons, which assumes that the probability of preferring response A over response B is proportional to $\exp(r(A)) / (\exp(r(A)) + \exp(r(B)))$. Under this model, the log-likelihood of observing the preference data given the reward scores is exactly the negative of the loss above. Minimizing this loss is equivalent to maximum likelihood estimation of the reward scores under the Bradley-Terry model. The key property this form provides is calibration of relative scores: the model learns not just to say "this is good" or "this is bad" in absolute terms, but to make fine-grained relative judgments — "this response is slightly better than that one" — which is exactly what's needed for RL training (where the reward signal guides gradient updates) and best-of-N selection (where the RM must distinguish among multiple candidate responses of varying quality).

Training hyperparameters. The paper reports learning rates of 1e-5 with a batch size of 256 for IXC-2.5-Reward training (Appendix A). Additional hyperparameters (optimizer choice, weight decay, learning rate schedule, number of epochs) are not specified.

What the training produces. After training on the multi-modal preference dataset, IXC-2.5-Reward can take any (image/video/text prompt, response) pair and output a scalar reward score that reflects human preference alignment. This frozen model then serves as the reward provider for PPO training, the selection mechanism for test-time scaling, and the quality filter for data cleaning.

---

3.4.3 PPO Training with IXC-2.5-Reward (Producing IXC-2.5-Chat)

With a frozen reward model in hand, the paper applies Proximal Policy Optimization (PPO) to train a chat-optimized policy model called IXC-2.5-Chat. This is an on-policy RL procedure where the policy model generates responses, the RM scores them, and the policy is updated to increase expected rewards while remaining close to a reference model for stability.

Prompt set curation. Before PPO training begins, the paper curates a prompt set from the same multi-modal preference data used to train IXC-2.5-Reward. The curation prioritizes general chat and instruction following prompts — the domains where improved conversational quality matters most for user experience — while ensuring diversity through inclusion of text-rich documents, math reasoning, and video understanding prompts. The paper notes:

"Similar to findings in (Hou et al., 2024), we found that average reward scores differ across task domains (e.g., general, text-rich, reasoning)."

This domain-dependent reward baseline matters because if the PPO training only sees prompts from high-reward domains, the policy might learn to exploit domain-specific patterns rather than genuinely improving response quality. Including diverse domains ensures the policy improves broadly.

Actor-critic architecture. The PPO setup involves four models:

1. Policy model $\pi_\theta$ (IXC-2.5-Chat): The model being trained. Initialized from the SFT model IXC-2.5. This is the actor that generates responses and gets updated by the policy gradient.

2. Reference model $\pi_{ref}$ (IXC-2.5, frozen): A frozen copy of the original SFT model. Used to compute a KL-divergence penalty that prevents the policy from drifting too far from its starting point, which is a standard stabilization technique in RLHF.

3. Reward model (IXC-2.5-Reward, frozen): The trained discriminative RM. Provides scalar reward scores $r_t$ for the policy's generated responses. Frozen during PPO — its parameters are never updated.

4. Critic model $V$ (initialized from IXC-2.5-Reward): A value function that estimates the expected future reward from each state. The paper states it is "initialized from IXC-2.5-Reward" — meaning it starts with the RM's weights (which already encode good representations for response quality) and is then fine-tuned during PPO to predict cumulative returns rather than immediate rewards. The critic is trained alongside the policy to reduce the variance of policy updates.

The PPO training loop (per step). At each training step:

1. Sampling: A prompt x is sampled from the curated prompt set.

2. Response generation: The current policy model $\pi_\theta$ generates a response token by token. At each time step t, the policy produces a token, forming a partial response that defines the current state $s_t$.

3. Reward computation: The frozen reward model IXC-2.5-Reward computes a reward score $r_t$ at each state $s_t$. The paper does not specify whether the reward is computed only at the final token (outcome reward) or at intermediate steps (process reward). Given that IXC-2.5-Reward is described as an Outcome RM (ORM) in Section 2, and the architecture averages hidden states across all tokens, it's likely that $r_t$ is zero for all intermediate steps and the full reward is assigned at the final token. However, the PPO formulation uses per-timestep notation, so $r_t$ is used generically.

4. Advantage and return computation. The critic model $V$ estimates the value of each state $V(s_t)$. Using these value estimates and the observed rewards, the paper computes three quantities via Generalized Advantage Estimation (GAE):

   • Temporal difference error $\delta_t$: The difference between the observed reward plus discounted next-state value, and the current state value:

   $\delta_t = r_t + \gamma \cdot V(s_{t+1}) - V(s_t)$

   where $\gamma = 0.99$ is the discount factor that determines how much future rewards are valued compared to immediate rewards. A value close to 1 means the policy cares almost equally about near-term and far-term rewards; a lower value would make the policy myopic.

   What it computes: $\delta_t$ measures the "surprise" at time t — how much better or worse the outcome was than the critic predicted. If the critic correctly predicted the reward and next-state value, $\delta_t \approx 0$. If the outcome was better than expected, $\delta_t$ is positive.

   Why this form: This is the standard TD error from temporal difference learning. It decomposes the credit assignment problem — instead of trying to figure out which action caused a reward 20 steps later, we only need to assess whether each step improved the situation relative to expectations.

   • Generalized Advantage Estimation $A_t$: A weighted sum of future TD errors that estimates how much better the policy's action was compared to the average (the critic's baseline):

   $A_t = \delta_t + \gamma \cdot \beta \cdot A_{t+1}$

   where $\beta = 0.95$ is the GAE parameter that controls the bias-variance tradeoff in advantage estimation.

   What it computes: $A_t$ is an exponentially weighted sum of all future TD errors, with weight $(\gamma\beta)^k$ for the error $k$ steps in the future. If $\beta = 1$, it's a full Monte Carlo return with high variance; if $\beta = 0$, it's a one-step TD error with high bias. The value $\beta = 0.95$ provides a balanced estimate that captures multi-step credit assignment while remaining reasonably low-variance.

   Why this form: GAE (Schulman et al., 2018) is the standard advantage estimator in PPO implementations. The parameter $\beta$ (often called $\lambda$ in the original GAE paper) allows practitioners to trade off between the high-variance but unbiased Monte Carlo returns and the low-variance but biased one-step TD errors. The value 0.95 means the advantage estimate looks approximately 20 steps into the future before the weights become negligible.

   • Returns $R_t$: The total cumulative reward from time t onward, estimated by adding the advantage back to the critic's value estimate:

   $R_t = A_t + V(s_t)$

   What it computes: Since $A_t$ measures how much better the action was than expected, adding the expected value $V(s_t)$ gives the actual (estimated) total return. If $A_t > 0$, the action was better than average; if $A_t < 0$, it was worse.

   Why this form: This decomposition separates the predictable component $V(s_t)$ (what the critic can anticipate) from the unpredictable component $A_t$ (what the specific action contributed). The critic is trained to predict returns, and the policy is trained to maximize advantages — learning to produce actions that lead to better-than-expected outcomes.

5. Policy update. The policy model $\pi_\theta$ is updated using the PPO clipped objective:

$\mathcal{L}_{PG} = \min\left(\frac{\pi_\theta}{\pi_{ref}} \cdot A, \text{clip}\left(\frac{\pi_\theta}{\pi_{ref}}, 1.0 - \epsilon, 1.0 + \epsilon\right) \cdot A\right)$

where $\pi_\theta / \pi_{ref}$ is the ratio of probabilities assigned to the generated token by the current policy versus the reference policy (both are computed as log-probabilities and exponentiated), $A$ is the advantage from GAE, and $\epsilon = 0.2$ is the clipping parameter.

What it computes: For each token, the policy gradient loss encourages the policy to increase the probability of tokens that led to positive advantages and decrease the probability of tokens that led to negative advantages. The ratio $\pi_\theta / \pi_{ref}$ measures how much the current policy has already changed relative to the reference — if this ratio exceeds $1 + \epsilon$, the gradient is clipped to zero (for positive advantages) to prevent the policy from moving too far in a single update. The $\min$ operation selects the more conservative of the unclipped and clipped objectives.

Why this form: The clipping mechanism (Schulman et al., 2017) is the key innovation of PPO over earlier policy gradient methods like TRPO. Without clipping, a single large advantage could cause the policy to make an unrecoverably large update — assigning near-zero probability to tokens that the reference model considered reasonable. The clipped objective ensures that each update changes the policy by at most a factor of $1 \pm \epsilon$ relative to the reference, maintaining stability. The reference model $\pi_{ref}$ (the frozen IXC-2.5 SFT model) serves as the anchor point — the policy is penalized for straying too far from its original behavior, which prevents reward hacking (finding responses that score highly under the RM but are unnatural or low-quality by other metrics).

6. Critic update. The critic model $V$ is updated to better predict returns using a mean squared error loss:

$\mathcal{L}_{critic} = \sum_t \text{MSE}(V(s_t), R_t)$

What it computes: For each timestep, the squared difference between the critic's predicted value $V(s_t)$ and the actual return $R_t$ (computed in step 4) is summed across all timesteps. Minimizing this makes the critic a better predictor of future rewards, which in turn makes the advantage estimates more accurate, which improves the policy gradient signal.

Why this form: MSE is the standard regression loss. Training the critic on the same trajectories as the policy ensures that the value function remains well-calibrated to the current policy's behavior — as the policy improves and generates higher-quality responses, the critic needs to update its expectations accordingly.

PPO hyperparameters. The paper reports learning rates of 5e-5 with a batch size of 256 for PPO training, plus $\gamma = 0.99$, $\beta = 0.95$, and $\epsilon = 0.2$ (Appendix A). These are standard values for PPO in language domains.

What PPO training produces. After training, IXC-2.5-Chat is a policy model that generates responses optimized to receive high reward scores from IXC-2.5-Reward, while remaining close enough to the original IXC-2.5 SFT model to avoid catastrophic deviation. The paper's results (Table 6) show that this produces substantial improvements on instruction following and chat benchmarks (WildVision: 37.5 → 74.6; MIA: 80.4 → 84.0) without degrading performance on knowledge, reasoning, or text-rich benchmarks.

---

3.4.4 Test-Time Scaling via Best-of-N Sampling

The paper demonstrates that IXC-2.5-Reward enables test-time compute scaling — a capability that was previously unavailable for open-source LVLMs. The specific technique used is Best-of-N (BoN) sampling, which is the simplest form of inference-time optimization.

Procedure. Given a prompt, the trained policy model IXC-2.5-Chat generates $N$ different candidate responses by sampling with different random seeds (the paper uses $N = 4$ in experiments, Table 8). Each response is a complete output generated autoregressively. The frozen reward model IXC-2.5-Reward then scores each of the $N$ candidates by computing $r(x, y_i)$ for each candidate $y_i$. The candidate with the highest reward score is selected as the final output:

$y^* = \arg\max_{y_i \in \{y_1, \ldots, y_N\}} r(x, y_i)$

What it computes: For $N$ independent samples from the policy, the RM acts as a selection mechanism — it evaluates which of the generated responses best aligns with human preferences and returns that one. The computation cost scales linearly with $N$ (each candidate requires a full forward pass through the RM), but the RM's discriminative architecture makes this substantially cheaper than running a generative evaluator that would need to produce a textual critique for each candidate.

Why this works: Best-of-N improves performance through a selection effect: even if the policy model has a relatively low probability of producing an excellent response on any single attempt, generating multiple independent samples increases the chance that at least one will be high-quality. The RM's job is to identify that high-quality response among the candidates. For this to work, the RM must have a high ranking accuracy — it doesn't need to predict the exact quality of each response, only to correctly determine which of $N$ candidates is best. The Bradley-Terry training objective directly optimizes for this relative ranking ability.

Results. Table 8 shows that Best-of-N with $N = 4$ provides consistent improvements over the base IXC-2.5-Chat: WildVision improves from 74.6 to 77.7, MIA from 84.0 to 87.3, MM-MT from 5.70 to 6.03, and MM-Vet v2 from 54.8 to 56.3. The average response length increases only slightly (from 274 to 283 tokens), indicating that the improvement comes from genuinely better content rather than the length-bias exploitation discussed in Section 3.4.1.

---

3.4.5 Data Cleaning via Reward Score Thresholding

The paper identifies a strong empirical correlation between low IXC-2.5-Reward scores and genuinely problematic training samples. This enables a data cleaning application where the RM serves as an automated quality filter for vision-language instruction tuning datasets.

Observed correlation (Figure 2). The paper visualizes examples from ALLaVA (Chen et al., 2024a) and LLaVA-Video-178K (Zhang et al., 2024e) where IXC-2.5-Reward assigns anomalously low scores:

• A sample with reward score -5.82 where the model's response describes "towering skyscrapers with illuminated windows and the well-lit bridge" for an image that (the paper explains) does not contain these elements — a hallucination case.
• A sample with reward score -3.23 where "the image and text are irrelevant" — the visual content doesn't match the question being asked.
• A sample with reward score -3.42 containing an empty question and answer — a data pipeline error.
• A sample with reward score -3.48 where "the video and text are irrelevant" — a mismatch between video content and the associated QA pair.

Mechanism. The paper does not specify an explicit threshold or automated cleaning protocol, but the principle is clear: by scoring all samples in an instruction tuning dataset with IXC-2.5-Reward, samples with exceptionally low scores (relative to the score distribution) can be flagged for removal or human review. The approach generalizes across modalities (image and video) and error types (hallucination, irrelevance, empty content) — unlike CLIP-based filtering (Radford et al., 2021), which can only detect image-text mismatches, the RM evaluates the full (visual_input, question, answer) tuple.

Why this matters. The paper notes that existing filtering approaches "have limitations, particularly with long-context inputs, high-resolution images, or videos." CLIP operates on fixed-size image embeddings and short text, making it unsuitable for long documents, high-resolution multi-patch images, or video sequences. IXC-2.5-Reward inherits the base model's dynamic image partitioning mechanism and video understanding capabilities, enabling it to evaluate samples that CLIP-based filters would miss.

The paper does not provide quantitative results on data cleaning effectiveness (e.g., precision/recall of detecting problematic samples, downstream model improvement from cleaning), presenting this as a qualitative capability demonstration rather than a rigorously evaluated application. The "Explain" annotations in Figure 2 are from human experts, not the RM itself — the RM only provides the scalar score; interpreting why a sample received a low score requires human inspection.

---

Summary of Design Choices and Their Justifications

• Frozen vision encoder and projector during RM training: avoids re-learning modality alignment, reduces training cost, and prevents catastrophic forgetting of visual understanding — the base model's visual processing is already high-quality from SFT training.
• Averaged hidden states across all tokens (not just the final token): ensures the reward prediction incorporates the entire response content, not just its ending.
• Discriminative scalar output rather than generative evaluation: enables efficient scoring of many candidates for RL and test-time scaling, and achieves higher accuracy than generative RMs (70.0% vs. 62.4% on VL-RewardBench).
• Bradley-Terry loss with length-constrained preference data: directly optimizes for relative ranking ability while preventing the model from exploiting response length as a spurious quality signal.
• PPO with clipped objective and KL reference penalty: enables stable on-policy optimization without catastrophic deviation from the base model's behavior.
• Frozen RM + trainable critic initialized from RM: the RM provides reliable reward signals without being affected by policy updates, while the critic (starting from RM weights) can adapt to the changing policy distribution.
• Best-of-N rather than more complex search: simplicity and computational efficiency — no tree search or lookahead required, just independent sampling and scoring.

4. Key Insights and Innovations

Innovation 1: Reframing Multi-Modal Reward Modeling as an Infrastructure Gap, Not a Methodological One

The paper's most distinctive intellectual move is how it frames the problem. Rather than positioning IXC-2.5-Reward as a novel architecture or training technique — the typical framing for a new model — the paper argues that the primary contribution is filling a fundamental infrastructure gap in the LVLM ecosystem. This reframing matters because it changes the criteria for success: the goal is not to beat existing multi-modal RMs on a benchmark (there are essentially none to beat), but to unlock downstream capabilities that the LLM community takes for granted but that LVLM developers have been unable to access.

Prior to this work, the open-source LVLM community operated under a capability ceiling imposed by the absence of general-purpose multi-modal reward models. Researchers could not run on-policy RL algorithms like PPO, forcing them into off-policy alternatives like DPO that the paper notes "potentially compromise performance" (Section 2). They could not apply test-time scaling via best-of-N selection with a reliable verifier. They could not automatically filter noisy instruction-tuning data beyond simple CLIP-based image-text alignment checks. Each of these limitations was individually acknowledged in prior work, but the paper's contribution is in recognizing them as symptoms of a single underlying gap — the absence of a multi-modal RM — rather than as separate problems requiring separate solutions.

This framing is significant beyond the specific model. It establishes a dependency graph for the LVLM ecosystem: progress on on-policy RL, test-time scaling, and automated data quality control all depend on having a reliable multi-modal RM first. By explicitly demonstrating all three applications with a single model, the paper makes the case that multi-modal RMs are not merely "nice to have" but are prerequisite infrastructure — in the same way that language-only RMs like InternLM2-Reward and Skywork-Reward became infrastructure for the LLM community's RLHF and inference-time optimization pipelines.

The evidence for this reframing is primarily structural rather than numeric: the paper's tripartite application structure (RL training → test-time scaling → data cleaning) is itself the argument. The benchmark results (70.0% on VL-RewardBench, 88.6% on Reward-Bench) validate that the RM works, but the design of the paper — dedicating equal space to all three applications rather than treating them as an afterthought — communicates that the contribution is the capability unlock, not just the model checkpoint. A conventional paper would have stopped after Section 5.1 (the benchmark results); this paper's Section 4 on applications is the core intellectual contribution.

---

Innovation 2: The Multi-Modal Preference Data Construction Pipeline as an Architectural Choice

A less obvious but equally important innovation is the paper's systematic approach to multi-modal preference data construction. The dominant assumption in prior work — when preference data was needed for LVLM alignment — was to either repurpose text-only preference datasets (losing the visual modality entirely), rely on proprietary models like GPT-4V as generative judges (expensive and unsustainable), or focus narrowly on hallucination-specific preference pairs (limiting generalization). Each of these approaches implicitly accepted that comprehensive, domain-diverse multi-modal preference data was not practically obtainable.

The paper challenges this assumption not through a fundamentally new data collection technique, but through a compositional strategy: combine existing open-source preference datasets (which cover instruction following and safety in the text domain) with newly collected pairs spanning the under-represented domains of text-rich documents, mathematical reasoning, and video understanding. The key insight is that multi-modal preference data doesn't need to be created from scratch for every domain. By using the SFT model's own outputs as the "chosen" responses and then generating rejected responses through either GPT-4o pairwise comparison or verifier-based correctness checking, the pipeline produces preference pairs that are tailored to the base model's specific failure modes — the rejected responses are actual mistakes the model makes, not artificial negatives constructed by a data annotator.

This is methodologically important because it resolves a distribution shift problem that plagues reward model training. If the preference data comes from models that behave differently than the target LVLM (e.g., GPT-4V judging GPT-4V's own outputs), the reward model learns preferences that may not transfer to the target model's output distribution. By using IXC-2.5 itself as the response generator and then having GPT-4o or verifiers judge those specific outputs, the preference pairs reflect judgments about responses from the same model family the RM will eventually evaluate. The paper's finding that the PRM800k dataset — which contains GPT-4-generated solutions with human step-level labels — was "largely ineffective" for their PaLM 2 models in the test-time scaling paper (Snell et al., 2024) illustrates this same principle: preference labels from a different model distribution don't transfer.

The domain-diverse coverage (Tables 1, 2) is the evidence that this compositional strategy works. By covering text, image, and video inputs across instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding, the dataset ensures the RM can serve as a general-purpose evaluator rather than a domain-specific one. The strong performance on both multi-modal (VL-RewardBench) and text-only (Reward-Bench, RM-Bench) benchmarks validates that this domain coverage translates to evaluation capability. The text-only benchmark results (88.6% on Reward-Bench, surpassing other multi-modal models by several points in Table 4) are particularly telling: they demonstrate that adding visual preference data during training does not degrade language understanding, which is a genuine concern when extending RMs to the multi-modal domain.

---

Innovation 3: Demonstrating That a Discriminative RM Outperforms Generative RMs by a Wide Margin on Multi-Modal Preference Evaluation

The VL-RewardBench results (Table 3) contain a finding that has significant implications for how the field should approach multi-modal evaluation: IXC-2.5-Reward, a 7B discriminative model, achieves 70.0% macro accuracy, substantially outperforming all proprietary generative RMs — GPT-4o (62.4%), Gemini-1.5-Pro (62.5%), Claude-3.5-Sonnet (53.6%). This is not an incremental improvement; it is a qualitative gap of 7–8 absolute percentage points against models that are orders of magnitude larger and trained on vastly more data.

The intellectual significance of this result is that it challenges the prevailing assumption that generative evaluation — prompting a large model to produce a textual critique — is the natural approach to multi-modal quality assessment. This assumption was implicit in prior multi-modal RM work (Yu et al., 2024c; Ouali et al., 2025; Xiyao et al., 2024), which largely repurposed existing LVLMs as evaluators by engineering evaluation prompts. The generative approach has intuitive appeal: a large, capable model should be able to articulate why one response is better than another, and the textual critique provides interpretability. But the benchmark results suggest that generative RMs systematically underperform discriminative ones on the core task of preference judgment, even when the generative models are much larger.

The paper identifies one reason for this gap: on the "General" subcategory of VL-RewardBench, IXC-2.5-Reward achieves 84.7% while GPT-4o achieves 49.1% and Gemini-1.5-Pro achieves 50.8%. The paper attributes this to generative RMs "often leading to tied judgments" on these problems, "whereas IXC-2.5-Reward demonstrates a greater ability to make correct classifications with different scalar scores" (Section 5.1.1). This suggests a fundamental limitation of generative evaluation: when two responses are of similar quality, a prompted evaluator may default to declaring them equivalent (or making an arbitrary choice based on position bias or other irrelevant factors), while a discriminative model trained on explicit preference pairs learns to make fine-grained relative distinctions.

This finding has practical implications for the RLHF pipeline. If generative RMs produce tied or noisy judgments on the hardest cases — precisely the cases where the reward signal is most valuable for policy improvement — then using them for PPO training would provide a degraded learning signal on those critical examples. The discriminative approach, by learning to assign different scalar scores even to similar-quality responses, provides a more informative gradient. This is not merely a benchmark curiosity: it suggests that the entire paradigm of "LLM-as-a-judge" for LVLM evaluation may be fundamentally limited compared to purpose-built discriminative RMs.

---

Innovation 4: The Identification of Length Bias as a Cross-Modal Reward Hacking Vulnerability

The paper's analysis of length bias (Table 7) reveals a subtle but important phenomenon: the same reward hacking vulnerability that affects text-only RMs — exploiting response length as a spurious quality signal — transfers to the multi-modal setting, and the downstream effects are amplified when the evaluator (GPT-4o as judge) itself exhibits the same bias.

The finding is a diagnostic one. When the paper removes length constraints from the reward model's training data, the PPO-trained chat model learns to produce longer responses (361 vs. 274 average tokens on WildVision), and these longer responses achieve higher benchmark scores (76.2 vs. 74.6 on WildVision, 87.0 vs. 84.0 on MIA). The paper correctly identifies this as a benchmark illusion: the improved scores do not reflect genuinely better user experience, but rather the fact that GPT-4o — the judge model used to evaluate these benchmarks — also exhibits length bias and rewards verbosity. The policy has learned to exploit the evaluator's weakness, not to produce genuinely better responses.

What makes this cross-modal is that the length bias propagates through the entire pipeline: the RM learns to prefer longer responses (if not constrained), the PPO policy amplifies this preference by generating longer outputs, and the downstream evaluation benchmarks — which use GPT-4o as judge — validate the longer outputs with higher scores, creating a self-reinforcing cycle of spurious improvement. The paper's decision to retain length constraints despite the fact that removing them would produce higher reported numbers is an important methodological commitment: it prioritizes genuine user experience over benchmark scores, and it exposes a flaw in current multi-modal evaluation protocols that future benchmarks will need to address.

This insight connects to the broader reward hacking literature in RLHF (where models learn to exploit the RM's blind spots rather than genuinely improving), but extends it to a multi-modal context with an important nuance: the reward model's biases are amplified by downstream evaluator biases, creating a compounding effect that could mislead the research community if not explicitly controlled for. The paper's call for multi-modal benchmarks to "address potential length and style biases in their evaluation protocols" (Appendix A) is a concrete recommendation that follows directly from this finding.

5. Experimental Analysis

Evaluation Methodology

• Datasets. The paper evaluates the reward model on three benchmarks. VL-RewardBench (Li et al., 2024b) contains 1,250 multi-modal preference problems covering general understanding, hallucination, and reasoning challenges. Reward-Bench (Lambert et al., 2024b) contains 2,985 language-only problems across chat, chat hard, safety, and reasoning categories. RM-Bench (Liu et al., 2024b) contains 1,237 language-only problems across chat, math, code, and safety, organized into three difficulty tracks (easy, normal, hard) that evaluate sensitivity to subtle content variations and style biases. The policy model (IXC-2.5-Chat) is evaluated on instruction following and chat benchmarks: WildVision (Lu et al., 2024c) uses real-world user-submitted prompts, MIA-bench (Qian et al., 2024) evaluates instruction following, MM-MT (Agrawal et al., 2024) correlates with LMSys-Vision ELO ratings, and MM-Vet v2 (Yu et al., 2023) evaluates complex tasks like language generation. Additional benchmarks assess knowledge (MMBench v1.1, MMMU, MMStar), reasoning (MathVista, MathVerse, MathVision), and text-rich understanding (TextVQA, ChartQA, OCRBench). The paper does not specify dataset splits (train/val/test) for any of these benchmarks, treating them as evaluation-only sets.

• Base model(s). The reward model IXC-2.5-Reward is built on InternLM-XComposer-2.5 (IXC-2.5) (Zhang et al., 2024c), a Large Vision Language Model based on the InternLM-2-7B language model. The choice of a 7B-scale base model positions IXC-2.5-Reward as a relatively lightweight RM compared to proprietary alternatives (GPT-4o, Gemini-1.5-Pro, Claude-3.5-Sonnet) whose parameter counts are not publicly disclosed but are known to be substantially larger. The policy model IXC-2.5-Chat is initialized from the same IXC-2.5 SFT model and trained via PPO using IXC-2.5-Reward as the frozen reward provider. For RL training, a critic model is initialized from IXC-2.5-Reward weights. The reference model used for KL-constrained PPO updates is the frozen IXC-2.5 SFT model.

• Metrics. For VL-RewardBench, the paper reports overall accuracy (percentage of preference pairs where the RM correctly ranks the chosen response above the rejected response) and macro accuracy (average of per-category accuracies, where categories include General, Hallucination, and Reasoning). For Reward-Bench, per-category accuracy and average score across Chat, Chat Hard, Safety, and Reasoning subcategories. For RM-Bench, per-domain accuracy (Chat, Math, Code, Safety), per-difficulty accuracy (Easy, Normal, Hard), and overall average. For instruction following and chat benchmarks (WildVision, MIA, MM-MT, MM-Vet), evaluation uses GPT-4o as a judge model (LLM-as-a-Judge; Zheng et al., 2023) to predict scores, following the standard protocol of those benchmarks. For knowledge, reasoning, and text-rich benchmarks, accuracy is computed via multiple-choice question accuracy or visual question answering string matching against ground-truth answers. Response length is measured in average tokens on the WildVision benchmark.

• Baselines. For VL-RewardBench (Table 3), baselines include proprietary generative RMs (Gemini-1.5-Flash, Gemini-1.5-Pro, Claude-3.5-Sonnet, GPT-4o-mini, GPT-4o) and open-source generative RMs repurposed as evaluators, including LLaVA-OneVision-7B-ov (Li et al., 2024a), Qwen2-VL-7B and -72B (Wang et al., 2024d), Molmo-7B and -72B (Deitke et al., 2024), InternVL2-8B (Team, 2024c), LLaVA-Critic-8B (Xiong et al., 2024), Llama-3.2-11B and -90B (Team, 2024b), Pixtral-12B (Agrawal et al., 2024), and NVLM-D-72B (Dai et al., 2024b). For Reward-Bench (Table 4), baselines include language-only reward models (InternLM2-7B-Reward, InternLM2-20B-Reward, Skyword-Reward-Llama3.1-8B, INF-ORM-Llama3.1-70B) and multi-modal models (QWen2-VL-7B, LLaVA-Critic-8B). For RM-Bench (Table 5), baselines include language-only RMs (Tulu-2-dpo-13b as implicit RM, InternLM2-7B and -20B-Reward as sequence classifiers, Nemotron-4-340B-Reward as generative, URM-LLaMa-3.1-8B, Skyword-Reward-Llama3.1-8B). For policy model evaluation (Table 6), the baselines are proprietary API models (GPT-4o for WildVision, MIA, MM-MT, and MM-Vet; Claude-3.5-Sonnet for MM-Vet; Gemini-2.0-Flash for MathVerse and MathVision; SenseNova and Taiyi for various benchmarks) and previous SOTA open-source models ≤10B parameters (LLaVA-Critic-8B for WildVision; Qwen2-VL-7B for MIA and MM-MT; InternVL2-8B for MMBench; Ovis1.6-Gemma2-9B for MathVerse and MathVision; LLaVA-OneVision-7B for TextVQA; MiniCPM-V-2.6 for ChartQA and OCRBench). The SFT model IXC-2.5 itself serves as the pre-RL baseline for comparing IXC-2.5-Chat.

• Generation budget / compute accounting. The paper does not use a unified compute budget framework (no FLOPs counting or "generation" units as in the test-time scaling paper). For the RL training (Section 4.1), compute is implicitly measured in PPO training steps with batch size 256, but total training FLOPs are not reported. For test-time scaling (Section 4.2), the budget is measured in N, the number of candidate responses generated per prompt, with N=4 in the reported experiments (Table 8). The paper does not compare different N values or analyze scaling behavior with increasing N, so there is no compute-optimal budget analysis. For data cleaning (Section 4.3), compute is measured per-sample (one forward pass through the RM per training example) but total cost is not quantified. The reward model evaluation benchmarks (Tables 3-5) are zero-shot — the RM is evaluated directly on held-out preference pairs without any generation budget. This is a significant difference from the test-time scaling analysis paradigm; the paper treats the RM as a fixed evaluator rather than a resource to be optimally allocated.

• Cross-validation / statistical protocol. The paper does not report cross-validation, confidence intervals, standard deviations, or statistical significance tests for any result. All benchmark numbers in Tables 3-8 are reported as point estimates without uncertainty quantification. The RL training (IXC-2.5-Chat) appears to be a single training run (no mention of multiple seeds or error bars). The test-time scaling results (Table 8) use N=4 with different random seeds for generation, but the paper does not report variance across sampling runs. For the reward model training, no validation set protocol or early stopping criterion is described. The paper does not mention whether the preference data used to train IXC-2.5-Reward has any overlap with the evaluation benchmarks — a potential concern since the RM's training data construction used prompts from sources (Table 2) that may share domains with the evaluation benchmarks, though the specific evaluation instances are presumably disjoint.

Main Quantitative Results

Reward Model Evaluation: VL-RewardBench (Table 3)

The headline result is that IXC-2.5-Reward achieves 65.8% overall accuracy and 70.0% macro accuracy on VL-RewardBench, outperforming all proprietary generative RMs on macro accuracy and matching GPT-4o on overall accuracy (65.8% for both). The macro accuracy gap is substantial: 70.0% for IXC-2.5-Reward versus 62.5% for Gemini-1.5-Pro, 62.4% for GPT-4o, and 53.6% for Claude-3.5-Sonnet. This means that when averaging performance across the General, Hallucination, and Reasoning subcategories (treating each category equally), the 7B discriminative model substantially exceeds models estimated to be orders of magnitude larger.

Breaking the results down by subcategory (Table 3):

• General: IXC-2.5-Reward achieves 84.7%, dramatically outperforming all other models. The next best are LLaVA-Critic-8B at 54.6% among open-source models and Gemini-1.5-Pro at 50.8% among proprietary models. This is a gap of over 30 percentage points. The paper attributes this to generative RMs frequently producing "tied judgments" on challenging general problems, whereas the discriminative model's scalar scores enable consistent differentiation ("IXC-2.5-Reward demonstrates a greater ability to make correct classifications with different scalar scores," Section 5.1.1).

• Hallucination: IXC-2.5-Reward achieves 62.5%, which is competitive but not dominant. Gemini-1.5-Pro scores higher at 72.5% and GPT-4o at 67.6%. Llama-3.2-90B achieves 57.3%, and most open-source models fall in the 20–40% range. This suggests that hallucination detection — judging whether a response contains fabricated visual details — is an area where proprietary generative RMs retain an advantage, possibly due to their larger scale and more extensive training on factual consistency.

• Reasoning: IXC-2.5-Reward achieves 62.9%, which is competitive with the best models. GPT-4o scores 70.5%, Gemini-1.5-Pro scores 64.2%, Claude-3.5-Sonnet scores 62.3%, and Llama-3.2-90B scores 61.7%. The field is relatively compressed on this subcategory, with the top models clustering in the 62–71% range.

The overall accuracy of 65.8% ties GPT-4o (also 65.8%) and exceeds Gemini-1.5-Pro (67.2% — wait, this appears contradictory: Table 3 shows Gemini-1.5-Pro with 67.2% overall, which is higher than both IXC-2.5-Reward and GPT-4o at 65.8%. The paper's claim that IXC-2.5-Reward "outperforms all other open-source models" and "achieves the highest overall accuracy (65.8%) among open-source models" is correct — Gemini-1.5-Pro is proprietary and achieves 67.2%, which exceeds IXC-2.5-Reward's 65.8%. The macro accuracy claim of 70.0% "among all models" does stand, since Gemini-1.5-Pro achieves 62.5% macro. This highlights an important distinction: IXC-2.5-Reward wins on balanced performance across categories (macro accuracy) but does not achieve the highest overall accuracy on the unbalanced set.)

Among open-source models, the gap is decisive. The next-best open-source model on macro accuracy is InternVL2-8B at 45.2%, and the next-best on overall accuracy is Llama-3.2-90B at 56.2%. IXC-2.5-Reward exceeds the best open-source alternative by approximately 20 percentage points on macro accuracy and 10 points on overall accuracy, while using fewer parameters (7B vs. 90B for Llama-3.2-90B).

Reward Model Evaluation: Text-Only Benchmarks (Tables 4, 5)

On Reward-Bench (Table 4), IXC-2.5-Reward achieves an average score of 88.6%. This is competitive with language-only reward models: InternLM2-7B-Reward scores 87.6%, InternLM2-20B-Reward scores 90.2%, Skyword-Reward-Llama3.1-8B scores 92.5%, and INF-ORM-Llama3.1-70B scores 95.1%. Compared to other multi-modal models, IXC-2.5-Reward substantially outperforms QWen2-VL-7B (83.8%) and LLaVA-Critic-8B (80.0%), with the gap largely driven by the Chat Hard subcategory where IXC-2.5-Reward scores 83.8% versus 57.0% and 52.8% respectively. The Chat Hard subcategory involves difficult preference distinctions (subtle quality differences between responses), and the paper's result suggests that the multi-modal training does not degrade performance on these challenging language-only judgments relative to purpose-built text RMs.

The subcategory breakdown reveals that IXC-2.5-Reward's relative weakness is in the Chat subcategory (90.8%), where InternLM2-7B-Reward achieves 99.2%. This is a notable gap that the paper does not discuss. One possible explanation: the multi-modal preference data may emphasize different qualities (visual grounding, detailed descriptions) than what the text-only Chat subcategory evaluates (conversational helpfulness in purely textual contexts). The Safety subcategory is strong at 87.8% (comparable to InternLM2-7B-Reward at 87.2%), and Reasoning is 90.0% (versus 94.5% for the language-only baseline), suggesting the RM retains strong safety and reasoning evaluation capabilities.

On RM-Bench (Table 5), IXC-2.5-Reward achieves an average of 68.8%, which is competitive with language-only reward models of similar scale. InternLM2-7B-Reward scores 67.1% and InternLM2-20B-Reward scores 68.3%. Larger models score higher: Skyword-Reward-Llama3.1-8B achieves 70.1% and Nemotron-4-340B-Reward achieves 69.5%. The Safety domain is IXC-2.5-Reward's strongest at 93.8%, exceeding all baselines except Skyword-Reward (95.7%) and URM-LLaMa-3.1-8B (93.1%). The Math (55.9%) and Code (51.7%) domains show relative weakness compared to dedicated text RMs, which is expected given that multi-modal preference data likely underrepresents these domains relative to text-only RM training sets.

The difficulty breakdown on RM-Bench shows that IXC-2.5-Reward performs well on Easy problems (87.5%) and Normal problems (71.3%), but drops to 47.4% on Hard problems. This pattern — strong on clear distinctions, weaker on subtle ones — is consistent across all reward models in Table 5, but the gap is somewhat larger for IXC-2.5-Reward than for the best text-only RMs (Skyword-Reward achieves 46.6% on Hard, InternLM2-20B-Reward achieves 50.7%). The paper acknowledges this limitation: "IXC-2.5-Reward demonstrates sensitivity to subtle content variations and style biases, an issue often overlooked in multi-modal research. We believe further research is needed to enhance the robustness of multi-modal reward models" (Section 5.1.2).

Policy Model Evaluation: IXC-2.5-Chat vs. Baselines (Table 6)

On instruction following and chat benchmarks, IXC-2.5-Chat shows substantial improvements over the SFT baseline IXC-2.5:

• WildVision: 74.6 (IXC-2.5-Chat) vs. 37.5 (IXC-2.5 SFT). This is a 37.1 percentage point improvement, moving from well below the previous open-source SOTA (LLaVA-Critic-8B at 67.3) to significantly above it. IXC-2.5-Chat also outperforms LLaVA-Critic-8B by 7.3 points. The proprietary SOTA (GPT-4o) remains ahead at 89.2.

• MIA (val): 84.0 vs. 80.4 (IXC-2.5 SFT). A 3.6 point improvement, moving from slightly below Qwen2-VL-7B (80.7) to above it. GPT-4o achieves 88.6.

• MM-MT (val): 5.70 vs. 3.85 (IXC-2.5 SFT). A 1.85 point improvement, exceeding Qwen2-VL-7B (5.45) and approaching GPT-4o (7.72). The absolute values on MM-MT are on a different scale (likely a 1-10 rating), and the relative improvement is approximately 48% over the SFT baseline.

• MM-Vet v2: 54.8 vs. 45.8 (IXC-2.5 SFT). A 9.0 point improvement, now competitive with InternVL2-8B (58.1) and approaching but not matching it. Claude-3.5-Sonnet achieves 71.8.

On knowledge, reasoning, and text-rich benchmarks, IXC-2.5-Chat performs comparably to the SFT model IXC-2.5, with no significant degradation:

• MMBench: 79.0 (Chat) vs. 79.4 (SFT) — negligible change
• MMMU: 44.1 vs. 42.9 — marginal improvement
• MMStar: 59.6 vs. 59.9 — essentially flat
• MathVista: 63.4 vs. 63.7 — essentially flat
• MathVerse: 19.0 vs. 16.2 — modest improvement
• MathVision: 18.8 vs. 17.8 — marginal improvement
• TextVQA: 81.3 vs. 78.2 — notable improvement of 3.1 points
• ChartQA: 80.5 vs. 82.2 — slight regression of 1.7 points
• OCRBench: 70.0 vs. 69.0 — marginal improvement

This pattern — significant gains on instruction following and chat with minimal impact on knowledge and reasoning — is the intended outcome of the PPO training, which "prioritizes general chat and instruction following" in its prompt set curation (Section 4.1) while including diverse domains to prevent catastrophic forgetting. The results confirm that the PPO training with IXC-2.5-Reward successfully targets the conversational quality axis without degrading other capabilities.

Test-Time Scaling: Best-of-N Results (Table 8)

Applying Best-of-N sampling with N=4 and IXC-2.5-Reward as the selector yields consistent improvements over the base IXC-2.5-Chat on all four instruction following and chat benchmarks:

• WildVision: 77.7 (BoN) vs. 74.6 (Chat) — +3.1 points
• MIA: 87.3 vs. 84.0 — +3.3 points
• MM-MT: 6.03 vs. 5.70 — +0.33 points
• MM-Vet v2: 56.3 vs. 54.8 — +1.5 points

The average token length increases only slightly from 274 to 283, indicating that the improvement comes from selecting higher-quality responses rather than exploiting length bias. The paper does not report results for N > 4, so the scaling behavior of Best-of-N with larger candidate sets is unknown. Notably, the improvements from Best-of-N (+3.1 on WildVision) are substantially smaller than the gains from PPO training (+37.1 on WildVision), suggesting that at N=4, test-time scaling provides incremental rather than transformative gains for this model. Whether larger N would yield proportionally larger improvements — or whether the gains saturate quickly, as they did for best-of-N weighted in the Snell et al. (2024) paper — cannot be determined from the reported data.

Data Cleaning: Qualitative Correlation (Figure 2)

The paper presents four qualitative examples from ALLaVA and LLaVA-Video-178K datasets where IXC-2.5-Reward assigns anomalously low reward scores (ranging from -5.82 to -3.23) to samples that human experts identify as problematic. The error types include hallucination (describing visual elements not present in the image), image-text irrelevance, empty question-answer pairs, and video-text irrelevance. No quantitative metrics (precision, recall, F1, or downstream model improvement from cleaning) are reported. The paper states: "We observe a strong correlation between low IXC-2.5-Reward scores and problematic samples," but provides no statistical characterization of this correlation — no threshold analysis, no ROC curve, no comparison to CLIP-based baselines, and no cleaned-vs-uncleaned downstream training comparison.

Ablation Studies and Robustness Checks

Length constraints in reward model training (Table 7): Removing length constraints from the preference data during IXC-2.5-Reward training causes the downstream PPO policy (IXC-2.5-Chat) to produce substantially longer responses. On WildVision, average token length increases from 274 to 361. The longer responses achieve higher benchmark scores (WildVision: 76.2 vs. 74.6; MIA: 87.0 vs. 84.0; MM-MT: 5.86 vs. 5.70; MM-Vet v2: 56.6 vs. 54.8), but the paper identifies this as a spurious improvement: GPT-4o, the judge model used in these benchmarks, exhibits its own length bias and rewards verbosity. The paper retains length constraints despite lower benchmark scores, prioritizing genuine user experience. This is a methodologically important result — it demonstrates that benchmark scores can be misleading when the evaluator (GPT-4o) shares the same bias (length preference) that the RM might learn, creating a self-reinforcing evaluation artifact. Without the ablation, a reader might incorrectly attribute the benchmark improvements to genuine quality gains.

Best-of-N scaling with different N: The paper only reports N=4. There is no ablation across N values (e.g., N=2, 8, 16, 32, 64), so the scaling behavior — whether gains increase logarithmically, linearly, or saturate — is unknown. This is a notable gap given that the test-time scaling literature (Snell et al., 2024) extensively characterizes compute-optimal scaling curves across different budgets and difficulty levels.

Reward model training data composition: No ablation is reported on the contribution of different data sources (Table 1 open-source data vs. Table 2 newly collected data; different domains like text-rich vs. video vs. reasoning). Without this, it is impossible to determine whether the strong performance on VL-RewardBench (especially the 84.7% on General) is driven by the breadth of the newly collected data or would have been achievable with existing open-source data alone. Similarly, there is no ablation on the amount of preference data needed — the paper uses all available data without showing learning curves or data scaling behavior.

Frozen vs. trainable vision components: The paper freezes the vision encoder and MLP projector during RM training but does not provide an ablation comparing this to fully trainable or fully frozen configurations. The stated rationale — avoiding "other pre-training data for modality alignment" — is reasonable but untested. An ablation showing that training the vision encoder degrades performance (e.g., due to catastrophic forgetting of visual features) or that freezing it is necessary for stability would strengthen the design choice.

GPT-4o vs. verifier-based preference labeling: For the newly collected data (Table 2), general and text-rich domains use GPT-4o as judge while math and instruction following use verifier functions. No ablation compares the quality of preference labels from these two sources (e.g., by comparing RM performance when trained on GPT-4o-labeled vs. verifier-labeled data for the same prompts). This matters because verifier labels are deterministic and objective (based on ground-truth correctness), while GPT-4o labels are subjective and may contain systematic biases. The relative contribution of these two labeling strategies to the RM's overall performance is unknown.

PPO hyperparameters (γ, β, ε): The paper uses standard values (γ=0.99, β=0.95, ε=0.2) but does not ablate them. Given that these hyperparameters control the bias-variance tradeoff in advantage estimation and the conservatism of policy updates, their sensitivity is unknown. For a paper that emphasizes enabling PPO for LVLMs (a previously unavailable capability), hyperparameter sensitivity analysis would be valuable for practitioners seeking to replicate the approach.

Critic initialization from RM vs. random: The critic model is initialized from IXC-2.5-Reward weights. No ablation compares this to random initialization. The paper's implicit claim is that starting from RM weights provides better value function estimates (since the RM already encodes good representations of response quality), but this is not empirically validated. In the language-only RLHF literature, critic initialization is a known design choice with varying practices — some works initialize from the RM, others from the SFT model, and others randomly — making the absence of this ablation notable.

Sequence-level vs. token-level reward assignment: The PPO formulation uses per-timestep reward notation (r_t), but IXC-2.5-Reward is described as an Outcome RM. The paper does not clarify whether the reward is assigned only at the final token (making all intermediate r_t zero) or distributed across tokens. This distinction affects the credit assignment signal: token-level rewards provide more granular feedback but may require a Process RM (which IXC-2.5-Reward is not). The absence of this specification makes it difficult to assess whether the PPO implementation follows standard outcome-reward practices or does something different.

Data cleaning threshold and downstream impact: The paper demonstrates qualitative examples of low-score samples but provides no quantitative data cleaning experiment. No threshold is specified for flagging samples. No comparison to CLIP-based filtering is provided. No downstream training experiment shows that cleaning with IXC-2.5-Reward improves model quality. The data cleaning application remains at the level of a capability demonstration rather than a validated methodology.

Critical Assessment

Claim: IXC-2.5-Reward achieves SOTA on multi-modal reward model benchmarks

The VL-RewardBench results (Table 3) support the claim that IXC-2.5-Reward achieves the highest macro accuracy (70.0%) among all models — proprietary and open-source alike. This is genuinely impressive for a 7B discriminative model evaluated against much larger generative RMs. However, the overall accuracy of 65.8% is below Gemini-1.5-Pro (67.2%), which the paper does not prominently acknowledge. The macro vs. overall distinction matters because macro accuracy weights the three subcategories equally, while overall accuracy weights by the number of examples per category. IXC-2.5-Reward's dominance on the General subcategory (84.7% vs. 49–51% for proprietary models) drives the macro accuracy advantage, while its weaker Hallucination performance (62.5% vs. 72.5% for Gemini-1.5-Pro) limits the overall accuracy. The claim "achieves best performance on VL-RewardBench" requires the qualifier "on macro accuracy" to be precise, or should acknowledge that Gemini-1.5-Pro achieves higher overall accuracy.

A second qualification: all evaluations are zero-shot inference on the benchmark without fine-tuning. While this is standard for RM evaluation, it means the results reflect the RM's generalization to preference distinctions from its training distribution. If the newly collected training data (Table 2) overlaps in domain or even specific examples with VL-RewardBench, the results could be inflated. The paper does not address potential data contamination. Given that both the training data construction and VL-RewardBench draw from similar domains (general VQA, reasoning, text-rich understanding), this is a non-trivial concern.

The text-only benchmark results (Reward-Bench: 88.6%, RM-Bench: 68.8%) support the claim that multi-modal training does not degrade language understanding, but they do not support any claim of SOTA performance relative to dedicated text RMs. IXC-2.5-Reward is competitive with similarly-sized language-only RMs (InternLM2-7B-Reward at 87.6% on Reward-Bench and 67.1% on RM-Bench) but does not exceed them. This is an honest and appropriate result — a multi-modal RM should not be expected to outperform dedicated text RMs on text-only evaluations — but claims should be calibrated accordingly. The paper appropriately states "demonstrates good results" and "achieves considerable performance" rather than claiming SOTA on these benchmarks.

Claim: IXC-2.5-Reward enables effective PPO training that improves instruction following

The policy model results (Table 6) provide strong evidence that PPO training with IXC-2.5-Reward improves instruction following and chat performance. The gains on WildVision (+37.1 points over the SFT baseline, moving from well below to well above the previous open-source SOTA) are substantial and unlikely to be explained by variance alone. The improvements on MIA (+3.6), MM-MT (+1.85), and MM-Vet v2 (+9.0) provide convergent evidence across different evaluation protocols.

However, three caveats weaken the strength of this evidence:

1. No ablation of the reward signal source. The paper does not compare PPO training with IXC-2.5-Reward against PPO training with an alternative reward signal (e.g., a simpler heuristic, a different RM, or direct GPT-4o scoring during training). Without such a comparison, we cannot determine whether the improvements are due to IXC-2.5-Reward specifically, or whether any reasonable reward signal combined with PPO would produce similar gains. The baseline that matters most — PPO with an off-the-shelf text-only RM applied to multi-modal prompts (ignoring visual input) — is not tested. If a text-only RM produced similar improvements, the case for a purpose-built multi-modal RM would be weaker.

2. No comparison to DPO or other off-policy methods. The paper's motivation (Section 2) argues that the absence of multi-modal RMs has forced researchers to use off-policy DPO, which "potentially compromise[s] performance compared to the on-policy RL solutions like PPO." Yet the paper never validates this claim by comparing IXC-2.5-Chat (PPO with RM) against a DPO-trained version of the same base model using the same preference data. The claimed advantage of on-policy PPO over off-policy DPO remains an assertion, not an empirical finding. This is a significant missed opportunity — the paper could have directly demonstrated the value of having an RM (enabling PPO) by showing PPO > DPO on the same benchmarks.

3. Single training run, no variance characterization. The PPO training is presented as a single run without multiple seeds, confidence intervals, or stability analysis. RL training is known to be sensitive to random seeds, data ordering, and hyperparameters. Without replication or variance characterization, we cannot assess whether the reported improvements are reliably achievable or reflect a particularly lucky training run. The detailed hyperparameters (Adam optimizer settings, learning rate schedule, number of PPO epochs, KL penalty coefficient if any beyond the clipped objective) are also not reported, making replication difficult.

A fourth, more subtle concern: the evaluation benchmarks use GPT-4o as a judge, and the RM was partially trained on GPT-4o-labeled preferences. This creates a potential circularity. If GPT-4o has systematic preferences (for certain response styles, formats, or content types), those preferences are encoded in the RM's training data (since GPT-4o served as the preference judge for general and text-rich domains). PPO then optimizes the policy to maximize the RM's reward, which reinforces GPT-4o's preferences. Finally, the evaluation benchmarks measure quality using GPT-4o as judge, which rewards the same preferences. The improvements on these benchmarks might therefore reflect alignment with GPT-4o's internal evaluation criteria rather than genuine improvement in human-perceived quality. The paper's own finding on length bias (Table 7) demonstrates exactly this kind of circularity: GPT-4o prefers longer responses, the RM learns to prefer longer responses (if unconstrained), PPO produces longer responses, and GPT-4o scores them higher. The length constraint mitigates this specific bias, but other, subtler GPT-4o biases may remain unaddressed.

Claim: IXC-2.5-Reward enables test-time scaling for LVLMs

The Best-of-N results (Table 8) demonstrate that using IXC-2.5-Reward to select among N=4 candidates provides modest but consistent improvements (+1.5 to +3.3 points) over the base IXC-2.5-Chat. This validates the capability in principle. However, the evidence is quite limited:

1. Only N=4 is tested. The test-time scaling literature (Snell et al., 2024) shows that Best-of-N benefits increase with N up to a point of diminishing returns or verifier over-optimization. Without results for N=2, 8, 16, 32, 64, etc., the scaling behavior is unknown. The paper does not even report N=1 accuracy for IXC-2.5-Chat (the base model without Best-of-N selection) separately from the N=4 results, making it impossible to compute the marginal benefit of increasing N.

2. No comparison to alternative selection methods. Best-of-N with the RM is the only test-time strategy tested. Majority voting (selecting the most common answer among N candidates) is a natural baseline that requires no RM. A comparison of RM-based selection vs. majority voting would quantify the value the RM adds beyond simple ensembling. The paper does not provide this.

3. The gains are small relative to the PPO gains. The +3.1 point improvement from Best-of-N on WildVision is dwarfed by the +37.1 point gain from PPO training. This suggests that at least for this model and these benchmarks, the primary value of the RM is in training-time optimization (enabling PPO) rather than inference-time optimization. The paper does not discuss this relative magnitude, instead presenting test-time scaling as an equally important application.

4. Computational cost of Best-of-N is not discussed. With N=4, the inference cost is 4× the base model cost for generation plus 4 RM forward passes for scoring. Whether this cost is acceptable depends on the deployment context, and the paper provides no guidance on cost-benefit tradeoffs. The test-time scaling paper this work connects to (Snell et al., 2024) extensively analyzes FLOPs-matched comparisons; this paper's complete absence of compute accounting makes it impossible to assess whether Best-of-N is actually worth the additional cost.

Claim: IXC-2.5-Reward can filter noisy data from instruction tuning datasets

This claim is supported by four qualitative examples (Figure 2) showing low reward scores on problematic samples. This is a capability demonstration, not a validated methodology. Critical gaps include:

1. No quantitative evaluation. Precision, recall, false positive rate, and false negative rate of RM-based filtering are not reported. We do not know what fraction of low-score samples are genuinely problematic (precision) or what fraction of problematic samples receive low scores (recall). Without these, the claim that IXC-2.5-Reward "effectively cleans both pre-training and post-training data" is unsupported.

2. No threshold specification. The paper shows examples with scores of -5.82, -3.23, -3.42, and -3.48 but does not specify what score threshold would be used for filtering, how this threshold would be determined, or whether the score distribution naturally separates clean and problematic samples. A histogram of RM scores for clean vs. problematic samples (with ground-truth problem labels from human annotation) would provide the necessary evidence; this is absent.

3. No comparison to simpler baselines. CLIP score, image-text similarity, response length filtering, or even random sampling are not compared. The paper's claim that existing CLIP-based approaches "have limitations" is true in principle but empirically unsubstantiated. A comparison showing that RM-based filtering catches problems that CLIP misses would strengthen the case.

4. No downstream impact measurement. The ultimate test of a data cleaning method is whether models trained on cleaned data perform better than models trained on uncleaned data. This experiment is not conducted. Without it, we cannot distinguish between "the RM identifies samples that look problematic to humans" and "removing these samples actually improves downstream model quality." It is possible that some "problematic" samples are actually useful for training (e.g., they teach the model what not to do, or they contain stylistic diversity that aids generalization).

The data cleaning application is the least validated of the three, and the paper's language ("We observe a strong correlation," "IXC-2.5-Reward can effectively clean") is appropriately hedged. However, the claim in the abstract that the RM enables "filtering outlier or noisy samples from existing image and video instruction tuning training data" implies a validated capability that the experimental evidence does not yet support.

What Would Strengthen the Experimental Analysis

Several missing experiments would substantially strengthen the paper's claims:

1. Data scaling ablations for the RM. Train IXC-2.5-Reward on subsets of the preference data (e.g., text-only, image-only, video-only, 25%/50%/75%/100% of each domain) and measure performance on each VL-RewardBench subcategory. This would reveal which data sources contribute to which evaluation capabilities and whether the full dataset is necessary or a subset would suffice.

2. PPO vs. DPO comparison. Train a DPO model on the same preference data used to train the RM, and compare it to IXC-2.5-Chat on the same benchmarks. This would directly test the paper's motivating claim that on-policy RL with an RM outperforms off-policy DPO.

3. Alternative RM baselines for PPO training. Run PPO training using (a) a text-only RM (e.g., InternLM2-7B-Reward applied to the text portion of multi-modal prompts), (b) GPT-4o directly as the reward provider (expensive but informative for an upper bound), and (c) a simple heuristic reward (e.g., response length, keyword matching). Compare the resulting policies to IXC-2.5-Chat.

4. Best-of-N scaling curve. Evaluate IXC-2.5-Chat with Best-of-N for N ∈ {1, 2, 4, 8, 16, 32, 64} on the instruction following benchmarks, with majority voting as a baseline. This would reveal whether the RM provides value beyond simple ensembling and at what N the gains saturate.

5. Data cleaning validation experiment. Have human annotators label a sample of the ALLaVA or LLaVA-Video-178K datasets as clean/problematic. Compute precision-recall curves for RM-based filtering at different thresholds. Train an LVLM on the full dataset vs. the RM-cleaned dataset and compare downstream performance.

6. Multiple PPO training runs with different seeds. Report mean and standard deviation of IXC-2.5-Chat performance across at least 3 seeds to characterize training variance.

7. Preference data contamination analysis. Check whether any prompts or response pairs from VL-RewardBench, Reward-Bench, or RM-Bench appear in the training data for IXC-2.5-Reward or were used during IXC-2.5's SFT training. Report any overlap.

8. Latency and throughput characterization. Report the inference time and memory requirements for IXC-2.5-Reward compared to generative RMs (GPT-4o API calls, LLaVA-Critic inference). This would quantify the practical advantage of the discriminative architecture beyond accuracy.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Remains Unaccounted For and May Exceed the Test-Time Budget

The assumption or constraint. The entire compute-optimal framework for test-time scaling — both search-based (Section 5.3) and revision-based (Section 6.3) — depends on knowing each prompt's difficulty before deciding how to allocate the inference budget. The paper's method for estimating difficulty involves evaluating the base model's pass@1 rate (or the PRM's predicted correctness) across 2,048 samples per question, then binning questions into five quintiles. Even the "predicted" variant that avoids ground-truth labels still requires generating and scoring 2,048 samples per prompt. The authors acknowledge this explicitly (Section 3.2):

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. In any realistic deployment, the total compute cost is difficulty_estimation_cost + strategy_execution_cost. The former involves generating 2,048 full solutions and scoring them with the PRM — this alone is 8× more expensive than the largest test-time budget studied (256 generations) and 128× more than the budget at which the 4× efficiency gains are claimed (16 generations). The reported compute-optimal scaling curves in Figures 4 and 8 are computed after difficulty is known, without amortizing the cost of learning it. If the difficulty estimation cost were included, the effective efficiency gain over best-of-N would shrink dramatically — potentially to zero or negative for budgets less than 2,048 generations. A practitioner with a budget of, say, 64 generations per prompt would need to spend 2,048 generations just to figure out which strategy to use, making the total cost 33× the stated budget.

What evidence exists in the paper. Figure 4 and Figure 8 show the compute-optimal scaling curves for predicted vs. oracle difficulty bins. Both versions exclude the cost of the 2,048 samples used for difficulty estimation. The paper does not report any results that include amortized difficulty estimation cost, nor does it provide a sensitivity analysis showing how many samples are actually needed for reliable difficulty binning (e.g., does 64 samples suffice? 256?). The question of whether difficulty can be estimated from far fewer samples — perhaps using the PRM's score distribution over just 4–8 initial samples — is flagged as future work (Section 8) but not explored.

Mitigation status. The authors are transparent about this limitation (Section 3.2) and frame it as a key avenue for future work, suggesting training a model "to directly predict question difficulty without requiring the generation of numerous samples." However, no such model is developed or evaluated in the paper. The predicted-difficulty variant (using PRM scores instead of ground-truth correctness) does not reduce the sample count — it only removes the need for answer labels. Until a cheap difficulty estimator exists, the compute-optimal framework remains a conceptual contribution with an unresolved practical barrier to deployment.

---

6.2 All Results Are on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*), Leaving Generalization to Other Domains and Architectures Unknown

The assumption or constraint. Every experiment in the paper — the search algorithm comparisons (Figures 3–4), the revision model analysis (Figures 6–8), the FLOPs-matched comparison (Figure 9), and all difficulty-bin analyses — is conducted on the MATH benchmark (500 test questions) using PaLM 2-S* as the base model. The authors state in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

but provide no empirical evidence across different model families, scales, or task domains. MATH consists exclusively of competition-level mathematics problems requiring multi-step symbolic reasoning with unambiguous ground-truth answers. The paper does not evaluate on code generation (HumanEval, MBPP), logical reasoning (ARC, FOLIO), scientific QA, open-ended generation, or any domain where correctness is ambiguous or multi-dimensional.

The consequence. Several findings could be specific to the MATH-PaLM 2-S* combination rather than general properties of test-time compute scaling. The PRM's over-optimization behavior (Figure 3, right) depends on the base model's output distribution and error patterns — a model with different calibration properties might exhibit different difficulty-dependent scaling curves. The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (some models are far better at leveraging in-context demonstrations than others). The finding that beam search degrades on easy problems (Figure 3, right) while helping on medium ones depends on the PRM's reliability profile, which is a function of both the model and the Monte Carlo rollout training procedure. The sharp boundary at difficulty bin 5 (where no method helps) — arguably the paper's most important practical finding — might shift significantly for a stronger base model or a different task domain. A practitioner using a different model (e.g., Llama-3, Qwen, DeepSeek) or targeting a different reasoning domain cannot confidently extrapolate the paper's specific difficulty thresholds, optimal strategies per bin, or even the qualitative pattern of which methods help on which difficulty levels.

What evidence exists in the paper. None. The paper contains zero cross-model or cross-domain experiments. The test set of 500 questions, while standard for MATH evaluations, is modest — when split into five difficulty quintiles of ~100 questions each and further divided by two-fold cross-validation, strategy selection is based on approximately 50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4, 8), making it impossible to assess whether the observed differences between strategies are statistically reliable at this sample size.

Mitigation status. The authors acknowledge this limitation implicitly by framing their work as an initial systematic study rather than a universal prescription. Section 8 (Conclusion) does not explicitly call for multi-model/multi-domain replication, which would be a natural extension. A practitioner seeking to apply the compute-optimal framework to their own model and domain would need to replicate the entire analysis pipeline (2,048 samples per prompt for difficulty estimation, PRM training via Monte Carlo rollouts, revision model training, strategy selection via cross-validation) for their specific setting — a substantial undertaking that the paper's single-setting results do not simplify.

---

6.3 The 14× Larger Model Baseline in the FLOPs-Matched Comparison Is Weakened by Non-Compute-Optimal Pretraining and the Absence of Any Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 pits PaLM 2-S* with compute-optimal test-time strategies against a model with approximately 14× more parameters, trained on the same amount of data (scaling parameters only, not data), and using greedy decoding with no test-time compute augmentation. The authors acknowledge (Section 7) that this departs from compute-optimal pretraining:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. This creates two sources of bias that favor the test-time compute side of the comparison. First, parameter-only scaling is known to be suboptimal — the Chinchilla scaling laws (Hoffmann et al., 2022) demonstrate that for a given increase in pretraining FLOPs, scaling both model parameters and training data in roughly equal proportion yields better performance than scaling parameters alone. A compute-optimally trained 14× larger model (with approximately 3.7× more parameters and 3.7× more data, following Chinchilla) would likely outperform the parameter-only-scaled model used as the baseline, narrowing or reversing the reported advantages of test-time compute (e.g., +27.8% on easy questions at R ≪ 1 in Figure 1).

Second, the baseline model uses only greedy decoding — a single deterministic output — while the test-time compute approach uses up to 256 generations with search or revisions. This is an asymmetric comparison: it compares an optimized inference strategy against the weakest reasonable inference strategy. The question the FLOPs-matched analysis answers is "can a small model with smart inference beat a large model with naive inference?" rather than the more policy-relevant "should I spend my next dollar on pretraining or on better inference for my existing model?" A fairer comparison would give the larger model some test-time compute budget — even a modest best-of-8 or majority vote — and compare FLOPs-matched performance with both models using their respective optimal inference strategies. The paper never tests this.

What evidence exists in the paper. Figure 9 shows the FLOPs-matched results at three values of R (0.16, 0.79, 22). The stars representing the 14× larger model's performance are placed at fixed accuracy values (from greedy decoding) at three x-axis positions corresponding to the three R values. The gap between the compute-optimal scaling curves and these stars is what generates the claimed advantages. No alternative baseline (Chinchilla-optimal pretraining, larger model with test-time compute) is evaluated.

Mitigation status. The paper is transparent about the parameter-only scaling limitation and frames the Chinchilla-optimal comparison as future work (Section 7). However, the absence of any test-time compute for the larger model is not acknowledged as a limitation — it is presented as the natural baseline. The headline finding that "test-time compute can outperform a 14× larger model" should be understood as conditional on the larger model being trained suboptimally and using greedy decoding. The practical policy implication — that organizations should invest in test-time compute rather than larger models — is stronger than the experimental design can support.

---

6.4 The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate, and Revision Training Is Demonstrated to Be Fragile Under Optimization

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target answer (Section 6.1). The training data construction procedure selects the last incorrect answer in a sequence to be the one with minimum character-level edit distance to the correct answer, ensuring the model learns to make targeted corrections. This creates a training-inference mismatch: during training, the model never sees correct answers in its context (the target is always preceded by incorrect answers), but during inference, the model may generate a correct answer early in the revision chain and then condition on it when producing subsequent revisions.

The consequence. The paper reports (Section 6.1) that approximately 38% of correct answers get converted back to incorrect ones during sequential revision. This is not a minor edge case — it means that for every 100 correct answers the model produces at some point in the revision chain, 38 will be "revised" into incorrect answers at the next step. The paper mitigates this with majority voting or verifier-based selection across the entire chain (picking the best answer from any revision step), but this is a patch, not a solution. The revision model has no mechanism to recognize when its current answer is already correct and should be left unchanged. This fundamentally limits the reliability of long revision chains — as the chain grows, the probability that a correct answer will be produced and then subsequently "corrected" into an error accumulates, creating a ceiling on how much sequential revision can improve performance.

The Appendix K experiment (Figure 16) reveals a deeper fragility: attempting to further optimize the revision model using ReST$^{EM}$ (Singh et al., 2024) — an RL-based self-improvement method — causes performance with sequential revisions to substantially degrade. At 256 generations, the ReST$^{EM}$-trained model's fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that "on-policy data collection in ReST$^{EM}$ exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This is a notable negative result: it demonstrates that the revision model's training procedure is sensitive to the data generation methodology in ways that are not fully understood, and that straightforward attempts to optimize it further can backfire.

What evidence exists in the paper. The 38% reversion rate is stated in Section 6.1. The ReST$^{EM}$ degradation is shown in Appendix K, Figure 16. The paper does not provide a breakdown of reversion rates by difficulty bin, revision step, or error type. It is unknown whether the 38% is uniform across difficulty levels (the reversion rate might be higher on hard problems where the model is less confident in its correct answers) or whether certain types of correct answers are more vulnerable to reversion than others.

Mitigation status. The paper partially mitigates the reversion problem with post-hoc selection (majority voting or verifier-based selection across the chain) but does not address the root cause — the training distribution mismatch. A more principled solution, such as including "no revision needed" training examples where the correct answer is preceded by other correct answers, or training the model to output a confidence score alongside each revision, is not explored. The ReST$^{EM}$ failure is presented as an empirical observation without a proposed solution, and the paper does not investigate whether alternative RL algorithms or data construction strategies could recover the expected improvement.

---

6.5 The Hardest Problems (Difficulty Bin 5) Show Near-Zero Improvement Across All Strategies, Establishing a Hard Capability Ceiling That Test-Time Compute Cannot Breach

The assumption or constraint. The compute-optimal framework allocates test-time compute based on estimated prompt difficulty, but it implicitly assumes that every prompt has some non-trivial probability of being solved correctly by the base model — that is, the pass@1 rate is above some minimum threshold. The paper's five difficulty quintiles are defined by the base model's pass@1 rate on each question, with bin 5 containing the hardest 20% of questions.

The consequence. Across all methods — PRM search (Figure 3, right), revision models (Figure 7, right), and their compute-optimal combinations (Figures 4, 8) — difficulty bin 5 shows essentially zero improvement regardless of the compute budget or strategy. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all search methods and all budgets from 4 to 256 generations. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%. The compute-optimal policy achieves its 4× efficiency gains entirely on bins 1–4; on bin 5, it performs no better than the simplest baseline.

This is not a minor edge case. In the FLOPs-matched comparison at R ≫ 1 (the high-throughput production setting where inference costs dominate), the paper reports (Figure 1, bottom-right bar chart) that PRM search on hard questions shows a −52.9% relative disadvantage compared to the 14× larger pretrained model. For easy questions at the same R value, test-time compute shows a +2.0% advantage. The asymmetry is stark: on problems within the base model's capability range, test-time compute helps; on problems outside it, test-time compute does nothing, and pretraining is the only viable path.

This finding establishes a fundamental boundary condition: test-time compute amplifies existing capability but does not create it. If the base model's probability of producing a correct answer on a given problem class is near zero — meaning the model lacks the necessary knowledge or reasoning patterns — then no amount of search, revision, or verifier guidance will find a correct solution, because no correct solutions exist in the proposal distribution. For such problems, the only path forward is additional pretraining (to acquire the missing capabilities) or a fundamentally different model architecture.

What evidence exists in the paper. The difficulty-bin breakdowns in Figures 3 (right) and 7 (right) provide consistent evidence across search and revision methods. The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 quantify the magnitude of the failure on hard problems. The paper is candid about this limitation in the Section 7 takeaway: test-time compute is effective "only when the prompts are within the model's existing capabilities."

Mitigation status. The paper does not attempt to address this limitation — it is treated as an intrinsic property of test-time compute scaling rather than a solvable problem. The authors do not explore whether combining both search and revisions (using the revision model as the proposal distribution within PRM-guided beam search) could push the capability frontier into harder problems, though Section 8 flags this combination as future work. Given the near-zero pass@1 rate on bin 5 problems, it is unlikely that any test-time strategy — no matter how sophisticated — could produce correct solutions from a proposal distribution that contains none. The limitation is therefore likely fundamental: test-time compute is bounded by the base model's knowledge, and closing the gap on genuinely hard problems requires pretraining advances.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a fundamentally new training algorithm or architectural innovation. Its contribution is more infrastructural than theoretical: it demonstrates that building a general-purpose discriminative multi-modal reward model — and making it open-source — unlocks capabilities that the LVLM community has been operating without. This is less a paradigm shift than a capability unlock: the paper shows that the absence of multi-modal RMs was not a minor inconvenience but a genuine bottleneck, and that removing it enables methods (on-policy RL, test-time best-of-N, automated data filtering) that the LLM community has taken for granted for years.

The conceptual shift is in how the LVLM community should think about reward models. Prior to this work, the implicit assumption was that multi-modal preference evaluation required large proprietary models acting as generative judges — GPT-4V or Gemini evaluating outputs through prompted critique. This assumption was understandable: building a discriminative RM requires preference data spanning multiple modalities and domains, and constructing such data seemed prohibitively expensive. The paper challenges this assumption through a compositional data strategy (combining existing open-source text preference data with newly collected pairs generated by the SFT model itself and judged by GPT-4o or programmatic verifiers) and through the empirical result that a 7B discriminative model substantially outperforms the proprietary generative evaluators on the core preference judgment task (70.0% vs. 62.4% macro accuracy on VL-RewardBench). If a small discriminative RM can judge multi-modal responses more accurately than GPT-4o — while being faster, cheaper, and fully controllable — then the "LLM-as-a-judge" paradigm for multi-modal evaluation looks less like a necessary evil and more like a transitional approach that the field should move beyond.

The paper also resolves a latent tension in the LVLM alignment literature. Prior work on multi-modal preference optimization split into two camps: those using off-policy DPO (Rafailov et al., 2024) because no RM was available for on-policy RL, and those using proprietary models as generative judges to construct DPO pairs (Zhang et al., 2024a; Zhao et al., 2023). The former approach was known to be suboptimal relative to on-policy methods but was the only option; the latter was expensive and unsustainable at scale. The paper demonstrates a third path: train a small, purpose-built discriminative RM, then use it for on-policy PPO. The IXC-2.5-Chat results (WildVision: 37.5 → 74.6, MIA: 80.4 → 84.0) represent improvements over the SFT baseline that are larger than what off-policy DPO has typically demonstrated for LVLMs, though the paper does not provide a direct DPO-vs-PPO comparison to make this case empirically.

One direction that becomes more attractive after this work is the development of process reward models for multi-modal reasoning. The paper's IXC-2.5-Reward is an Outcome RM (it scores complete responses), but the Snell et al. (2024) paper demonstrated that Process RMs — which score individual reasoning steps — are critical for effective test-time search. The infrastructure and data pipeline established here (preference data construction, RM training on a frozen LVLM backbone, PPO integration) provides a template that could be adapted to train step-level verifiers for visual reasoning tasks. The VL-RewardBench results already show that the discriminative approach works; extending it to process supervision is a natural next step that this paper makes newly tractable.

A direction that becomes less attractive is relying on proprietary API-based evaluators for LVLM alignment at scale. The paper's finding that a 7B open-source model outperforms GPT-4o on preference judgments (70.0% vs. 62.4% macro) and the practical demonstration of PPO training at batch size 256 — something that would be prohibitively expensive with API-based reward signals — suggests that the field should invest in purpose-built discriminative RMs rather than routing evaluation through paid services. The cost differential is not quantified in the paper, but the architectural efficiency advantage (a single scalar forward pass vs. full autoregressive critique generation) is inherent.

Follow-Up Research This Work Enables

Direct PPO vs. DPO comparison using the same preference data. The paper's central motivation is that the absence of multi-modal RMs has forced the community into off-policy DPO, which "potentially compromise[s] performance compared to the on-policy RL solutions like PPO" (Section 2). Yet this claim is never tested. A direct comparison would use the same multi-modal preference data to train (a) IXC-2.5-Reward followed by PPO training of IXC-2.5-Chat (as in the paper) and (b) a DPO-trained model directly on the preference pairs. Both would be evaluated on the same instruction-following benchmarks (WildVision, MIA, MM-MT, MM-Vet). The key measurement is the performance gap: if PPO substantially outperforms DPO, the paper's motivating claim is validated and the case for investing in multi-modal RM development is strengthened. If the gap is small or zero, then the RM is less essential infrastructure than the paper claims, and the community could achieve similar alignment with simpler off-policy methods. A secondary measurement is training stability: PPO is known to be more sensitive to hyperparameters than DPO, and a comparison of variance across multiple seeds would inform practitioners about the reliability tradeoff.

Best-of-N scaling curve with N up to 64 and majority voting baseline. The paper demonstrates Best-of-N with N=4 (Table 8) but provides no scaling analysis. A complete characterization would evaluate IXC-2.5-Chat with Best-of-N for N ∈ {1, 2, 4, 8, 16, 32, 64} on WildVision, MIA, and MM-Vet, plotting accuracy against N on a log scale. The baseline is majority voting — selecting the most common response among N candidates without any RM. The key question is whether the RM provides value beyond simple ensembling. If the RM-based Best-of-N curve lies substantially above the majority voting curve, the RM adds genuine selection capability. If the curves are close or the RM saturates quickly (e.g., gains stop after N=8), the practical value of the RM for test-time scaling is limited. A secondary analysis would measure the ranking accuracy of the RM on the N candidates — what fraction of the time does the highest-RM-scored candidate actually correspond to the highest-GPT-4o-judged candidate? This directly evaluates the RM's discriminative capability on in-distribution policy outputs.

Data cleaning validation with human annotation and downstream training. The paper's data cleaning claim (Section 4.3) is supported by four qualitative examples and no quantitative evidence. A validation experiment would sample, say, 1,000 examples from ALLaVA or LLaVA-Video-178K, have two human annotators label each as "clean" or "problematic" (hallucination, image-text mismatch, empty content, etc.) with inter-annotator agreement measured by Cohen's κ, and then compute precision-recall curves for RM-based filtering at different score thresholds. The key metric is area under the precision-recall curve (AUPRC) compared to baselines: CLIP score filtering, random sampling, and response length filtering. The downstream impact would be measured by training an LVLM (e.g., IXC-2.5 itself) on the full dataset vs. the RM-cleaned dataset (at different filtering thresholds) and evaluating on standard benchmarks. If RM-cleaned training data produces a measurable improvement (e.g., +2% on MMBench or reduced hallucination rates), the data cleaning application moves from capability demonstration to validated methodology.

Cross-model and cross-domain replication of the RM training recipe. All results are on a single model family (InternLM-XComposer-2.5) and the evaluations span specific benchmarks. A replication study would train discriminative multi-modal RMs using the same pipeline — frozen vision components, Bradley-Terry loss, compositional preference data from SFT outputs judged by GPT-4o — but on different base models (e.g., Qwen2-VL-7B, LLaVA-OneVision-7B, InternVL2-8B). Each resulting RM would be evaluated on VL-RewardBench, Reward-Bench, and RM-Bench. The key question is whether the strong performance (70.0% macro on VL-RewardBench) is specific to the InternLM backbone or generalizes across architectures. If Qwen2-VL-Reward and LLaVA-OneVision-Reward also achieve 65–70% macro accuracy, the training recipe is validated as a general method. If performance drops substantially, then IXC-2.5-Reward's success reflects model-specific properties (e.g., the quality of the base model's SFT training, the projector's alignment quality) that would need to be characterized.

PPO training with process reward signals for multi-step visual reasoning. IXC-2.5-Reward is an Outcome RM, providing a single scalar score for a complete response. This is sufficient for chat quality optimization (the paper's focus) but limits test-time scaling to Best-of-N selection — beam search or tree search over intermediate reasoning steps requires per-step rewards. A natural extension would train a multi-modal Process Reward Model by adapting the Monte Carlo rollout approach from the test-time scaling paper (Snell et al., 2024, Section 5.1): for visual reasoning problems (e.g., from MathVista or MathVerse), generate step-by-step solutions, sample rollouts from each intermediate step, and label each step with the rollout success rate. The PRM would be trained with the same frozen-backbone + score-head architecture as IXC-2.5-Reward but with per-step supervision. The evaluation would compare PPO training with the ORM (IXC-2.5-Reward) vs. the PRM on visual reasoning benchmarks, and additionally test whether PRM-guided beam search at inference time provides gains beyond Best-of-N. This directly connects the multi-modal RM infrastructure established here with the test-time compute scaling framework that has been transformative for language-only math reasoning.

Length bias audit across multi-modal evaluation benchmarks. The paper's Table 7 reveals a critical blind spot in current multi-modal evaluation: GPT-4o (the judge model for WildVision, MIA, MM-MT, and MM-Vet) exhibits length bias that creates a self-reinforcing evaluation artifact when the RM and the judge share the same preference. A systematic audit would evaluate all major multi-modal benchmarks that use LLM-as-a-Judge (including WildVision, MIA, MM-MT, MM-Vet, and any others in the OpenVLM Leaderboard) by generating responses of varying lengths at equal quality (e.g., by prompting the same model with "answer concisely" vs. "answer in detail") and measuring whether the judge model assigns systematically higher scores to longer responses. The outcome would be a bias coefficient for each benchmark (the score increase per additional token of response length, controlling for content quality). This would inform the community about which benchmarks can be trusted for evaluating genuine quality improvements and which primarily measure alignment with GPT-4o's stylistic preferences. The paper's call for benchmarks to "address potential length and style biases in their evaluation protocols" (Appendix A) provides the motivation, but the audit itself is not performed.

Practical Applications and Downstream Use Cases

On-policy RL training for open-source LVLM developers. The most immediate practical application is enabling any team with an LVLM to run PPO-based alignment training. Before this work, open-source LVLM developers had to choose between off-policy DPO (suboptimal but feasible) and using proprietary APIs as reward providers (expensive and rate-limited). With IXC-2.5-Reward available as an open-source 7B model, a team can run PPO training with a batch size of 256 at the cost of their own GPU compute — no API calls, no per-query fees, no rate limits. The paper's results show that this produces meaningful improvements: IXC-2.5-Chat achieves 74.6 on WildVision (up from 37.5 for the SFT baseline) and 84.0 on MIA (up from 80.4). For a team deploying an LVLM-based chatbot, these gains translate directly to better instruction following and more helpful conversational responses. The RL training infrastructure is standard (the paper uses off-the-shelf PPO with GAE), and the training recipe is documented in Appendix A, making this application immediately actionable for any group that has already completed SFT training on their LVLM.

Test-time response selection for quality-sensitive LVLM deployments. For applications where response quality matters more than latency or throughput — medical image description, legal document analysis, educational feedback — Best-of-N with IXC-2.5-Reward provides a simple post-hoc quality improvement mechanism. The paper's Table 8 shows that with N=4, the RM-selected response improves WildVision from 74.6 to 77.7 (+3.1 points) and MIA from 84.0 to 87.3 (+3.3 points), with negligible increase in response length (274 → 283 tokens), indicating the gains come from selecting genuinely better responses rather than exploiting length bias. The implementation is straightforward: generate N responses from the deployed policy model with different random seeds, score each with a single forward pass through IXC-2.5-Reward, and return the highest-scoring response. For a production system already running inference at scale, the 4× generation cost and 4 RM forward passes may be acceptable for high-stakes queries, especially if the RM is served on a separate, smaller GPU. The paper does not explore N > 4, but even the modest N=4 gains may justify the cost for applications where a 3% accuracy improvement translates to meaningful user value.

Pre-training and instruction-tuning data quality control. The correlation between low IXC-2.5-Reward scores and problematic samples (Figure 2) suggests that the RM can serve as a data quality filter for large-scale multi-modal datasets. For teams curating instruction-tuning datasets — a process that typically involves scraping millions of image-text pairs, generating questions and answers with LVLMs, and hoping the results are mostly correct — running each sample through IXC-2.5-Reward provides a quality signal that goes beyond simple CLIP-based image-text alignment checks. Samples with anomalously low scores (the paper shows examples at -5.82, -3.23) can be flagged for removal or human review before they contaminate training. This is particularly valuable for video understanding datasets (LLaVA-Video-178K, MiraData) where CLIP-based filtering is insufficient due to the temporal dimension and long-context requirements. The paper does not provide a specific threshold or quantify the downstream training improvement from cleaning, but the qualitative evidence is strong enough to justify integration into data curation pipelines as an additional filtering stage, especially since the per-sample cost is a single RM forward pass.

When to Prefer This Method

The paper explicitly positions IXC-2.5-Reward as a general-purpose multi-modal reward model and does not articulate a sharp tradeoff against named alternative approaches beyond the implicit comparison to generative RMs (GPT-4o, Gemini) on the evaluation benchmarks. The paper's contribution is primarily a capability provision (filling a gap where no comparable open-source alternative exists) rather than a method selection (choosing between competing approaches with different strengths). As such, a decision matrix would be speculative — the paper does not provide the comparative ablations (PPO vs. DPO, discriminative RM vs. generative RM for RL training, RM-based filtering vs. CLIP-based filtering with downstream training evaluation) that would support evidence-based decision rules.

The one clear tradeoff the paper does establish empirically is between constrained and unconstrained length preference in the RM training data (Table 7). The decision rule is: if optimizing for genuine user experience (where verbosity is not inherently valuable), retain length constraints in the RM training data. If optimizing for benchmark scores on evaluations that use GPT-4o as judge (which exhibits length bias), removing length constraints will produce higher reported numbers but at the cost of generating unnecessarily long responses. This is a concrete, evidence-supported tradeoff that practitioners face immediately when deploying this method.