Inference-Time Scaling for Generalist Reward Modeling

🎯 Pitch

A generative reward model trained with self-generated principles, given 32× more inference compute, can outperform a 671B MoE model at reward evaluation—but only on tasks within its generalist range, while scalar reward models still dominate verifiable reasoning. The method introduces Self-Principled Critique Tuning (SPCT) via online RL and a meta reward model to filter sampled critiques, pushing 27B-param performance past larger static judges.

1. Executive Summary

This paper proposes Self-Principled Critique Tuning (SPCT), a novel method that improves inference-time scalability for generalist reward modeling by training generative reward models (GRMs) to adaptively generate their own evaluation principles and critiques through rule-based online RL. Using Gemma-2-27B as the base and evaluating across multiple benchmarks—Reward Bench, PPE, RMB, and ReaLMistake—the approach enables pointwise GRMs to output richer, more diverse reward signals via parallel sampling (generating multiple sets of principles and corresponding scores per query, then voting), and introduces a meta RM to filter low-quality generations during voting. SPCT yields substantial inference-time scaling gains: DeepSeek-GRM-27B improves from 69.9% (greedy) to 70.6% with 8-sample voting and 71.0% with 32-sample voting—and further to 72.8% with meta RM guidance—outperforming larger models like the 671B MoE DeepSeek-V3 at greedy decoding, while establishing that inference-time scaling can surpass training-time model size scaling only on tasks within the GRM's generalist capability range, with scalar RMs retaining advantage on verifiable reasoning questions.

2. Context and Motivation

The Core Problem: How to Get Better Rewards for General Queries by Spending More Compute at Inference

This paper addresses a fundamental bottleneck in the modern LLM post-training pipeline: how to produce high-quality, reliable reward signals for arbitrary, unverifiable prompts when you're willing to spend extra computation at inference time. The problem sits at the intersection of two major trends in LLM development: the widespread adoption of reinforcement learning (RL) as a post-training method, and the growing realization that inference-time compute can be scaled to improve output quality.

The challenge is deceptively simple. Reinforcement learning requires reward signals — evaluations of how good a model's response is. In constrained domains with ground-truth answers (math problems with final answer checking, coding tasks with unit tests, formal verification environments), obtaining high-quality rewards is relatively straightforward: run the code and see if the tests pass, or check whether the final answer matches the expected value. The paper explicitly acknowledges this, noting that prior work has achieved "strong performance in specific domains" through "human-designed environments with clear conditions or from hand-crafted rules for verifiable questions, e.g., mathematical problems and coding tasks" (Section 1).

But the vast majority of real-world queries — creative writing, multi-step reasoning through open-ended problems, safety evaluations, instruction-following assessment — have no ground truth and no verifiable right answer. The criteria for what constitutes a good response are "more diverse and complex, and there are often no explicit reference or ground truth" (Section 1). This is the domain of generalist reward modeling, and it's precisely where RL for LLMs needs to work if we want to improve model behavior beyond narrowly scoped benchmarks.

The paper frames this as a two-part challenge. First, how do you build a reward model that works across arbitrary domains (the "generalist" requirement)? Second, and more novel, how do you make that reward model improve when you give it more inference-time compute (the "inference-time scalability" requirement)? The second question mirrors what has been studied extensively for policy models — you can sample many times, vote, use verifier-guided search, etc. — but the paper argues that reward models themselves have received almost no attention in this regard:

"few of them focus on inference-time scalability and study the interconnection between the learned reward generation behaviors and the effectiveness of inference-time scaling of RMs, resulting in marginal performance improvement" (Section 1).

Why This Problem Matters Now

The timing is not accidental. Three converging developments make this question urgent:

1. RL has become the dominant post-training paradigm at scale. From RLHF for alignment (Ouyang et al., 2022) to GRPO-based reasoning incentivization (DeepSeek-AI, 2025), the quality of the reward signal directly determines the ceiling of what RL can achieve. If rewards are noisy, biased, or domain-limited, the policy model inherits those limitations. As the paper puts it, reward modeling is "a crucial component in RL, essential for generating accurate reward signals for LLM responses" (Section 1). A generalist RM that produces better rewards with more compute would directly translate to better RL outcomes across the board.

2. Inference-time scaling has been validated for policy models, creating a natural question about whether the same principles apply to reward models. Recent work (Snell et al., 2025; Wu et al., 2025) demonstrated that for problem-solving, scaling test-time compute — through repeated sampling, verifier-guided search, or chain-of-thought expansion — can outperform scaling model parameters. The paper draws an explicit connection to this literature, asking whether similar gains are possible for the verifier itself. If you're going to spend more compute at inference to generate better policy outputs, shouldn't you also be able to spend more compute to generate better evaluations of those outputs?

3. The dominant RM paradigm (scalar reward models) is architecturally incompatible with inference-time scaling. This is perhaps the most technically specific and important motivation. Scalar reward models — the workhorse of RLHF — map a query and response to a single numerical score. When you sample from such a model multiple times with the same input, you get the same output (or trivially different outputs with minimal variance). There's no mechanism to "spend more compute and get a better answer" — you've already consumed all the computation you can. The paper explicitly notes this limitation: "scalar RMs could hardly generate diverse reward signals for the same response, which obstructs getting better rewards through sampling-based inference-time scaling methods" (Section 2.1). If we want inference-time scalability, we need a different kind of reward model.

Where Prior Approaches Fall Short

The paper organizes its critique of prior work along two axes (visualized in Figure 2): the reward generation paradigm (how the model represents rewards) and the scoring pattern (how the model handles multiple responses).

Scalar RMs: Fast but Inflexible and Unscalable

Scalar reward models (Cobbe et al., 2021; Wang et al., 2024e; Liu et al., 2024; the Bradley-Terry model class) output a single numerical score per response. They are computationally efficient and straightforward to train, making them the default choice for large-scale RLHF pipelines. However, they suffer from three specific limitations that the paper identifies:

No inference-time diversity: The scalar output is deterministic or near-deterministic. You cannot meaningfully "sample multiple times and aggregate" because each sample produces essentially the same number. This is the architectural barrier to inference-time scaling — there's no knob to turn.
Expressive limitations: A single scalar cannot decompose evaluation into sub-criteria, explain its reasoning, or adapt its judgment framework to different types of queries. This limits faithfulness and interpretability.
Domain biases: The paper observes empirically that scalar RMs tend to overperform on specific narrow domains (like verifiable tasks) while underperforming on others, a bias pattern that limits their generalist utility.

Pairwise RMs: Cannot Handle Single Responses

Pairwise reward models (Jiang et al., 2023; Zheng et al., 2023; Liu et al., 2025) compare two responses and select which is better. This is sufficient for many RLHF data collection scenarios (where annotators compare pairs) but fundamentally breaks for single-response evaluation: you can't ask a pairwise RM "how good is this response on its own?" without an artificial comparison response. The paper identifies this as a critical flexibility limitation: "pairwise RMs only consider the relative preference of paired responses, lacking flexibility to accept single or multiple responses as input" (Section 2.1). This matters because many RL algorithms require absolute quality estimates, not just relative rankings, and because many evaluation use cases (e.g., detecting errors in individual outputs) require single-response scoring.

LLM-as-a-Judge: Ties Are Ignored

The LLM-as-a-Judge paradigm (Zheng et al., 2023) uses LLMs themselves to generate textual judgments, typically selecting the best response from a pair. While this approach is generative (enabling some diversity through sampling), it has a specific structural flaw: "this approach defaults to neglecting ties of the paired responses" (Section 2.1). Each judgment must pick a winner, even when responses are substantively equivalent. Moreover, the standard pairwise variant suffers from the same single-response limitation as pairwise scalar RMs.

Semi-Scalar RMs: A Half-Measure That Doesn't Fully Scale

Semi-scalar approaches (Ankner et al., 2024; Zhang et al., 2025a; CLoud) generate textual critiques alongside scalar scores. This seems like it should enable inference-time scaling — you sample multiple critiques and aggregate the scores. But the paper shows empirically that the gains are limited: "the performance increase is limited, mainly due to the lack of variance in scalar reward generation, even though the critique has changed a lot" (Section 5.2, referring to CLoud-Gemma-2-27B in Table 3). The critique changes, but the final scalar score doesn't change enough for aggregation to help. This is a subtle finding: text generation alone isn't sufficient for inference-time scalability. You need the final reward itself to be part of the generative process, not a post-hoc regression on top of generation.

The paper also notes a variant of semi-scalar RMs that extracts the probability of a preference-indicating token as a scalar reward (Mahan et al., 2024; Zhang et al., 2025a). This injects some variance into the reward signal, which the paper confirms can help (LLM-as-a-Judge with TokenProb shows a performance increase from 67.0 to 68.1 with 8-sample voting in Table 3), but the improvement is modest compared to what SPCT achieves.

The Missing Link: Learning Scalable Reward-Generation Behaviors

The paper's key observation about prior work is not just that existing RM approaches have architectural limitations. It's that no prior learning method has been designed with the explicit goal of teaching the model to produce rewards that improve when you sample more. This is the critical gap:

"different learning methods have been proposed to improve the quality of rewards, but few of them focus on inference-time scalability and study the interconnection between the learned reward generation behaviors and the effectiveness of inference-time scaling" (Section 1).

This means that even if you take a generative RM (which architecturally could scale) and train it with standard methods, there's no guarantee it will actually learn scalable behaviors. The model needs to learn that generating diverse, high-quality principle-critique pairs, such that voting over many such pairs produces better aggregate judgments, is a good strategy. Standard supervised fine-tuning doesn't teach this, because it trains the model to produce one correct output per input rather than a distribution of useful outputs.

How This Paper Positions Itself

The paper positions itself at the confluence of two research currents and proposes a synthesis that addresses limitations in both.

First current: Generative reward modeling. Recent work (Li et al., 2024a; Kim et al., 2024; Vu et al., 2024; Cao et al., 2024; Alexandru et al., 2025) has shown that representing rewards as text rather than scalars enables richer, more interpretable evaluations. These models can explain their reasoning, break down judgments by criteria, and flexibly handle different numbers of responses. The paper explicitly adopts this paradigm, choosing pointwise generative reward modeling as the foundation for DeepSeek-GRM: "pointwise generative reward modeling (GRM) could unify the scoring of single, paired, and multiple responses within pure language representation" (Section 1). This resolves the input flexibility challenge — the same model can rate one response, compare two, or rank many, all in a consistent format.

Second current: Principle-guided evaluation. The idea that evaluation should be guided by explicit criteria goes back to Constitutional AI (Bai et al., 2022b), where hand-crafted principles were used to guide models in self-improvement loops. Other work explored principle-driven self-alignment (Sun et al., 2023; 2024) and rule-based rewards for safety (Mu et al., 2024). However, these approaches all share a critical limitation: principles are static, hand-crafted, and limited in scope. The paper argues that for truly generalist reward modeling, principles must be generated adaptively based on the specific query and responses, not pre-specified by humans.

The paper's preliminary experiment (Table 1) provides the empirical foundation for this positioning. Testing on the Chat Hard subset of Reward Bench and the IFEval subset of PPE, the authors find that:

Self-generated principles (without filtering) barely help — GPT-4o goes from 76.1 to 75.9, Gemma-2-27B-it from 59.1 to 64.0.
Filtered principles (keeping only those that led to correct judgments) significantly boost performance — GPT-4o reaches 77.8, Gemma-2-27B-it reaches 68.0.

This is the key insight that drives the entire SPCT design: "Current LLMs could generate diverse principles, but not all of them are proper for reward generation. A subset of generated principles could better guide reward generation under correct criteria, indicating a potential of self-bootstrapping" (Section 2.2). The model can generate many candidate principles; some of them are good and lead to correct evaluations; if you can train the model to preferentially generate the good ones, you get better rewards. And if you can do this via online RL — where the model learns from its own generated principles, guided by a correctness signal — you get a self-reinforcing cycle of improvement.

The gap this paper fills. Prior work on GRMs focused on making them produce good single outputs. Prior work on principle-guided evaluation used fixed principles. Prior work on inference-time scaling focused on policy models, not reward models. This paper combines all three: it proposes a training method (SPCT) that teaches a GRM to adaptively generate principles and critiques such that inference-time scaling — sampling multiple times and voting — produces substantially better aggregate rewards than any single sample. The method is designed from the ground up to create scalable behavior, not just to improve single-sample quality.

The paper explicitly frames this as answering a specific research question: "Can we design a learning method aiming to enable effective inference-time scaling for generalist reward modeling?" (Section 1). This positions the contribution not as "we built a better reward model" (though that's true) but as "we identified what architectural and training choices enable reward models to benefit from additional inference compute, and we demonstrated a method that achieves this."

3. Technical Approach

3.1 Reader Orientation

The paper constructs a training pipeline that transforms a base pretrained LLM (Gemma-2-27B) into a generative reward model (DeepSeek-GRM) whose outputs become more accurate as you sample them more times. The core problem is that standard reward models produce either a fixed scalar (which cannot benefit from repeated sampling) or a single judgment (which doesn't improve with aggregation). The solution's shape is a two-stage training recipe — rejective supervised fine-tuning followed by rule-based online RL — that teaches the model to produce diverse, independently useful evaluations (each consisting of self-generated criteria plus a scored critique), such that voting over many such evaluations yields substantially better aggregate judgments than any single sample. A separate meta reward model then learns to filter out low-quality samples before voting, further boosting scaling efficiency.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, built sequentially through a training pipeline:

Pretrained GRM (starting point). A base LLM (Gemma-2-27B) already capable of generating text. Before SPCT, it can produce principle-guided critiques but with low accuracy — it serves as the raw material to be shaped.
Rejective Fine-Tuning (RFT) stage. The pretrained GRM is prompted to generate principles and critiques for training queries. A rejection filter discards any trajectory where the predicted best response doesn't match the ground-truth label. The surviving (correct) trajectories form a curated dataset used to supervised-fine-tune the model. This produces DeepSeek-GRM-RFT, a cold-start model that formats outputs correctly and achieves non-trivial accuracy.
Rule-Based Online RL stage. DeepSeek-GRM-RFT is further trained with GRPO, a policy-gradient RL algorithm. For each training query, the model samples multiple principle-critique outputs, a binary reward signal (+1 for correct preference identification, -1 otherwise) is computed by comparing extracted scores against ground-truth labels, and the policy is updated to increase the probability of correct outputs. This produces DeepSeek-GRM, which learns to generate diverse, high-quality principles adaptively.
Meta Reward Model. A separate pointwise scalar RM, trained on trajectories sampled from DeepSeek-GRM, learns to predict whether a given principle-critique pair will lead to a correct judgment. At inference time, this model scores each of the k sampled outputs, and only the top k_meta (by default, k/2) are included in the final vote. This filters out low-quality samples, improving scaling efficiency.

Inference-time flow: A query and set of responses enter DeepSeek-GRM → the model is sampled k times in parallel (temperature = 0.5), each sample generating its own set of principles, a critique, and pointwise scores (1–10) for each response → the meta RM scores each sample's quality → the top k_meta samples' scores are summed per response → the response with the highest total score is selected as best.

3.3 Roadmap for the Deep Dive

First, the pointwise GRM scoring format, because every subsequent training step — RFT rejection, RL reward computation, and inference-time voting — depends on extracting and comparing structured scores from generated text.
Second, the RFT stage, including the hinted versus non-hinted sampling distinction, the unified rejection criterion, and why this cold start is necessary before RL can work.
Third, the rule-based RL stage, covering the GRPO objective, how the binary reward signal is defined from ground-truth labels, and the critical role of the KL penalty in preventing format collapse.
Fourth, the meta RM, including how it is trained, what data it uses, and how it performs guided voting at inference time.
Fifth, inference-time scaling mechanics, detailing the voting equations for pointwise GRMs, why summing discrete scores expands the effective reward space, and the computational cost model.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology and systems paper whose core idea is that by training a generative reward model to produce diverse, self-principled evaluations and then aggregating them at inference time via voting (optionally filtered by a learned quality estimator), you can achieve inference-time scalability — more compute produces monotonically better rewards — with gains that can surpass scaling to much larger models.

Pointwise Generative Reward Format

The entire SPCT pipeline hinges on a specific output format: the model generates a structured text containing (a) self-generated evaluation principles with associated weights, (b) a comparative analysis against those principles, and (c) discrete pointwise scores (integers 1–10) for each response, enclosed in \boxed{}. This format is not an arbitrary choice — it is the architectural prerequisite for every subsequent mechanism in the paper.

Why pointwise rather than pairwise. In a pairwise format, the model selects which of two responses is better, often by outputting something like "Response 1 is better." This approach has two structural problems for inference-time scaling. First, each sample produces a categorical choice (win/lose), and aggregating categorical choices via majority vote discards information about how much better one response is — two samples where Response A wins narrowly and two where Response B wins overwhelmingly produce the same majority outcome as four samples where Response A wins narrowly, even though the underlying quality signal differs substantially. Second, pairwise judgments cannot naturally handle ties or single-response evaluation.

The pointwise format avoids both problems. For n responses, each sample outputs n independent scores on a 1–10 scale. This provides quantitative granularity — a response that consistently scores 8–9 across samples is clearly distinguished from one that scores 5–6, even if both "win" in pairwise comparison against the alternative. The paper formalizes the extraction:

$$\{S_i\}_{i=1}^n = f_{\text{point}}(R, \{y_i\}_{i=1}^n) = f_{\text{extract}}(C)$$

where $R = C$ is the generated critique text, $\{y_i\}_{i=1}^n$ are the responses being evaluated, $f_{\text{point}}$ is the scoring function, and $f_{\text{extract}}$ extracts the integer scores from the generated text (specifically from the \boxed{} delimiters).

What it computes: Given a model-generated critique text, the extraction function parses out n integer scores, one per response, each in the range $[1, 10]$ . These scores represent the model's absolute quality assessment of each response independently.

Why this form: The discrete 1–10 range is small enough to be learnable (the model can reliably distinguish 10 quality levels) but large enough to provide meaningful granularity for aggregation — when scores are summed across k samples, the effective range expands to $[k, 10k]$ , creating a much finer-grained composite score. A continuous scalar (e.g., a floating-point value from a regression head) would lack the diversity needed for voting, because independent samples would produce nearly identical numbers. A binary correct/incorrect judgment would discard too much information per sample.

The role of principles in the format. The output is not just scores — it includes a "Specific Criteria" section listing weighted evaluation principles (e.g., "Instruction Adherence (Weight: 40%), Level of Detail (Weight: 30%), ...") followed by an "Analysis" section applying those principles to each response, and finally the boxed scores. This structure is what enables the model to produce diverse outputs across samples. When sampled with non-zero temperature, the model generates different sets of principles (different criteria, different weightings, different numbers of principles), which leads to different analyses and different final scores. This diversity is the engine of inference-time scaling — it creates the variance that makes voting useful.

The preliminary experiment (Table 1) established that diverse principles can lead to better rewards, but only when they are well-chosen. The SPCT training pipeline is designed to teach the model to generate principles that are both diverse and likely to produce correct judgments.

Rejective Fine-Tuning (Cold Start)

The RFT stage addresses a chicken-and-egg problem: to train the model with online RL (where it learns from its own generated outputs), you need a model that already produces outputs in the correct format with non-trivial accuracy. A randomly initialized or purely pretrained model would produce mostly garbage outputs, providing no useful learning signal. RFT bootstraps this by creating a clean supervised dataset through rejection sampling.

Data construction procedure. The training set for RFT comprises 1,256K examples: 1,070K general instruction-tuning examples (from in-house datasets) that teach basic formatting and language capabilities, plus 186K rejective-sampled RM examples. The RM examples are constructed as follows:

For each training data point (a query and one or more responses with a ground-truth label indicating the best response), a pretrained GRM (DeepSeek-V2.5-0905) is prompted to generate principles and critiques $N_{\text{RFT}} = 3$ times.
Each generated trajectory is scored: the extracted pointwise scores are compared against the ground-truth label. A trajectory is correct if the response with the highest predicted score matches the ground-truth best response.
Trajectories where the predicted best response is wrong are rejected (discarded). Trajectories where all $N_{\text{RFT}}$ samples are correct for a given query are also discarded — these are considered "too easy" and do not provide useful learning signal.
The surviving trajectories form the RFT training dataset.

The rejection criterion (Equation 10). For n responses $y_1, ..., y_n$ with ground-truth rewards $r_1, ..., r_n$ (where exactly one response is best), the predicted pointwise scores $\{S_i\}_{i=1}^n$ are correct if:

$\begin{cases} \forall i \neq j, S_j > S_i, \quad j = \arg\max_l r_l, & \text{if } n \geq 2 \\ S_1 = r_1, & \text{if } n = 1 \end{cases}$
where $j$ is the index of the ground-truth best response, $S_j$ is the predicted score for that response, and $S_i$ are the predicted scores for all other responses.

What it computes: For multi-response cases ( $n \geq 2$ ), the condition checks that the response labeled as best in the ground truth receives a strictly higher predicted score than every other response. For single-response cases ( $n = 1$ ), it checks that the predicted score exactly matches the ground-truth label (0 for incorrect, 1 for correct — single-response data is only used for verifiable questions where correctness is known).

Why this form: The strict inequality $S_j > S_i$ (rather than $S_j \geq S_i$ ) ensures that the model learns to discriminate between responses, not just assign them all the same score. The single-response case uses exact equality because there is no comparison to make — the only question is whether the model correctly identifies the response as right or wrong. The unified formulation allows mixing single-response, pairwise, and multi-response data in the same training pipeline without format switching.

Hinted versus non-hinted sampling. The RFT stage uses two distinct sampling strategies:

Non-hinted sampling: The model generates principles and critiques given only the query and responses, with no additional guidance. Each data point is sampled multiple times ( $N_{\text{RFT}} = 3$ ) and rejected if incorrect.
Hinted sampling: An additional segment is appended to the input: "The best response is: Response $\arg\max_l r_l$ ". The model is told which response is correct and must generate principles and critiques consistent with that answer. Hinted trajectories are sampled once per data point and rejected only if incorrect (i.e., if the model fails to align with the hint).

The paper observes that hinted sampling generates higher-quality data but introduces a problem: "hinted sampled trajectories sometimes take shortcuts in the generated critique, especially for reasoning tasks" (Section 3.2). The model learns to produce critiques that justify the known answer without genuine analysis. This motivates the subsequent RL stage, which optimizes for accurate outputs when the answer is not provided.

The ablation (Table 4, bottom) quantifies the relative importance of these components in the RFT stage:

Removing all rejective sampling (using only general instruction data) drops greedy accuracy from 68.8% to 63.3% (RFT stage) or from 69.9% to 68.7% (after RL follows). This confirms that RM-specific data matters but RL can partially compensate.
Removing non-hinted sampling drops greedy accuracy to 67.4% (RFT stage); removing hinted sampling drops it to 68.0%. Non-hinted sampling is slightly more important, consistent with the "shortcuts" observation — hinted data alone produces a model that relies on receiving the answer rather than learning to judge independently.
Removing both (keeping only general instruction data) drops RFT accuracy to 66.1%.

Training hyperparameters for RFT. The Gemma-2-27B model is fine-tuned with learning rate $5 \times 10^{-6}$ , batch size 1024, for 900 steps. The training uses 128 A100 GPUs and takes 19.2 hours. All DeepSeek-GRM models are initialized from the pretrained (not instruction-tuned) version of the base LLM — the instruction-tuning capability comes entirely from the RFT stage's general instruction data and the RM-specific trajectories.

What RFT produces. The output is DeepSeek-GRM-RFT, which achieves an overall score of 68.8% across all RM benchmarks with greedy decoding. Notably, it already outperforms some baseline methods (LLM-as-a-Judge at 67.0% with greedy, DeepSeek-BTRM-27B at 68.6%) but trails the strongest scalar baselines and shows limited inference-time scalability: voting with 8 samples improves from 67.8% to only 69.3% (+1.5%), compared to DeepSeek-GRM's +2.7% improvement from the same starting point. The RFT model knows the right format and achieves baseline competence, but it hasn't learned to generate the diverse, independently-useful evaluations that make voting effective.

Rule-Based Online Reinforcement Learning

The RL stage takes DeepSeek-GRM-RFT and further optimizes it using GRPO (Group Relative Policy Optimization), the same algorithm used in DeepSeek-R1 to incentivize reasoning capabilities. The key insight is that the binary accuracy signal — did the model correctly identify the best response? — provides a clean, automatic reward that requires no human annotation during training. This enables the model to learn from its own generated outputs at scale.

The GRPO objective (Equation 15). The model is optimized to maximize:

$J_{\text{GRPO}}(\theta) = \mathbb{E}\left[q \sim P(Q), \{o_i\}_{i=1}^G \sim \pi_{\theta_{\text{old}}}(O|q)\right] \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \left[ \min\left( \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})} \hat{A}_{i,t}, \text{clip}\left( \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q, o_{i,<t})}, 1-\epsilon, 1+\epsilon \right) \hat{A}_{i,t} \right) - \beta D_{\text{KL}}[\pi_\theta || \pi_{\text{ref}}] \right]$

where $q = (x, \{y_i\}_{i=1}^n)$ is the input (query and responses), $G$ is the group size (number of outputs sampled per query), $o_i$ is the $i$ -th sampled output (a complete principle-critique-score text), $|o_i|$ is its length in tokens, $\pi_\theta$ is the current policy, $\pi_{\theta_{\text{old}}}$ is the frozen policy from the previous iteration, $\hat{A}_{i,t}$ is the advantage estimate for the $t$ -th token of the $i$ -th output, $\epsilon$ is the clipping parameter, $\beta$ is the KL penalty coefficient, and $D_{\text{KL}}[\pi_\theta || \pi_{\text{ref}}]$ is the KL divergence between the current policy and a reference policy (the RFT model).

What it computes: For each batch of queries, the model samples $G$ outputs, computes a scalar reward $\hat{r}_i$ for each based on accuracy (described below), standardizes these rewards to zero mean and unit variance to produce advantages $\hat{A}_{i,t}$ , then takes a policy gradient step that increases the probability of outputs with positive advantages while clipping updates that would move the probability ratio too far from 1. The KL penalty term prevents the policy from drifting too far from the RFT starting point.

Why this form: GRPO is chosen over PPO because it eliminates the need for a separate value function (critic) — the advantages are computed from within-group reward comparisons rather than from learned value estimates. This is computationally simpler and avoids the instability of training a value function on the same limited reward signal. The clipping mechanism ( $\min$ with ratio bounded by $1 \pm \epsilon$ ) is standard in PPO-family algorithms and prevents destructively large policy updates. The KL penalty is critical for this specific application, as discussed below.

Hyperparameters. The paper performs a grid search over $\beta \in \{0.00, 0.01, 0.02, 0.08\}$ and finds that $\beta = 0.08$ is "the most stable configuration for DeepSeek-GRM-27B." Lower KL coefficients cause the model to "collapse on a few subsets in benchmarks, e.g., Chat in the Reward Bench benchmark and Harmlessness in the RMB benchmark, and shows biases towards some other domains" (Appendix C.1). For the smaller DeepSeek-GRM-16B, a much lower coefficient $\beta = 0.002$ works because the smaller model is "less vulnerable to the KL loss coefficient." The group size $G = 4$ is chosen as "a better trade-off between efficiency and performance." The learning rate is $4 \times 10^{-7}$ , batch size 512, trained for 900 steps on 128 A100 GPUs, taking 15.6 hours. The RL training data comprises 237K examples (the same RM datasets as RFT, with "too easy" examples removed).

The rule-based reward (Equation 11). For each sampled output, a binary reward is computed by comparing the extracted scores to the ground truth:

$\hat{r}_i = \begin{cases} 1, & \text{if } n \geq 2 \text{ and } \forall i' \neq j', S_{j'} > S_{i'}, \quad j' = \arg\max_l r_l \\ 1, & \text{if } n = 1 \text{ and } S_1 = r_1 \\ -1, & \text{otherwise} \end{cases}$

where $\{S_i\}_{i=1}^n$ are the pointwise scores extracted from the $i$ -th output, and $j'$ is the index of the response predicted as best by that output.

What it computes: A single scalar +1 (correct) or -1 (incorrect) per generated output, based on whether the extracted scores correctly identify the ground-truth best response. For multi-response cases, this means the predicted best response must strictly outscore all others. For single-response cases, the predicted score must exactly match the ground-truth binary label.

Why this form: The binary +1/-1 reward (rather than, say, a continuous score based on how close the predicted scores are to some ideal) provides a clean, unambiguous signal. It is also automatically computable from any preference dataset — no human annotation is needed during RL training. The paper notes that "no format rewards are used" (Section 3.2), unlike DeepSeek-R1 which used format rewards to encourage structured outputs. Instead, the large KL penalty ( $\beta = 0.08$ ) serves double duty: it prevents format collapse (the model stays close enough to the RFT model to maintain correct output structure) and prevents domain bias collapse.

What the RL stage learns. The key transformation is from generating one type of output to generating a diverse distribution of useful outputs. Before RL, the model tends to produce similar principles and critiques across samples — it's optimized to produce a single "correct" output per input, as is typical for supervised fine-tuning. After RL, the model learns that different principles can all lead to correct judgments, and that generating diverse principle sets is beneficial because the RL objective rewards any output that produces correct scores, regardless of which specific principles were used.

The response length analysis (Figure 7) provides evidence for adaptive compute allocation: after RL, the model's output length increases substantially for Reasoning tasks (from 218 to 376 tokens) but barely changes for Chat tasks (241 to 259 tokens) and even decreases slightly for Safety tasks (245 to 260 to 259 tokens). This suggests the model learns to "adaptively use more inference compute on reasoning extensive tasks, and the compute could be saved for some other domains" (Appendix E.4). This adaptive behavior is crucial for inference-time scaling — it means the model doesn't waste tokens generating elaborate principles for simple chat comparisons, but invests more analysis in complex reasoning tasks where deeper evaluation helps.

The training data includes a mix of response counts. The RL training data incorporates "the preference for single, paired, and multiple responses, constructed from internal data and open-source datasets" (Appendix C.1), including MATH, UltraFeedback, OffsetBias, Skywork-Reward-Preference-80K-v0.2, and HelpSteer2-Preference. For UltraFeedback, the authors "re-tagged the preference label of a part of UltraFeedback due to its quality issues." For MATH, they "sampled and filtered trajectories on MATH by rule-based ground-truth matching, resulting in pairwise preference data." For single-response rating, they use only verifiable questions where correctness is known, setting "the ground-truth reward to 1 for correct responses and 0 for incorrect ones." This mixed training is essential for the model's input flexibility — it learns to handle any number of responses using the same format.

Ablation confirming RL's necessity. The paper shows (Table 4) that starting from general-instruction-tuned GRMs without any RFT rejective sampling, RL still improves performance from 66.1% to 68.7%. This is a surprising finding: "without the cold start with rejective sampled critique data, general-instruction-tuned GRMs still improve significantly after undergoing the online RL." It means RL alone, given enough training, can bootstrap reasonable reward generation behavior even without curated RFT data, though the combination of RFT cold start + RL achieves the best results (69.9%).

Meta Reward Model (Meta RM)

The meta RM addresses a practical problem with naive voting: some fraction of sampled outputs are low-quality — they contain poorly chosen principles, incorrect analyses, or random scores — and including them in the vote degrades the aggregate. The meta RM learns to identify and filter out these bad samples before voting.

Training data construction. The meta RM is trained on trajectories generated by DeepSeek-GRM. The training set reuses the rejective-sampled data from the RFT stage (where DeepSeek-V2.5-0905 generated outputs that were checked against ground truth) and adds fresh trajectories sampled from the final DeepSeek-GRM-27B (post-RL). The latter is important to "alleviate the gap between training and inference policy" (Section 4) — the meta RM needs to evaluate outputs from the same distribution it will see at inference time.

Each training example consists of: the query, the candidate responses, the generated principles text, and the generated critique text. The label is binary: 1 if the extracted scores from that trajectory correctly identify the ground-truth best response (Equation 10), 0 otherwise. The meta RM is trained as a pointwise scalar RM with binary cross-entropy loss.

Meta RM training details. The meta RM is based on Gemma-2-27B, trained with learning rate $1 \times 10^{-5}$ and batch size 512. To avoid bias, the authors "use DeepSeek-GRM-27B to perform rejective sampling with $N_{\text{RFT}} = 3$ " — that is, they sample 3 times from the GRM, check correctness, and use all trajectories (both correct and incorrect) for training the meta RM, ensuring a balanced mix of positive and negative examples.

Guided voting procedure. At inference time, given k sampled outputs from DeepSeek-GRM:

The meta RM processes each output (query + responses + that output's principles + that output's critique) and produces a scalar meta-score.
The k outputs are sorted by meta-score, and only the top $k_{\text{meta}} \leq k$ are retained.
The final reward for each response is computed by summing the pointwise scores from only the retained outputs:

$S_i^* = \sum_{j \in \text{top-}k_{\text{meta}}} S_{i,j}$

where $S_i^*$ is the final score for response $i$ , and $S_{i,j}$ is the score assigned to response $i$ by the $j$ -th retained sample.

What it computes: A filtered sum — only the samples that the meta RM deems likely to be correct contribute to the final scores. Low-quality samples (those with likely incorrect judgments) are excluded entirely.

Why this form: Filtering before summing (rather than, say, weighting each sample by its meta-score) is more robust because it completely removes bad samples rather than dampening them. A bad sample with an incorrect judgment could still distort weighted sums if its scores are extreme, even with a low weight. By default, $k_{\text{meta}} = \frac{k}{2}$ — half the samples are filtered out.

Ablation on $k_{\text{meta}}$ (Table 4, bottom). The paper tests different meta RM filtering fractions with k = 32:

$k_{\text{meta}} = 1$ (keep only the single best sample): 71.5% overall — better than greedy (69.9%) but worse than naive voting (71.0%).
$k_{\text{meta}} = 8$ (keep top 8 of 32): 72.7% — substantial improvement over naive voting.
$k_{\text{meta}} = 16$ (keep top 16 of 32, the default half): 72.8% — best performance.

The small gap between $k_{\text{meta}} = 8$ and $k_{\text{meta}} = 16$ suggests that the meta RM's ranking is well-calibrated — the top 8 samples capture most of the benefit, and including more adds diminishing returns. The poor performance at $k_{\text{meta}} = 1$ confirms that even the "best" single sample according to the meta RM is less reliable than aggregating several good ones.

Comparison to naive voting. The meta RM provides the largest gains on the PPE Correctness benchmark specifically (Table 9): with k = 8, meta RM (k_meta = 4) achieves 63.0% vs. 60.3% for naive voting at the same k. This suggests the meta RM is particularly effective at filtering out samples that make errors on verifiable tasks, where correctness is unambiguous and the gap between good and bad samples is starker.

Inference-Time Scaling Mechanics

The inference-time scaling process builds on the pointwise scoring format and combines it with parallel sampling and voting. The key insight is that summing discrete scores across samples expands the effective reward space, transforming coarse individual judgments into fine-grained aggregate assessments.

Naive voting (Equation 14). The basic aggregation method is:

$S_i^* = \sum_{j=1}^k S_{i,j}, \quad \{p_{i,j}\}_{i=1}^{m_j} \sim p_\theta(x, \{y_i\}_{i=1}^n), \quad R_j = C_j \sim r_\theta\left(x, \{y_i\}_{i=1}^n, \{p_{i,j}\}_{i=1}^{m_j}\right)$

where $k$ is the number of sampled outputs, $S_i^*$ is the final score for response $i$ , $S_{i,j}$ is the score from the $j$ -th sample, $\{p_{i,j}\}_{i=1}^{m_j}$ are the principles generated in the $j$ -th sample, $m_j$ is the number of principles in that sample, and $R_j = C_j$ is the critique text for the $j$ -th sample.

What it computes: For each response, sum the integer scores (1–10) assigned to it across all $k$ sampled evaluations. The response with the highest total is predicted as best. Each sample independently generates its own principles and critique before producing scores.

Why this form: Summation (rather than averaging or majority vote over best-response indices) preserves the quantitative information in each sample. If Response A consistently scores 8–9 while Response B scores 6–7, the summed difference grows with $k$ , making the aggregate more confident. Averaging would not increase confidence — the mean stays the same regardless of $k$ . Majority vote over categorical "best response" selections would discard the magnitude of preference entirely.

Expanding the reward space. Since each sample produces scores in $\{1, ..., 10\}$ , the summed score across $k$ samples for a single response falls in $\{k, ..., 10k\}$ . This means the effective resolution of the final score scales linearly with $k$ — with $k = 32$ , the possible score range spans 320 distinct integer values. The paper explains this as: "if each principle could be viewed as a proxy of judgement perspectives, a larger number of principles may reflect the real distribution more accurately, leading to scaling effectiveness" (Section 4). Each sample brings a different set of evaluation perspectives (principles), and aggregating across many perspectives produces a more robust final judgment than any single perspective could.

Practical details. To avoid positional biases (the model might systematically prefer the first response listed), responses are shuffled before being presented to the model in each sample. The temperature is set to 0.5 for inference-time scaling (0 for greedy decoding). The paper notes that "due to the universality of the pointwise GRM approach, DeepSeek-GRM could potentially serve as a process RM in addition to the outcome RM" (Appendix B), though this capability is not explored in depth.

Computational cost. The paper acknowledges the efficiency challenge: "the efficiency of the generative RMs is largely lagging behind the scalar RMs at the same scale by nature, which inhibits its large-scale usage in online RL pipelines" (Appendix B). Each sample requires generating potentially hundreds of tokens (principles + analysis + scores). However, because sampling is done in parallel, "the latency of reward generation with a reasonable amount of, e.g., eight samplings will not increase significantly." This is a latency-throughput tradeoff: parallel sampling increases total FLOPs but, with sufficient hardware parallelism, the wall-clock time is bounded by the single longest generation.

Comparison with alternative voting methods (Table 3). The paper explicitly compares DeepSeek-GRM's voting with other voting-amenable methods:

LLM-as-a-Judge (majority voting over discrete best-response indices): 67.0% (greedy) → 67.6% (+0.6) at k=8. The small gain reflects the information loss from categorical voting — each sample provides only a binary win/lose signal per response, and majority voting over these binary signals saturates quickly.
LLM-as-a-Judge with TokenProb (using the probability of the preference-indicating token as a scalar): 67.0% (greedy) → 68.1% (+1.1) at k=8. Better than majority voting because the token probability provides a continuous weight, but limited by the pairwise format — each sample still only compares two responses, so the token probability only captures relative preference, not absolute quality.
CLoud-Gemma-2-27B (semi-scalar, averaging scalar scores): 68.5% (greedy) → 68.8% (+0.3) at k=8. The near-zero gain demonstrates the paper's key architectural claim: "the performance increase is limited, mainly due to the lack of variance in scalar reward generation, even though the critique has changed a lot" (Section 5.2). The scalar head produces nearly identical outputs regardless of how the critique varies, so averaging doesn't help.
DeepSeek-GRM-27B (pointwise, summing): 67.9% (greedy) → 70.6% (+2.7) at k=8 → 71.0% (+3.1) at k=32. The gains are substantially larger and continue to increase with k, confirming that the pointwise format + diverse principle generation creates genuine inference-time scalability.

Why SPCT specifically enables this scaling. The RFT-only model (DeepSeek-GRM-RFT) achieves only +1.5% improvement at k=8, compared to +2.7% for the RL-trained model. This difference — RL adding roughly 1.8× the scaling benefit of RFT alone — is the paper's core methodological result. The RFT model produces correct but relatively uniform outputs; the RL model learns that diversity across samples is beneficial and produces genuinely varied evaluations. The meta RM provides a further boost by filtering out the small fraction of samples where this diversity produces incorrect results, yielding a total improvement of +4.1% at k=8 (from 67.9% greedy to 72.0% with meta RM) and +4.9% at k=32.

Training-time versus inference-time scaling (Figure 4). The paper demonstrates that inference-time scaling of the 27B model can match or exceed training-time scaling to much larger models:

DeepSeek-GRM-27B with naive voting at k=32 achieves 88.5% on Reward Bench, comparable to the 671B MoE DeepSeek-V3 at greedy decoding (88.4%).
DeepSeek-GRM-27B with meta RM voting at k=8 achieves 89.8% on Reward Bench, surpassing both the 671B MoE model and DeepSeek-R1 (84.9% on the downsampled test set).
DeepSeek-GRM-27B with meta RM voting at k=32 achieves 90.4% on Reward Bench, the best result overall.

This is a strong result, but it is primarily demonstrated on Reward Bench specifically, and the paper acknowledges that on verifiable tasks (PPE Correctness), scalar models retain an advantage that inference-time scaling cannot fully close — DeepSeek-GRM-27B at k=32 with meta RM achieves only 63.2% on PPE Correctness, compared to Nemotron-4-340B-Reward at 60.8% and several scalar models in the mid-60s.

4. Key Insights and Innovations

Innovation 1: Inference-Time Scalability as a Training Objective for Reward Models, Not an Architectural Afterthought

The dominant conceptual move in this paper is not the use of generative reward models (GRMs) per se — prior work already explored GRMs (Li et al., 2024a; Kim et al., 2024) — nor is it the idea of principle-guided evaluation (Constitutional AI did that with hand-crafted rules, Bai et al., 2022b). The novel framing is that inference-time scalability should be treated as a learned behavior that must be explicitly incentivized during training, rather than an emergent property of using generative outputs.

Before this work, the field implicitly assumed that if a reward model produces text, it automatically becomes amenable to sampling-based aggregation — just generate multiple critiques and average or vote. The paper shows this assumption is false in two complementary ways. First, the CLoud semi-scalar baseline (Table 3) generates varied critiques but nearly identical scalar scores, yielding only +0.3% improvement with 8-sample voting — text diversity alone is insufficient. Second, DeepSeek-GRM-RFT (the same architecture, but trained only with supervised fine-tuning) achieves only +1.5% improvement at k=8, compared to +2.7% for the RL-trained DeepSeek-GRM. The architecture is identical in both cases; the difference is entirely in what the training procedure taught the model to do.

The conceptual reframing is this: standard supervised fine-tuning optimizes the model to produce a single correct output per input. An architecture that could produce diverse outputs does not automatically learn to produce usefully diverse outputs. The RL stage, by rewarding any output that produces correct scores regardless of which principles were used, teaches the model a different objective — generate a distribution of independently useful evaluations, where diversity across samples is beneficial because different principles capture different facets of response quality. This is fundamentally different from training for single-sample accuracy and represents a shift from viewing inference-time scaling as an inference technique to viewing it as a capability that must be learned.

The significance extends beyond reward modeling. This insight suggests that many "scalable" generative systems — verifiers, evaluators, debate participants — may need to be explicitly trained for scalable behavior rather than having it emerge from architecture alone. It parallels the finding in DeepSeek-R1 (DeepSeek-AI, 2025) that long-chain reasoning emerges from RL incentivization, not from prompting alone, and extends that logic to the evaluator side of the RL pipeline. The paper is essentially arguing that inference-time scaling laws, like pretraining scaling laws, have a training methodology prerequisite — you can't just take any model and scale its inference; you need to train it in a way that makes scaling effective.

Innovation 2: Self-Generated Principles as a Mechanism for Creating Structured Diversity

The paper's use of principles is superficially similar to prior work — Constitutional AI uses principles to guide model behavior, LLM-as-a-Judge prompting sometimes includes evaluation criteria. But the paper makes a specific diagnostic move that distinguishes its approach: it identifies that not all generated principles are equally useful (Table 1), and that the distribution of useful principles can be bootstrapped through online RL with only a binary outcome signal.

The preliminary experiment (Table 1) is deceptively simple but carries a non-obvious implication. When GPT-4o generates its own principles, performance drops slightly (76.1% → 75.9%). When only "filtered" principles — those that happened to lead to correct judgments — are provided, performance improves (77.8%). This means the model has the capability to generate good principles and to use them effectively, but it cannot reliably select which generated principles are good. The gap between "self-generated" and "filtered" is a measure of the model's failure to self-regulate its own evaluation criteria.

Prior work addressed this gap through human curation (hand-crafted constitutions, expert-written rubrics) or through separate classifier training. The paper's key conceptual move is to collapse principle generation and principle selection into a single RL objective. By rewarding the GRM only when its final scores are correct (Equation 11), the RL signal implicitly favors principle sets that lead to accurate judgments without requiring any explicit signal about which principles are good. The model learns to generate principles that are likely to produce correct scores because generations with poor principles receive negative rewards and get down-weighted by the policy gradient.

This creates a self-bootstrapping dynamic that the paper explicitly identifies: "the principles generated by DeepSeek-GRM-27B could be transferred to other models, and are even slightly better than manually filtered principles from GPT-4o" (Appendix E.2). The RL-trained GRM doesn't just get better at using its own principles — it gets better at generating principles in the first place, to the point where its principles transfer as effective evaluation guides to other models. This is evidence of genuine learning at the meta-cognitive level — the model learns what makes a good evaluation criterion, not just how to apply a fixed set of criteria.

The diversity created by this principle-generation process is qualitatively different from the diversity created by simple temperature sampling of a flat critique. Temperature sampling produces surface-level variation — different wordings, different ordering of the same points. SPCT produces structural variation — different evaluation frameworks, different weighting of criteria, different decomposition of the quality construct. This structural diversity is why voting works better for DeepSeek-GRM than for simpler generative approaches: aggregating genuinely different perspectives produces a more robust estimate than aggregating paraphrases of the same perspective.

Innovation 3: Pointwise Generative Scoring as a Unifying Format That Resolves the Flexibility-Scalability Tension

The paper identifies and resolves what amounts to a trilemma in reward model design: previous approaches could achieve at most two of {input flexibility, inference-time scalability, quantitative granularity}. Scalar RMs provide quantitative scores and handle single responses (flexibility ✓, granularity ✓), but cannot scale (scalability ✗). Pairwise RMs can potentially scale via majority voting (categorical aggregation), but cannot handle single responses (flexibility ✗) and provide only relative rankings (granularity ✗). LLM-as-a-Judge handles pairwise comparisons with text (inference diversity ✓ to some degree), but ignores ties and lacks single-response capability (flexibility ✗).

The pointwise GRM format — generating integer scores (1–10) for each response independently, within a structured text that includes principles and analysis — resolves all three simultaneously. The discrete score range provides quantitative granularity (the summed scores across k samples span [k, 10k], enabling fine distinction). The pointwise nature handles any number of responses including single responses (input flexibility ✓). And the generative format combined with SPCT training creates structured diversity that enables inference-time scaling (scalability ✓).

What makes this a conceptual innovation rather than just an engineering choice is the recognition that the scoring format directly determines the effectiveness of aggregation. The paper provides a clear comparative analysis (Table 3) showing that different voting mechanisms — majority over categorical indices, averaging over near-invariant scalars, summing over discrete pointwise scores — produce dramatically different scaling behavior even when the underlying model quality is similar at k=1. The pointwise format's advantage is not that individual samples are more accurate (DeepSeek-GRM at greedy is 67.9%, comparable to alternatives), but that the aggregation function extracts more signal from multiple samples.

This has implications for how we should think about designing generative systems intended for inference-time scaling. The choice of output representation — categorical vs. ordinal vs. cardinal, coarse vs. fine-grained — is not just a matter of convention. It determines the information-theoretic efficiency of aggregation. A representation that discards quantitative information per sample (e.g., picking a winner rather than scoring) cannot recover that information through aggregation, no matter how many samples are taken. The paper demonstrates this empirically but the principle is more general: inference-time scaling requires lossless or near-lossless per-sample representations to be effective.

Innovation 4: Meta Reward Modeling as On-Policy Quality Filtering for Generative Evaluators

The meta RM is superficially a straightforward component — train a classifier to predict which samples are correct, filter before voting. But the conceptual contribution is more significant: it introduces the notion that generative evaluators produce an implicit distribution over evaluation qualities, and a learned filter over this distribution can improve scaling efficiency beyond what naive aggregation achieves.

Prior work on verifier-guided voting (e.g., using a PRM to weight samples in best-of-N) assumed access to a verifier trained on the same distribution as the policy model's outputs. The meta RM faces a harder problem: it must evaluate the evaluator's outputs, which are themselves evaluations of arbitrary responses. This is a meta-evaluation task — judging judgments — and it requires the meta RM to assess not the quality of responses to a query, but the quality of a critique about responses to a query.

The paper's specific contribution here is the on-policy training methodology: the meta RM is trained on trajectories sampled from the final DeepSeek-GRM, not just the RFT-stage model used for initial data construction. The authors explicitly justify this as necessary to "alleviate the gap between training and inference policy" (Section 4), referencing Chow et al. (2025). This is a conceptually important point: a filter trained only on RFT-stage outputs would be filtering a different distribution than what the final model produces, potentially discarding good samples that the RFT model couldn't generate or retaining samples that the RL model has learned to generate but which happen to look like RFT failures.

The ablation on k_meta (Table 4) reveals a non-obvious property: keeping only the single best sample (k_meta = 1) performs worse than naive voting (71.5% vs. 71.0%), while keeping the top half (k_meta = 16) performs best (72.8%). This means the meta RM is not simply identifying one "correct" evaluation — it's identifying a subset of evaluations that collectively produce a better aggregate than any individual. The meta RM's value is not in finding the single best evaluation, but in identifying which evaluations are safe to include in an ensemble. This reframes the problem from "select the best evaluator" to "curate a set of evaluators whose errors are uncorrelated enough that aggregation cancels them out."

This connects to broader themes in ensemble methods and uncertainty quantification, but applies them to a setting — generative evaluation — where the "base learners" are different principles and analyses applied to the same responses. The meta RM is learning something like "this evaluation used a reasonable set of criteria and applied them consistently," which correlates with correctness without requiring the meta RM to itself judge the responses — it only judges the judging.

Innovation 5: A Clear Empirical Characterization of Where Inference-Time Scaling of Reward Models Helps and Where It Doesn't

The paper provides unusually specific boundary conditions for its claims, and this specificity is itself a contribution. Rather than claiming universal improvement, the paper characterizes:

Where inference-time scaling helps most: General-domain preference judgments (Reward Bench Chat Hard, PPE Preference), where the improvement from k=1 to k=32 with meta RM is substantial (Table 8: Chat Hard goes from 78.3% to 85.7% with k=8 meta RM; overall Reward Bench from 86.0% to 90.4% with k=32 meta RM).
Where it helps modestly: Verifiable correctness tasks (PPE Correctness), where the improvement is present but limited (59.8% → 63.2% with k=32 meta RM), and scalar RMs retain competitive or superior performance.
Where training-time scaling dominates: The performance comparison across model sizes (Figure 4) shows that inference-time scaling can match or exceed model size scaling on Reward Bench, but the pattern is not universal — on some benchmarks (PPE Correctness, certain RMB subsets), larger scalar models maintain advantages.
Where the approach fundamentally underperforms: The failure mode analysis (Appendix F.2) identifies specific categories — complex reasoning requiring expert knowledge, pattern matching, counting — where DeepSeek-GRM's generated principles lead to incorrect critiques even after inference-time scaling. The paper also notes that DeepSeek-R1's much longer chain-of-thought reasoning achieves the highest Reasoning subset score (95.6% on Reward Bench Reasoning, Table 8), suggesting that explicit multi-step reasoning may be necessary for certain evaluation tasks in ways that principle-guided critique alone cannot match.

The significance of this characterization is that it provides a more honest and practically useful picture than a simple "we beat baselines" claim. It tells practitioners: if your evaluation domain is general chat preference, inference-time scaling of a well-trained GRM is highly effective; if it's math verification, stick with scalar models or add reference-based evaluation (Table 12 shows DeepSeek-GRM achieves 91.6% on PPE Correctness when given the ground truth answer as reference); if it's complex reasoning evaluation, expect limitations. This kind of boundary-drawing is rare in methods papers and represents a conceptual contribution to understanding when generative reward modeling with inference-time scaling is the right tool.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on four reward modeling benchmarks spanning different domains and response formats: Reward Bench (Lambert et al., 2024) — a widely used benchmark with Chat, Chat Hard, Safety, and Reasoning subsets, each containing paired responses requiring preference ranking; PPE (Frick et al., 2025) — a large-scale benchmark split into Preference (crowdsourced pairwise data) and Correctness (verifiable tasks with known right answers); RMB (Zhou et al., 2025) — a comprehensive benchmark with Helpfulness and Harmlessness subsets, each containing both pairwise and best-of-N (BoN) multiple-response data; and ReaLMistake (Kamoi et al., 2024) — a benchmark for evaluating single-response error detection, using ROC-AUC as the metric. For Reward Bench, the "Prior Sets" (Bai et al., 2022a; Askell et al., 2021; Ethayarajh et al., 2022; Stiennon et al., 2020) are excluded from overall score calculations. Dataset sizes and splits are not explicitly stated for all benchmarks, but the Reward Bench downsampled subset for DeepSeek-R1 testing contains 300 samples, and PPE is described as "large-scale."
Base model(s). The primary experiments use Gemma-2-27B (Team, 2024) as the base pretrained model for all 27B-scale DeepSeek-GRM variants and for all re-implemented baselines (LLM-as-a-Judge, DeepSeek-BTRM-27B, CLoud-Gemma-2-27B, DeepSeek-PairRM-27B). For scaling experiments, additional base models include DeepSeek-V2-Lite (16B MoE) (DeepSeek-AI, 2024a), DeepSeek-V2.5 (236B MoE), and DeepSeek-V3 (671B MoE) (DeepSeek-AI, 2024b). The meta RM is also trained on Gemma-2-27B. Gemma-2-27B is chosen as the primary base for its "representative" capabilities and practical training cost, while the DeepSeek MoE models provide training-time scaling comparison points. All DeepSeek-GRM models are trained from the pretrained (not instruction-tuned) version of each base LLM — instruction-following capability is acquired entirely through SPCT training.
Metrics. The primary metric is accuracy of selecting the best response from a candidate set, used for Reward Bench, all PPE subsets, and all RMB subsets. For ReaLMistake (single-response evaluation), the metric is ROC-AUC. For RMB BoN subsets, the default evaluation protocol requires correctly identifying the best response among n responses; baseline methods evaluate (n-1) pairwise comparisons (each pair contains the best response and one other), while DeepSeek-GRM directly scores all n responses and selects the highest-scored one via argmax. When predicted scores produce ties between responses, responses are shuffled and the best is determined by arg max_i S_i, where S_i is the predicted score for the i-th response after shuffling. "Overall" scores are computed as the average accuracy across Reward Bench (excluding Prior Sets), PPE Preference, PPE Correctness, and RMB, using the standard metrics reported by each benchmark.
Baselines. Four re-implemented baselines, all based on Gemma-2-27B with the same compatible training data and settings as DeepSeek-GRM: LLM-as-a-Judge (Zheng et al., 2023) — generative pairwise evaluation with the same RFT and RL training protocol, but restricted to pairwise data in the RL stage; DeepSeek-BTRM-27B — a Bradley-Terry scalar RM (Kendall & Smith, 1940), trained with pairwise preference data and outputting scalar pointwise rewards; CLoud-Gemma-2-27B (Ankner et al., 2024) — a semi-scalar RM generating pointwise critiques with a separate value head for scalar reward extraction, fine-tuned on the same general instruction data and sampled critiques but without rejective sampling (since no rewards can be extracted without the trained value head); DeepSeek-PairRM-27B (Jiang et al., 2023) — a pairwise scalar RM comparing response pairs via the sign of the scalar reward. Additionally, publicly reported results are cited for Nemotron-4-340B-Reward, GPT-4o, Gemini-1.5-Pro, Claude-3.5-sonnet, LLaMA-3.1-70B-Instruct, ArmoRM-8B-v0.1, Skywork-Reward-Gemma-2-27B, and InternLM2-20B-Reward, though these use different base models, training data, and evaluation protocols (the versions of GPT-4o differ slightly across benchmarks: GPT-4o-2024-08-06 for Reward Bench and PPE, GPT-4o-2024-05-13 for RMB).
Generation budget / compute accounting. The unit of inference compute is the number of sampled rewards (k) — each sample consists of one complete forward pass producing principles, critique, and scores for all responses. Greedy decoding (k=1, temperature=0) represents the minimum compute baseline. Inference-time scaling experiments use temperature=0.5, with k swept across powers of 2: 1, 2, 4, 8, 16, 32. For meta RM guided voting, the effective budget is k (samples generated) + the meta RM's forward passes over k samples, with k_meta = k/2 (default) determining how many pass through to voting. For baseline voting comparisons: LLM-as-a-Judge uses majority voting over discrete best-response indices; LLM-as-a-Judge w/ TokenProb averages the generation probability of the preference-indicating token; CLoud-Gemma-2-27B averages scalar scores. For the training-time scaling comparison (Figure 4), compute is measured by model size (parameter count) at fixed greedy decoding, not by FLOPs — the paper compares DeepSeek-GRM-27B with inference-time scaling against larger models at greedy decoding without a FLOPs-matched accounting (unlike the Snell et al., 2025 framework).
Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The paper uses fixed train/test splits as provided by each benchmark. For the DeepSeek-R1 evaluation on Reward Bench, a downsampled subset of 300 samples is used due to inference cost, and the result (84.9%) is compared against other models evaluated on the same subset, but the paper does not report variance across multiple random downsampling runs. No confidence intervals, standard deviations, or significance tests are reported for any result — all comparisons rely on point estimates. For ablation studies, hyperparameter sweeps (e.g., β for the KL penalty, k_meta for the meta RM) are reported as point estimates.

Main Quantitative Results

Overall Benchmark Performance (Table 2)

Table 2 reports the overall performance of all methods across Reward Bench, PPE Preference, PPE Correctness, and RMB, along with the average across these four benchmarks ("Overall"). Results are reported for greedy decoding (k=1) and, for DeepSeek-GRM, inference-time scaling with Voting@32 and MetaRM (k=32, k_meta = 16).

Greedy decoding results. DeepSeek-GRM-27B (post-RL) achieves 69.9% overall, outperforming all re-implemented baselines at greedy decoding: DeepSeek-PairRM-27B at 69.0%, CLoud-Gemma-2-27B at 68.7%, DeepSeek-BTRM-27B at 68.6%, DeepSeek-GRM-27B-RFT at 68.8%, and LLM-as-a-Judge at 67.8%. Against publicly reported models, DeepSeek-GRM-27B is competitive with Nemotron-4-340B-Reward (70.5%) and trails GPT-4o (71.3%), despite being 12.6× smaller than Nemotron-4-340B and GPT-4o being a proprietary model of undisclosed scale.

Domain-specific patterns. The scalar and semi-scalar baselines exhibit clear domain biases visible in Table 2. DeepSeek-BTRM-27B achieves 68.3% on PPE Correctness — substantially outperforming all generative methods (DeepSeek-GRM-27B: 59.8%, LLM-as-a-Judge: 57.4%) — but drops to 57.9% on RMB. CLoud-Gemma-2-27B shows the reverse pattern: 62.4% on PPE Correctness but only 63.4% on RMB. DeepSeek-PairRM-27B achieves 87.1% on Reward Bench but 58.2% on RMB. In contrast, DeepSeek-GRM-27B's performance is more balanced across benchmarks: 86.0% (Reward Bench), 64.7% (PPE Preference), 59.8% (PPE Correctness), 69.0% (RMB). The paper interprets this as "significantly fewer biases compared to scalar and semi-scalar RMs" (Section 5.2).

Inference-time scaling to 32 samples. With Voting@32, DeepSeek-GRM-27B reaches 71.0% overall, surpassing Nemotron-4-340B-Reward (70.5%) and approaching GPT-4o (71.3%). With meta RM guided voting (MetaRM, k=32, k_meta=16), DeepSeek-GRM-27B achieves 72.8% overall — the highest result in the table, surpassing GPT-4o (71.3%) by 1.5 percentage points. The meta RM provides the largest boost on Reward Bench (86.0% → 88.5% voting → 90.4% MetaRM) and PPE Correctness (59.8% → 60.4% voting → 63.2% MetaRM).

Average rank analysis. The "Avg. Rank (↓)" column in Table 2 computes ranks (1–6, best to worst) across the four benchmarks for the re-implemented methods only (excluding publicly reported models). DeepSeek-GRM-27B and DeepSeek-PairRM-27B tie for the best average rank at 2.75, while DeepSeek-GRM-27B-RFT ranks worst at 4.00. This suggests that while DeepSeek-GRM-27B has the highest overall accuracy, its per-benchmark rank advantage is more modest — its strength lies in avoiding the very low scores that hurt other methods' averages rather than dominating every benchmark.

Inference-Time Scalability Analysis (Table 3, Figure 1)

Table 3 isolates the scaling behavior by showing performance at k=1, k=8, and k=32 (for DeepSeek-GRM only), with the improvement over k=1 shown in parentheses. Figure 1 visualizes the scaling curves across the tested k values for key methods.

DeepSeek-GRM's scaling trajectory. Starting from 67.9% at greedy decoding (k=1), DeepSeek-GRM-27B reaches 70.6% at k=8 (+2.7 points) and 71.0% at k=32 (+3.1 points). The meta RM variant achieves 72.0% at k=8 (with k_meta = 4) and 72.8% at k=32 (with k_meta = 16), representing improvements of +4.1 and +4.9 points respectively over the k=1 baseline. The scaling shows diminishing but persistent returns — the marginal gain from k=8 to k=32 is +0.4 points for naive voting and +0.8 points for meta RM, suggesting the curve has not fully saturated at k=32.

Comparison with alternative voting methods at k=8. The scaling effectiveness varies dramatically by method:

LLM-as-a-Judge (majority voting): 67.0% → 67.6% (+0.6). The near-flat scaling confirms the paper's claim that categorical aggregation of pairwise judgments provides minimal benefit — each sample contributes only a binary best-response index, and majority voting over these indices saturates quickly.
LLM-as-a-Judge w/ TokenProb: 67.0% → 68.1% (+1.1). Better than majority voting because the token probability provides a continuous weight, but still limited by the pairwise format — each sample only compares two responses, so the token probability captures relative preference, not absolute quality.
CLoud-Gemma-2-27B: 68.5% → 68.8% (+0.3). The near-zero gain is the paper's key evidence that semi-scalar RMs fail at inference-time scaling: the critique text varies across samples, but the scalar value head produces nearly identical outputs, so averaging doesn't help. The paper states this explicitly: "the performance increase is limited, mainly due to the lack of variance in scalar reward generation" (Section 5.2).
DeepSeek-GRM-27B-RFT: 67.8% → 69.3% (+1.5). The RFT-only model does benefit from voting, but substantially less than the RL-trained model (+2.7). This is direct evidence that RL improves scaling behavior beyond what supervised training alone achieves — the RFT model generates more uniform outputs (having been trained to produce a single correct answer per input), while the RL model learns to generate diverse, independently-useful evaluations.
DeepSeek-GRM-27B (MetaRM): 67.9% (k=1, same as naive) → 72.0% (+4.1). The meta RM variant shows the largest absolute improvement, confirming that filtering low-quality samples before voting further amplifies the scaling benefit.

Further scaling to k=32. Only DeepSeek-GRM-27B is tested at k=32: naive voting reaches 71.0% (+3.1 over k=1) and MetaRM reaches 72.8% (+4.9 over k=1). The continued improvement from k=8 to k=32, though modest, suggests that the scaling curve has not plateaued — more samples could yield further gains, though the diminishing marginal returns imply practical limits.

Detailed Per-Benchmark Scaling Results (Tables 6, 8, 9, 10)

The appendix tables provide per-benchmark breakdowns that reveal non-uniform scaling behavior.

Reward Bench (Table 8). DeepSeek-GRM-27B at greedy achieves 86.0% overall. The Chat Hard subset shows the largest scaling gain: 78.3% (greedy) → 80.9% (k=8) → 81.8% (k=32) with naive voting, and to 85.7% (k=8 MetaRM). This is notable because Chat Hard is the most challenging subset — it contains "tricky" preference pairs where responses are similar in quality, requiring fine discrimination. The Reasoning subset shows strong RL improvement (79.2% RFT → 83.8% RL) with limited further scaling gain at k=32 (86.9% with voting, 91.0% with MetaRM at k=32). The Safety subset starts high (88.0% greedy) and reaches 90.0% at k=32 voting. DeepSeek-R1-0120 achieves 84.9% overall on the downsampled 300-sample subset (with individual subset scores: Chat 97.1%, Chat Hard 73.7%, Safety 73.3%, Reasoning 95.6%), with remarkably high Reasoning performance (95.6%) but notably low Safety (73.3%), suggesting that long chain-of-thought reasoning helps with structured reasoning tasks but may not transfer to safety-related judgments. DeepSeek-GRM-230B (RFT-only, no RL) achieves 85.3% and DeepSeek-GRM-671B (RFT-only) achieves 88.4%, both surpassed by DeepSeek-GRM-27B with MetaRM at k=8 (89.8%) and k=32 (90.4%).

PPE Correctness (Table 9). DeepSeek-GRM-27B at greedy achieves 59.8% overall, with individual task scores ranging widely: MMLU-Pro (64.8%), MATH (68.8%), GPQA (55.6%), MBPP-Plus (50.1%), IFEval (59.8%). Voting at k=8 provides negligible improvement (60.3%), but MetaRM at k=8 boosts IFEval dramatically to 69.9% while barely affecting other tasks — the 9.6-point gain on IFEval drives most of the MetaRM improvement on PPE Correctness. DeepSeek-BTRM-27B (scalar) achieves 66.7% — a 6.9-point advantage over DeepSeek-GRM-27B at greedy — confirming the paper's acknowledgment that "scalar RMs capture hidden features of reasoning queries and responses, while GRMs need stronger reasoning capabilities to examine responses thoroughly" (Appendix B). With reference provided (ground truth answer), DeepSeek-GRM-27B achieves 91.6% on PPE Correctness (Table 12), indicating that the GRM's limitation on verifiable tasks is primarily in determining correctness, not in applying principles — given the answer, it judges accurately. The paper positions this as evidence that "both reference-based reward generation and long-horizon reasoning could mitigate this limitation" (Appendix B).

RMB (Table 10). DeepSeek-GRM-27B at greedy achieves 69.0%. Performance differs notably between the two RMB domains: Helpfulness (pairwise) at 80.5% and Harmlessness (pairwise) at 76.1% are much higher than Helpfulness (BoN) at 62.3% and Harmlessness (BoN) at 57.0% — the multiple-response (BoN) setting is substantially harder, requiring the model to identify the best among more than two candidates. Voting at k=8 provides modest gains on the pairwise subsets (Helpfulness pairwise goes from 80.5% to 79.5% — actually a decrease, though this may be noise — and Harmlessness pairwise from 76.1% to 77.1%) and small gains on BoN subsets. MetaRM at k=8 actually performs worse than naive voting on Harmlessness BoN (56.8% vs. 57.6%) and Helpfulness pairwise (80.5% vs. 79.5%), suggesting the meta RM is not uniformly beneficial and may sometimes filter out valid samples. DeepSeek-BTRM-27B performs very poorly on RMB (57.9% overall), with especially low scores on Harmlessness subsets (33.6% BoN, 51.0% pairwise) — this represents the domain bias problem the paper attributes to scalar RMs.

Training-Time vs. Inference-Time Scaling (Figure 4)

Figure 4 compares scaling behavior along two axes: inference-time scaling (Figure 4a) and training-time scaling (Figure 4b), both evaluated on the Reward Bench benchmark.

Inference-time scaling (Figure 4a). DeepSeek-GRM-27B with voting scales from approximately 85.2% at k=1 to 88.5% at k=32, while with MetaRM (k_meta = k/2) it scales from 85.2% to 90.4%. DeepSeek-GRM-16B with voting scales from approximately 82.5% to 84.2% — the smaller model benefits less from additional samples, suggesting a minimum base capability threshold for effective inference-time scaling. The gap between 27B and 16B widens with more samples, indicating that model capacity and inference-time compute are complementary rather than substitutable at these scales.

Training-time scaling (Figure 4b). Greedy decoding performance increases with model size: DeepSeek-GRM-16B ≈ 82.9%, DeepSeek-GRM-27B ≈ 86.0%, DeepSeek-GRM-230B (RFT-only) ≈ 85.3%, DeepSeek-GRM-671B (RFT-only) ≈ 88.4%. The 230B and 671B models are only RFT-trained, not RL-trained, so they represent a lower bound on what fully SPCT-trained larger models could achieve. DeepSeek-R1-0120 achieves 84.9% on the downsampled test set — worse than DeepSeek-GRM-230B (RFT) at 85.3%, suggesting that "expanding long chain-of-thoughts for reasoning tasks could not significantly improve the performance of generalist RM" (Section 5.2). This is a notable negative result: the same RL approach that dramatically improved reasoning for policy models (DeepSeek-R1) does not transfer gains to generalist reward modeling, at least not on Reward Bench overall.

Crossing point. DeepSeek-GRM-27B with MetaRM at k=8 achieves 89.8%, matching or exceeding the 671B MoE model at greedy (88.4%). With MetaRM at k=32 (90.4%), it clearly exceeds the largest model. This demonstrates that, on Reward Bench, inference-time scaling can substitute for approximately 25× more parameters (27B → 671B) when the larger model receives no additional inference compute. However, this comparison is not FLOPs-matched — the paper acknowledges this implicitly by reporting only parameter counts, not total FLOPs consumed during training or inference.

Ablation Studies and Robustness Checks

Principle generation: Removing principle generation from DeepSeek-GRM-27B drops greedy accuracy from 69.9% to 67.5% and reduces inference-time scaling gain at k=8 from +2.7 to +0.5 (Table 4, "w/o Principle Generation"). The greedy degradation (-2.4 points) and the near-elimination of scaling benefit together demonstrate that self-generated principles are not merely an interpretability feature — they are the mechanism that creates useful output diversity. Without principles, the model presumably generates flat critiques that, while textually varied, lack the structural diversity (different evaluation frameworks, different criteria weightings) that makes voting effective.

Rejective sampling (RFT cold start): Removing rejective sampling from DeepSeek-GRM-27B (i.e., starting from a general-instruction-tuned model and applying RL directly) drops greedy accuracy from 69.9% to 68.7% (Table 4, "w/o Rejective Sampling"). The small degradation (-1.2 points) is notable — RL can largely compensate for the absence of curated RFT data, supporting the claim that online RL with binary accuracy rewards is a powerful training signal even without careful cold-start data construction. However, the paper does not report inference-time scaling performance for this ablation, so it is unknown whether the scaling behavior (the +2.7 gain at k=8) depends on RFT.

RFT data components: Further decomposing the RFT stage (Table 4, bottom): removing hinted sampling drops RFT greedy accuracy from 68.8% to 68.0% (-0.8); removing non-hinted sampling drops it to 67.4% (-1.4); removing both (keeping only general instruction data) drops it to 66.1% (-2.7). Non-hinted sampling is more important than hinted sampling, consistent with the paper's claim that "hinted sampled trajectories sometimes take shortcuts in the generated critique" (Section 3.2) — hinted data teaches the model to justify known answers rather than to independently evaluate responses. Removing general instruction data from RFT causes a catastrophic drop to 63.3% (Table 7), confirming prior work (Cao et al., 2024) that broad instruction-following capability is essential for GRM performance.

Meta RM k_meta: At k=32 with the meta RM, varying k_meta (the number of top-scored samples retained before voting) produces: k_meta = 1 (keep only the single best sample): 71.5% overall; k_meta = 8: 72.7%; k_meta = 16 (default, half): 72.8% (Table 4, bottom). The large jump from k_meta=1 to k_meta=8 (+1.2 points) indicates that the meta RM's top-ranked sample is substantially less reliable than an ensemble of several highly-ranked samples. The tiny gain from k_meta=8 to k_meta=16 (+0.1) suggests that samples ranked roughly 9–16 by the meta RM add little marginal value — they are neither harmful enough to exclude nor beneficial enough to change the outcome. This is encouraging for practical deployment, as it means k_meta = 8 provides nearly all the benefit at half the voting cost.

Training data generalization (Table 15): Removing all MATH training data from the RL stage and re-training DeepSeek-GRM-27B causes a modest drop on Reward Bench overall (86.0% → 83.0%, -3.0) but a severe drop on the Chat Hard subset (78.3% → 70.4%, -7.9). The MATH data — which contains reasoning-heavy preference pairs — appears to provide transferable benefits to general chat evaluation, particularly for difficult discrimination tasks. The effect is even more pronounced for the smaller model: DeepSeek-GRM-16B drops from 82.9% to 77.4% (-5.5) overall and from 74.3% to 63.4% (-10.9) on Chat Hard when MATH data is removed. This suggests that reasoning-intensive training data improves generalist RM capability, consistent with the paper's observation that reasoning ability is a bottleneck for GRM performance on complex evaluation tasks.

Response input format (Table 11): On the RMB BoN benchmarks (where each query has n > 2 responses), DeepSeek-GRM-27B shows negligible sensitivity to whether responses are presented in pairs (the default protocol: evaluate n-1 pairs, each containing the best response and one other) or as a single list of all n responses evaluated simultaneously. The performance difference is 0.2% on Helpfulness BoN and 0.5% on Harmlessness BoN — essentially noise-level. This confirms the paper's claim about input flexibility: the pointwise GRM format genuinely handles any number of responses without format switching or performance degradation.

Response length analysis (Figure 7): Comparing DeepSeek-GRM-27B-RFT and DeepSeek-GRM-27B (post-RL) on Reward Bench subsets, the RL stage increases response length on Reasoning (218 → 376 tokens, +72%) and Chat Hard (259 → 265 tokens, +2%), while actually decreasing length on Safety (245 → 260 tokens for RFT → 259 tokens for RL, -0.4%). The Chat subset remains essentially unchanged (241 → 259 tokens). This adaptive allocation — spending more tokens on tasks that benefit from deeper analysis — is an emergent property of the RL training, which rewards correct outputs regardless of length. DeepSeek-R1-0120 uses dramatically more tokens (1690 on Chat, 4405 on Chat Hard, 4210 on Safety, 5224 on Reasoning) but achieves lower overall accuracy (84.9% vs. 86.0% for DeepSeek-GRM-27B greedy on the same subset, Table 8), demonstrating that raw token expansion without SPCT-style training does not convert to better generalist reward modeling.

Transferability of generated principles (Table 14): Principles generated by DeepSeek-GRM-27B and then used as fixed evaluation criteria by other models improve their performance: GPT-4o-2024-08-06 with DeepSeek-GRM-generated principles achieves 78.1% on Chat Hard (vs. 76.1% with self-generated principles and 77.8% with filtered GPT-4o principles), and 58.3% on IFEval (vs. 56.0% self-generated and 57.5% filtered). DeepSeek-GRM-27B with its own generated principles achieves 78.3% (Chat Hard) and 59.8% (IFEval). With filtered principles, DeepSeek-GRM-27B's performance drops slightly to 77.0% and 58.5% — providing external "correct" principles partially overrides the model's adaptive principle generation and slightly hurts performance. This validates that the principles learned by SPCT are genuinely high-quality and generalize beyond the model that generated them, but that the adaptive principle selection (choosing principles based on the specific query) is itself part of what makes DeepSeek-GRM effective.

ReaLMistake single-response evaluation (Table 13): DeepSeek-GRM-27B achieves 72.2% ROC-AUC on ReaLMistake, outperforming Gemma-2-27B-it (65.8%) and DeepSeek-V2-Lite-Chat (61.9%), and matching DeepSeek-V2.5-0905 (69.4%). With voting at k=8, DeepSeek-GRM-27B reaches 74.4%, approaching GPT-4o-2024-08-06 (74.3%). DeepSeek-GRM-16B achieves 64.9%, and DeepSeek-BTRM-27B achieves 69.3%. This confirms the input flexibility claim: the same model architecture, trained with the same protocol, evaluates single responses (error detection) and multi-response comparisons (preference ranking) without task-specific adaptation.

Critical Assessment

Claim: "SPCT significantly improves the quality and inference-time scalability of GRMs over existing methods and several strong public models." The evidence for improved quality (greedy decoding) is solid but modest: DeepSeek-GRM-27B (69.9%) beats re-implemented baselines by 0.9–2.1 points (Table 2) — a real but not dramatic improvement. The evidence for improved scalability is stronger and more distinctive: the +2.7 gain at k=8 is 1.8–9.0× larger than alternative methods' scaling gains at the same budget (Table 3). The claim against "strong public models" is qualified: DeepSeek-GRM-27B with MetaRM at k=32 (72.8%) surpasses GPT-4o (71.3%), but GPT-4o is evaluated at greedy decoding without inference-time scaling — the paper does not test whether GPT-4o (which can also generate principle-guided evaluations) would similarly benefit from sampling and voting. This is a missing comparison: running GPT-4o with the same voting protocol would reveal whether SPCT's advantage is in training methodology or simply in the application of inference-time scaling to any capable generative model. If GPT-4o also gains 2–3 points with voting, then SPCT's contribution narrows to making this capability available in a 27B open-source model rather than a proprietary API, which is significant but a different claim.

Claim: "Inference-time scaling could outperform model size scaling in training time" (Figure 4). The evidence supports this for the specific comparison shown (27B + inference-time scaling vs. 671B greedy), but the comparison has three significant weaknesses. First, the 671B model is RFT-only, not RL-trained — the paper acknowledges this in Appendix C.1: "Due to resource constraints, DeepSeek-GRM models larger than 27B does not undergo the rule-based RL and only trained with 50K rejective sampled data." The 671B model therefore represents a lower bound; an RL-trained 671B GRM might perform substantially better, potentially reversing the crossover. Second, the comparison is not FLOPs-matched — 32 samples from a 27B model and 1 sample from a 671B model consume very different amounts of total compute, and the paper does not attempt to normalize for this. At 27B parameters with 32 samples at roughly 260 tokens per sample (Figure 7 average across subsets), the inference FLOPs are approximately 27B × 32 × 260 × 2 ≈ 450T FLOPs; the 671B model at 1 sample of similar length uses 671B × 1 × 260 × 2 ≈ 349T FLOPs. These are within roughly 30% of each other on this metric, but the paper does not perform this accounting, making the comparison suggestive rather than rigorous. Third, Figure 4b shows only Reward Bench — the scaling patterns on other benchmarks (especially PPE Correctness where scalar models dominate) are not shown, so the "outperform" claim may not generalize.

Claim: "DeepSeek-GRM shows fewer biases on different domains" (compared to scalar/semi-scalar RMs). The evidence in Table 2 is suggestive but not systematic. DeepSeek-GRM-27B's performance ranges from 59.8% (PPE Correctness) to 86.0% (Reward Bench), a gap of 26.2 points. DeepSeek-BTRM-27B ranges from 57.9% (RMB) to 81.7% (Reward Bench), a gap of 23.8 points — actually less variable. DeepSeek-PairRM-27B ranges from 58.2% (RMB) to 87.1% (Reward Bench), a gap of 28.9 points — more variable. The "fewer biases" claim appears to be based on DeepSeek-GRM having no catastrophic failures (nothing below 59.8%), whereas scalar models have at least one very low score (DeepSeek-BTRM-27B at 57.9% on RMB, CLoud at 63.2% on RMB). This is a high-minimum rather than low-variance property, and the paper does not quantify "bias" formally (e.g., as the coefficient of variation or entropy across benchmarks). A more precise statement would be: "DeepSeek-GRM avoids the severe domain-specific failures observed in scalar and semi-scalar RMs, though it still underperforms on verifiable correctness tasks."

Missing baseline: GPT-4o or other strong generative models with the same voting protocol. This is the most significant missing experiment. The paper demonstrates that SPCT enables inference-time scaling for a 27B model, but it does not test whether larger, more capable generative models (GPT-4o, Claude-3.5-sonnet, DeepSeek-V3) would also benefit from sampling + voting when prompted to generate principles and pointwise scores. If they do, then SPCT's primary contribution is bringing this capability to smaller open-source models; if they don't (because they weren't explicitly trained for scalable behavior), then SPCT's training methodology is genuinely necessary for scalable reward generation. The preliminary experiment (Table 1) hints that GPT-4o with filtered principles can improve, but does not test voting over multiple GPT-4o samples.

Missing ablation: The effect of RL without rejective sampling on inference-time scaling. Table 4 reports that RL without RFT rejective sampling achieves 68.7% at greedy (close to 69.9% with RFT), but the paper does not report the inference-time scaling performance of this ablated model. If the RFT-less model also scales well with sampling, it would weaken the claim that RFT is an essential part of the SPCT recipe for scalability. If it scales poorly, it would strengthen the paper's narrative that RFT provides a necessary cold start for scalable behavior.

Missing analysis: Per-benchmark scaling curves. The paper provides detailed per-benchmark results in appendix tables, but only Figure 1 and Figure 4a show scaling curves and only for Reward Bench and overall average. Scaling curves for individual benchmarks — particularly PPE Correctness (where the gains are small) and RMB (where the meta RM sometimes hurts) — would reveal whether scaling is uniformly beneficial or whether there are benchmarks where additional compute provides zero or negative return. The per-benchmark point estimates in Tables 9 and 10 show some non-monotonic behavior (e.g., DeepSeek-GRM-27B voting at k=32 scores 65.5% on MMLU-Pro vs. 65.7% at k=8, Table 9), suggesting that scaling is not strictly monotonic on all tasks, but without curves it's unclear whether these are noise or genuine patterns.

Small n on some analyses. The failure mode analysis (Appendix F.2) examines only 10 incorrect samples per benchmark (40 total). While informative for qualitative understanding, this sample is too small to draw distributional conclusions about failure modes. The DeepSeek-R1 evaluation uses only 300 samples from Reward Bench, but the full Reward Bench test set size is not specified in the paper, making it difficult to assess how representative this downsampled evaluation is. The ReaLMistake benchmark is included in the main results but its sample size is not reported.

No statistical significance reporting. All results are point estimates without confidence intervals, standard deviations, or significance tests. For a test set of, say, 500 samples (the size of Reward Bench is not explicitly stated but is standard), a 0.5 percentage point difference (e.g., 71.0% vs. 70.5%) represents roughly 2.5 samples — well within the range of sampling noise. The paper's strongest claims (+2.7 gain at k=8, +3.1 at k=32) likely exceed reasonable noise thresholds for reasonably-sized test sets, but the exact reliability is unknown. The two-fold cross-validation protocol used in the referenced Snell et al. (2025) paper is notably absent here — there is no attempt to estimate variance or to perform strategy selection on held-out data.

Single training run. All results for DeepSeek-GRM-27B appear to be from a single training run. The hyperparameter sweep for β (the KL coefficient) found β = 0.08 optimal for stability, but it's unclear whether the reported results are from the best-performing checkpoint or from a fixed training duration (900 steps). Without multiple runs or checkpoint selection details, the reproducibility and variance of the reported numbers are unknown.

Latency not addressed. The paper acknowledges the efficiency limitation of GRMs ("largely lagging behind the scalar RMs at the same scale by nature") and claims that "the latency of reward generation with a reasonable amount of, e.g., eight samplings will not increase significantly" due to parallel sampling. However, no latency measurements are reported. Eight parallel generations from a 27B model, even batched, incur non-trivial GPU memory and time costs — and the meta RM adds k additional forward passes on top. The claim that latency doesn't "increase significantly" is unsubstantiated and depends heavily on hardware configuration. This is a practical weakness: if DeepSeek-GRM-27B with k=8 takes 8× the wall-clock time of a scalar RM, the accuracy advantages must be weighed against deployment feasibility.

Benchmark coverage. All four benchmarks focus on preference ranking and correctness verification of LLM responses. Missing are evaluations of: instruction-following quality beyond binary correctness (IFEval is included but only as a subset of PPE Correctness), safety evaluation beyond preference pairs (the RMB Harmlessness subset uses pairwise annotations, not open-ended safety judgments), multi-turn conversation quality, factual accuracy beyond verifiable tasks, and creativity/open-ended generation quality. The paper's claim to "generalist" reward modeling must be understood within the scope of these specific benchmark distributions.

The meta RM's domain specificity. The meta RM is trained on trajectories from DeepSeek-GRM-27B and evaluated on the same model's outputs. Its effectiveness when applied to outputs from a different GRM (e.g., a larger DeepSeek-GRM, or a GPT-4o judge) is not tested. If the meta RM learns model-specific failure patterns rather than general principles of judgment quality, it would not transfer — limiting its utility as a general-purpose filtering mechanism.

6. Limitations and Trade-offs

6.1 The Claims to "Inference-Time Scalability" Rest on a Single Model Family and a Narrow Set of Benchmarks

The assumption or constraint. All experiments demonstrating SPCT's effectiveness use Gemma-2-27B as the primary base model (with additional scaling experiments on DeepSeek-V2-Lite, DeepSeek-V2.5, and DeepSeek-V3 — all from the DeepSeek family). The paper states it "believe[s] this model is representative of the capabilities of many contemporary LLMs" (Section 4), but provides no evidence from other model families (e.g., LLaMA, Qwen, Mistral). The evaluation uses four benchmarks — Reward Bench, PPE, RMB, and ReaLMistake — all of which test preference ranking or correctness verification of LLM responses. Missing are evaluations of safety beyond pairwise comparisons, multi-turn conversation quality, factual accuracy outside verifiable tasks, creativity, and open-ended generation evaluation.

The consequence. It is unknown whether SPCT's core finding — that rule-based online RL on self-generated principles creates inference-time scalable behavior — transfers to other base model architectures, training distributions, or evaluation domains. The paper's preliminary experiment (Table 1) showed that GPT-4o's self-generated principles degraded performance (76.1% → 75.9%), while filtered principles helped (77.8%). This hints that model capability level may interact with whether principle generation is beneficial, but the paper never tests SPCT on models with substantially different pretraining or capability profiles. A practitioner adopting SPCT for a non-DeepSeek/Gemma model family cannot predict whether the +2.7 scaling gain at k=8 will replicate, or whether the method may underperform relative to simpler alternatives.

What evidence exists in the paper. The evidence for model-family specificity is entirely indirect. Table 2 shows that publicly reported models (Nemotron-4-340B-Reward, GPT-4o, Gemini-1.5-Pro) achieve competitive results with different architectures and training procedures, but none of them are tested with SPCT, so we cannot distinguish whether SPCT's advantage comes from the training method or from some interaction with the Gemma-2-27B base model. The training-time scaling results (Figure 4b) use only DeepSeek-family MoE models for larger scales, leaving open the question of whether a dense 70B model from another family would show different scaling behavior. The paper acknowledges that "due to resource constraints, DeepSeek-GRM models larger than 27B does not undergo the rule-based RL and only trained with 50K rejective sampled data" (Appendix C.1), meaning the scaling comparison is further confounded by incomplete training of larger models.

Mitigation status. Not addressed. The paper does not train SPCT on any model outside the DeepSeek/Gemma families, does not test on evaluation domains beyond preference/correctness benchmarks, and does not discuss model-family or domain generalization as an explicit limitation.

6.2 The Difficulty Estimation Cost for Scalable Behaviors Is Unaccounted for in the Headline Numbers

The assumption or constraint. The SPCT pipeline requires, for the RL stage, a training set of 237K examples, each containing a query, responses, and a ground-truth preference label. These labels come from human annotation or from automated verifiers for constrained tasks (e.g., MATH answer checking). The paper assumes access to this labeled data and does not account for its cost in any budget calculation. More critically, the RL stage's reward signal (Equation 11) is a binary +1/-1 based on whether the model correctly identifies the ground-truth best response — a signal that is only available when ground-truth labels exist. The meta RM, similarly, requires binary correctness labels for training (Section 4).

The consequence. The headline results (72.8% overall with MetaRM at k=32) represent a scenario where a practitioner has already invested in curating a large, high-quality preference dataset with ground-truth labels to train SPCT. For domains where such labels are expensive or unavailable — which is precisely the "generalist" setting the paper targets, where "there are often no explicit reference or ground truth" (Section 1) — the full SPCT pipeline cannot be replicated. The paper's framing as a solution for "general domains" where rewards are "more diverse and complex" with "no explicit reference or ground truth" (Section 1) is in tension with the training procedure's reliance on ground-truth preference labels. The claim that "the reward signal could be obtained seamlessly from any preference dataset and labeled LLM responses" (Section 3.2) is true but masks the fact that such datasets are themselves the product of substantial human annotation investment.

What evidence exists in the paper. The training data description (Appendix C.1) lists the datasets used: UltraFeedback, OffsetBias, Skywork-Reward-Preference-80K-v0.2, HelpSteer2-Preference, and MATH. These are all human-annotated or rule-verifiable datasets. The paper notes that for UltraFeedback, they "re-tagged the preference label of a part of UltraFeedback due to its quality issues," indicating that even existing datasets required additional curation effort. The ablation on removing MATH data (Table 15) shows a 3.0-point drop on Reward Bench overall and a 7.9-point drop on Chat Hard, demonstrating that performance is sensitive to the inclusion of specific high-quality labeled datasets. No experiment tests SPCT with reduced labeling budgets, synthetic labels, or weak supervision.

Mitigation status. Not addressed. The paper does not discuss the labeling cost as a limitation, does not experiment with label-efficient variants of SPCT, and does not estimate how much labeled data is minimally necessary for effective training. The future work section mentions using GRMs "as versatile interfaces of reward systems" (Section 7) but does not propose reducing annotation dependency.

6.3 DeepSeek-GRM Still Underperforms Scalar Models on Verifiable Correctness Tasks, Even After Inference-Time Scaling

The assumption or constraint. The paper positions pointwise GRMs as solving the "generalist" reward modeling problem, but the empirical results reveal a persistent capability gap on verifiable correctness tasks. On PPE Correctness, DeepSeek-GRM-27B achieves 59.8% at greedy (Table 2), compared to DeepSeek-BTRM-27B (scalar Bradley-Terry model) at 68.3% — an 8.5-point deficit. Even with MetaRM at k=32, DeepSeek-GRM-27B reaches only 63.2% (Table 9), still trailing the scalar baseline. The paper explicitly acknowledges this: "In specific domains such as verifiable tasks, DeepSeek-GRM still lags behind scalar models. This could be because the scalar RMs capture hidden features of reasoning queries and responses, while GRMs need stronger reasoning capabilities to examine responses thoroughly" (Appendix B).

The consequence. A practitioner choosing between a scalar RM and DeepSeek-GRM faces a genuine tradeoff that the paper's "generalist" framing somewhat obscures. If their evaluation workload is heavily weighted toward verifiable tasks (math, coding, factual QA with known answers), a well-trained scalar RM will substantially outperform DeepSeek-GRM at both greedy decoding and after inference-time scaling. The paper's solution — using reference-based evaluation (providing the ground-truth answer in the input) — boosts DeepSeek-GRM-27B on PPE Correctness from 59.8% to 91.6% (Table 12). But this transforms the task from "evaluate whether a response is correct" to "verify whether a response matches a known answer," which is a fundamentally easier problem that a simple string-match or unit test could often solve more reliably. The GRM is being used as an expensive substitute for deterministic verification, not as a genuine correctness evaluator.

What evidence exists in the paper. Table 9 provides the per-task breakdown on PPE Correctness. DeepSeek-GRM-27B at greedy achieves 64.8% on MMLU-Pro, 68.8% on MATH, 55.6% on GPQA, 50.1% on MBPP-Plus, and 59.8% on IFEval. The meta RM provides a large boost only on IFEval (59.8% → 69.9% at k=8 MetaRM), with minimal gains on other correctness tasks. The paper's failure mode analysis (Appendix F.2, Figure 8) identifies "Incorrect Critiques" as the dominant failure category on all benchmarks (7/10 on PPE Correctness), suggesting the model frequently produces plausible-sounding but wrong evaluations. DeepSeek-R1, despite its strong reasoning performance (95.6% on Reward Bench Reasoning subset, Table 8), achieves only 84.9% overall on Reward Bench — lower than the 27B GRM — and is not tested on PPE Correctness. This suggests that even long chain-of-thought reasoning, without SPCT-style principle-guided training, does not close the verifiable-task gap.

Mitigation status. Partially addressed. The paper explicitly acknowledges the limitation (Appendix B) and proposes two mitigation strategies: reference-based evaluation (shown to work, but at the cost of reducing the task's difficulty) and long-horizon reasoning (shown via DeepSeek-R1's Reasoning subset score, but not combined with SPCT). Neither mitigation is integrated into the main DeepSeek-GRM pipeline. The paper frames this as a direction for future work rather than a solved problem: "For GRMs, we found that both reference-based reward generation and long-horizon reasoning could mitigate this limitation" (Appendix B).

6.4 The Comparison Against Larger Models Is Not FLOPs-Matched, Undermining the "Inference-Time > Training-Time Scaling" Claim

The assumption or constraint. Figure 4 and the associated claims — "inference-time scaling could outperform model size scaling in training time" — compare DeepSeek-GRM-27B with k=32 sampling against models up to 671B parameters at greedy decoding (k=1). The paper does not perform a FLOPs-matched comparison in the style of Snell et al. (2025), where the total inference compute budget is held constant across model sizes. The comparison also uses RFT-only (not RL-trained) larger models, which the paper acknowledges in Appendix C.1: "Due to resource constraints, DeepSeek-GRM models larger than 27B does not undergo the rule-based RL and only trained with 50K rejective sampled data."

The consequence. The claim that inference-time scaling "outperforms" training-time scaling overstates what the evidence supports. What the experiments actually show is: (1) DeepSeek-GRM-27B with k=32 sampling outperforms a 671B MoE model at greedy decoding on Reward Bench specifically (90.4% vs. 88.4%), (2) when the 671B model is trained only with RFT (a weaker training procedure than the 27B model receives). This is a triple confound: different model sizes, different training procedures, and different inference budgets. A fair comparison would require either: (a) giving the larger model an equivalent inference compute budget (e.g., the 671B model at k=1 might use similar FLOPs to the 27B model at k=25, not k=32), or (b) giving the larger model the same RL training as the 27B model. Without either control, we cannot determine whether inference-time scaling is truly more efficient, or whether the 27B model simply received better training (RL vs. RFT-only) and more inference compute. The paper's modest scaling from k=8 to k=32 (+0.4 points for naive voting, Table 3) further suggests that the inference-time scaling curve is flattening, while the training-time scaling curve (Figure 4b) might steepen if the larger models were fully RL-trained.

What evidence exists in the paper. Figure 4b shows the training-time scaling data points: DeepSeek-GRM-16B at ~82.9%, 27B at ~86.0%, 230B at ~85.3%, 671B at ~88.4%. The 230B model actually underperforms the 27B model — strong evidence that RFT-only training does not scale monotonically, and that the absence of RL training for larger models is consequential. The paper does not report the number of RFT training steps or data volume for the larger models (only "50K rejective sampled data" is mentioned), making it impossible to assess whether they were undertrained relative to the 27B model (which received 1256K RFT examples plus 237K RL examples). The DeepSeek-R1 result (84.9% on a downsampled Reward Bench subset) further complicates the picture: a model explicitly trained for long-horizon reasoning underperforms the 27B GRM, suggesting that raw reasoning capability does not straightforwardly transfer to reward modeling.

Mitigation status. Partially addressed through transparency. The paper explicitly states the training procedure difference for larger models (Appendix C.1), which is more honest than many comparable papers. However, it does not adjust its claims to reflect this limitation — the abstract and Section 5.2 present the comparison as evidence of inference-time scaling's superiority without the caveat that the larger models were less thoroughly trained. The future work section does not mention FLOPs-matched comparison as a needed experiment.

6.5 The Meta RM Introduces an Additional Training Burden and May Not Generalize Across Model Versions

The assumption or constraint. The meta RM is a separate 27B-parameter model (based on Gemma-2-27B) that must be trained on trajectories sampled from the specific DeepSeek-GRM it will filter at inference time. The paper trains it on a mix of RFT-stage rejective-sampled data (from DeepSeek-V2.5-0905) and on-policy samples from the final DeepSeek-GRM-27B, explicitly to "alleviate the gap between training and inference policy as suggested by Chow et al. (2025)" (Section 4). This means the meta RM is tightly coupled to the specific GRM it evaluates.

The consequence. Every time the GRM is updated (e.g., through further RL training, adaptation to new domains, or fine-tuning on additional data), the meta RM may need to be retrained or fine-tuned to track the new output distribution. This doubles the training cost: the paper reports 15.6 hours for RL training of the GRM (Table 5), and the meta RM requires an additional round of sampling from the updated GRM plus classifier training (learning rate 1 × 10^{-5}, batch size 512, unspecified training duration). For a production pipeline where the GRM is periodically updated, this creates a maintenance burden. More subtly, the meta RM's performance may degrade if the GRM's output distribution drifts in deployment (e.g., due to changes in the types of queries or responses being evaluated, or due to distribution shift in the policy model whose outputs the GRM evaluates). The paper provides no evidence on whether the meta RM's filtering accuracy is stable under distribution shift.

What evidence exists in the paper. The paper's evidence for the meta RM's effectiveness comes entirely from evaluating it on the same benchmark distributions used to train the GRM. There is no out-of-distribution evaluation of the meta RM, no test of whether a meta RM trained on DeepSeek-GRM-27B outputs can effectively filter outputs from DeepSeek-GRM-16B or DeepSeek-GRM-230B, and no ablation on how much on-policy data is needed for effective meta RM training. The k_meta ablation (Table 4) shows that keeping only the top 1 sample (as ranked by meta RM) performs at 71.5% — worse than naive voting at 71.0% — while keeping top 8 reaches 72.7% and top 16 reaches 72.8%. This suggests the meta RM's ranking is imperfect at the very top (its #1 pick is not reliably the single best sample) but becomes useful when aggregating a set. The paper does not explore whether this pattern holds for GRMs at different capability levels or for different base models.

Mitigation status. Partially addressed. The paper's use of on-policy sampling for meta RM training (including trajectories from the final GRM, not just the RFT-stage model) is a deliberate mitigation against train-inference distribution gap, following established practice (Chow et al., 2025). However, the paper does not discuss the ongoing maintenance cost of keeping the meta RM aligned with an evolving GRM, does not test cross-model generalization, and does not propose lighter-weight alternatives (e.g., using the GRM's own confidence estimates or simple heuristics like output length or principle count).

6.6 The Pointwise 1–10 Scoring Rubric Is Arbitrary, and the Model's Calibration Across This Scale Is Not Validated

The assumption or constraint. DeepSeek-GRM outputs integer scores on a 1–10 scale for each response. The paper states that "a score of 1 means the response does not meet the criteria at all, a score of 6 means the response meets only some parts, and a score of 10 means the response perfectly meets the evaluation criteria" (prompt template, Appendix G). The summed voting procedure (Equation 14) treats these scores as cardinal values — a score of 8 contributes exactly twice as much to the aggregate as a score of 4. The paper provides no evidence that the model's scores are calibrated in this cardinal sense: does a score of 8 actually represent twice the quality of a score of 4? Are the intervals between adjacent scores (e.g., 6→7 vs. 9→10) used consistently?

The consequence. The voting procedure assumes that score magnitudes are comparable across samples and that the 1–10 scale is used linearly. If the model uses the scale non-linearly — for example, clustering most scores in the 6–9 range while rarely using 1–3 — then the effective discrimination comes from only a few scale points, and the "expanded reward space" (Section 4) is effectively much coarser than the theoretical [k, 10k] range. More problematically, if different principles lead to systematically different score distributions (e.g., a "safety" principle might produce more extreme scores than a "relevance" principle), then samples emphasizing different principles contribute with different effective weights to the summed aggregate, in ways not controlled by the stated principle weights. The paper's failure mode analysis identifies "Imbalanced Weights of Principles" as a recurring issue (Figure 8, 4/10 cases on Reward Bench, 3/10 on PPE Correctness), indicating that the model does not consistently translate principle importance into proportional score differences.

What evidence exists in the paper. The paper does not conduct any calibration analysis: no per-score-bin frequency distributions, no comparison of score distributions across different principle sets, and no test of whether the linear-sum aggregation is optimal versus alternatives (e.g., rank-based aggregation, weighted averaging by principle importance, or learning aggregation weights). The prompt template provides anchor descriptions only for scores 1, 6, and 10 (Appendix G: "a score of 1 means the response does not meet the criteria at all, a score of 6 means the response meets only some parts, and a score of 10 means the response perfectly meets the evaluation criteria"), leaving scores 2–5 and 7–9 unanchored. Whether the model uses these intermediate scores consistently is unexplored. The case study in Table 17 shows a sample where DeepSeek-GRM outputs "8, 8" for two responses (indicating a tie), another sample outputting "9, 5" (strong preference), and another outputting "10, 7" — the inconsistency in score spread across samples for the same query suggests the scale usage varies substantially.

Mitigation status. Not addressed. The paper treats the 1–10 scale as a given, does not experiment with alternative granularities (e.g., 1–5, 1–100, continuous values), does not test whether score distributions are well-behaved, and does not consider non-linear aggregation methods. The use of integer scores is motivated by practical extraction ease ("the rewards are discrete, and in this work we assign $S_i \in \mathbb{N}, 1 \leq S_i \leq 10$ by default," Section 2.1), not by empirical validation of the scale's properties.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around reward modeling from a static, single-pass evaluation paradigm toward one where reward quality is a function of inference compute, not just training data volume or model size. The conceptual move is subtle but significant: prior work treated reward models as fixed oracles — train them once, then query them greedily. This paper demonstrates that, with the right training methodology, a reward model can be treated as a generative distribution over evaluations, where spending more compute (sampling multiple times and aggregating) produces monotonically better rewards. This is not merely "voting helps" — voting is an old idea. The shift is in recognizing that the ability to benefit from voting is a learned behavior, not an automatic consequence of using a generative architecture.

The evidence for this is the comparison between DeepSeek-GRM-RFT and DeepSeek-GRM (post-RL): same architecture, same base model, same output format, but the RL-trained model gains +2.7 points from 8-sample voting while the RFT-only model gains only +1.5 (Table 3). The architecture could scale in both cases; the training procedure determined whether it did scale. This mirrors the finding in DeepSeek-R1 that long-chain reasoning emerges from RL incentivization, not from prompting alone, and extends it from policy models to evaluator models — inference-time scalability of evaluators is a capability that must be trained for, not assumed.

This reframes how the field should think about reward model development. The dominant paradigm has been: collect more preference data → train a larger scalar RM → deploy at greedy decoding. This paper suggests an alternative: train a generative RM with RL to produce diverse, independently-useful evaluations → deploy with parallel sampling and voting, optionally filtered by a learned meta-evaluator. The two paradigms are not mutually exclusive — the paper's scaling experiments (Figure 4) suggest they are complementary — but the second paradigm opens design space that was previously invisible. Researchers building reward models now have a new knob to tune: not just model capacity and data quality, but training methodology for scalable behavior.

The work also partially reconciles a tension in the prior literature. Scalar RMs dominate on verifiable tasks (Table 2: DeepSeek-BTRM-27B achieves 68.3% on PPE Correctness vs. 59.8% for DeepSeek-GRM-27B), while generative approaches offer flexibility and interpretability. The paper doesn't resolve this tradeoff — scalar models still win on correctness verification — but it provides a framework for understanding why: scalar models capture "hidden features of reasoning queries and responses" (Appendix B) that GRMs, relying on explicit text-based analysis, struggle to replicate. The paper's demonstration that providing a reference answer boosts DeepSeek-GRM on PPE Correctness from 59.8% to 91.6% (Table 12) suggests that the GRM's limitation is not in applying principles to judge, but in independently determining correctness — a capability that scalar models can learn implicitly from training data without needing to articulate their reasoning. This diagnostic — separation of "judgment capability" from "correctness determination capability" — is a useful decomposition that prior work had not made explicit.

The work also redirects research attention in two specific ways. First, it suggests that verifier over-optimization is not the only bottleneck for inference-time scaling of evaluators. In the policy-model scaling literature (Snell et al., 2025), the primary concern is that aggressive search exploits verifier weaknesses. Here, the bottleneck is different: the model's ability to generate usefully diverse evaluations, not just correct evaluations. The meta RM addresses a related but distinct problem — filtering out low-quality samples — rather than preventing adversarial exploitation of the verifier. This opens a new category of research: how to train models so that their output diversity is structured and beneficial rather than noisy or degenerate.

Second, it establishes that inference-time scaling of reward models can substitute for model size scaling, but with clear boundary conditions. Figure 4 shows DeepSeek-GRM-27B with MetaRM at k=8 (89.8% on Reward Bench) surpassing a 671B MoE model at greedy decoding (88.4%). But this result is specific to Reward Bench (a general preference benchmark) and to the comparison against an RFT-only (not RL-trained) larger model. On verifiable tasks, the substitution does not hold: DeepSeek-GRM-27B with k=32 MetaRM achieves only 63.2% on PPE Correctness (Table 9), trailing scalar models at much smaller scales. The practical implication: for general-domain preference evaluation, invest in inference-time scaling of a well-trained GRM; for correctness verification, invest in larger scalar models or provide reference answers. This is a more nuanced and useful takeaway than a blanket "inference-time scaling beats training-time scaling" claim.

Finally, the paper's failure to find benefits from long-horizon chain-of-thought reasoning for generalist RM — DeepSeek-R1 achieves only 84.9% on Reward Bench overall (Table 8), worse than DeepSeek-GRM-230B (RFT-only) at 85.3%, despite using 10–20× more output tokens (Figure 7) — is a notable negative result that should give pause to anyone assuming that "more reasoning" automatically improves evaluation. DeepSeek-R1 does excel on the Reasoning subset (95.6%), but its near-random performance on Safety (73.3%) suggests that long-horizon reasoning trained primarily on math and logic may not transfer to domains requiring value judgments or normative assessments. This cautions against a naive extrapolation of reasoning-model advances to the reward modeling domain without domain-specific training.

Follow-Up Research This Work Enables

SPCT applied to other model families and scales, with controlled RL training of larger models. The paper's most obvious gap is the incomplete training of larger models: DeepSeek-GRM-230B and 671B are RFT-only, not RL-trained, due to resource constraints. A direct follow-up would apply the full SPCT pipeline (RFT cold start + rule-based RL) to models at 70B, 230B, and 671B scales, measuring both greedy accuracy and inference-time scaling gains. The key question: does the +2.7 scaling gain at k=8 observed for the 27B model hold, shrink, or grow at larger scales? If larger models benefit less from voting (because their single-sample accuracy is already higher), the case for inference-time scaling weakens at scale. If they benefit equally or more, the case strengthens. A FLOPs-matched comparison — where a 671B model at k=1 is compared against a 27B model with enough samples to equalize inference FLOPs — would finally provide rigorous evidence for or against the paper's central scaling claim.

Combining SPCT-trained GRMs with process reward modeling for step-by-step verification. The paper notes in Appendix B that "DeepSeek-GRM could potentially serve as a process RM in addition to the outcome RM," and that its performance on the Reasoning subset of Reward Bench (which contains MATH-prm data from Lightman et al., 2024) "could partially support the potential of this application," but this direction is entirely unexplored. A concrete experiment: fine-tune DeepSeek-GRM to generate principle-guided step-level critiques for mathematical reasoning chains, then use voting across multiple sampled step-evaluations to produce a process reward signal. Compare against scalar PRMs on process-level benchmarks like ProcessBench (Zheng et al., 2025). The hypothesis: principle-guided step evaluation with voting might be more interpretable and robust than scalar PRMs, especially for reasoning steps that are partially correct or that have multiple valid approaches. The risk: the efficiency cost of generating full textual critiques per step may be prohibitive for online RL pipelines where millions of step evaluations are needed.

Testing whether SPCT's benefits transfer when the training data's ground-truth labels are synthetic rather than human-annotated. The SPCT pipeline requires ground-truth preference labels for both RFT rejection and RL reward computation (Equation 11). The paper uses human-annotated datasets (UltraFeedback, HelpSteer2, etc.) plus rule-verified MATH data. A practically important question: can SPCT work with labels generated by a strong LLM-as-a-Judge (e.g., GPT-4o or DeepSeek-V3 acting as the "ground truth")? This would dramatically reduce the annotation bottleneck and make SPCT applicable to domains where human labels are scarce. A concrete experiment: train DeepSeek-GRM-27B with SPCT using only GPT-4o-generated preference labels (no human annotations), and measure whether the inference-time scaling gains (+2.7 at k=8) are preserved, reduced, or eliminated. The risk is that synthetic labels introduce systematic biases that SPCT's RL amplifies rather than corrects — the "re-tagging" of UltraFeedback mentioned in Appendix C.1 suggests the authors are already aware of label quality issues in synthetic datasets.

Online RL for the meta RM, closing the loop between evaluation quality estimation and GRM training. The meta RM is currently a static filter: it's trained once and then used to score samples at inference time. A natural extension is to use the meta RM's scores as an additional reward signal during GRM training — not just filtering after the fact, but shaping the GRM's behavior to produce samples that the meta RM will judge as high-quality. This would create a two-player game (GRM generates evaluations, meta RM judges them) with potential for co-adaptation. A concrete experiment: add a meta RM-derived reward term to the GRPO objective (Equation 15), weighted by a small coefficient, and measure whether it improves scaling behavior beyond what the binary accuracy reward alone achieves. The risk: this could lead to reward hacking where the GRM learns to produce evaluations that look good to the meta RM without being actually correct — a known failure mode in RLHF that would need careful monitoring.

Using SPCT-trained GRMs to evaluate policy model outputs during online RL, closing the loop. The paper positions DeepSeek-GRM as a generalist reward model that could serve in RL pipelines, but never actually tests it in that role. A concrete experiment: use DeepSeek-GRM-27B with k=8 voting as the reward signal for GRPO training of a policy model (e.g., on a general chat benchmark like AlpacaEval or MT-Bench), and compare policy quality against using a scalar RM (e.g., DeepSeek-BTRM-27B) at equivalent inference cost. The key metric: does the GRM's inference-time scalability translate to better policy outcomes, or do the GRM's remaining biases and errors propagate in ways that negate the benefit? The paper's one-sentence mention of "inference-time co-scaling with policy models" (Section 7) suggests the authors see this as important but unexplored.

Ablation on the necessity of the "principle" structure for inference-time scaling. The paper claims that self-generated principles are the mechanism that creates useful output diversity. An alternative hypothesis: any structured output format that forces the model to decompose its evaluation into sub-components would produce similar scaling benefits, and "principles" are just one possible decomposition. A concrete experiment: train a GRM to output scores broken down by fixed, pre-defined criteria (e.g., always "Instruction Adherence, Usefulness, Level of Detail, Relevance") rather than self-generated principles, apply the same RL training, and measure inference-time scaling. If the fixed-criteria model scales equally well, the paper's emphasis on adaptive principle generation is overstated — the benefit comes from structured decomposition, not from adaptivity. If the fixed-criteria model scales poorly, it confirms that adaptive principle selection is genuinely important.

Practical Applications and Downstream Use Cases

Cost-efficient evaluation of LLM outputs in production pipelines. Organizations running large-scale LLM applications need to evaluate model outputs for quality, safety, and policy compliance. The standard approach — using a single forward pass through a scalar RM or calling GPT-4 as a judge — provides a point estimate with unknown reliability for any individual query. DeepSeek-GRM with inference-time scaling offers a direct upgrade: for high-stakes evaluations (e.g., checking outputs before showing them to users, evaluating candidate model checkpoints), spend k=8 or k=32 samples and use meta RM-guided voting. The paper's numbers suggest this shifts aggregate accuracy from ~70% (greedy) to ~73% (MetaRM at k=32), a 3-point improvement that, in a production setting evaluating millions of outputs, translates to thousands fewer misclassified responses. The latency cost is bounded if sampling is parallelized — the paper notes that "the latency of reward generation with a reasonable amount of, e.g., eight samplings will not increase significantly" (Appendix B) — making this feasible for asynchronous evaluation pipelines even if not for real-time filtering.

Training data curation for RLHF and preference optimization. A persistent challenge in RLHF is identifying high-quality preference pairs for training — human annotation is expensive, and using a weak RM to generate synthetic preferences propagates noise. DeepSeek-GRM with inference-time scaling can serve as a high-recall filter: for a batch of candidate response pairs, run DeepSeek-GRM with k=8 voting and keep only pairs where the predicted preference is high-confidence (large score margin between responses). The expanded reward space from voting ([8, 80] at k=8) provides a natural confidence measure: if Response A totals 68 and Response B totals 24, the margin is substantial; if both are in the 40s, the preference is uncertain and the pair can be discarded or sent for human review. This filtering step, applied upstream of human annotation or policy training, could improve data quality at lower cost than annotating all pairs.

Interpretable evaluation reports for model debugging. Because each DeepSeek-GRM sample produces explicit principles and analysis, aggregating across k samples yields not just a final score but a distribution over evaluation criteria. A practitioner evaluating a new model checkpoint can examine which principles were most frequently generated, how scores varied across principles, and where the model's evaluations were inconsistent (high variance across samples). This is substantially more informative than a single scalar score — it tells you why the model is being scored a certain way, not just what the score is. For model developers iterating on training recipes, this kind of interpretable feedback could accelerate debugging cycles. The paper's case study (Table 16) demonstrates the format — a query about behavioral neuroscience review yields principles like "Depth of Analogies and Metaphors (Weight: 30%)," with specific analysis of how each response meets or fails each criterion. Aggregating such analyses across multiple samples would surface which aspects of response quality the model consistently identifies as differentiating.

Offline benchmark evaluation with adaptive compute budgets. The paper's finding that inference-time gains are non-uniform across domains — large for Chat Hard (78.3% → 85.7% with MetaRM at k=8, Table 8), modest for PPE Correctness (59.8% → 63.0% with MetaRM at k=8, Table 9) — suggests a natural resource-allocation strategy: spend more inference compute on domains where it helps, less where it doesn't. A benchmark evaluation platform could estimate the expected scaling benefit per domain from a small pilot study, then allocate a fixed total compute budget across test examples to maximize aggregate accuracy. This is the same logic as compute-optimal test-time scaling for policy models (Snell et al., 2025) but applied to the evaluator rather than the solver. DeepSeek-GRM's pointwise format makes this straightforward: the cost per sample is roughly proportional to output length, which Figure 7 shows varies by domain (Chat: ~259 tokens, Reasoning: ~376 tokens), so the optimal allocation would account for both per-sample cost and per-domain scaling benefit.