P-GenRM: Personalized Generative Reward Model with Test-time User-based Scaling

ArXiv: 2602.12116

🎯 Pitch

Existing personalized reward models collapse diverse user preferences into a handful of fixed rubrics, but P-GenRM instead generates scenario-adaptive personas and scoring criteria on the fly from user interaction history. By further clustering users into prototypes and aggregating signals from similar users at test time—much like collaborative filtering—it delivers an additional 3% accuracy boost over static models and generalizes to new users with sparse feedback.

---

1. Executive Summary

This paper introduces P-GenRM, the first personalized generative reward model that transforms heterogeneous user preference signals—both implicit interaction histories and explicit criteria—into structured evaluation chains that derive scenario-adaptive personas and scoring rubrics, then evaluates candidate responses against those dynamically generated standards. Built on LLaMA-3.1-8B and 70B backbones and evaluated on the Chatbot Arena-Personalized, PRISM-Personalized, and LaMP-QA benchmarks, P-GenRM integrates a three-stage training pipeline—Persona-guided Scoring Induction via supervised fine-tuning, Criteria-based Reasoning Enhancement through reinforcement learning with process and outcome rewards, and hard-negative-aware curriculum learning—followed by a novel Test-time User-based Scaling mechanism that performs dual-granularity scaling: individual-level parallel sampling to aggregate multiple scoring schemes for the same user, and prototype-level incorporation of preference signals from similar users within learned clusters (analogous to collaborative filtering from recommendation systems). P-GenRM-8B achieves state-of-the-art results with an average 2.31% improvement over prior personalized reward models, while test-time user-based scaling delivers an additional 3% boost (e.g., Ind-16, Pro-8 reaching 75.92% on Chatbot Arena-Personalized versus 72.68% without scaling), and further demonstrates strong generalization to unseen users under sparse feedback on the out-of-distribution LaMP-QA benchmark (Spearman correlation of 0.638 versus 0.619 for Qwen3-235B-A22B), establishing that prototype-based transfer can compensate for limited individual histories—but only when the reference prototypes are properly tuned, as excessive partitioning introduces inference noise that degrades performance.

2. Context and Motivation

The Core Problem: Personalization Requires Signals We Don't Know How to Use

The fundamental question this paper tackles arises from a shift in what "good" alignment means for large language models. Standard RLHF (reinforcement learning from human feedback) optimizes for universal preferences — values like helpfulness, harmlessness, and factual accuracy that are assumed to apply broadly across all users. But real users are not a monolith. One user wants concise, bullet-pointed technical answers; another wants warm, conversational explanations with examples; a third wants the model to challenge their assumptions and play devil's advocate. A single model optimized for average helpfulness will satisfy none of them fully.

Personalized alignment — adapting model behavior to individual user preferences — has therefore emerged as a critical frontier. The problem is that personalization requires the model to understand what a specific user values, and that understanding must be inferred from signals that are fundamentally messier and sparser than the clean pairwise preference data used in standard RLHF. The paper identifies two categories of signal:

• Explicit preference criteria ($E^{(u)}$): Direct statements from the user such as "I prefer concise answers" or "use a professional tone." These are clear when available but are often sparse — most users never articulate their preferences explicitly.
• Implicit signals ($H_t^{(u)}$): The user's historical interaction patterns — which responses they preferred over others in past conversations. These are richer and more abundant but also noisier, since a single preference choice may reflect mood, context, or topic rather than a stable underlying value.

The paper's core insight is that these two signal types are complementary: implicit history provides volume, while explicit criteria provide precision. But existing methods fail to integrate them effectively, and the paper argues this integration failure is the central bottleneck in personalized alignment.

Why This Problem Matters: The Real-World Deployment Gap

The paper motivates the importance of personalized reward modeling through several interconnected practical concerns that go beyond academic benchmark scores.

Open-ended tasks amplify subjectivity. In domains like dialogue, creative writing, and long-form question answering, there is often no single "correct" response. Quality depends on alignment with the specific user's values, communication style, and contextual needs. A response that is factually accurate but delivered in a dry, technical register may be perfectly aligned for one user and misaligned for another who values warmth and accessibility. Without a reward model that can score responses through a user-specific lens, reinforcement learning cannot optimize for personalized quality.

Preferences are dynamic, not static. The paper highlights a critical nuance that simpler approaches miss: a single user's preferences shift with context. The introduction gives the example of a user who prefers brevity while driving (when cognitive load is high and interaction time is limited) but expressiveness in casual settings (when depth of engagement is the goal). A reward model that collapses a user into a single static persona — "this user likes short answers" — will mis-score responses in roughly half their interactions. The paper explicitly frames this as the failure mode of static preference modeling:

"Most methods reduce diverse and dynamic user preferences to a fixed set of evaluation rules, failing to capture scenario-dependent variability, even within a user"

This insight — that preferences are context-dependent even within the same person — is what motivates the paper's scenario-aware evaluation chain design, where personas and rubrics are generated fresh for each query rather than cached.

Cold-start users are the norm, not the exception. In any deployed system, new users arrive with no interaction history. A personalized reward model that requires extensive per-user data to function is useless at onboarding time — precisely when first impressions matter most. The paper identifies this as the weak generalization to new users problem (Section 1):

"Current models struggle to adapt to new users with sparse feedback, limiting effectiveness in cold-start scenarios."

This is not merely an annoyance; it is an adoption barrier. If personalization only kicks in after dozens of interactions, users may abandon the system before it ever adapts to them. The paper's prototype-based scaling mechanism is explicitly designed to address this: by clustering users into preference archetypes, a new user can immediately benefit from the learned preferences of similar users, providing a form of collaborative filtering for reward modeling.

Real-world deployment requires both fidelity and efficiency. The paper acknowledges a tension: generating structured evaluation chains with persona analysis and criterion-weighted scoring is more computationally expensive than a scalar reward model that outputs a single number. But it argues — through the test-time scaling results — that the fidelity gains justify the cost, especially since the scaling mechanism adds only modest additional inference time (00:14:16 for P-GenRM-8B without scaling versus 00:23:05 for the strongest scaling configuration, as shown in Table 11, Appendix A.9).

Where Existing Approaches Fall Short

The paper identifies a landscape of prior work and systematically catalogs their limitations along several axes. Understanding these limitations is essential because P-GenRM's design choices are direct responses to specific failure modes.

Static Persona Methods: SynthesizeMe and Its Limitations

The most directly comparable prior work is SynthesizeMe (Ryan et al., 2025), which the paper treats as its primary baseline and most recent competitor. SynthesizeMe infers a synthetic persona from a user's historical preferences and uses that persona to construct personalized prompts for reward models or LLM-as-a-judge systems. The approach has several strengths that P-GenRM builds on: it operates without identity data or fixed preference axes, and it shows that persona-based prompting improves personalization accuracy.

However, the paper identifies a fundamental limitation: the persona is static. Once inferred, it is treated as a fixed prior applied uniformly across all the user's interactions. This means it cannot capture the scenario-dependent variability the paper's preliminary experiments demonstrate. In Table 6 (Appendix A.2), the paper shows that providing an inferred persona improves accuracy from 62.42% to 64.02%, but adding scenario-specific explicit criteria (Output Style Requirements and Scoring Dimensions) pushes it further to 66.17%. This gap — roughly two percentage points between static persona and persona-plus-contextual-criteria — is the signal that dynamic, context-aware preference inference matters.

P-GenRM's approach is structurally different: rather than inferring a persona once and reusing it, the model generates a fresh persona and rubric for each query conditioned on both the interaction history and the current context. This is what the paper means by "adaptive" versus "static" personas — the evaluation chain is produced per-query, allowing the model to interpret the same user's preferences differently depending on what they're asking about and how they're asking it.

Fixed-Dimension Preference Models: PAL, GPO, VPL, and the Dimensionality Bottleneck

Several prior methods model user preferences as combinations of a small, fixed set of evaluation dimensions. PAL (Chen et al., 2024) represents each user as a mixture of "prototypical preference points" in a learned embedding space. GPO (Zhao et al., 2023) uses a separate Transformer module to predict group preferences from limited data. VPL (Poddar et al., 2024) treats preference as a latent variable inferred through variational inference. These methods share a common philosophy: preferences are captured by learning how a user weights a finite set of underlying dimensions.

The paper's critique is implicit but sharp: in open-domain settings, a fixed dimension set cannot cover the full space of possible preferences. The PRISM dataset provides predefined criteria like {Style, Values, Fluency, Factuality, Safety, Diversity, Helpfulness}, but P-GenRM's generated rubrics surface dimensions those definitions miss entirely — things like Philosophical Engagement, Openness, Structure, Depth, Nuance, Sensitivity, Breadth of Ideas (Section 5.2, "Broader Preference Space"). The paper does not just assert this; it visualizes concrete examples in Figures 6 and 7 (Appendix A.6) showing a single user's dramatically different preference patterns in a music recommendation scenario versus a serious discussion scenario.

The limitation is not that these methods produce wrong scores — PAL, GPO, and VPL all achieve reasonable accuracy in the 56–59% range on PersonalRewardBench (Table 1) — but rather that they ceiling early because their representational capacity is bounded by their predefined dimension set. A generative reward model that produces free-text evaluation chains has, in principle, unlimited representational capacity for preference dimensions.

LLM-as-a-Judge Approaches: Prompting Alone Is Not Enough

The paper also evaluates a family of in-context LLM-as-a-judge baselines (Table 1, top section) where different types of personalization information are appended to the prompt. The baseline accuracy is instructive:

• Default (no personalization): 56.37% on Chatbot Arena, 52.04% on PRISM
• Adding Chain-of-Thought reasoning: marginal improvement (~0.7%)
• Adding Demographics: limited benefit on PRISM (52.96% → 54.21%)
• Adding Preference History: more substantial gain (58.53% on Chatbot Arena, 56.24% on PRISM)
• Adding SynthesizeMe-style persona prompting: further gain (61.07% on Chatbot Arena, 54.70% on PRISM)

The pattern reveals two things. First, information matters: the more preference-relevant context provided, the better the judgment. Second, prompting alone plateaus: even the best in-context configuration (Persona-guided Scoring Induction) reaches only 62.20% on Chatbot Arena, while fine-tuned methods reach 72.68%. The gap — roughly 10 percentage points — represents what the model can learn from explicit training on the structured evaluation chain format that it cannot recover through zero-shot or few-shot prompting alone. This motivates the supervised fine-tuning stage of P-GenRM's training pipeline.

Bradley-Terry Reward Models: The Universal Preference Ceiling

A Bradley-Terry reward model trained on unfiltered PersonalRewardBench data achieves surprisingly strong results: 67.21% on Chatbot Arena and 63.27% on PRISM (Table 1, "Finetuned Reward Models"). This is competitive with some personalized methods and demonstrates that simply learning the average preference of the data distribution captures a substantial portion of the signal.

But the paper's framing makes clear why this is insufficient. A Bradley-Terry model optimized for average preference will systematically under-serve users whose preferences deviate from the norm — particularly minority preference groups. The paper addresses this concern explicitly in Appendix A.5, reporting macro accuracy (accuracy computed per-prototype-group and then averaged) of 65.21% for P-GenRM versus 60.64% for a fine-tuned Bradley-Terry 70B model. The 4.57% gap in macro accuracy is larger than the gap in standard accuracy, confirming that personalized methods disproportionately benefit underrepresented preference patterns.

How This Paper Positions Itself

The paper positions P-GenRM at the intersection of two research threads that had not been combined before: personalized reward modeling and generative reward models (GenRM).

The generative reward model paradigm — where a language model produces a textual evaluation rather than a scalar score — has recently gained traction for its interpretability, flexibility, and test-time scalability (Zhang et al., 2024a; Li et al., 2023b; Liu et al., 2025). These models can explain why a response is good or bad, generate principled critiques, and improve through additional computation at inference time. But prior GenRM work focused entirely on universal evaluation criteria — assessing general helpfulness, correctness, or harmlessness. No prior work had applied the GenRM framework to personalized evaluation, where the scoring criteria themselves must be user-specific.

The paper's positioning therefore has two components:

1. Extending GenRM to personalization. P-GenRM is "to the best of our knowledge, the first personalized generative reward model" (Section 1, Contribution 1). This is not merely a domain shift — it requires solving new technical problems. A universal generative evaluator can rely on broadly shared criteria; a personalized one must infer criteria from heterogeneous, noisy, and sparse per-user signals. The paper's three-stage training pipeline (SFT for persona-guided scoring, RL for criteria-based reasoning, curriculum learning for robustness) is designed to address exactly this challenge.

2. Introducing test-time scaling to personalization. Prior test-time scaling work in reward modeling (Liu et al., 2025) focused on generating multiple critique principles and aggregating them. P-GenRM adapts this idea but adds a crucial personalization-specific mechanism: prototype-based scaling. Rather than just generating multiple evaluation chains for the same user (individual-level scaling), it also incorporates evaluations from similar users (prototype-level scaling). This is a novel twist inspired by collaborative filtering — the idea that users who agreed in the past will agree in the future — and it directly addresses the cold-start generalization problem that prior personalized methods struggled with.

The paper explicitly frames the prototype mechanism as solving two problems simultaneously (Section 4.2):

"How to mitigate the noise inherent in inferred user preference; How to develop a transfer mechanism that enhances generalization to new users by adapting learned preferences?"

The dual-granularity design — individual scaling to reduce variance, prototype scaling to enable transfer — means the same mechanism serves both the accuracy-improvement goal (reducing noise) and the generalization goal (handling new users).

The Paper's Relationship to Prior Work: A Map

To situate P-GenRM precisely, here is how it relates to each category of prior work:

The paper does not claim to invent any of its individual components from scratch. Its contribution is the integration — showing that combining dynamic persona inference, generative evaluation chains, multi-stage RL training, and dual-granularity test-time scaling creates a system that substantially outperforms all prior personalized reward models on both accuracy and generalization, while also providing interpretability through explicit personas and criterion-weighted rubrics.

3. Technical Approach

3.1 Reader orientation

P-GenRM is a fine-tuned large language model that performs personalized reward modeling by generating structured evaluation chains—textual outputs containing a user persona analysis, scenario-specific scoring rubrics, and criterion-weighted scores for candidate responses—rather than outputting a single scalar. The system solves the problem of evaluating open-ended LLM responses according to individual user preferences by transforming heterogeneous preference signals (historical interaction data and explicit stated criteria) into a comprehensive, interpretable scoring process, adaptively re-deriving the user's persona and evaluation criteria for each new query to capture scenario-dependent preference shifts.

3.2 Big-picture architecture

The P-GenRM system can be understood as a pipeline with five major stages:

1. Preference Signal Ingestion: The system receives three inputs—the current user query $q_t$, a limited set of historical preference pairs $H_t^{(u)}$ (up to $\mathbf{h}=3$ randomly sampled past interactions showing which responses the user preferred), and optional explicit preference criteria $E^{(u)}$ (e.g., style requirements, desired tone).

2. Persona-guided Scoring Induction (Training Stage 1): A supervised fine-tuning stage where an instruction-tuned LLM distills an instructor model's ability to transform preference signals into evaluation chains. The instructor model (OpenAI o3) is prompted with implicit and explicit signals to produce a persona analysis and scoring rubric; these outputs are filtered via rejection sampling to create the SFT dataset, then used to fine-tune the base LLaMA model to acquire initial personalized scoring capability.

3. Criteria-based Reasoning Enhancement (Training Stage 2): A reinforcement learning stage using GRPO that improves the quality of evaluation chains, especially when explicit feedback is missing. The model is rewarded for both producing correct final scores (outcome reward) and generating evaluation chains that faithfully cover the user's known preference criteria (process reward via LLM-as-a-judge).

4. Hard-Negative-Aware Curriculum Learning (Training Stage 3): A second RL phase that progressively increases the proportion of difficult-to-distinguish response pairs, improving robustness on challenging cases while disabling process rewards to allow broader exploration.

5. Test-time User-based Scaling (Inference-Time Mechanism): A dual-granularity scaling strategy applied after training completes. At the individual level, P-GenRM generates multiple parallel scoring schemes for the current user and aggregates them. At the prototype level, the user is assigned to a pre-learned cluster of similar users, and preference signals from those neighbors are incorporated into the final score. Both levels run concurrently at inference time and their outputs are averaged.

Information flows: preference signals → SFT-trained base model produces evaluation chains → RL refines chain quality → curriculum learning hardens against difficult cases → at deployment, individual and prototype scaling run in parallel → final score is the average of all individual and prototype-level extracted scores.

3.3 Roadmap for the deep dive

• First, the formal problem formulation (Equation 2) and the structure of the evaluation chain, since the evaluation chain is the central data structure that every subsequent component produces, consumes, or refines.
• Second, the three-stage training framework in order—Persona-guided Scoring Induction (SFT), Criteria-based Reasoning Enhancement (RL with process and outcome rewards), and hard-negative-aware curriculum learning—because each stage builds on the output of the previous one, and understanding the RL reward design requires understanding what the SFT stage already learned.
• Third, the offline prototype initialization and optimization process, since prototype-based scaling at test time depends on having learned meaningful user clusters; this includes the embedding, clustering, and history-aware attentive refinement steps, along with the pairwise loss and regularization terms used to update prototypes.
• Fourth, the test-time dual-granularity scaling mechanism itself, covering individual-level parallel sampling, prototype-level nearest-neighbor selection, and the final score aggregation formula, since this is the deployment-stage mechanism that delivers the reported 3% improvement.
• Fifth, a summary of key design choices and their justifications—why SFT before RL, why process and outcome rewards separately, why edit-distance-like similarity for prototype matching, why averaging rather than weighted voting—since these choices collectively define what is novel in the system.

3.4 Detailed, sentence-based technical breakdown

This is primarily a systems-building paper whose core idea is that personalized evaluation can be decomposed into structured, interpretable evaluation chains generated by a fine-tuned language model, and that test-time scaling across both individual samples and similar-user references improves accuracy and generalization beyond what any single evaluation chain can achieve.

---

The Structured Evaluation Chain: Data Structure and Problem Formulation

The central unit of computation throughout P-GenRM is the Structured Evaluation Chain (SEC). It is not a separate module or architecture component—it is a text generation format that the language model is trained to produce, and the structure is enforced through prompting and training, not through hard constraints on decoding.

What the SEC contains. As shown in Figure 2(b), a completed evaluation chain consists of two concatenated components:

1. $P_t^{(u)}$ — the personalized preference modeling, a textual analysis that describes the user's scenario-specific persona and derives the corresponding preference criteria. This persona is not static; it is generated fresh for each query $q_t$ conditioned on the user's available signals.
2. $S_t^{(u)}$ — the scoring process, which applies the derived criteria to each candidate response $y_t^i$ and produces a numerical score $s_t^i$ along with textual justification for why that score was assigned.

The formal definition (Equation 2 in the paper) captures this generation process:

$[P_t^{(u)}; S_t^{(u)}] \sim R_\theta\big(q_t, H_t^{(u)}, E^{(u)}, y_t^i\big), \quad \{s_t^i\}_{i=1}^b = \mathrm{Extract}(S_t^{(u)})$

where $R_\theta$ is the personalized generative reward model parameterized by $\theta$, $q_t$ is the current user query, $H_t^{(u)}$ is the user's sampled historical interaction records (limited to $\mathbf{h}=3$ pairs by random selection, i.e., $|H_t^{(u)}| = \mathbf{h}$), $E^{(u)}$ is the user's explicit preference criteria when available, $y_t^i$ is the $i$-th candidate response being evaluated, $b$ is the number of candidates (typically $b=2$), and $\mathrm{Extract}$ is a post-processing operation that parses the generated text to isolate the numerical score for each response.

What it computes. The model $R_\theta$ takes as input a prompt that includes the query, the historical preference pairs, any explicit criteria, and the candidate responses, then autoregressively generates the entire evaluation chain—persona analysis followed by criterion-weighted scoring—as a single continuous text. The $\mathrm{Extract}$ operation then scans the generated text for the score values assigned to each response index and converts them to scalars. The output is a set of $b$ scores $\{s_t^i\}$ that can be compared to determine which of the $b$ candidate responses better aligns with the inferred user preferences.

Why this form over a scalar reward model. A traditional outcome reward model (ORM) would take the same inputs and output a single scalar per response, compressing all reasoning into an opaque numeric value. The SEC format has three structural advantages: (1) it produces interpretable intermediate outputs—the persona and criteria are human-readable and auditable; (2) it allows test-time scaling because multiple independent evaluation chains can be sampled for the same query, each exploring different hypotheses about the user's preferences, and the resulting scores can be aggregated to reduce variance; (3) it allows the model to reason openly about which criteria matter in the specific context, rather than being forced to collapse that reasoning into a fixed-dimensional embedding. A scalar model trained on the same data would need to learn an implicit mapping from user history to preference weights, but it could not explain those weights or be debugged when they are wrong.

The history sampling mechanism. A practical detail that matters for reproducibility: when constructing the prompt, the system randomly samples exactly $\mathbf{h}=3$ historical preference pairs from the user's interaction history $H_t^{(u)}$. The paper reports in Appendix A.10 (Table 12) that performance improves significantly when moving from 1 to 3 preference pairs (59.78% → 72.68% accuracy on Chatbot Arena) but plateaus thereafter (72.50% for 4 pairs). This is explained by the fact that a single preference choice is a one-off decision that may not reflect stable preferences, two choices are insufficient to establish a pattern, but three provide the minimal structure to assess consistency. More than three primarily improves evaluation stability without substantial accuracy gains. The random selection (rather than selecting the most recent or most informative pairs) is a deliberate choice to prevent the model from overfitting to temporal patterns that may not generalize.

---

Persona-guided Scoring Induction: The SFT Stage

The first training stage equips the base LLaMA model with the fundamental capability to produce structured evaluation chains. The key design choice is not to train from scratch on human annotations—which the paper notes would be prohibitively expensive at scale—but rather to distill from a stronger instructor model using a rejection sampling pipeline.

The preliminary experiment that motivated the design. Before building the full pipeline, the authors conducted a controlled experiment (Table 6, Appendix A.2) to determine which types of preference information most improve personalized scoring accuracy when provided in the prompt. Using OpenAI o3 as the judge model and a 15% sample of the PRISM dataset, they tested various preference indicators appended to the baseline prompt. The results established a clear hierarchy:

• Baseline (no personalization, Chain-of-Thought only): 62.42%
• Adding Self-Description from the user: 63.22%
• Adding Basic Demographic Information: 63.34%
• Adding Choice Attributes (user-defined scoring dimension weights): 63.63%
• Adding Inferred Persona: 64.02%
• Adding Output Style Requirements (system-level rules): 64.24%
• Combining Persona, Output Style Requirements, and Choice Attributes: 66.17%

Two conclusions drive the SFT design: (1) user persona inferred from interaction histories is the single most impactful individual signal (64.02%), and (2) augmenting it with explicit preference criteria produces the strongest result (66.17%). This directly motivates the evaluation chain structure—persona generation first, followed by criterion-weighted scoring.

The distillation pipeline. The SFT training data is constructed as follows:

1. Instructor model prompting: OpenAI o3 is prompted with the user's implicit preference signals $H_t^{(u)}$ (historical preference pairs) and explicit preference signals $E^{(u)}$ (if available). The prompt instructs o3 to first induce the user's preference modeling $P_t^{(u)}$ (consisting of a scenario-specific persona and derived preference criteria), then evaluate each candidate response $y_t^i$ against those criteria, producing a personalized score $s_t^i$.

2. Structured output format: o3's output follows the SEC template—persona analysis, criteria derivation, criterion-by-criterion scoring for each candidate, and final numerical scores.

3. Rejection sampling filter: Not all o3-generated evaluation chains are retained. The paper states that the data is "filtered through rejection sampling" before being used for SFT. While the paper does not explicitly detail the filtering criteria in the main text, the implied filter is correctness of the final pairwise judgment: only evaluation chains where the extracted scores correctly rank the known-preferred response above the known-dispreferred response are kept. This ensures the SFT data teaches the model to produce chains that lead to correct conclusions.

4. SFT training: The LLaMA base model (3.1-8B or 3.1-70B) is fine-tuned via standard next-token prediction on the filtered SEC dataset. The model learns to map (query, history, explicit criteria, candidate responses) to the full evaluation chain text. The training objective is standard autoregressive language modeling loss on the instructor-generated targets.

What this stage produces. After SFT, the model has acquired "the initial capability of transforming hybrid preference factors into an adaptive personalized scoring scheme" (Section 4.1). Concretely, it can now: take a new user's query and preference signals, generate a plausible persona and criteria set, and assign scores that reflect those criteria. However, the SFT-only model is limited in two ways: (1) it relies entirely on the quality of the instructor model's outputs and inherits any biases or errors from the distillation process; (2) it has no mechanism to improve when explicit preference criteria are missing—it can only imitate what the instructor would have done with whatever signals were provided.

The adaptive vs. static persona distinction. The paper explicitly contrasts the PSI (Persona-guided Scoring Induction) approach with SynthesizeMe's static persona approach in Table 4. When both are implemented as LLM-as-a-judge prompting methods (no fine-tuning), PSI consistently outperforms SynthesizeMe: on Chatbot Arena, Qwen3-8B + PSI achieves 64.22% versus 62.57% for Qwen3-8B + SynthesizeMe; on PRISM, the gap is 58.01% versus 56.33%. The same holds for the o3 instructor model (69.14% versus 67.73% on Chatbot Arena). This validates that dynamic, per-query persona generation captures context-dependent preference shifts that a static persona misses.

---

Criteria-based Reasoning Enhancement: The RL Stage

The SFT stage produces a model that can generate evaluation chains when given preference signals, but it has no explicit incentive to produce better chains when measured against ground-truth preferences. The RL stage addresses this by directly optimizing the quality of generated evaluation chains using a composite reward signal.

The core problem this stage solves. In deployment, the most common scenario is one where the user has provided implicit signals (historical preference choices) but no explicit criteria. The model must infer what the user values purely from which responses they preferred in the past. The SFT-trained model can do this through the distilled behavior of the instructor, but it has no mechanism for systematic improvement—it either gets the pairwise judgment right or wrong, with no feedback on whether its reasoning process (the persona and criteria) was sound. The RL stage introduces process-level feedback that explicitly rewards the model for generating evaluation chains that faithfully cover the user's actual preferences, even when those preferences must be inferred.

Synthetic explicit preference generation. The key insight is to treat explicit preferences as a training-time supervision signal rather than an inference-time dependency. The training procedure works as follows:

1. When a training sample includes explicit user criteria $E^{(u)}$, those are used directly as the supervision for the process reward.
2. When a training sample lacks explicit criteria, the system first instructs P-GenRM to synthesize plausible explicit preferences from the available historical interactions. The prompt for this synthesis is shown in Figure 9 (Appendix C). The model generates, based on the pattern of which responses the user preferred, a textual description of what criteria the user likely values (e.g., "Based on the user's preference for concise, fact-dense responses over elaborate explanations, the user likely values brevity, factual accuracy, and directness").

This synthetic criteria generation serves as an intermediate reasoning step: the model first states what it believes the user wants, then evaluates the candidate responses against those stated criteria. The process reward judges how well the evaluation chain covers these stated or synthesized criteria.

The composite reward function. The total reward for each generated evaluation chain is:

$\operatorname{R}_{t} = \alpha \cdot \operatorname{PR}_{t} + \beta \cdot \operatorname{OR}_{t}$

where $\alpha$ and $\beta$ are hyperparameters controlling the relative weight of process and outcome rewards (set to $\alpha = 0.5$ and $\beta = 1.0$ in the main experiments, as stated in Section 5.1), $\operatorname{PR}_{t}$ is the process reward, and $\operatorname{OR}_{t}$ is the outcome reward.

The process reward $\operatorname{PR}_t$: An LLM-as-a-judge module (a separate, frozen evaluation model) is tasked with assessing how well the generated evaluation chain covers the user's stated or synthetic explicit preferences. The judge outputs a score in the range $[0, 1]$ representing the coverage quality—a score of 1.0 means the evaluation chain thoroughly addresses all the relevant preference criteria, while lower scores indicate missing or superficial treatment of important criteria. This is a process-level reward because it evaluates the quality of the reasoning, not just the correctness of the output.

The outcome reward $\operatorname{OR}_t$: A rule-based binary indicator of whether the final score ordering is correct:

$\operatorname{OR}_{t} = \mathbf{1}\{s_t^c > s_t^r\}$

where $c$ is the index of the human-labeled "chosen" (preferred) response and $r$ is the index of the "rejected" (dispreferred) response. This is 1 if the model correctly assigns a higher score to the chosen response, and 0 otherwise. Additionally, if the model produces a formatting error (output that cannot be parsed to extract valid scores), a penalty of $-0.1$ is applied to the total reward.

What the composite reward computes. For a single training sample, the process reward captures whether the model's reasoning is faithful to the user's known preferences, and the outcome reward captures whether that reasoning leads to the correct conclusion. The weighted sum $\operatorname{R}_t$ is a scalar that reflects both the quality of the process and the correctness of the result.

Why this form over outcome-only reward. An outcome-only reward (setting $\alpha = 0$) would teach the model to produce correct score orderings without learning to articulate valid reasoning—it could achieve high reward by generating plausible-sounding but irrelevant evaluation chains that happen to assign the right scores. Conversely, a process-only reward (setting $\beta = 0$) would reward faithful reasoning but provide no signal about whether that reasoning produces correct judgments. The composite reward balances these: the model must both reason about the user's preferences and reach the right conclusion. The paper's ablation (Table 7, Appendix A.3) confirms that both components matter: removing the process reward ($\alpha = 0, \beta = 1$) drops Chatbot Arena accuracy from 71.07% to 70.22%, while removing the outcome reward ($\alpha = 1, \beta = 0$) drops it further to 69.05%. The weights $\alpha = 0.5, \beta = 1.0$ provide the best balance; increasing $\alpha$ to 1.0 (overweighting process) degrades performance, suggesting the model can overfit to specific criteria at the expense of overall judgment quality.

Integration with GRPO. The reward $\operatorname{R}_t$ is used to compute advantages in the Group Relative Policy Optimization (GRPO) algorithm. The GRPO objective function, as stated in the paper, is:

$J_{GRPO}(\theta) = \mathbb{E}_{(q_t, H_t^{(u)}, y_t^i) \sim \mathcal{D}, \{c_t^{(k)}\}_{k=1}^K \sim \pi_{\theta_{\text{old}}}} \left[ \frac{1}{K} \sum_{k=1}^K \frac{1}{|c_t^{(k)}|} \sum_{j=1}^{|c_t^{(k)}|} \left\{ \min\left( \frac{\pi_\theta(c_{t,j}^{(k)} | q_t, H_t^{(u)}, y_t^i, c_{t,<j}^{(k)})}{\pi_{\theta_{\text{old}}}(c_{t,j}^{(k)} | q_t, H_t^{(u)}, y_t^i, c_{t,<j}^{(k)})} A_t^{(k)}, \text{clip}\left( \frac{\pi_\theta(c_{t,j}^{(k)} | q_t, H_t^{(u)}, y_t^i, c_{t,<j}^{(k)})}{\pi_{\theta_{\text{old}}}(c_{t,j}^{(k)} | q_t, H_t^{(u)}, y_t^i, c_{t,<j}^{(k)})}, 1 - \varepsilon, 1 + \varepsilon \right) A_t^{(k)} \right) \right\} - \beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}}) \right]$

where $c_t^{(k)} = [P_t^{(u)}; S_t^{(u)}]$ is the $k$-th sampled structured evaluation chain (out of $K$ samples generated by the old policy $\pi_{\theta_{\text{old}}}$), $|c_t^{(k)}|$ is its token length, $c_{t,j}^{(k)}$ is the $j$-th token of that chain, $A_t^{(k)}$ is the relative advantage computed from the reward $R_t^{(k)}$ for the $k$-th sample, $\varepsilon$ is the clipping threshold, $\pi_\theta$ is the current policy being optimized, $\pi_{\theta_{\text{old}}}$ is the frozen policy from the previous iteration, $\pi_{\text{ref}}$ is a reference policy (typically the SFT model), and $\beta D_{\text{KL}}(\pi_\theta \parallel \pi_{\text{ref}})$ is a KL-divergence penalty that prevents the policy from drifting too far from the reference.

What the GRPO objective computes. For each training example, the model generates $K$ different evaluation chains (sampled from the old policy). Each chain receives a reward $R_t^{(k)}$, and the relative advantage $A_t^{(k)}$ is computed by comparing each reward to the mean reward across the $K$ samples (chains that score above average get positive advantage; below average get negative). The objective then increases the probability of tokens that appeared in high-advantage chains (while clipping to prevent too-large updates) and penalizes divergence from the reference policy. The $\min$ and $\text{clip}$ operations together implement the standard PPO-style conservative policy update.

Why GRPO over standard PPO or DPO. GRPO is a variant of PPO that operates on groups of samples rather than individual rollouts. For this task, generating multiple chains per input is natural because quality can vary substantially across samples (different chains may hypothesize different personas and criteria). GRPO provides a stable way to compare samples and reinforce the better ones. DPO (Direct Preference Optimization) would require a different formulation since it operates on pairwise preference data rather than per-sample rewards; the process reward component ($\operatorname{PR}_t$) does not have a natural pairwise analog.

The practical effect of the RL stage. Before RL, the model can generate evaluation chains but has no training signal distinguishing good chains from bad ones beyond the SFT imitation objective. After RL, the model learns that: (1) evaluation chains that explicitly cover the user's preference criteria are rewarded (process reward), so the model develops a tendency to generate more thorough and criterion-aligned analyses; (2) evaluation chains that produce correct score orderings are rewarded (outcome reward), so the model's reasoning becomes more calibrated to actual preferences; (3) the KL penalty keeps the model from drifting into degenerate behaviors like producing overly short chains or repetitive text to game the reward. The ablation in Table 3 shows the magnitude of this contribution: removing the RL stage entirely (keeping only SFT and curriculum learning) drops Chatbot Arena accuracy from 72.68% to 66.76%—a 5.92 percentage point decrease, which is the single largest drop from any ablation.

---

Hard-Negative-Aware Curriculum Learning

The third training stage addresses a specific failure mode of the RL-trained model: it performs well on clear-cut preference distinctions (where the chosen response is obviously better than the rejected one) but struggles on hard negatives—pairs where both responses are plausible, the quality difference is subtle, and distinguishing them requires nuanced understanding of the user's preferences.

What "hard negative" means in this context. In personalized reward modeling, a hard negative is a rejected response that is superficially similar to the preferred response but differs along a specific preference dimension. For example, if a user values conciseness, both a short-but-incomplete response and a short-and-complete response may initially appear similar; the model must learn to penalize incompleteness without penalizing conciseness. The SFT and RL stages, which operate on uniformly sampled training data, may under-expose the model to these edge cases.

The curriculum mechanism. The training procedure progressively increases the proportion of hard negative examples during the curriculum learning phase. The paper does not specify the exact curriculum schedule (e.g., linear increase, step function, or the specific proportion trajectory), but the principle is: early in this stage, the training distribution resembles the original data; later, an increasing fraction of examples are drawn from a hard-negative subset.

Process reward is disabled. A significant design choice: during this stage, the reward function omits the process reward $\operatorname{PR}_t$, meaning:

$\operatorname{R}_t = \beta \cdot \operatorname{OR}_t \quad \text{(with implicit } \operatorname{PR}_t = 0 \text{)}$

The paper states this is done "to enable a larger exploration space for hard negative samples." The reasoning is that process reward—which evaluates how well the chain covers explicit criteria—may constrain the model to generate conservative, criteria-literal evaluations. On hard negatives, where the correct distinction relies on subtle preference interpretation rather than explicit criteria matching, this constraint could be harmful. Disabling process reward gives the model freedom to explore evaluation strategies that may not map cleanly to stated criteria but still produce correct judgments.

The objective function. The GRPO objective retains the same form as in the RL stage, but with the modified reward function (outcome-only). The KL penalty against the reference policy continues to apply, preventing the model from diverging too far from the behavior learned in the RL stage.

What this stage produces. After curriculum learning, the model is more robust on difficult pairwise comparisons. The ablation in Table 3 quantifies this: removing curriculum learning (keeping SFT and RL) drops Chatbot Arena accuracy from 72.68% to 71.07%—a 1.61 percentage point decrease. While smaller than the RL ablation effect, it is consistent across both datasets (71.07% → 72.68% on Chatbot Arena; 63.82% → 65.32% on PRISM), confirming that hard-negative training provides a reliable, if modest, improvement.

Summary of the three-stage training effect. The full ablation in Table 3 traces the cumulative contributions:

• Base model (no training, in-context only): 56.37% (Chatbot Arena)
• SFT only: 66.76%
• SFT + RL: 71.07%
• SFT + RL + Curriculum Learning (full P-GenRM): 72.68%

Each stage provides a non-trivial improvement, with RL contributing the largest single gain (4.31%) and curriculum learning adding an additional 1.61%. The progression validates the paper's claim that each stage addresses a distinct aspect of the personalized evaluation problem: SFT provides basic capability, RL improves reasoning quality and calibration, and curriculum learning enhances robustness on difficult cases.

---

Offline Prototype Initialization and Optimization

Before test-time scaling can operate, the system must learn a mapping from users to prototypes—clusters of similar users whose preferences can be transferred. This is an offline process completed after training and before deployment.

Why prototypes rather than per-user models. Training a separate reward model for each user is infeasible for two reasons: (1) most users have too few preference annotations (the paper requires only $\mathbf{h}=3$ per user, which is insufficient to train a model from scratch); (2) even if enough data existed, maintaining thousands of per-user models would be computationally prohibitive. Prototypes provide a middle ground: learn a moderate number of preference archetypes (50 in the paper's experiments), then assign each user to the nearest archetype at test time.

Step 1: Preference embedding. For each user, after training completes, P-GenRM is used to generate the user's preference modeling $P_t^{(u)}$ for each scenario in which the user appears in the training data. This is a cross-scenario set of persona descriptions. Each $P_t^{(u)}$ is fed into Qwen3-Embedding-0.6B (a dedicated embedding model, not P-GenRM itself) to produce a dense vector in $\mathbb{R}^d$. All embeddings for all users are concatenated into a matrix $\mathbf{P}$ representing the overall cross-scenario user-preference embedding space.

Step 2: K-means clustering. K-means is applied to $\mathbf{P}$ to obtain $k$ centroids. These centroids are the initial user prototypes, denoted as $\mathbf{A} \in \mathbb{R}^{k \times d}$, where each row $a_j = \mathbf{A}_{[j,:]}$ represents the $j$-th prototype as a vector in the same embedding space.

The number $k = 50$ is determined through PCA analysis (Figure 3, left panel): the authors perform dimensionality reduction on $\mathbf{P}$ and record the proportion of singular values retained as a function of the number of leading singular vectors. Beyond 50 vectors, additional components provide only marginal information gains while substantially increasing the number of prototypes (and hence the inference cost of prototype-level scaling). Validation experiments in Figure 3 (right panel) confirm that performance improves substantially from 0 to 50 prototypes, plateaus, and slightly degrades at 100 prototypes—the degradation is attributed to overly fine-grained partitioning introducing inference noise, where small prototype groups may have noisy or unrepresentative preference patterns.

Step 3: History-aware attentive prototype refinement. The initial K-means centroids are purely semantic—they represent the average embedding of users assigned to each cluster. But for personalized reward modeling, a good prototype should be predictive: knowing that a user belongs to prototype $j$ should help the model correctly judge which responses the user prefers. The refinement step transforms each prototype from a semantic average into a discriminative prior.

For a given prototype $a_j$ and a user $u$ assigned to it, the user's historical records at turn $t$ are:

$H_t^{(u)} = \{(q_\tau, y_\tau^+, y_\tau^-) \mid \tau \in \mathrm{Random}(t-1, \mathbf{h})\}^{(u)}$

where $\mathrm{Random}(t-1, \mathbf{h})$ denotes that $\mathbf{h}$ items are randomly sampled from the history up to step $t-1$. For the refinement computation, the paper states "for brevity, we assume that these letters represent their embeddings," meaning each $q_\tau$, $y_\tau^+$, and $y_\tau^-$ is represented as a $d$-dimensional embedding vector.

Each historical triple is encoded into a single interaction vector:

$o_\tau = \sigma\left(W \cdot \mathrm{concat}(q_\tau, y_\tau^+ - y_\tau^-)\right)$

where $W$ is a learnable weight matrix, $\mathrm{concat}$ concatenates the query embedding and the difference between positive and negative response embeddings, and $\sigma$ is a nonlinear activation function. The vector $o_\tau$ encodes the information from one historical interaction: what the query was about and how the preferred and dispreferred responses differed. The subtraction $y_\tau^+ - y_\tau^-$ captures the direction of preference—which features distinguish good from bad responses for this user.

A prototype-augmented attention mechanism then assigns importance weights to each historical record:

$v_H = \sum_{\tau=1}^{\mathbf{h}} \alpha_\tau o_\tau, \quad \alpha_\tau = \mathrm{softmax}_\tau \left( \frac{o_\tau^\mathsf{T} q_t}{\sqrt{d}} + \rho \frac{o_\tau^\mathsf{T} a_j}{\sqrt{d}} \right)$

where $\alpha_\tau$ is the attention weight for the $\tau$-th historical interaction, $q_t$ is the embedding of the current query, $a_j$ is the prototype embedding, $d$ is the embedding dimension, and $\rho$ is a hyperparameter controlling the influence of the prototype term.

What the attention mechanism computes. The weight $\alpha_\tau$ has two components: the first term $o_\tau^\mathsf{T} q_t / \sqrt{d}$ measures relevance between the historical interaction and the current query (standard query-key attention—records similar to the current query get higher weight), and the second term $\rho \cdot o_\tau^\mathsf{T} a_j / \sqrt{d}$ measures relevance between the historical interaction and the prototype (records that are informative about the prototype's preference pattern get additional weight). The $\sqrt{d}$ scaling is the standard Transformer attention scaling to prevent dot products from growing too large in high dimensions. The $\mathrm{softmax}$ normalizes these combined relevance scores across the $\mathbf{h}$ historical records. The output $v_H$ is a weighted sum of the encoded interactions, representing the most relevant historical information for the current query, biased toward interactions that best characterize the prototype.

Why the prototype term matters. Without the $\rho$ term, the attention mechanism would be pure query-based relevance and would ignore the prototype structure. The prototype term ensures that the model selectively exploits historical records that are informative about the prototype's general preference pattern—even if those records are not the most semantically similar to the current query. This is the mechanism by which prototype-level preferences influence the individual-level evaluation.

Step 4: Discriminative prior update. The salient historical information $v_H$ is integrated with the prototype representation to form a prior vector $z_t$:

$z_t = a_j + \lambda_q W_q q_t + \lambda_s W_s v_H, \quad z \in \mathbb{R}^d$

where $W_q$ and $W_s$ are learnable projection matrices, and $\lambda_q$ and $\lambda_s$ are hyperparameters controlling the influence of the current query and the historical summary, respectively. The prior $z_t$ starts from the prototype base $a_j$, then adds query-specific and history-specific adjustments—it is the prototype adapted to the current context.

This prior is then used to score the positive and negative responses:

$\Delta_t = z_t^\top y_t^+ - z_t^\top y_t^-, \quad \mathcal{L}_{\text{pair}} = -\log \sigma(\Delta_t)$

where $\Delta_t$ is the discriminative score difference (positive if the prior $z_t$ assigns a higher score to the preferred response), $\sigma$ is the sigmoid function, and $\mathcal{L}_{\text{pair}}$ is the pairwise logistic loss. Minimizing $\mathcal{L}_{\text{pair}}$ maximizes the log-probability that the preferred response scores higher than the dispreferred one—this is the standard Bradley-Terry ranking objective applied to the prototype-augmented prior.

The full prototype update objective. The prototype $a_j$ is updated to minimize:

$\mathcal{L} = \mathcal{L}_{\text{pair}} + \lambda_{\text{cent}} \|a_j - \mu_j\|_2^2 + \lambda_{\text{tr}} \|a_j - p_j\|_2^2$

where $\mu_j$ is the center (mean) of the $j$-th cluster's associated sample embeddings, $p_j$ is the state of prototype $a_j$ from the previous update step, and $\lambda_{\text{cent}}$ and $\lambda_{\text{tr}}$ are regularization hyperparameters.

What this loss computes. The first term $\mathcal{L}_{\text{pair}}$ pushes the prototype toward being discriminatively useful—its associated prior should correctly rank responses. The second term $\lambda_{\text{cent}} \|a_j - \mu_j\|_2^2$ acts as a centering regularizer that prevents the prototype from drifting too far from the semantic center of its assigned users—this ensures the prototype remains representative of the group rather than overfitting to the specific training samples used in the update. The third term $\lambda_{\text{tr}} \|a_j - p_j\|_2^2$ is a trust-region regularizer that prevents the prototype from changing too much in a single update step—this smooths the optimization trajectory.

Why these regularizers are necessary. Without the centering term, the prototype could drift to optimize $\mathcal{L}_{\text{pair}}$ at the expense of semantic meaningfulness—it might learn to discriminate well on the training data but fail to represent a coherent preference archetype that generalizes to new users. Without the trust-region term, the prototype could oscillate or collapse during iterative updates. Together, they produce prototypes that are both discriminatively useful and semantically stable.

Step 5: Reassignment and iteration. After updating all prototypes, users are reassigned to their nearest prototype (in embedding space). The refinement process (steps 3–5) can be iterated, though the paper does not specify the number of iterations used.

Output of this offline phase. A set of $k$ optimized prototypes $\{a_j\}_{j=1}^k$ and a user-to-prototype assignment mapping. This mapping is frozen and used at test time.

---

Test-time Dual-Granularity Scaling

With the trained P-GenRM model and the optimized prototype assignments in place, the deployment-time mechanism applies two parallel scaling strategies and aggregates their results.

Individual-level scaling (parallel sampling). Given the current user's query $q_t$, historical signals $H_t^{(u)}$, any explicit criteria $E^{(u)}$, and the candidate responses $y_t^i$, the model performs $m$ independent forward passes—each sampling a potentially different evaluation chain:

$S_{t,x}^i \sim R_\theta\big(q_t, H_t^{(u)}, y_t^i, P_{t,x}^{(u)}\big)$

where $x$ indexes the $x$-th sample (out of $m$ total, e.g., $m=8$, $m=16$, or $m=32$ in the experiments), and $P_{t,x}^{(u)}$ is the preference analysis obtained from that specific sampling run. The stochasticity comes from the model's own sampling process (temperature-based token sampling), so each run may generate a slightly different persona, criteria set, or score assignment. The score for response $y_t^i$ from each individual-level sample is extracted:

$\mathrm{Extract}(S_{t,x}^i)$

All $m$ extracted scores are then averaged—this is the individual-level contribution to the final score.

What individual-level scaling accomplishes. By sampling multiple evaluation chains, the model explores multiple hypotheses about the user's preferences. Some chains may overemphasize certain criteria or misinterpret ambiguous signals; averaging across chains reduces the variance from any single chain. This is directly analogous to majority voting or best-of-N in standard LLM evaluation, but applied to the structured evaluation chain rather than the final answer.

Prototype-level scaling (similar-user transfer). The user's preference embedding (computed at test time using the same Qwen3-Embedding-0.6B model used during offline prototype construction) is used to assign the user to the nearest prototype. Within that prototype, the $n$ most similar users $\{u_w\}_{w=1}^n$ are selected based on embedding proximity. For each similar user $u_w$, P-GenRM generates an evaluation chain using that user's preference signals:

$(S_t^i)^{(u_w)} \sim R_\theta\big(q_t, H_t^{(u_w)}, y_t^i, P_t^{(u_w)}\big)$

where $H_t^{(u_w)}$ and $P_t^{(u_w)}$ are the preference signals of the similar user, not the current user. The model evaluates the current user's candidate responses through the lens of what the similar user would prefer. The score for each similar user's evaluation is extracted and averaged across all $n$ similar users.

What prototype-level scaling accomplishes. This is a form of collaborative filtering for reward modeling: the preferences of similar users serve as a prior for the current user's preferences. For a new user with sparse feedback (few or no historical interactions), the prototype-based inferences provide the primary signal, enabling generalization to cold-start scenarios. For an established user with ample history, the prototype signals serve as a regularizer, preventing the model from overfitting to noisy individual preference patterns.

The final score aggregation. The two granularity levels are combined through simple averaging:

$s_t^i = \frac{1}{m} \sum_{x=1}^m \mathrm{Extract}(S_{t,x}^i) + \frac{1}{n} \sum_{w=1}^n \mathrm{Extract}\big((S_t^i)^{(u_w)}\big)$

Note on the formula. The paper presents this as a sum of two averages, not an average of all $m+n$ scores. This means the individual-level and prototype-level contributions are weighted equally regardless of the values of $m$ and $n$. If $m=16$ and $n=8$, the 16 individual scores collectively contribute half the total, and the 8 prototype scores collectively contribute the other half. This is a deliberate design choice that gives prototype-level signals substantial influence even when many individual samples are generated.

Scaling configurations explored. Table 2 reports results for various $(m, n)$ combinations:

• Ind-8, Pro-4 ($m=8, n=4$): 74.30% on Chatbot Arena (up from 72.68% base, a +1.62% gain)
• Ind-16, Pro-8 ($m=16, n=8$): 75.92% (best configuration, +3.24% gain)
• Ind-32 ($m=32, n=0$): 75.59% (individual-only at high samples, but underperforms Ind-16 + Pro-8 despite more total samples)
• Ind-16, Pro-16 ($m=16, n=16$): 72.59% (overscaling at prototype level degrades performance)
• Ind-0, Pro-8 ($m=0, n=8$): 66.90% (prototype-only, underperforms the base model, indicating that prototype signals alone are insufficient without individual-level base scores)

Why prototype overscaling degrades performance. The drop at Ind-16, Pro-16 (72.59% vs. 75.92% for Ind-16, Pro-8) is attributed to the highly user-specific nature of the personalized scoring task. Incorporating too many similar users introduces noise—users who are similar in embedding space may still have meaningfully different preferences on specific queries, and aggregating their judgments can pull the score away from the current user's true preference. The optimal $n$ depends on the homogeneity of the prototype: tightly clustered prototypes can benefit from more neighbors; loosely clustered ones cannot.

Inference time cost. As reported in Table 11 (Appendix A.9), the scaling adds modest inference time. P-GenRM-8B without scaling takes 00:14:16 wall-clock time on the full Chatbot Arena-Personalized test set. The Ind-8, Pro-4 configuration takes 00:18:22 (+4:06), and Ind-16, Pro-8 takes 00:23:05 (+8:49). The paper attributes the modest increase to two factors: (1) the long input prompt (which includes 3 historical preference pairs) makes KV-cache construction the dominant cost, and this is done once per query and shared across all samples; (2) parallel sampling via the OpenAI-compatible $n$ parameter and efficient batching in vLLM allow multiple outputs to be generated with latency comparable to single-output generation. Notably, P-GenRM-8B + Ind-16, Pro-8 (00:23:05, 75.92%) achieves higher accuracy with lower inference time than the previous SOTA SynthesizeMe + FT RM-70B (01:29:59, 72.05%).

---

Summary of Design Choices and Their Justifications

• Structured evaluation chains over scalar outputs: Interpretability enables debugging and auditing; the free-text format allows the model to surface preference dimensions not captured by predefined criteria sets (e.g., Philosophical Engagement, Nuance, Sensitivity—dimensions absent from the PRISM dataset's original definitions). The text format also enables test-time scaling through parallel sampling, which a deterministic scalar model cannot support.

• Dynamic, per-query persona over static, cached persona: Preliminary experiments (Table 6, Appendix A.2) showed that scenario-dependent variability exists even within a single user (e.g., the same user wanting brevity while driving but expressiveness in casual settings). A static persona would collapse these context-dependent preferences into a single average, mis-scoring responses in roughly half the contexts. The per-query generation allows the model to interpret the same user's history differently depending on the current query's topic, tone, and implicit context.

• Three-stage training pipeline (SFT → RL with process+outcome rewards → curriculum learning) rather than end-to-end training: Each stage addresses a distinct limitation. SFT provides the basic capability (generating structured chains) through distillation from a stronger instructor, avoiding the need for expensive human-annotated chain data. RL improves chain quality by explicitly rewarding faithful preference coverage (process) and correct conclusions (outcome), addressing the SFT model's inability to distinguish good chains from mediocre ones. Curriculum learning progressively introduces hard negatives to improve robustness, while disabling process reward to prevent conservative, criteria-literal evaluations on subtle distinctions. The cumulative ablation (Table 3) validates that each stage provides non-overlapping improvements.

• Process reward as LLM-as-a-judge coverage score rather than a learned verifier: A learned process verifier (e.g., a separately trained model that scores chain quality) would require training data mapping chains to quality scores, which is expensive to collect and may not generalize to new preference dimensions. Using an LLM-as-a-judge leverages the judge model's general language understanding to evaluate how well the chain covers stated preferences, without requiring chain-specific training. The weight $\alpha = 0.5$ (versus $\beta = 1.0$ for outcome reward) keeps the process reward from dominating; excessive process weight ($\alpha = 1.0$) causes overfitting to specific criteria at the expense of overall judgment accuracy (Table 7, Appendix A.3).

• Prototype-based collaborative filtering over per-user model training or purely individual scaling: Per-user models are infeasible with only $\mathbf{h}=3$ preference pairs per user. Pure individual-level scaling (without prototypes) cannot generalize to cold-start users—if a new user has no history, individual-level sampling has nothing to condition on. Prototypes provide a transfer mechanism: new users inherit the learned preferences of similar existing users, enabling personalization from the first interaction. The number $k=50$ is chosen based on PCA singular value retention (Figure 3); more prototypes introduce noise from overly fine-grained partitioning, while fewer under-represent the diversity of preference patterns.

• Prototype refinement with dual regularization (centering + trust region) over static K-means centroids: Raw K-means centroids represent the semantic average of a cluster but are not optimized for discriminative utility (predicting which response a user will prefer). The refinement step with pairwise logistic loss makes prototypes predictive. The centering regularizer prevents prototypes from drifting into discriminatively useful but semantically unrepresentative regions (which would harm generalization to new users assigned to the prototype). The trust-region regularizer ensures smooth optimization, preventing oscillation when prototypes are iteratively updated.

• Equal-weight averaging of individual and prototype contributions (sum of two means) over weighted voting or full average: By giving the individual-level average and prototype-level average equal weight in the sum, the design ensures prototype signals have substantial influence even at small $n$ (e.g., $n=4$ contributes as much as $m=8$ individual samples). This prioritizes the collaborative filtering benefit—the prototype signal serves as a prior that regularizes the individual samples, preventing overfitting to noise in the user's limited history. A full average across all $m+n$ samples would dilute the prototype signal when $m$ is large; a weighted scheme would introduce an additional hyperparameter to tune.

• Random sampling of $\mathbf{h}=3$ historical preference pairs per query over using the most recent or most informative pairs: Three pairs provide the minimal structure needed to assess preference consistency (one pair = one-off, two pairs = insufficient pattern, Table 12 confirms three is sufficient). Random selection prevents the model from overfitting to recency bias (assuming the most recent preference is the most relevant) or selection bias (the "most informative" pairs may be systematically different from typical interactions). In practice, this means each inference call sees a potentially different subset of the user's history, contributing to the diversity of individual-level samples during test-time scaling.

4. Key Insights and Innovations

Innovation 1: Evaluation as Structured Generation Rather than Scalar Regression

The most fundamental intellectual move in this paper is reframing personalized reward modeling from a regression problem (learn a function that maps user signals and responses to a scalar) into a generation problem (learn a model that produces a textual evaluation chain containing persona analysis, criteria, and scores). This is not merely an architectural choice—it is a conceptual shift with cascading implications for interpretability, expressiveness, and scalability that scalar reward models cannot match.

What the field did before. Prior personalized reward models—PAL (Chen et al., 2024), VPL (Poddar et al., 2024), GPO (Zhao et al., 2023), and the Bradley-Terry baselines in Table 1—all operate by compressing user preferences and candidate responses into a fixed-dimensional representation, then producing a scalar score through a learned mapping. The scoring function is opaque: you can see what score was assigned but not why. If the model gives a low score to a response you believe should be high, you cannot inspect its reasoning—there is no intermediate representation to audit. The dimensionality of the preference representation is baked into the architecture (latent variables, mixture weights, prototype coefficients), meaning the model can only represent preference nuance up to its representational capacity.

What P-GenRM changes. By making reward modeling a text generation task, P-GenRM inherits the representational flexibility of natural language. The model is not limited to a fixed set of preference dimensions; it can surface any criterion expressible in text—the paper demonstrates this concretely by showing that P-GenRM's generated rubrics include dimensions like "Philosophical Engagement," "Nuance," and "Breadth of Ideas" that are absent from the PRISM dataset's predefined criteria (Section 5.2). More importantly, the generated evaluation chain is self-documenting: it produces the reasoning alongside the score, making failures diagnosable. If the model assigns the wrong score, you can trace whether the error came from mis-inferring the persona, mis-weighting criteria, or misapplying criteria to the response.

This shift also enables test-time scaling in a way scalar models cannot support. A deterministic scalar reward model produces one score per input; generating multiple scores would require architectural changes (ensembles, dropout at inference) that are not part of the standard training pipeline. A generative model, by contrast, can simply be sampled multiple times with temperature-based decoding, each sample exploring a different hypothesis about the user's preferences. The individual-level scaling mechanism (Section 4.2.2) is a direct consequence of this generative framing—it would have no analog in a scalar architecture.

Why this is fundamental rather than incremental. The paper does not claim to have invented generative reward models—Zhang et al. (2024a) and Liu et al. (2025) established the GenRM paradigm for universal evaluation. The innovation is the extension to personalization, which is non-trivial because it requires solving a problem universal GenRMs do not face: the evaluation criteria themselves must be inferred from user signals rather than being shared across all evaluations. A universal generative judge can rely on broadly agreed-upon standards (correctness, helpfulness); a personalized one must construct its rubric from sparse, noisy per-user data. The paper's contribution is showing that this inference can be done within the generation process itself—persona induction and rubric construction are not pre-processing steps but are part of the autoregressive chain—and that doing so yields substantial accuracy gains over both scalar personalized models and prompting-based generative approaches. The evidence anchors: P-GenRM-8B (72.68% on Chatbot Arena) outperforms the best scalar personalized model (SynthesizeMe + FT RM-8B, 69.78%) by 2.90 percentage points and the best in-context generative approach (o3 + PSI, 69.14%) by 3.54 percentage points (Table 1 and 2).

---

Innovation 2: Difficulty-Conditioned Training via Disentangled Process and Outcome Supervision

The paper's three-stage training pipeline is not a grab-bag of techniques but a principled progression addressing a specific diagnostic insight: personalized evaluation requires both faithful reasoning and correct conclusions, and these two capabilities are learned best in sequence rather than jointly. The insight is made concrete through the paper's separation of process reward (does the reasoning cover the user's criteria?) from outcome reward (does the score ordering match the ground truth?), and the deliberate staging where process reward is active during RL but disabled during curriculum learning.

What the field did before. Standard reward model training (Bradley-Terry, preference optimization) optimizes solely for outcome: maximize the probability that the preferred response scores higher than the dispreferred one. The internal reasoning that produces the score is unobserved and unrewarded—the model may learn spurious shortcuts that produce correct rankings for wrong reasons (e.g., exploiting dataset-specific surface features rather than understanding user preferences). Prior personalized methods (PAL, VPL, GPO) add structural priors (prototypes, latent variables) but still train with outcome-only objectives. SynthesizeMe uses persona prompting but does not train the underlying model to produce better personas—the persona is generated once and used as static context.

What P-GenRM changes. The paper diagnoses that for personalized evaluation, outcome-only training is insufficient because the space of possible preference criteria is vast and context-dependent. A model that learns to rank responses correctly on training data without learning to articulate why will fail to generalize when the preference context shifts. The process reward—using an LLM-as-a-judge to score how well the evaluation chain covers known or synthesized user criteria—provides a training signal that directly incentivizes the model to generate faithful, criterion-aware reasoning.

The key intellectual move is not that process+outcome rewards outperform outcome alone (Table 7 confirms they do, but the margin is modest: 71.07% vs. 70.22% on Chatbot Arena). Rather, it is the staging across training phases: process reward is active during RL to build the reasoning capability, then disabled during curriculum learning to prevent it from constraining exploration on the hardest cases. This reveals a nuanced understanding of the interaction between reward signals and training difficulty—process reward helps the model learn to reason faithfully on typical examples, but on hard negatives where the distinction is subtle and may not map cleanly to explicit criteria, process reward becomes a straitjacket. The paper's decision to disable it during curriculum learning is a diagnostic move: it recognizes that the same signal that helps during early training can hurt during late training when the model needs to handle edge cases that defy clean criteria-based reasoning.

Why this is a reframing rather than just a training trick. The paper is not simply stacking RL on top of SFT—that is standard practice. The innovation is the disentanglement of the reward into two interpretable components and the stage-dependent application of those components based on an analysis of what each training phase needs to accomplish. This reframes RL for reward model training from "optimize the final output" to "first learn to reason, then learn to be robust." The evidence for this reframing comes not just from the performance numbers (Table 3) but from the diagnostic ablation pattern: removing process reward alone (keeping outcome) loses 0.85 points; removing outcome alone (keeping process) loses 2.02 points; removing both loses 4.31 points. The non-additive interaction—removing both hurts more than the sum of removing each individually—suggests the two rewards are complementary, with process reward providing a scaffolding that makes outcome optimization more effective.

---

Innovation 3: Collaborative Filtering for Reward Models via Prototype-Based Preference Transfer

The paper's prototype mechanism is, at first glance, technically straightforward: embed users, cluster them, and use cluster membership to share preference information. The innovation is not the mechanics but the recognition that personalized reward modeling in cold-start regimes is structurally identical to the collaborative filtering problem in recommendation systems, and that techniques from that mature field—specifically, prototype-based user clustering—can be adapted to the very different domain of LLM evaluation.

What the field did before. Prior personalized reward models handled new users in one of three ways, none fully satisfactory. Some methods (VPL, PAL) learn per-user latent variables or mixture weights but require at least a few interactions to estimate them—they provide no mechanism for zero-history personalization. Others (SynthesizeMe) infer a persona from whatever history is available but degrade when that history is sparse or absent—a user with zero interactions gets no persona and therefore no personalization. A third category (Bradley-Terry reward models fine-tuned on population data, majority voting) ignores individual differences entirely and optimizes for average preference, which the paper shows in Appendix A.5 under-serves minority preference groups (macro accuracy of 60.64% for BT-70B versus 65.21% for P-GenRM-8B).

What P-GenRM changes. The paper frames the cold-start problem as a transfer learning challenge: the model needs to predict a new user's preferences without any direct signal from that user. The solution—assign the user to a prototype and use the preferences of similar existing users as a proxy—is directly inspired by collaborative filtering algorithms (Goldberg et al., 1992; Sarwar et al., 2001), which the paper explicitly cites in Section 1. The transfer works because preferences are not randomly distributed across users; they cluster into archetypes (the "concise technical reader," the "warm conversational partner," the "critical devil's advocate"). By learning these archetypes during training and mapping new users to them at test time, the model can provide meaningful personalization even before the user has expressed any preferences.

Why this is more than an engineering solution. The paper validates that this transfer works not just in aggregate but specifically addresses the cold-start scenario the field has struggled with. On LaMP-QA (Table 5)—where reward models are given only sparse user history and must generalize to unseen users—P-GenRM-8B with test-time scaling achieves a Spearman correlation of 0.638, outperforming even Qwen3-235B-A22B (0.619), a model roughly 30× larger. This is striking because P-GenRM-8B was trained only on PersonalRewardBench and never saw LaMP-QA data; the generalization comes entirely from the prototype mechanism's ability to match new users to learned archetypes.

The paper also identifies a boundary condition that is as informative as the positive result: prototype overscaling degrades performance. Increasing from Pro-8 to Pro-16 drops accuracy from 75.92% to 72.59% on Chatbot Arena (Table 2). The paper attributes this to the "highly user-specific nature of the personalized scoring task"—beyond a certain number, similar users are not similar enough, and their aggregated judgments introduce noise. This is a diagnostic finding: it tells us that prototype transfer is powerful but bounded, and that the optimal neighborhood size depends on the homogeneity of the preference clusters. The paper's 50-prototype configuration (chosen via PCA singular value analysis in Figure 3) appears to hit a sweet spot where intra-prototype preferences are consistent enough for transfer but inter-prototype preferences are distinct enough to warrant separation.

---

Innovation 4: Adaptive Personas as a Resolution to the Static-vs-Dynamic Preference Tension

A recurring tension in personalized systems is whether to model users with stable dispositions (e.g., "this user prefers concise answers") or situationally adaptive behavior (e.g., "this user prefers concise answers when asking technical questions but elaborate answers when discussing philosophy"). Prior work largely defaulted to the former: SynthesizeMe infers a single persona from a user's history and reuses it across all queries; PAL and VPL learn stable per-user preference representations. The paper's preliminary experiment (Table 6, Appendix A.2) provides concrete evidence that this default is wrong: adding scenario-specific criteria (Output Style Requirements + Scoring Dimensions) to an already-persona-informed prompt improves accuracy from 64.02% to 66.17%, a 2.15-point gain that cannot be explained by the static persona alone.

What P-GenRM changes. Rather than choosing between static and dynamic modeling, P-GenRM collapses the distinction by making persona generation part of the per-query evaluation chain. Every time the model scores a response, it first generates a persona—but that persona is conditioned on the current query and available signals, not cached from a previous inference. This means the same user with the same history can receive different personas for different queries if the context warrants it. The paper's Figures 6 and 7 (Appendix A.6) show a concrete example: the same user's inferred preferences shift dramatically between a music recommendation scenario (where "openness to new experiences, creativity, and emotional resonance" dominate) and a serious discussion scenario (where "factual precision, logical coherence, and depth of analysis" dominate).

Why this is a conceptual contribution rather than just a better feature. The paper is not claiming that context-dependent preferences are a new discovery—the idea is intuitive and the paper cites prior work acknowledging it (Guan et al., 2025). The contribution is operationalizing dynamic persona inference within a generative reward model and showing that it can be learned through training rather than engineered through prompting. Table 4 demonstrates that the Persona-guided Scoring Induction (PSI) approach—which generates a fresh persona per query—consistently outperforms SynthesizeMe's static persona approach across multiple base models (Qwen3-8B: 64.22% vs. 62.57%; o3: 69.14% vs. 67.73% on Chatbot Arena). The gap persists after fine-tuning (P-GenRM-8B: 72.68% vs. SynthesizeMe + FT RM-8B: 69.78% in Table 1), confirming that training on dynamic persona generation produces better personalization than training on static personas.

The deeper implication is that test-time compute and dynamic personalization are mutually reinforcing. Individual-level scaling (multiple parallel evaluation chains) is beneficial precisely because each chain may produce a slightly different persona, exploring the space of possible preference interpretations. If the persona were static and deterministic, scaling would provide no benefit—all chains would produce identical reasoning and identical scores. The 3.24% gain from test-time scaling (72.68% → 75.92% on Chatbot Arena) is therefore not just a compute-efficiency result; it is evidence that the model's preference inferences have meaningful variance that can be reduced through aggregation, which in turn implies that per-query persona generation is genuinely non-deterministic and context-sensitive.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses three datasets: Chatbot Arena-Personalized (a subset of PersonalRewardBench with data from 131 users, derived from the Chatbot Arena platform where users engage in open-ended conversations with anonymous LLMs and provide pairwise preference judgments); PRISM-Personalized (the PRISM subset of PersonalRewardBench with data from 720 users, where participants rate multiple candidate completions on a 1–100 scale and provide fine-grained feedback on attributes such as factuality, creativity, and value alignment); and LaMP-QA (Salemi and Zamani, 2025), an out-of-distribution benchmark for personalized long-form question answering covering Arts & Entertainment, Personal Life & Development, and Society & Culture categories, used to evaluate generalization to unseen users. The PersonalRewardBench datasets were constructed by Ryan et al. (2025), who devised a data-filtering pipeline to extract challenging and highly personalizable user data. For the PRISM dataset, all $\binom{N}{2}$ comparisons from each turn are extracted to form pairwise data, and pairs with less than a 10% quality difference are removed.

• Base model(s). Experiments use LLaMA-3.1-8B and LLaMA-3.1-70B as the base models. The 8B model is trained with full fine-tuning, while the 70B model is trained using LoRA (Hu et al., 2022) to manage computational cost. These models were chosen to evaluate P-GenRM's effectiveness across different scales and to enable fair comparison with prior work that used the same model family. The 8B/70B experiments utilize 8/32 GPUs respectively. Additional experiments use Qwen3-8B as a base model for the adaptive-vs-static persona comparison (Table 4). For the LaMP-QA evaluation, six LLMs (Qwen3-8B, Qwen3-32B, Qwen-235B-A22B, GPT-5, GPT-4o, and Gemini-2.5-pro) are used to generate candidate responses, and three highly-advanced LLMs (Gemini-2.5-pro, Claude-3.7-Sonnet, and GPT-4o) serve as judges to establish ground-truth rankings.

• Metrics. The primary metric throughout is accuracy (%) — the fraction of test instances where the reward model correctly ranks the chosen response above the rejected response. This is reported as "mean ± standard error" over 5 independent runs on both Chatbot Arena-Personalized and PRISM-Personalized. For the LaMP-QA out-of-distribution evaluation, the metric is Spearman's rank correlation coefficient between the reward model's response rankings and ground-truth rankings (established by aggregating scores from three advanced judge LLMs). In Appendix A.5, the paper additionally reports macro accuracy, where accuracy is computed separately for each prototype group and then averaged across all groups, ensuring minority preference groups are equally weighted rather than dominated by majority groups.

• Baselines. The paper evaluates against multiple categories of baselines, all summarized in Table 1:

  • In-Context LLM as a Judge (no fine-tuning): (i) Default prompt without personalization; (ii) + CoT (Chain-of-Thought reasoning); (iii) + Demographics (basic demographic information appended); (iv) + Preference History (the user's historical preference pairs provided as context); (v) + SynthesizeMe (static persona inferred from history, per Ryan et al., 2025); (vi) + Persona-guided Scoring Induction (the paper's own PSI prompt, but applied in-context rather than fine-tuned). These baselines use LLaMA-3.1-8B and LLaMA-3.1-70B as the base models.
  • Finetuned Reward Models: A Bradley-Terry Reward Model (Bradley and Terry, 1952) trained on unfiltered PersonalRewardBench data to fit the overall preference distribution of the population.
  • Existing Personalized Reward Models: GPO (Group Preference Optimization, Zhao et al., 2023), VPL (Variational Preference Learning, Poddar et al., 2024), PAL (Pluralistic Alignment Framework, Chen et al., 2024), and SynthesizeMe (Ryan et al., 2025) combined with a fine-tuned reward model. All are implemented at both 8B and 70B scales for fair comparison.
  • Proprietary Model: OpenAI o3 evaluated both with standard prompting and with Persona-guided Scoring Induction prompting (Table 2).

• Generation budget / compute accounting. Test-time compute is measured in terms of scaling operations: $m$ individual-level parallel samples and $n$ prototype-level similar-user evaluations. The total scaling cost is $m + n$ forward passes through P-GenRM. The paper reports configurations such as Ind-8, Pro-4 ($m=8, n=4$), Ind-16, Pro-8 ($m=16, n=8$), and Ind-32 ($m=32, n=0$). A single evaluation chain generation constitutes one unit. The paper does not report FLOP counts but provides wall-clock inference times (Table 11, Appendix A.9) measured on the full Chatbot Arena-Personalized test set using vLLM on 8 NVIDIA A100 GPUs, showing that the Ind-16, Pro-8 configuration takes 00:23:05 compared to 00:14:16 for P-GenRM-8B without scaling.

• Cross-validation / statistical protocol. All results are reported as "mean ± standard error" over 5 independent runs to account for the inherent stochasticity in LLM generation (different sampling runs may produce different evaluation chains). This is applied uniformly to both P-GenRM and all baselines. For the policy model training evaluation (Appendix A.12), 95% confidence intervals are additionally reported to assess statistical significance. No explicit cross-validation protocol is described for hyperparameter selection on PersonalRewardBench; the paper appears to use the standard test set evaluation directly. For LaMP-QA, all reward model scoring is normalized by repeating the scaling procedure eight times and taking the average to enhance stability.

Main Quantitative Results

Personalized Alignment Performance on PersonalRewardBench

The headline result from Table 1: P-GenRM-8B achieves 72.68% accuracy on Chatbot Arena-Personalized and 65.32% on PRISM-Personalized, while the P-GenRM-8B with Ind-16, Pro-8 test-time scaling achieves 75.92% on Chatbot Arena and 68.06% on PRISM — establishing new state-of-the-art results on both benchmarks.

The full comparison across model scales reveals several patterns worth examining:

In-context methods plateau below fine-tuned methods. The best in-context configuration (LLaMA-3.1-8B + Persona-guided Scoring Induction) reaches 62.20% on Chatbot Arena and 58.33% on PRISM. The best fine-tuned baseline (SynthesizeMe + FT RM-8B) reaches 69.78% and 62.84% respectively — a gap of 7.58 and 4.51 percentage points. This gap represents what cannot be recovered through prompting alone and validates the need for training.

P-GenRM outperforms all prior personalized methods at both scales. At 8B, P-GenRM beats the best prior method (SynthesizeMe + FT RM-8B) by 2.90 points on Chatbot Arena (72.68% vs. 69.78%) and 2.48 points on PRISM (65.32% vs. 62.84%). At 70B (LoRA-trained), P-GenRM achieves 73.42% on Chatbot Arena and 66.21% on PRISM, surpassing SynthesizeMe + FT RM-70B by 1.37 and 2.47 points respectively. The paper reports these as an average improvement of 2.77% on the 8B model and 1.99% on the 70B model over prior SOTA, which the abstract rounds to a 2.31% average across scales.

P-GenRM-8B surpasses the 70B prior SOTA. P-GenRM-8B (72.68%) outperforms SynthesizeMe + FT RM-70B (72.05%) by 0.63 points on Chatbot Arena, and on PRISM, P-GenRM-8B (65.32%) similarly exceeds the 70B prior SOTA (63.74%) by 1.58 points. The paper notes this as an average of 1.04% improvement over the previously best-performing 70B model. This is practically significant because it means the computational cost of a 70B model is not necessary to achieve top personalized reward modeling performance.

Bradley-Terry baselines are surprisingly competitive but limited. The Bradley-Terry reward model fine-tuned on unfiltered population data achieves 67.21% (8B) and 71.12% (70B) on Chatbot Arena — substantially better than the in-context methods and competitive with some personalized methods. However, the paper's macro accuracy analysis in Appendix A.5 (Table 10) reveals a different picture: BT-70B achieves only 60.64% macro accuracy versus P-GenRM-8B's 65.21%, a 4.57-point gap. This confirms that population-optimized models under-serve preference minorities, and personalized methods recover this loss.

Existing personalized reward models (GPO, VPL, PAL) underperform even the Bradley-Terry baseline. GPO achieves 57.87–59.16%, VPL achieves 58.12–59.70%, and PAL achieves 56.74–59.40% — all substantially below the BT baseline and the SynthesizeMe approaches. The paper does not provide extensive analysis of why these methods underperform in this setting, but the implication is that fixed-dimensional preference representations (latent variables, mixture weights) are insufficient compared to methods that condition on richer preference context (SynthesizeMe's personas, P-GenRM's evaluation chains).

Test-time User-based Scaling Results

The second major set of results comes from Table 2, which reports the effect of different scaling configurations on P-GenRM-8B:

Individual-level scaling alone provides diminishing returns. Ind-8 achieves 73.61% (+0.93 over base), Ind-16 achieves 73.87% (+1.19), and Ind-32 achieves 75.59% (+2.91). The jump from 16 to 32 individual samples yields a larger gain than from 8 to 16, suggesting that the benefit of individual scaling is not monotonic and may depend on reaching a critical sample mass.

Prototype-level scaling amplifies individual scaling efficiently. The best overall result (Ind-16, Pro-8: 75.92%) outperforms Ind-32 (75.59%) despite using fewer total forward passes (24 vs. 32), while the Ind-8, Pro-4 configuration (74.30% with 12 passes) outperforms Ind-16 (73.87% with 16 passes). This demonstrates that prototype-level information is not merely additive but complementary — it provides signal that individual sampling alone cannot recover.

Prototype-only scaling underperforms the base model. Ind-0, Pro-8 achieves only 66.90% on Chatbot Arena, which is below P-GenRM without any scaling (72.68%). This confirms that prototype-level signals serve best as a supplement to individual-level evaluation, not as a replacement. Without the user's own preference signals, similar-user transfer lacks the anchoring needed to make accurate judgments.

Prototype overscaling degrades performance. Ind-16, Pro-16 drops to 72.59%, which is actually below the base P-GenRM (72.68%). This negative result — that too many similar-user references hurt rather than help — is attributed to the user-specific nature of the task: beyond a certain neighborhood size, the "similar" users are not similar enough, and their preferences introduce conflicting signals.

Comparison with proprietary models. OpenAI o3 with the best prompting (o3 + PSI) achieves 69.14% on Chatbot Arena and 63.87% on PRISM. P-GenRM-8B without scaling (72.68% / 65.32%) already substantially outperforms this, and the Ind-16, Pro-8 configuration (75.92% / 68.06%) extends the lead to 6.78 and 4.19 points respectively. This is noteworthy because o3 is a substantially larger and more capable base model than LLaMA-3.1-8B, suggesting that the architectural choices (structured evaluation chains, three-stage training, test-time scaling) compensate for base model capability.

Out-of-Distribution Generalization on LaMP-QA

Table 5 reports Spearman correlation on LaMP-QA under sparse-feedback conditions, where reward models trained only on PersonalRewardBench are given limited historical interactions and evaluated on ranking responses from six generation models:

P-GenRM-8B with test-time scaling (Ind-8, Pro-4) achieves the highest average correlation (0.638), outperforming:

• Qwen3-235B-A22B (0.619), a model ~30× larger
• LLaMA-3.1-70B (0.600)
• SynthesizeMe-8B (0.581)
• LLaMA-3.1-8B (0.524)

The performance varies across LaMP-QA's three categories: P-GenRM achieves 0.543 (Arts & Entertainment), 0.714 (Personal Life & Development), and 0.657 (Society & Culture). The particularly large gap on Personal Life & Development (0.714 vs. 0.657 for Qwen3-235B-A22B) suggests that prototype-based transfer is especially effective when preference patterns within clusters are consistent and informative.

The generalization result is especially strong because P-GenRM was never trained on LaMP-QA data. The model was trained solely on PersonalRewardBench (Chatbot Arena + PRISM) and evaluated on LaMP-QA with sparse user histories — a true out-of-distribution test. The fact that it outperforms much larger models that were also not trained on LaMP-QA demonstrates that the prototype mechanism successfully transfers learned preference archetypes to unseen users and domains.

P-GenRM for Policy Model Training

Table 14 (Appendix A.12) reports the downstream performance of policy models trained with P-GenRM as the reward signal:

Llama-3.1-8B-Instruct fine-tuned with GRPO using P-GenRM achieves a mean score of 3.354 (on a 1–5 scale judged by GPT-4o, Claude-Sonnet-4, and Gemini 2.5-Pro), compared to:

• Llama-3.1-8B-Instruct (base): 2.954
• Llama-3.1-70B-Instruct: 3.156
• Qwen2.5-72B-Instruct: 3.214
• Llama-3.1-8B-Instruct-DPO (trained with P-GenRM as implicit reward): 3.316

The 95% confidence intervals for the GRPO-trained policy model ([3.334, 3.374]) and DPO-trained model ([3.303, 3.329]) have lower bounds exceeding the upper bounds of both 70B-scale models (3.174 and 3.228), confirming that the 8B policy models trained with P-GenRM significantly outperform 70B baselines — there is no interval overlap, establishing statistical significance.

This result closes the loop: P-GenRM is not just an evaluation tool but an effective training signal. An 8B policy model optimized against P-GenRM's personalized reward signal produces responses that three independent advanced judges rate as more personalized than those from 70B base models, demonstrating that P-GenRM captures preference signal that translates to downstream behavior.

Ablation Studies and Robustness Checks

Training stage ablation (Table 3). Removing curriculum learning reduces Chatbot Arena accuracy from 72.68% to 71.07% (−1.61 points) and PRISM from 65.32% to 63.82% (−1.50 points). Removing both curriculum learning and process reward further reduces Chatbot Arena to 70.22% (−2.46 from full) and PRISM to 62.70% (−2.62). Removing curriculum learning and outcome reward (keeping process reward only) drops to 69.05% (−3.63) and 60.94% (−4.38), which is substantially worse than process-reward-only, suggesting that outcome correctness provides the stronger signal for this task. Removing the entire RL stage (keeping only SFT) drops to 66.76% (−5.92) and 57.08% (−8.24), the single largest ablation effect. Removing all training (base in-context model) yields 56.37% and 52.04% — the full P-GenRM pipeline thus accounts for 16.31 and 13.28 percentage points of improvement over the untrained baseline.

Alpha-beta weight sensitivity (Table 7, Appendix A.3). The paper sweeps $(\alpha, \beta)$ combinations for the composite RL reward. The chosen $(0.5, 1.0)$ achieves 71.07% (Chatbot Arena) and 63.82% (PRISM). Equal weights $(0.5, 0.5)$ are slightly worse (70.65%, 63.33%). Removing outcome reward $(1.0, 0.0)$ drops to 69.05% and 60.94% — consistent with the Table 3 ablation showing outcome reward matters more. Removing process reward $(0.0, 1.0)$ drops to 70.22% and 62.70%. The non-additive interaction — removing either alone hurts less than the sum of their individual contributions would predict — confirms the two reward signals are complementary and partially overlapping.

Adaptive vs. static persona (Table 4). This ablation proves critical to the paper's central claim about dynamic preference inference. On Qwen3-8B, Persona-guided Scoring Induction (PSI, the paper's dynamic method) achieves 64.22% (Chatbot Arena) and 58.01% (PRISM), while SynthesizeMe (static persona) achieves 62.57% and 56.33% — gaps of 1.65 and 1.68 points. On the o3 instructor model, the gap widens: 69.14% vs. 67.73% on Chatbot Arena (+1.41) and 63.87% vs. 58.49% on PRISM (+5.38). The larger gap on PRISM — the dataset with more diverse and culturally varied users (720 users vs. 131) — supports the claim that dynamic personas particularly benefit heterogeneous user populations where static profiles collapse meaningful preference variation.

Prototype number sensitivity (Figure 3, right; Table 13, Appendix A.11). With the Ind-8, Pro-4 setting, Chatbot Arena accuracy improves from 72.68% (0 prototypes, i.e., no prototype scaling) to 73.23% (25 prototypes) to 74.30% (50 prototypes), then declines to 73.69% (100 prototypes) and 73.45% (125 prototypes). PRISM follows a similar pattern: 65.32% → 66.29% → 67.54% → 67.28% → 67.19%. The performance peak at 50 prototypes across both datasets validates the PCA-based selection method (which showed diminishing singular value returns beyond 50), and the degradation beyond 50 supports the paper's claim that overly fine-grained clustering introduces inference noise from small, unrepresentative prototype groups.

Preference sample size analysis (Table 12, Appendix A.10). P-GenRM-8B trained with 1 preference pair achieves 59.78% accuracy; 2 pairs reach 64.62%; 3 pairs reach 72.68%; 4 pairs reach 72.50%. The jump from 2 to 3 pairs is substantial (+8.06 points) while 3 to 4 is essentially flat (−0.18). This validates the paper's choice of $\mathbf{h}=3$ as the minimum number needed to establish preference consistency, with the authors' interpretation that 1 pair is a one-off decision, 2 are insufficient to form a pattern, and 3 provide the minimal structure for reliable inference.

Macro accuracy across prototypes (Figure 5, Tables 8–10, Appendix A.5). The PRISM dataset exhibits a pronounced long-tail distribution of user groups (minimum 18 samples per prototype, maximum 930, median 105). Despite this, P-GenRM-8B achieves prototype-level macro accuracy of 65.21% with only 0.11% difference from sample-level average accuracy (65.32%), indicating the model does not overfit to majority personas. Dispersion metrics (median 0.6500, standard deviation 0.0401, IQR 0.0544) show consistent performance across both large and small groups. Compared to baselines, P-GenRM's macro accuracy (65.21%) substantially exceeds LLaMA-3.1-70B (58.27%), o3 (63.33%), fine-tuned BT-70B (60.64%), and SynthesizeMe 70B (61.51%), with the 3.70-point gap over the best baseline (o3 at 63.33%) larger than the gap in standard accuracy, confirming that personalized methods disproportionately benefit minority preference groups.

Inference time scaling (Table 11, Appendix A.9). P-GenRM-8B inference time scales from 00:14:16 (no scaling) to 00:18:22 (Ind-8, Pro-4) to 00:23:05 (Ind-16, Pro-8). The increase is sublinear relative to the number of forward passes because: (1) the long input prompt makes KV-cache construction the dominant cost, and this is done once and shared; (2) parallel sampling and efficient vLLM batching allow multiple outputs to be generated with latency comparable to single-output generation. Notably, P-GenRM-8B + Ind-16, Pro-8 (00:23:05) achieves higher accuracy (75.92%) than the prior SOTA SynthesizeMe + FT RM-70B (01:29:59, 72.05%) with roughly 4× lower inference time, demonstrating that the approach is not just more accurate but more efficient.

Broader preference space visualization (Figures 6–7, Appendix A.6). The paper provides qualitative evidence through a non-cherry-picked user example showing distinct preference patterns: in a music recommendation scenario, the model infers preferences for "openness to new experiences, creativity, and emotional resonance"; in a serious discussion scenario, it infers "factual precision, logical coherence, and depth of analysis." These figures are not quantitative ablations but serve as existence proofs that the model's dynamic persona generation captures scenario-dependent preference shifts — a claim that would be difficult to validate through aggregate metrics alone.

Critical Assessment

Claim 1: P-GenRM is the first personalized generative reward model and achieves state-of-the-art results on personalized reward benchmarks.

The experiments in Table 1 provide strong support. P-GenRM-8B achieves 72.68% (Chatbot Arena) and 65.32% (PRISM), outperforming all baselines including the prior SOTA SynthesizeMe + FT RM across both model scales. The average improvement of 2.31% over prior SOTA is consistent across benchmarks and model scales. The "first personalized generative reward model" claim is more about positioning — the paper is the first to apply the GenRM paradigm to personalization — and is supported by the literature review showing no prior work combining these concepts.

However, the gap between P-GenRM and the Bradley-Terry population baseline (72.68% vs. 67.21% at 8B on Chatbot Arena) is 5.47 points, which while statistically significant, is not enormous given the substantially greater complexity of P-GenRM (three-stage training, generative evaluation chains, test-time scaling infrastructure). A practitioner might ask whether the added engineering complexity justifies 5.47 points of accuracy on this specific benchmark. The paper's stronger argument is the macro accuracy result (Table 10: 65.21% vs. 60.64%), which shows P-GenRM's gains are disproportionately concentrated on underserved preference minorities — a point that matters ethically but may not be captured by standard accuracy metrics in most deployment contexts.

Claim 2: Test-time user-based scaling provides an additional 3% boost, demonstrating personalized alignment with test-time scalability.

Supported with nuance. The best result (Ind-16, Pro-8, 75.92% on Chatbot Arena) is 3.24 points above P-GenRM without scaling (72.68%), consistent with the "3% boost" claim. However, the paper does not report confidence intervals for the unscaled base P-GenRM separately from the baselines, and the standard errors for both are overlapping at ±1.85% (base) and ±1.70% (Ind-16, Pro-8). A formal statistical test of the difference would require paired comparisons across the same 5 runs, which are not provided.

More concerning: the scaling results are only reported for the 8B model (Table 2), not for the 70B model. The 70B results in Table 1 show "—" for all scaling configurations, with no explanation for why. If test-time scaling does not benefit the 70B model (e.g., because larger models have lower sampling variance, or because LoRA training reduces the diversity needed for effective scaling), this would significantly limit the claim's generality. If it was simply not run due to computational constraints, this should be stated explicitly.

Additionally, the scaling benefit varies substantially by benchmark. On Chatbot Arena, Ind-16, Pro-8 achieves +3.24 over base. On PRISM, the same configuration achieves +2.74 over base (68.06% vs. 65.32%). The gap is small but consistent, and the paper does not analyze why PRISM benefits less — potentially because PRISM has more diverse users (720 vs. 131), making prototype-level transfer less reliable per user, or because PRISM's predefined criteria already provide richer signals that reduce the marginal benefit of additional scaling.

Claim 3: P-GenRM exhibits strong generalization to new users with sparse feedback, demonstrated on the out-of-distribution LaMP-QA benchmark.

The LaMP-QA results (Table 5) are genuinely impressive for a model trained only on PersonalRewardBench. Achieving 0.638 Spearman correlation versus 0.619 for a ~30× larger model is strong evidence of transfer. However, several aspects of the evaluation framework merit scrutiny.

First, the ground-truth rankings on LaMP-QA are established by aggregating scores from three advanced LLM judges (Gemini-2.5-pro, Claude-3.7-Sonnet, GPT-4o). These judges are not human — they are themselves models with their own biases. The paper's claim about personalization quality is therefore relative to what these judges consider personalized, not to actual human user satisfaction. This is a limitation of the benchmark rather than the paper, but it means the LaMP-QA results should be interpreted as "P-GenRM's rankings agree with advanced LLM judges more than baselines do," which is a weaker claim than "P-GenRM provides better personalization for real users."

Second, the sample size is unclear. The paper reports Spearman correlation across three categories but does not state how many users or queries per category are evaluated. A Spearman correlation computed over, say, 10 queries has wide confidence intervals; over 100 queries it is more reliable. Without this information, the statistical reliability of the LaMP-QA results cannot be assessed.

Third, P-GenRM is the only model in the LaMP-QA comparison that uses test-time scaling (Ind-8, Pro-4). The other baselines (Qwen3 models, LLaMA models, SynthesizeMe) are evaluated without any scaling. A fairer comparison would give baselines some form of aggregation (e.g., majority voting across multiple runs) — though implementing this for reward models that output scalar scores rather than generative chains is not straightforward, which itself highlights P-GenRM's architectural advantage.

Claim 4: The three-stage training pipeline (SFT → RL → Curriculum Learning) is necessary, with each stage contributing non-overlapping improvements.

The cumulative ablation in Table 3 supports this for the specific ordering tested. However, the paper does not explore alternative orderings (e.g., curriculum learning before RL, or RL before SFT), nor does it ablate the SFT stage alone versus the RL stage alone (all ablations remove curriculum learning first, then components of RL, then all RL, then both RL and SFT). This means we cannot determine from Table 3 whether RL without SFT would achieve comparable performance — the SFT stage might be providing an essential initialization that RL alone cannot recover from.

The paper also does not report whether the SFT-only model benefits from test-time scaling. If SFT-only + scaling matched or approached full P-GenRM + scaling, it would suggest the RL and curriculum learning stages are less important than the scaling mechanism itself. This experiment would have been informative for isolating the contribution of the training pipeline versus the inference-time mechanism.

Missing experiments that would strengthen the paper:

• Test-time scaling on the 70B model. The absence of these results (Table 1 shows "—" for all scaling configurations on 70B) is the most significant gap. Without them, we cannot know whether scaling benefits are architecture-dependent or scale-dependent.

• Per-difficulty or per-user-history-length analysis. The paper would benefit from breaking down accuracy by the amount of user history available (0 interactions, 1 interaction, 3 interactions) to directly validate the cold-start claim. Does P-GenRM's advantage over baselines grow as history length shrinks? This is the core use case for prototypes.

• Human evaluation of the generated evaluation chains. The paper claims interpretability as a benefit of the generative format, but the only evaluation is accuracy of the final pairwise judgment. An experiment where human raters assess the quality, relevance, and faithfulness of the generated personas and criteria would substantiate the interpretability claim.

• Comparison with ensemble-based baselines. Since test-time scaling is fundamentally an ensemble method (multiple evaluation chains aggregated), a fair baseline would be an ensemble of Bradley-Terry reward models or SynthesizeMe models with different random seeds. The paper does not include such baselines, making it impossible to determine whether the scaling benefit comes from the generative format specifically or from ensembling generally.

• Ablation of the prototype refinement process. The paper describes a sophisticated prototype optimization procedure (history-aware attention, pairwise loss, dual regularization). An ablation comparing refined prototypes versus raw K-means centroids would reveal whether this complexity is necessary.

• Runtime and memory profiling beyond wall-clock time. Table 11 reports inference time but not GPU memory usage or throughput (queries per second). For practitioners deciding whether to deploy P-GenRM, these practical metrics matter as much as accuracy.

Where the claims genuinely hold: P-GenRM does outperform prior personalized reward models on PersonalRewardBench, and test-time scaling does improve performance beyond the base model. The prototype mechanism does enable generalization to unseen users on LaMP-QA. The macro accuracy analysis does show that personalized methods benefit minority preference groups.

Where the claims need qualification: The "3% boost" applies to the specific Ind-16, Pro-8 configuration on the 8B model and may not generalize to other scales. The LaMP-QA results demonstrate agreement with LLM judges, not necessarily with human users. The training pipeline's necessity is supported only for the specific ordering tested. The interpretability benefit is asserted but not directly evaluated.

The most robust finding: Prototype-based transfer works for personalization. The combination of the LaMP-QA generalization results (Table 5), the macro accuracy analysis showing benefits for minority groups (Table 10), and the prototype overscaling result (Ind-16, Pro-16 degrades performance, Table 2) paints a consistent picture: learned user archetypes enable meaningful preference transfer, but only when the archetypes are properly sized and not overused. This is a genuinely novel finding with practical implications for any system that needs to personalize with limited per-user data.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in Headline Efficiency Claims

The assumption or constraint. The test-time user-based scaling mechanism depends on an offline phase that constructs user preference embeddings, clusters them into prototypes, and refines those prototypes through iterative optimization. This phase is computationally non-trivial: it requires generating cross-scenario preference analyses $P_t^{(u)}$ for every user in the training set using the fully trained P-GenRM, embedding those analyses via Qwen3-Embedding-0.6B, running K-means clustering over the full embedding matrix, and then performing the history-aware attentive prototype refinement with pairwise loss and dual regularization. None of this cost is included in the reported inference-time comparisons.

The paper is aware of this gap but treats it as outside scope. Section 5.1 describes the prototype initialization and optimization as part of the "offline" pipeline, and Section 4.2.1 details the multi-step refinement procedure, but nowhere does the paper quantify the computational cost of this phase—not in FLOPs, not in GPU-hours, not in wall-clock time relative to inference. Table 11 (Appendix A.9) reports inference-time costs at various scaling configurations, but these measurements start after the prototype infrastructure is already built. The paper does not claim this cost is negligible; it simply does not measure it.

The consequence. The reported gains from test-time scaling are computed assuming the prototype infrastructure is available at zero cost. In a realistic deployment, the total cost of personalization is: (1) train P-GenRM (three-stage pipeline), (2) run the offline prototype construction and refinement pipeline on the entire user base, (3) serve inference with scaling. The headline "3% boost from test-time scaling" and the efficiency comparisons versus larger models (e.g., P-GenRM-8B outperforming SynthesizeMe + FT RM-70B while using less inference time) omit cost (2). For a system with many users, the per-user amortized cost of prototype construction may be small. For a system with few users, or one that must frequently update prototypes as the user base changes, this cost could dominate. The paper provides no guidance on when the offline cost is worth paying.

What evidence exists in the paper. The paper does not report any measurement of prototype construction cost. The only proxy is the inference-time measurements in Table 11, which show that a single forward pass of P-GenRM-8B takes approximately 14 minutes on the full Chatbot Arena-Personalized test set (131 users). The offline phase requires generating preference analyses for every user across all their scenarios, which for the training set (far larger than 131 users) would be substantially more expensive. The prototype refinement itself (Equations 4–7 in Section 4.2.1) involves attention computation over historical records, pairwise loss optimization, and iterative reassignment, but the paper provides no scale estimates—number of iterations, convergence time, or memory requirements for the full embedding matrix.

Mitigation status. The paper does not attempt to mitigate this limitation. The offline phase is described as a necessary prerequisite for test-time scaling, and the cost is not discussed or amortized. The paper also does not analyze whether simpler, cheaper prototype construction methods (e.g., using only SFT-model outputs rather than full P-GenRM outputs, or skipping the refinement step and using raw K-means centroids) would achieve comparable test-time scaling benefits. The prototype refinement ablation—comparing performance with refined prototypes versus raw K-means centroids—is a notable omission that would directly address whether this cost is justified.

---

Hard Problems Remain Effectively Unsolved: No Mechanism for Genuinely Novel Preference Inference

The assumption or constraint. P-GenRM personalizes by conditioning on a user's historical preference signals $H_t^{(u)}$ (past pairwise choices) and any explicit criteria $E^{(u)}$. The prototype mechanism extends this by borrowing signals from similar users. Both mechanisms assume that the information needed to correctly evaluate a response is present, at least implicitly, in the available signals—either the user's own history or the history of prototype neighbors. The model can infer, interpolate, and aggregate, but it cannot create preference knowledge that is absent from all available signals.

The paper's problem formulation (Section 3) formalizes this dependency: the model generates $[P_t^{(u)}; S_t^{(u)}] \sim R_\theta(q_t, H_t^{(u)}, E^{(u)}, y_t^i)$. When $H_t^{(u)}$ is empty (a genuinely cold-start user with no history assigned to a prototype that is not genuinely representative) and $E^{(u)}$ is absent, the model has no signal to condition on. The prototype mechanism mitigates this by providing a prior from similar users, but the prior is only useful to the extent that the user genuinely resembles their assigned prototype. If a user's preferences are genuinely novel—not well-represented by any learned prototype—the system has no mechanism to discover this or adapt.

The consequence. The system will silently produce confident but incorrect personalization for users whose preferences fall outside the learned prototype distribution. The generated evaluation chain will present a plausible-looking persona and criteria analysis, and the extracted score will be precisely computed, but both will be wrong in ways the system cannot detect. This is an unknown-unknown failure mode: the model does not know when it does not know. A traditional recommendation system might flag low-confidence predictions; P-GenRM has no calibrated uncertainty estimate.

More concretely, Appendix A.5 shows that the PRISM dataset has a pronounced long-tail distribution of user groups (minimum 18 samples per prototype, maximum 930, median 105). The smallest prototype groups—those with the least data for the refinement process to learn from—are exactly the ones where the prototype prior is least reliable. For users assigned to these small prototypes, the prototype-level scaling signal may be actively misleading, pulling the score toward the average of a small, potentially unrepresentative reference group. The paper's prototype overscaling result (Ind-16, Pro-16 degrades performance, Table 2) can be read as partial evidence: even for users with established prototypes, too many neighbor references hurt. For users with poorly estimated prototypes, even a few references may hurt.

What evidence exists in the paper. The LaMP-QA generalization experiment (Table 5) tests out-of-distribution generalization but evaluates aggregate correlation, not per-user or per-prototype breakdown. We cannot see whether P-GenRM's performance on LaMP-QA is uniformly good or driven by users who happen to resemble PersonalRewardBench prototypes, with poor performance on genuinely novel users masked by averaging. The prototype-wise accuracy analysis in Appendix A.5 (Figure 5, Tables 8–9) shows consistent performance across prototypes of different sizes (median 0.6500, standard deviation 0.0401, IQR 0.0544), but this is measured on PRISM test data, which is drawn from the same distribution as the training data used to build the prototypes. It does not test generalization to users with genuinely novel preference patterns not represented in the prototype set.

Mitigation status. The paper does not address this limitation. There is no mechanism for detecting out-of-distribution users, no uncertainty quantification, no fallback to population-level scoring when prototype confidence is low. The prototype overscaling result suggests the authors are aware that prototype transfer has boundaries, but they frame this as a tuning issue (find the right $n$) rather than a fundamental capability bound. The paper also does not explore whether increasing the number of prototypes beyond 50 (Figure 3) could capture more fine-grained preference patterns and reduce the novelty problem, though the degradation at 100 prototypes suggests this approach has limits.

---

Single Model Family and Benchmark Suite: Generalization to Other Architectures, Domains, and Preference Granularities Is Unvalidated

The assumption or constraint. All experiments use LLaMA-3.1 (8B and 70B) as the base model family for P-GenRM training and evaluation. The primary benchmarks—Chatbot Arena-Personalized and PRISM-Personalized—are both subsets of PersonalRewardBench, constructed by the same authors (Ryan et al., 2025) using a shared data-filtering pipeline. Both benchmarks evaluate pairwise preference judgments on open-ended conversational responses. The LaMP-QA out-of-distribution test uses long-form question answering, a related but distinct task, but uses the same base model (LLaMA-3.1-8B for P-GenRM) and the same training data (PersonalRewardBench only).

The paper does not test P-GenRM on other base model families (e.g., Qwen, Mistral, Gemma) except for a limited in-context comparison in Table 4 where Qwen3-8B is used as a base for the adaptive-vs-static persona ablation. No full P-GenRM training is reported on non-LLaMA models. The paper does not test on other types of preference expression—cardinal ratings versus pairwise judgments, structured preference surveys versus organic interaction history, fine-grained aspect ratings versus holistic preferences—despite the PRISM dataset containing rich cardinal rating and attribute-level feedback that the paper reduces to pairwise comparisons.

The consequence. The core architectural choices—structured evaluation chain format, three-stage GRPO-based training, prototype construction via Qwen3-Embedding-0.6B—may depend on properties of LLaMA-3.1 that do not transfer. LLaMA-3.1 models may have particular strengths or weaknesses in generating structured text formats, following multi-step instructions, or learning from process-level reward signals that other model families do not share. Without replication on at least one other model family, practitioners cannot assess whether P-GenRM's gains are methodological (the techniques work generally) or model-specific (the techniques compensate for LLaMA-3.1-specific weaknesses).

Similarly, the PersonalRewardBench benchmarks are constructed through a specific filtering pipeline that extracts "challenging and highly personalizable" data. The characteristics of this filtered data—average conversation length, diversity of topics, noisiness of user feedback, prevalence of explicit versus implicit preference signals—may not match the preference data available in other deployment contexts. A system trained and evaluated on PersonalRewardBench may perform differently on organic, unfiltered user feedback from a different platform.

What evidence exists in the paper. The only cross-model-family evidence is Table 4, where PSI prompting is applied to both Qwen3-8B and o3 and shows consistent improvement over SynthesizeMe. This validates the dynamic persona idea across models but does not validate the full P-GenRM training pipeline (SFT + RL + curriculum learning + test-time scaling) on non-LLaMA architectures. The LaMP-QA results (Table 5) show generalization to a different dataset but with the same base model and training data. The paper does not report any experiment where P-GenRM is trained on one dataset and evaluated on a held-out dataset with different characteristics (e.g., trained on Chatbot Arena, tested on PRISM, or vice versa).

Mitigation status. The paper does not claim broader model-family or domain generalization. The authors are transparent about their experimental scope—they use LLaMA-3.1 as the base, PersonalRewardBench as the primary benchmark, and LaMP-QA for OOD testing—but do not discuss how these choices bound the generality of their findings. The Related Work section (Section 2) cites generative reward models across different domains (code verification, stepwise reasoning) but does not position P-GenRM within that broader landscape or discuss domain-transfer considerations.

---

The Revision and Search Analogy from Standard Test-Time Scaling Papers Does Not Apply: Scaling Only Reduces Variance, Not Bias

The assumption or constraint. The paper frames test-time user-based scaling as improving personalization accuracy through aggregation—individual-level parallel sampling reduces variance in the inferred preferences, and prototype-level incorporation adds a transfer signal from similar users. This framing implicitly assumes that the individual evaluation chains generated by P-GenRM are unbiased estimates of the true personalized score, and that aggregation improves accuracy by reducing variance around that unbiased mean.

This assumption is not tested. In standard test-time scaling for reasoning tasks (e.g., majority voting, best-of-N, beam search with verifiers), the base model generates multiple candidate solutions, and aggregation selects the best one. The improvement comes from exploring a space of possible answers where at least some are correct. If the base model has zero probability of generating a correct answer (the hardest difficulty bin in the reference example paper), no amount of scaling helps—the proposal distribution has no correct support.

P-GenRM's scaling mechanism is different: it does not search for a correct answer among candidates; it averages numerical scores from multiple evaluation chains. Averaging reduces variance if the chains are noisy but unbiased, but it cannot correct systematic bias. If P-GenRM systematically misinterprets a certain type of preference (e.g., consistently underweights creativity when evaluating technical users), generating 16 evaluation chains and averaging them will produce 16 consistently wrong scores whose average is still wrong. The paper provides no analysis of whether P-GenRM's errors are dominated by variance (which scaling helps) or bias (which it does not).

The consequence. The test-time scaling benefit may be saturated at the reported configurations. If the remaining error is predominantly bias rather than variance, additional scaling—more individual samples, more prototype references—will yield diminishing or zero returns. The paper's own data contains hints of this: Ind-32 achieves 75.59% on Chatbot Arena, only marginally better than Ind-16, Pro-8 at 75.92% despite using 33% more forward passes (32 vs. 24). The prototype overscaling result (Ind-16, Pro-16 drops to 72.59%, below the unscaled baseline) is even starker—additional scaling actively hurts, which is inconsistent with a pure variance-reduction story and suggests systematic bias introduction from prototype mismatch.

For deployment, this means the scaling benefit observed at moderate configurations (Ind-8, Pro-4; Ind-16, Pro-8) may not extrapolate to larger budgets. A practitioner expecting to trade additional inference compute for proportionally higher accuracy would be disappointed. The paper's experiments stop at Ind-32 and Ind-16, Pro-8; there is no Ind-64 or Ind-32, Pro-16 data point, so the scaling curve beyond ~32 forward passes is unknown.

What evidence exists in the paper. The scaling results in Table 2 show a non-monotonic pattern: performance improves from base (72.68%) to Ind-8 (73.61%) to Ind-16 (73.87%) to Ind-32 (75.59%), but Ind-16, Pro-8 (75.92%) outperforms Ind-32 (75.59%) with fewer passes. This is consistent with the idea that prototype-level scaling introduces a different kind of information (transfer from similar users) that individual-level scaling (more samples from the same user's signals) cannot replicate. But it also means the "scaling" story is not a simple "more compute → better": the composition of the scaling budget (individual vs. prototype) matters as much as the total budget. The paper does not frame this as a bias-variance decomposition problem and does not provide the analysis that would allow a practitioner to predict optimal budget allocation for a new domain.

Mitigation status. The paper partially addresses this through the prototype number analysis (Figure 3) and the scaling configuration sweep (Table 2), which show that both under-scaling and over-scaling exist. But the underlying mechanism—whether over-scaling introduces bias, or whether individual samples are biased in correlated ways—is not investigated. The paper does not measure bias-variance decomposition, does not report per-sample score distributions to assess whether individual chains agree or disagree systematically, and does not analyze error patterns to determine whether certain preference types are consistently misjudged regardless of scaling.

---

The Generative Evaluation Chain Format Introduces a Latency-Accuracy Tradeoff That Is Incompletely Characterized

The assumption or constraint. P-GenRM produces a structured evaluation chain—persona analysis, criteria derivation, criterion-by-criterion scoring—as a single autoregressive generation before extracting numerical scores via post-processing. This format is the source of the model's interpretability, its ability to surface novel preference dimensions, and its support for test-time scaling through parallel sampling. But it is also inherently slower than a scalar reward model, which would map the same inputs to a single numerical output in one forward pass without generating intermediate text.

The paper reports wall-clock inference times in Table 11 (Appendix A.9) and argues that the overhead is modest: P-GenRM-8B without scaling takes 00:14:16 on the Chatbot Arena-Personalized test set, compared to 00:14:06 for the LLaMA-3.1-8B-Instruct + PSI in-context baseline. The paper attributes this to the long input prompt (which includes 3 historical preference pairs) making KV-cache construction the dominant cost, shared across all samples. With Ind-16, Pro-8 scaling, the time increases to 00:23:05—still faster than SynthesizeMe + FT RM-70B (01:29:59) while achieving higher accuracy.

The consequence. The inference time comparison in Table 11 has several properties that limit its generalizability. First, it is measured on the Chatbot Arena-Personalized test set, which has only 131 users. The per-query time depends on prompt length (which includes the historical preference pairs), and prompt length varies across datasets and deployment contexts. A deployment with longer conversation histories or more preference pairs per query would increase the KV-cache cost, potentially changing the relative efficiency picture.

Second, the comparison against SynthesizeMe + FT RM-70B contrasts an 8B model with a 70B model—the 8B model being faster is unsurprising and does not isolate the cost of the evaluation chain format itself. A fairer comparison would be P-GenRM-8B versus a scalar personalized reward model at 8B scale (e.g., a Bradley-Terry RM-8B fine-tuned with preference history in the prompt). The paper does not report inference time for such a baseline, so the marginal cost of the generative format over an equivalently-sized scalar model is unknown.

Third, the paper does not report throughput (queries per second) or GPU memory usage. Wall-clock time for a single test set pass is informative but incomplete: a deployment serving many concurrent users needs to understand how P-GenRM scales under load, whether batching multiple users' queries degrades latency, and whether the memory footprint of generating long evaluation chains limits batch size.

Fourth, the generation length of the evaluation chain is not controlled. The model autoregressively generates persona analysis and criterion-weighted scoring; the length of this output depends on the complexity of the user's preferences and the model's generation behavior. In adversarial or edge cases, the model might generate very long chains (verbose justifications, redundant criteria), increasing latency unpredictably. A scalar reward model has constant output length. The paper provides no statistics on evaluation chain length distribution—mean, variance, or maximum observed.

What evidence exists in the paper. Table 11 provides the only latency measurements, reporting wall-clock time for the full test set under specific hardware conditions (8 NVIDIA A100 GPUs, vLLM serving). The paper notes two factors mitigating the scaling cost: KV-cache sharing and parallel sampling efficiency. No throughput, memory, or chain-length statistics are reported.

Mitigation status. The paper does not frame this as a limitation. The authors present the inference time data as evidence that the generative format is practical, and the comparison against the 70B prior SOTA supports this framing. But the incomplete characterization leaves practitioners without the information needed to make deployment decisions: What is the p50 and p99 latency per query? How does throughput scale with batch size? What is the GPU memory required per concurrent query? These are the metrics that determine whether a system can be deployed in a latency-sensitive application, and the paper does not provide them.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes personalized reward modeling as a generation problem rather than a regression problem, and in doing so opens a new axis for test-time compute scaling that the personalization community had not previously explored. The shift is methodological but has conceptual weight: by reframing preference evaluation as structured text generation—persona analysis, criteria derivation, criterion-weighted scoring—rather than as a mapping from user signals to a scalar, the paper makes personalization auditable, scalable, and transferable in ways that scalar reward models structurally cannot support.

The magnitude of the shift is a reframing with immediate practical consequences, not a paradigm revolution. The paper does not claim that prior personalized reward models are fundamentally wrong—they work, and the Bradley-Terry population baseline in Table 1 (67.21% on Chatbot Arena at 8B) is respectably strong. Rather, it identifies a ceiling those methods share: they compress user preferences into fixed-dimensional representations (latent variables, mixture weights, prototype coefficients) that cannot express preference dimensions the designer did not anticipate. P-GenRM's generated rubrics surface criteria like "Philosophical Engagement," "Nuance," and "Breadth of Ideas" (Section 5.2) that are absent from PRISM's predefined dimensions, and the adaptive persona experiment (Table 4) shows that dynamic, per-query preference inference consistently outperforms static persona methods (+1.41 to +5.38 points depending on model and dataset). These are not marginal improvements from better optimization—they are structural gains from removing a representational bottleneck.

The paper also resolves a latent tension between static and dynamic preference modeling that had been treated as a binary choice in prior work. SynthesizeMe (Ryan et al., 2025) infers a persona once and reuses it; PAL (Chen et al., 2024) and VPL (Poddar et al., 2024) learn stable per-user representations. P-GenRM collapses the distinction by making persona generation part of the autoregressive evaluation chain—the same user with the same history can receive different personas for different queries, and the model is trained (via the three-stage pipeline) to condition this generation on the current context. The preliminary experiment in Table 6 (Appendix A.2) provides the diagnostic: adding scenario-specific criteria to an already-persona-informed prompt improves accuracy from 64.02% to 66.17%, a gap that static methods cannot close. This reframes the field's question from "should we model users as stable or adaptive?" to "how do we train models that adaptively decide when stability versus adaptation is appropriate?"

The most consequential shift for future research is the paper's demonstration that collaborative filtering principles transfer to LLM evaluation. The prototype mechanism—cluster users by preference embeddings, use cluster membership to share preference information—is a direct adaptation of techniques from recommendation systems (Goldberg et al., 1992; Sarwar et al., 2001), but applied to a domain where the "items" being rated are evaluation criteria rather than products. This opens a bridge between two research communities that had minimal overlap: the collaborative filtering literature has decades of work on cold-start recommendation, user clustering, and preference elicitation that is now directly applicable to personalized alignment. The LaMP-QA result (Table 5) provides the existence proof: a model trained only on PersonalRewardBench, using prototype-based transfer, outperforms a ~30× larger model on out-of-distribution personalization (0.638 vs. 0.619 Spearman correlation). This is not a marginal improvement—it is a qualitative demonstration that learned preference archetypes generalize across domains.

The paper also redirects research attention away from architectural complexity and toward training signal design. Prior personalized methods invested heavily in model architecture—separate Transformer modules for preference prediction (GPO, Zhao et al., 2023), latent-variable inference (VPL, Poddar et al., 2024), prototype mixture weights (PAL, Chen et al., 2024). P-GenRM uses a standard LLaMA backbone with no architectural modifications; all gains come from the training pipeline (SFT distillation, GRPO with process+outcome rewards, curriculum learning) and the inference-time mechanism (dual-granularity scaling). This suggests that the bottleneck in personalized reward modeling is not model capacity but how preference signals are structured, rewarded, and aggregated. Future work on personalization should prioritize training data construction, reward design, and test-time strategies over novel architectures—a lesson that parallels findings in the broader LLM alignment literature (where RLHF reward design and data quality have repeatedly proven more impactful than policy architecture innovations).

One prior contradiction the paper partially resolves: the field had conflicting evidence on whether user personas improve personalization. SynthesizeMe showed strong gains from persona-based prompting, but the personas were static and their benefit was context-independent. Other work (implicitly) showed that preference dimensions alone (GPO, VPL) were insufficient without persona context. P-GenRM's adaptive persona ablation (Table 4) and the preliminary experiment (Table 6) together establish that personas help, but only when they are scenario-specific—a static persona provides a prior that improves accuracy by ~1.6 points over no persona, but a dynamic persona provides an additional ~1.7–5.4 points beyond that. This explains why prior work showed mixed results: methods that used static personas (SynthesizeMe) captured part of the benefit; methods that used no personas (PAL, GPO, VPL) missed it entirely; and no prior method captured the full dynamic benefit that P-GenRM demonstrates.

Research directions that become more attractive after this paper:

• Training signal design for subjective evaluation. The paper's disentanglement of process and outcome rewards (Equation 3) and the staged training (RL with both rewards → curriculum learning with outcome only) provides a template for how to train models that must both reason faithfully and judge correctly. The finding that process reward helps during early training but constrains exploration on hard cases (Section 4.1) is a concrete, testable hypothesis about the interaction between reward components and training difficulty.
• Test-time compute scaling for subjective tasks. The 3.24% gain from Ind-16, Pro-8 scaling (Table 2) demonstrates that parallel sampling and aggregation improve personalized evaluation—a result that extends the test-time scaling paradigm from reasoning (math, code) to subjective judgment (preference, style, values). This opens a new class of tasks where scaling may help, and the prototype-level mechanism (borrowing signals from similar users) is a novel scaling dimension with no analog in standard reasoning benchmarks.
• Preference elicitation through evaluation chain design. The structured evaluation chain format is not just an output format—it is an implicit preference elicitation mechanism. By generating a persona and criteria before scoring, the model surfaces what it believes the user values, which could be used for active preference elicitation (asking the user to confirm or correct the inferred criteria) or for user-facing explanations ("I scored this response lower because I inferred you value conciseness—is that correct?"). The paper does not explore this, but the infrastructure is in place.

Research directions that become less attractive:

• Fixed-dimension preference representations. The paper shows that PAL, GPO, and VPL all underperform the Bradley-Terry population baseline (Table 1: 56–59% vs. 67.21% on Chatbot Arena at 8B), and all substantially underperform P-GenRM. While there may be domains where fixed dimensions are adequate (e.g., tasks with well-defined, enumerable preference axes), the paper's evidence suggests that open-ended personalization requires open-ended preference representation. Future work on fixed-dimension methods would need to demonstrate that the dimensionality bottleneck has been overcome—a higher bar than before this paper.
• Static persona methods without per-query adaptation. Table 4 demonstrates that dynamic personas consistently outperform static ones across base models (Qwen3-8B, o3) and datasets (Chatbot Arena, PRISM). A static persona approach would need to show that the remaining accuracy gap can be closed through other means (better persona inference, richer static descriptions) or that the gap is negligible in a specific deployment context.
• Purely individual-level personalization without collaborative signals. The prototype overscaling result (Ind-16, Pro-16 degrades performance, Table 2) shows that prototype transfer has limits, but the strong results at Ind-16, Pro-8 and the LaMP-QA generalization demonstrate that collaborative signals are valuable. A system that relies solely on per-user data without any transfer from similar users leaves performance on the table, especially in cold-start regimes.

---

Follow-Up Research This Work Enables

Bias-variance decomposition of P-GenRM's evaluation chain errors. The paper frames test-time scaling as reducing variance through aggregation, but never measures whether P-GenRM's errors are predominantly variance (random noise around an unbiased mean, which scaling helps) or bias (systematic misinterpretation of certain preference types, which scaling does not help). A follow-up study would: (1) generate a large number of evaluation chains (e.g., 100) for the same query from P-GenRM, extract the score distributions, and measure the variance across chains; (2) compare the mean of the chain scores to the ground-truth pairwise judgment to estimate bias; (3) determine whether the remaining error after Ind-16, Pro-8 scaling (roughly 24% on Chatbot Arena) is irreducible bias or residual variance that could be reduced with further scaling. The prototype overscaling result (Ind-16, Pro-16 drops to 72.59%, below the unscaled baseline) suggests that prototype-level scaling introduces bias when too many neighbors are included—explicitly measuring this would clarify whether prototype transfer has a fundamental ceiling or a tunable optimum. A strong follow-up would also measure whether different preference types (e.g., stylistic preferences vs. factual preferences vs. value-based preferences) exhibit different bias-variance profiles, which would inform adaptive scaling strategies that allocate more samples to high-variance preference types.

Active preference elicitation via evaluation chain feedback. P-GenRM generates an explicit persona and criteria set before scoring. This intermediate output could be shown to the user for verification or correction, closing the loop between inference and elicitation. A concrete experiment: for each query, generate the evaluation chain, extract the top-N inferred criteria (e.g., "You appear to value conciseness and factual precision"), present them to the user as a lightweight feedback prompt ("Is this accurate? Y/N"), and measure whether incorporating user corrections improves downstream scoring accuracy. The paper's finding that explicit criteria ($E^{(u)}$) improve accuracy when available (Table 6: +2.15 points for Persona+OSR+SDim over Persona alone) suggests that even coarse user feedback could provide meaningful gains. The key question is whether the inferred criteria are accurate enough that users mostly confirm them (making feedback low-cost) or whether they require substantial correction (making feedback burdensome). A related experiment would compare different elicitation formats: binary confirmation vs. criteria re-ranking vs. free-text correction, measuring both accuracy improvement and user effort.

Cross-model-family replication of the full P-GenRM training pipeline. The paper's experiments are confined to LLaMA-3.1 (8B and 70B) as the base model family, with only a limited in-context ablation on Qwen3-8B (Table 4) and o3. The three-stage training pipeline—SFT distillation from o3, GRPO with process and outcome rewards, curriculum learning with hard negatives—may depend on properties of LLaMA-3.1 that do not transfer. A replication study would train P-GenRM on at least two additional model families (e.g., Qwen2.5, Mistral, Gemma) at comparable scales, measuring: (1) whether the SFT distillation quality varies across base models (some models may be better at imitating structured output formats than others); (2) whether the RL stage provides consistent gains or whether some base models benefit more from process reward than others; (3) whether the optimal alpha-beta weights (0.5, 1.0) transfer across model families. The hypothesis to test is that the training pipeline's effectiveness is architecture-independent and driven by the training signal design; a negative result (large variation across model families) would suggest that P-GenRM's gains are partially model-specific and that practitioners should benchmark multiple base models before adopting the pipeline.

Human evaluation of generated evaluation chains for faithfulness and completeness. The paper claims interpretability as a benefit of the generative format, but evaluates only accuracy of the final pairwise judgment—not whether the intermediate reasoning (persona, criteria, criterion-by-criterion analysis) is faithful to the user's actual preferences or complete in covering relevant dimensions. A human evaluation study would: (1) recruit annotators who have provided their own preference data (e.g., from the PRISM dataset), present them with P-GenRM's generated evaluation chains for their own queries, and ask them to rate on Likert scales: (a) how accurately the inferred persona describes them, (b) how well the criteria capture what they actually value, (c) whether the criterion-by-criterion scoring reflects how they would evaluate the responses; (2) compare these ratings against a baseline (e.g., SynthesizeMe's static persona descriptions, or a simple criteria list generated by prompting); (3) test whether users who rate the chain as more faithful also show higher agreement with the final pairwise judgment. This would validate (or refute) the paper's implicit claim that better reasoning leads to better judgments—a correlation that is assumed but not measured. A negative result (high accuracy but poor faithfulness ratings) would suggest the model is producing plausible-sounding but unfaithful reasoning, which would undermine the interpretability claim.

Dynamic prototype reassignment during test-time based on query context. The paper assigns each user to a single prototype based on their cross-scenario preference embedding, then uses that prototype for all queries. But the adaptive persona results suggest that a user's preferences shift with context—a user who is in prototype A for technical queries might belong in prototype B for creative queries. A natural extension is context-dependent prototype assignment: rather than assigning a user to one prototype globally, compute a query-conditioned prototype weight vector that allows the model to draw on different prototypes for different queries. A concrete implementation: embed the current query $q_t$ alongside the user's preference embedding, compute similarity to all prototypes, and use a soft prototype weighting rather than hard assignment during prototype-level scaling. The experiment would compare context-dependent vs. context-independent prototype assignment on: (1) overall accuracy; (2) per-category accuracy (does the benefit concentrate in categories where users are most likely to shift preferences?); (3) performance on users whose inferred preferences in Table 6 shifted substantially across scenarios. The paper's Figure 6 and 7 provide existence proofs of within-user preference shifts; context-dependent prototype assignment would test whether the model can operationalize this insight at scale.

Scaling-law characterization of test-time compute for personalized evaluation. The paper sweeps scaling configurations (Ind-8, Ind-16, Ind-32, with various Pro combinations) but stops at ~32 total forward passes and does not fit a scaling law (e.g., accuracy as a function of $m$ and $n$). A scaling-law study would: (1) generate data points at many more configurations (Ind-1 through Ind-128, Pro-1 through Pro-64, and their cross-product), fit a parametric function relating accuracy to individual samples and prototype samples, and determine whether the relationship follows a power law (as in pretraining and inference-time reasoning scaling laws) or a different functional form; (2) estimate the irreducible error (the accuracy ceiling) and the rate at which scaling approaches it; (3) determine the optimal allocation ratio $m:n$ as a function of total budget and user history length—the paper's anecdotal evidence (Ind-16, Pro-8 outperforms Ind-32) suggests the ratio matters, but a systematic sweep would provide prescriptive guidance. This would also test the paper's implicit claim that prototype-level scaling provides a fundamentally different signal from individual-level scaling—if the scaling law shows that prototype samples are simply equivalent to some fraction of an individual sample (e.g., 1 prototype sample ≈ 0.7 individual samples), then the dual-granularity design is an efficiency improvement but not a qualitative innovation; if prototype samples show a different scaling exponent or interact non-additively with individual samples, the innovation is more fundamental.

---

Practical Applications and Downstream Use Cases

Cold-start personalization for deployed conversational agents. In any LLM-powered product with user-facing chat (customer support, therapy bots, tutoring systems, creative writing assistants), new users arrive with no interaction history. A standard personalized alignment pipeline would serve population-average responses until enough per-user data accumulates—a process that may take dozens of interactions and during which the user experiences generic, potentially misaligned responses. P-GenRM's prototype mechanism enables immediate personalization from the first interaction: the user's initial query is embedded, matched to the nearest prototype, and the system uses prototype-neighbor preferences to score and select responses. The paper's LaMP-QA results (Table 5) provide evidence that this works out-of-distribution: P-GenRM-8B with Ind-8, Pro-4 achieves 0.638 Spearman correlation on unseen users with sparse history, outperforming a ~30× larger model. The practical deployment architecture would: (1) run the offline prototype construction pipeline on the existing user base; (2) for each new user, compute their preference embedding from available signals (even minimal ones); (3) serve responses scored by P-GenRM with prototype-level scaling, gradually increasing the weight of individual-level scaling as the user's own history accumulates. The inference time overhead (00:18:22 for Ind-8, Pro-4 on the 131-user Chatbot Arena-Personalized set, Table 11) is ~4 minutes for the full test set, or roughly 1.8 seconds per query—acceptable for many non-streaming applications.

Training data filtering for personalized instruction tuning. When constructing training datasets for personalized LLM fine-tuning, the quality of preference labels determines downstream policy performance. P-GenRM can serve as a data quality filter: for each candidate response pair in a training corpus, score both responses through the lens of the user's inferred preferences, and retain only pairs where the score difference exceeds a threshold (indicating a clear preference signal) and where the preferred response's score is above a minimum (indicating acceptable quality). This would filter out ambiguous pairs (where the user's preference is uncertain) and low-quality pairs (where neither response is good), concentrating the training data on high-signal examples. The paper's policy model experiments (Appendix A.12, Table 14) show that an 8B policy model trained with P-GenRM under GRPO achieves a mean score of 3.354 (on a 1–5 scale), outperforming 70B baselines—this establishes that P-GenRM's reward signal translates to downstream policy quality. The filtering application extends this: rather than using P-GenRM as the online reward during RL, use it offline to curate the training data, potentially reducing the need for expensive online RL infrastructure. A practitioner could: (1) run P-GenRM with moderate scaling (Ind-8, Pro-4) over an existing preference dataset; (2) filter to the top 50% of pairs by score difference; (3) train a policy model on the filtered data; (4) evaluate whether the filtered-data policy outperforms the full-data policy, which would indicate that P-GenRM successfully identifies noisy or low-signal preference pairs.

Multi-turn dialogue systems with evolving user preference tracking. In long-running conversational systems (ongoing therapy, tutoring over weeks, persistent assistant relationships), user preferences evolve—a user who initially values detailed explanations may, as their expertise grows, shift toward preferring concise, technical responses. P-GenRM's per-query adaptive persona generation naturally captures this drift: each new query produces a fresh evaluation chain conditioned on recent history, so preferences inferred from earlier interactions gradually lose influence. A practical system could: (1) maintain a sliding window of the user's most recent $\mathbf{h}$ preference pairs (the paper uses $\mathbf{h}=3$, but a deployment might use 10–20 for longer-term tracking); (2) weight recent interactions more heavily in the prompt or in the prototype attention mechanism (the paper uses uniform random sampling, but recency-weighted sampling is a straightforward modification); (3) periodically re-compute the user's prototype assignment if their preference embedding shifts substantially. The paper's Table 6 shows that adding explicit criteria to persona-informed prompts improves accuracy by 2.15 points; in a deployed system, explicit criteria could be periodically elicited from the user ("Are you still finding detailed explanations helpful, or would you prefer more concise answers given your growing expertise?") and fed into P-GenRM's $E^{(u)}$ input, creating a feedback loop where the system adapts and the user calibrates. The inference cost per query is modest enough (~1.8 seconds amortized) to support interactive use, and the interpretability of the evaluation chain (generated persona and criteria are human-readable) means the system could explain its personalization choices to the user, building trust.

Minority preference protection in shared LLM services. When a single LLM deployment serves a diverse user population (public-facing chatbots, educational platforms, content generation APIs), population-average optimization systematically under-serves users with minority preferences. The paper's macro accuracy analysis (Appendix A.5, Table 10) quantifies this: a Bradley-Terry reward model fine-tuned on population data achieves 60.64% macro accuracy versus P-GenRM-8B's 65.21%—a 4.57-point gap that represents minority users being mis-scored by the population model. P-GenRM can be deployed as a fairness-aware scoring layer: before a response is served to a user, score it through P-GenRM with the user's prototype-based personalization, and only serve responses that exceed a personalized quality threshold. This ensures that minority-preference users are not served responses optimized for the majority. The paper's prototype-wise performance analysis (Figure 5, Table 9) shows that P-GenRM's accuracy is consistent across prototypes of different sizes (median 0.6500, standard deviation 0.0401, IQR 0.0544), indicating that the system does not degrade on small, minority-representing prototypes. A concrete deployment would: (1) train P-GenRM on the existing user base; (2) build prototypes and assign all users; (3) for each user query, generate multiple candidate responses from the policy LLM, score them via P-GenRM with appropriate scaling, and return the highest-scoring response; (4) monitor per-prototype satisfaction metrics to detect if any prototype group is systematically under-served. The additional inference cost of P-GenRM scoring (00:14:16–00:23:05 for the full test set) adds latency that may be acceptable for asynchronous or batch generation but would need optimization (model distillation, shorter evaluation chains, speculative scoring) for real-time streaming applications.

---

When to Prefer This Method

The paper does not articulate an explicit tradeoff against named alternatives with decision criteria. It positions P-GenRM as a new state-of-the-art method that outperforms all baselines in its evaluation setting—there is no "prefer P-GenRM when X, prefer SynthesizeMe when Y" framing. The ablation studies (Table 3) establish which components of P-GenRM contribute to its performance, but do not test conditions under which a simpler method might be preferable. The paper also does not compare P-GenRM against itself with different design choices (e.g., static vs. dynamic persona, prototype-based vs. individual-only scaling) in a way that yields prescriptive deployment guidance—the comparisons exist (Tables 2 and 4) but are presented as validation of P-GenRM's design rather than as a decision framework.

A forced decision matrix would therefore be speculative generalization beyond the paper's claims, and is omitted.