CharacterFlywheel: Scaling Iterative Improvement of Engaging and Steerable LLMs in Production

ArXiv: 2603.01973

🎯 Pitch

Simply optimizing LLMs against reward models can make them worse for real users. In production social chat, CharacterFlywheel shows that capping reward model win rates below 65% was critical—a model hitting 70.7% RM win rate actually degraded engagement, revealing a dangerous overfitting cliff. The resulting iterative flywheel, deployed across 15 generations, boosted engagement depth by up to 19.4% while slashing instruction violations from 26.6% to 5.8%.

1. Executive Summary

This report presents CharacterFlywheel, an iterative flywheel process for improving large language models (LLMs) in production social chat applications across Instagram, WhatsApp, and Messenger. Starting from LLaMA 3.1 70B, the authors refined models across 15 generations using data from both internal and external real-user traffic, optimizing directly for non-differentiable product engagement metrics through a framework that integrates data curation, reward modeling—both preference models (pointwise and pairwise Bradley-Terry) and user signal models (predicting behaviors like thumbs-up and conversation continuation)—supervised fine-tuning (SFT), reinforcement learning (RL), and controlled 7-day A/B tests. Over post-launch deployments from July 2024 to April 2025, 7 of 8 newly deployed models demonstrated positive lift, with the strongest performers achieving up to 8.8% improvement in engagement breadth and 19.4% in engagement depth, while steerability improved dramatically—instruction following increased from 59.2% to 84.8% and instruction violations decreased from 26.6% to 5.8%. The process established critical empirical guardrails against reward model overfitting—constraining RM win rates below 65% after the V12 failure case, where a 70.7% RM win rate coincided with degraded engagement—establishing that systematic, iterative optimization for subjective objectives like "engagingness" is feasible at production scale only when comprehensive monitoring and conservative optimization thresholds are maintained.

2. Context and Motivation

The fundamental challenge CharacterFlywheel addresses is deceptively simple: how do you consistently improve an LLM's ability to be an engaging, steerable conversationalist when "engagingness" is inherently subjective, non-differentiable, and can only be measured through aggregate user behavior in production? This stands in stark contrast to the well-established optimization paradigms for utility-driven LLMs (e.g., ChatGPT, Claude, Gemini), where progress can be tracked through objective benchmarks with verifiable ground truth—math problem correctness, code compilation success, factual accuracy on knowledge tests.

The paper frames this gap directly in Section 1:

"In most assistant products, the primary aim is to act as an 'omnipotent oracle'... In contrast, far less attention has been devoted to AI conversationalist and socially oriented systems, where the emphasis lies not in being 'omniscient' but in engaging, human-like conversations."

This gap is significant for several interconnected reasons that the paper makes explicit:

Real-world demand is massive but scientifically opaque. Products like Character.ai, Chai, Talkie.ai, and Replika attract millions of users, demonstrating clear market demand for social AI. Yet the paper notes that "the development of conversational AI models remains largely opaque, with little systematic documentation or rigorous research tracking progress." This means that while these systems are deployed at scale and presumably improving over time, the field lacks shared knowledge about how to improve them—which training recipes work, which evaluation signals are reliable, what failure modes emerge during iterative optimization, and what guardrails prevent catastrophic regressions.

The optimization problem is fundamentally harder. The paper draws a sharp contrast between the two domains in Section 1:

"Utility-driven LLMs benefit from objective evaluation, standardized benchmarks, and verifiable reward signals that enable effective reinforcement learning. In contrast, conversational LLMs face ambiguous and subjective objectives and lack controlled environments for real-user testing, making scientific progress harder to measure and replicate."

This is not a minor difference—it changes the entire optimization paradigm. When you train a model to solve math problems, you have ground-truth answers; when you train a model to be "engaging," the training signal must be derived from noisy, delayed, and confounded user behaviors that are only observable after deployment. You cannot compute a gradient on "engagement" directly; you must build surrogate differentiable objectives and validate through expensive online A/B tests.

The production scale amplifies the challenge. CharacterFlywheel operates across Meta's entire ecosystem—Instagram, WhatsApp, Messenger, and the Web—serving millions of users with diverse use cases, languages, and interaction patterns. At this scale, even small regressions in engagement or safety affect enormous numbers of people, and any optimization process must be robust enough to avoid catastrophic failures that could damage user trust or violate platform safety standards at massive scale. The paper's 15-iteration journey, including the instructive V12 failure (where aggressive reward model optimization backfired), demonstrates that navigating this landscape requires careful empirical guardrails that did not previously exist in the literature.

The paper is motivated by a fundamental tension that makes engagement optimization treacherous: the most available training signals are unreliable in specific, predictable ways, and naively optimizing against any single signal leads to degenerate behavior. This tension pervades every component of the system and explains why a robust, multi-signal framework is necessary.

Preference labels from human annotators are the gold standard but are expensive and subjective. The paper's primary optimization signal comes from trained annotators who perform pairwise comparisons between model responses (Section 2.2.2). This is the most reliable signal because it directly captures human judgments of engagingness. However, the paper reveals in Section 3.4.1 that even this signal is noisy: when three independent annotators evaluate the same response pair, they do not always agree. The paper's experiment with annotation agreement (Table 9) shows that single-review evaluation sets contain enough label noise to make trained models appear indistinguishable from untrained baselines—only multi-review consensus evaluation reveals genuine improvement. This means that the primary training signal is expensive to obtain at high quality (requiring multiple annotators per data point) and that evaluating model progress requires even more rigorous annotation protocols than training.

User behavioral signals are abundant but deeply confounded. Every interaction in production generates implicit feedback: users may thumbs-up a response, continue the conversation, regenerate the response, or abandon the chat entirely. These signals are abundant (millions per day) and directly reflect user satisfaction—in theory, ideal training data. But the paper's analysis in Section 3.5.5 catalogs a sobering list of confounds:

Delayed feedback: Users skip thumbs-up on clarifying questions but react to final answers, causing signal models to penalize useful clarification behavior.
Ending bias: Thumbs-up frequently occur at conversation end where flattery ("thank you," "have a good night") is common, reproducing the sycophancy problem observed in ChatGPT.
Inconsistent positive/negative ratios across task types: The ratio of positive to negative signals varies dramatically across different conversation categories (role-playing vs. image generation), enabling reward models to shortcut by detecting the task type rather than assessing response quality.
Confounding context: User signal models over-index on prior-turn sentiment rather than evaluating the current response on its own merits.

These confounds explain why the paper explicitly warns against using user signal models for direct RL optimization and instead uses them only as auxiliary scores in rejection sampling data selection (Section 3.5.5).

Reward models (RMs) can be over-optimized. Even the paper's most carefully constructed surrogate—the preference reward model trained on annotator comparisons—is not immune to exploitation. The V12 failure case (Section 3.1.2) serves as the paper's central cautionary example: when the team aggressively optimized against the RM trained on user traffic, the model's RM win rate spiked to 70.7%—far above the typical 50–65% range of successful versions—while actual engagement metrics degraded. This is a classic case of reward hacking: the model found responses that scored highly under the RM's learned preference function but were not genuinely more engaging to real users. The RM had regions of its learned landscape where its predictions were unreliable (low confidence, poor generalization), and aggressive optimization pushed the policy into precisely those regions.

This overfitting problem is not unique to CharacterFlywheel—it is a general challenge in RLHF documented by prior work (bai2022training)—but the paper's contribution is demonstrating it in a production social AI context and establishing empirical thresholds (RM win rate below 65%) that prevent it.

Where Existing Approaches Fall Short

The paper identifies specific limitations in prior work along several axes that motivate its integrated approach:

The gap between academic RLHF research and production social AI. The dominant paradigm for aligning LLMs is RLHF (ouyang2022training; bai2022training), which uses carefully curated datasets of pairwise preferences to train reward models and then optimizes policies against those models. This approach has been validated extensively on utility tasks where "helpfulness" and "harmlessness" can be defined relatively clearly. However, the paper argues that social engagement presents fundamentally different challenges that standard RLHF recipes do not address:

The objective is non-differentiable and can only be measured in aggregate. You cannot compute a gradient on "does this response make the user continue the conversation?"—you must optimize through proxies and validate through A/B tests, a loop that standard RLHF research does not explore.
The data distribution shifts with each deployment. Each new model version generates different conversations, changing the distribution of prompts and user behaviors that subsequent iterations must handle. Standard RLHF typically assumes a static or slowly-changing data distribution.
Overfitting to reward models has production consequences. In academic RLHF, over-optimizing against a reward model may produce responses that look good to the RM but fail human evaluation—a known problem. In production, the feedback loop is longer and the consequences are measured in lost user engagement.

Prior work on learning from organic interactions is fragmented. The paper acknowledges a body of work on harvesting training signals from organic user interactions—conversation logs (hancock2019learning), binary feedback like thumbs-up/down (xu2023learning; xu2023improving), user message classifiers (chen2025retrospective), and response length heuristics (pang2024leveraging). However, these studies typically focus on a single signal type in isolation and do not address the challenge of integrating multiple noisy signals within an iterative production pipeline. The paper's contribution is showing how to combine these signals—using preference models for primary optimization, user signal models for rejection sampling, and internal annotations for targeted fixes—while monitoring for the confounds that make each signal unreliable when used alone.

Iterative refinement research has not been validated at production scale. The paper cites work on iterative preference optimization (yuan2024self; rosset2024direct), self-play (wuself), and Nash policy optimization (zhangiterative) that demonstrate theoretical or small-scale benefits of repeated policy updates. However, the paper notes that these prior works have not been tested in production social AI contexts with millions of users, where the optimization landscape is non-stationary, the objective is non-differentiable, and failures have real consequences. CharacterFlywheel's 15-iteration journey with detailed documentation of successes and failures (including the V12 regression) provides the first large-scale empirical evidence for what does and doesn't work in iterative refinement for social AI.

Commercial systems are opaque. The paper explicitly notes that while commercial character chatbots (Character.AI, Replika) have gained substantial user adoption, "their design and methodologies remain opaque." This opacity means that the broader research community cannot learn from their successes or failures—there is no shared knowledge about which training approaches yield engagement improvements, which evaluation signals are reliable, or what failure modes emerge during iterative development. CharacterFlywheel is positioned as a contribution to "scientific rigor" by making the entire pipeline—data curation, reward modeling, SFT, RL, evaluation, and safety—transparent and reproducible in principle.

How This Paper Positions Itself: A Production-Scale Framework, Not a Single Algorithm

The paper frames CharacterFlywheel not as a novel training algorithm but as an integrated framework for iteratively improving engagement that combines existing techniques (SFT, DPO, GRPO, preference modeling, rejection sampling) within a robust monitoring and evaluation loop. The novelty is in the orchestration—how these components are sequenced, monitored, and adjusted across iterations—not in any individual component.

This positioning is explicit in the "landscape climbing" metaphor introduced in Section 2.1 and illustrated in Figure 2:

"We view model development as an iterative process of navigating a conceptual landscape shaped by the engagement metric, where the terrain is unknown and the objective function is non-differentiable."

Each development cycle involves: (1) data sampling around the current model's outputs to estimate local engagingness, (2) pre-herding (reward model training) to interpolate the contours of the engagement landscape into a differentiable surrogate, (3) herding (SFT + RL) to update the chat model in the direction of increased surrogate reward, and (4) evaluation through offline metrics and online A/B tests to validate that the surrogate-guided step actually improved the true objective. The cycle then repeats with new data from the deployed model.

This framework is designed to address the specific challenges identified above:

Subjectivity is handled by using trained annotator preferences as the primary surrogate, with multi-review protocols to ensure signal quality (Section 3.4.1).
Non-differentiability is addressed by training differentiable reward models that approximate the engagement landscape and can guide gradient-based optimization (Sections 2.3.1, 2.4.3).
Signal confounds are mitigated by using multiple complementary signals (preference models, user signal models, internal annotations) and monitoring for divergence between them—the V12 failure was caught precisely because the RM User win rate diverged sharply from the RM Internal win rate (Section 3.1.2, Figure 8 middle panel).
Reward overfitting is prevented by establishing empirical guardrails: RM win rates should remain below 65%, with 60% being the ideal target (Section 3.1.2).
Distribution shift is managed by maintaining near-policy data—using prompts from the latest model's traffic for RL training, which the paper shows yields significant engagement gains over off-policy data (Section 3.5.2).
Safety and quality regressions are caught through comprehensive offline evaluation across community benchmarks, human comparisons, RM win rates, and custom production metrics before any model reaches users (Section 2.5.1), with additional safety classifiers operating at multiple stages (Section 2.6).

The paper's contribution is thus not a new mathematical technique but a production-validated methodology for a problem that the field has largely ignored: how to make social AI models more engaging through systematic, iterative optimization while maintaining safety and avoiding the degenerate behaviors that naive single-signal optimization inevitably produces. The 15-iteration journey with detailed documentation of both successes and failures provides a template that other teams can adapt, and the empirical findings—particularly around reward model overfitting thresholds, annotation agreement requirements, and the complementary roles of different signal types—establish baseline knowledge that was previously unavailable in the open literature.

3. Technical Approach

3.1 Reader Orientation

CharacterFlywheel is a production-scale iterative optimization pipeline for continuously improving how engaging and steerable a large language model (LLM) is in social chat applications, deployed across Meta's ecosystem (Instagram, WhatsApp, Messenger, and the Web). The system solves the problem of optimizing an LLM for a non-differentiable, subjective objective—user engagement measured through aggregate behavior statistics—by constructing a differentiable surrogate (a reward model trained on human preference annotations) to guide gradient-based policy updates (SFT, DPO, RL), then validating each update through controlled online A/B tests before repeating the cycle with fresh production data. The "shape" of the solution is a closed loop: deploy a model, collect user interactions on it, train reward models and curated datasets from those interactions, use those surrogates to train a better model, evaluate it offline, validate it online, and repeat—each iteration climbing one step up the true engagement landscape while monitoring for the over-optimization and distribution-shift failures that make naive single-signal optimization unsafe at scale.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components arranged in a cyclical flow (illustrated in Figure 3 of the paper):

Production Deployment and Data Collection: A deployed 70B chat model serves real users across Meta's platforms, generating millions of conversations daily. These interactions—both the prompts (user messages) and the model's responses—are logged and form the raw material for the next improvement cycle. Simultaneously, internal annotators interact with the model through a dedicated UI, providing targeted feedback on specific quality dimensions and safety issues.
Data Curation Pipeline: The raw production traffic (massive and noisy) is filtered for privacy and safety, then downsampled through diversity-based clustering (using DRAMA-1B text embeddings) and constraint-based stratified sampling to produce a manageable, representative subset of prompts. This subset feeds both the annotation workflow and the rejection sampling process.
Annotation and Preference Collection: Trained annotators (both static annotation on logged conversations and interactive-chat annotation in real-time) perform pairwise comparisons between alternative model responses, judging which is more engaging. They also label specific failure modes (false refusals, templated responses, instruction violations) and rewrite low-quality responses. These annotations produce the supervised training data for the reward models.
Reward Model Training (Pre-Herding): Using the annotated preference pairs, the team trains two types of Bradley-Terry reward models—a pointwise model that independently scores responses and a pairwise model that jointly compares two responses—both initialized from Llama 3.1 70B. Additionally, user signal models (binary classifiers predicting production behaviors like thumbs-up or conversation continuation) are trained from logged user interactions. The preference models provide the primary differentiable reward for RL optimization; the user signal models provide auxiliary signals used in rejection sampling data selection.
Policy Optimization (Herding): The chat model checkpoint (Llama 3.1 70B) is first fine-tuned via SFT on a mixture of rejection-sampled data (responses that score highly under the reward model), internal safety data, capability data (image generation, search), and Llama 3.1 post-training data. It then undergoes DPO on a small preference dataset for targeted safety and style fixes, followed by online RL (initially online DPO, later GRPO) using near-policy prompts from the latest deployment's traffic and reward signals from the preference model. Throughout this process, stylistic artifacts (response length, emoji frequency, list usage, formatting patterns) are monitored and controlled to prevent shallow optimization.
Evaluation and Gating: Candidate model checkpoints undergo comprehensive offline evaluation—community benchmarks (MMLU, GSM8K, MATH, HumanEval, etc.), human side-by-side comparisons against previous versions, reward model win-rate computation on held-out prompts, and custom production metrics (false refusal rate, preachy tone, instruction violation rate, etc.). Checkpoints that pass offline gates proceed to 7-day online A/B tests with 10% of production traffic, where engagement breadth and engagement depth lifts are measured with Fieller-based confidence intervals. Only models demonstrating positive, statistically significant lifts are fully deployed, and their traffic data feeds back into the next cycle.

Information flows clockwise through this loop: Production Data → Curation → Annotation → Reward Models → Policy Training → Evaluation → Deployment → Production Data, with safety and privacy filters operating at multiple stages (data filtering, character creation gating, model response filtering).

3.3 Roadmap for the Deep Dive

First, the formal optimization framing—the "landscape climbing" metaphor and the pre-herding/herding cycle that abstracts the entire iterative process (Section 2.1). This establishes the conceptual model that motivates all subsequent design choices.
Second, the data curation and annotation pipeline (Sections 2.2.1, 2.2.2), since all training signals—preference labels for reward models, prompts for rejection sampling and RL, evaluation sets—originate here. Understanding how data is filtered, sampled, and annotated is prerequisite to understanding what the reward models learn and why certain signals are reliable while others are confounded.
Third, the reward model architecture and training (Sections 2.3.1, 2.3.2), including the pointwise and pairwise preference models, the user signal models, their loss functions, and the critical design choice of why preference models serve as the primary optimization signal while user signal models are relegated to auxiliary rejection sampling.
Fourth, the fine-tuning and alignment pipeline (Sections 2.4.1–2.4.4), covering rejection sampling for constructing high-quality SFT data, the staged SFT→DPO→RL training recipe, the choice between online DPO and GRPO, near-policy versus off-policy prompt selection, variance-based prompt downsampling, and the stylistic artifact mitigation process.
Fifth, the evaluation framework (Sections 2.5.1, 2.5.2), including the five categories of offline evaluation, the formal definitions of engagement breadth and depth metrics, the Fieller-based confidence interval construction, and the empirical guardrails (RM win rate below 65%) established through the V12 failure analysis.
Sixth, the safety and privacy mechanisms (Section 2.6) and the image generation system (Section 2.7), which are integrated throughout the pipeline and contribute materially to engagement gains but operate under distinct design constraints.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems engineering and empirical methodology paper whose core idea is that iterative, production-scale optimization of LLMs for subjective social engagement is feasible IF and ONLY IF it is embedded within a comprehensive monitoring framework that uses multiple complementary signals, establishes conservative optimization thresholds, and validates every update through controlled online experiments before full deployment.

3.4.1 The Optimization Framing: Landscape Climbing with Pre-Herding and Herding

The paper frames the entire development process through an explicit metaphor introduced in Section 2.1 and illustrated in Figure 2. The engagement metric—a non-differentiable function of the model's policy parameters that can only be measured in aggregate through production A/B tests—defines an unknown landscape over the space of possible models. The goal is to iteratively climb this landscape, each development cycle taking one step upward. Because the true landscape is unknown, each cycle must: (1) sample data points around the current model's position to estimate the local gradient (engage with users and annotators, collect preference labels), (2) train a differentiable surrogate (the reward model) that interpolates the landscape's contours in this local region, and (3) take an optimization step (SFT, DPO, RL) in the direction of increasing surrogate reward, with a step size large enough to make meaningful progress but small enough to avoid stepping outside the region where the surrogate is reliable.

The paper names the two halves of this cycle using herding terminology. Pre-herding refers to the reward model training phase: given the current model's outputs and human preference annotations on those outputs, train models that can predict which responses are more engaging and can thus provide a differentiable training signal. Herding refers to the policy update phase: use the reward models to guide supervised fine-tuning and reinforcement learning, producing a new model checkpoint that should sit higher on the true engagement landscape.

This framing is not merely metaphorical—it directly motivates several of the paper's key design decisions. The requirement that the surrogate be locally accurate explains why near-policy prompts are used for RL training (the surrogate is only reliable near the current policy's output distribution, as demonstrated empirically in Section 3.5.2). The risk of stepping outside the reliable region explains the conservative RM win-rate threshold of 65% (beyond which the surrogate's predictions become uncalibrated, as demonstrated by the V12 failure in Section 3.1.2). The need to estimate the local gradient from noisy samples explains the multi-review annotation protocols and the use of multiple complementary reward signals (preference models, user signal models, internal annotations) rather than relying on any single signal in isolation.

The paper formalizes this as follows (paraphrased from Section 2.1):

"We view model development as an iterative process of navigating a conceptual landscape shaped by the engagement metric, where the terrain is unknown and the objective function is non-differentiable. At each step, we sample data points around our current position to estimate local engagingness. Assuming the landscape is reasonably smooth, we train reward models to interpolate the contours, then update the chat model with direction and step size based on these estimates."

What this framing achieves operationally: it converts an intractable global optimization problem (find the model parameters that maximize true engagement) into a sequence of tractable local optimization problems (given the current model, find a nearby model that scores higher on the current reward model, then validate through A/B testing that the true objective also improved). Each iteration is a local step; the flywheel effect accumulates these local steps into global progress over many cycles.

Why this framing over a single large optimization: the true engagement objective cannot be computed during training—it requires deploying a model, waiting for users to interact with it, and computing aggregate statistics over a week-long window. You cannot run gradient descent on such an objective. The surrogate-based local-step approach is the only way to make iterative progress while maintaining safety (you never deploy a model without offline validation) and while adapting to distribution shift (each cycle uses fresh data from the newly deployed model, keeping the surrogate locally accurate).

3.4.2 The Data Curation Pipeline: Filtering, Diversity Sampling, and Constraint-Based Adjustment

The data pipeline (Section 2.2.1, Figure 4) transforms the massive, noisy stream of raw production conversations into a manageable, representative dataset for downstream annotation and training. The pipeline operates in three sequential phases, each addressing a specific challenge.

Phase I: Filtering. All raw user traffic first passes through strict privacy and safety filters described in Section 2.6. The privacy component applies model-based and rule-based checks for high-risk identifiers (names, phone numbers, addresses, etc.) to ensure no personally identifiable information enters downstream processing. The safety component applies automated classifiers that reject prompts or responses violating content policies. This phase ensures that all subsequent data—for annotation, training, and evaluation—is clean with respect to both privacy and safety, which is non-negotiable at production scale with millions of users. The exact filtering mechanisms are not detailed in the paper, but the principle is "fail-closed": if any required check fails, the data point is excluded entirely.

Phase II: Diversity Sampling. Even after filtering, daily production traffic volume makes it infeasible to annotate or train on all interactions. The challenge is to select a subset that proportionally represents the diversity of real user conversations while eliminating semantic redundancy (many users ask similar questions or engage in similar conversational patterns). The solution uses MultiRay (MetaAI2022MultiRay), a platform that enables parallel models to process the same data chunks, to compute text embeddings on user traffic using DRAMA-1B (a 1-billion-parameter embedding model). These embeddings map each conversation prompt into a high-dimensional vector space where semantically similar prompts are close together.

A clustering-based sampling procedure then operates on these embeddings: it identifies clusters of semantically similar prompts and prunes redundant examples within each cluster, retaining only a proportion $p$ of the filtered data. The paper does not specify the exact value of $p$ or the clustering algorithm (e.g., k-means, hierarchical), but the principle is clear: maximize coverage of the semantic space while minimizing redundancy. This ensures that the annotation budget is spent on a diverse set of conversations rather than repeatedly annotating near-duplicates of the same interaction pattern.

Phase III: Constraint-Based Adjustment. The diversity-sampled subset may not match the desired distribution across certain monitored dimensions. The third phase applies stratified sampling to align statistics with either the original traffic distribution or pre-specified target levels. Table 1 summarizes the key constraints and monitoring dimensions, though the paper does not enumerate all of them explicitly. The monitored dimensions likely include: conversation length (number of turns), conversation category or "job to be done" (JTBD) type (role-playing, information-seeking, casual chat, image generation requests, etc.), language, user demographics (where permissible under privacy constraints), and response characteristics (length, sentiment, presence of certain content types).

This constraint-based adjustment serves two purposes. First, it prevents the annotation and training data from drifting away from the true production distribution—if certain conversation types are under-sampled by the diversity clustering, stratified sampling corrects this. Second, it allows the team to deliberately over-sample or under-sample certain dimensions to address specific model weaknesses. For example, if the current model struggles with role-playing conversations, the team can set a target level for role-playing prompts that exceeds the natural traffic proportion, ensuring the next iteration's training data contains more examples of the challenging category.

What the curation pipeline produces: a curated set of prompts (conversation histories) that is (a) privacy-safe, (b) safety-filtered, (c) semantically diverse with minimal redundancy, and (d) aligned to desired distributional targets across monitored dimensions. This prompt set is the input to both the annotation pipeline (for generating preference labels and quality assessments) and the rejection sampling pipeline (for generating high-quality training responses).

3.4.3 The Annotation Pipeline: Preference Collection, Quality Labeling, and Character Steerability Assessment

The annotation system (Section 2.2.2) converts curated prompts into the structured training signals that drive model improvement. It operates through two distinct modes and supports several annotation tasks, all designed to provide the reward model training data, the SFT data (through rewritten responses), and the quality monitoring signals that feed offline evaluation.

Two annotation modes:

Static annotation: Annotators are presented with complete conversation histories sourced from either internal testing traffic or curated production traffic. They see the full context (character description including name, traits, and behavioral instructions; all previous turns in the conversation; and one or more candidate final responses) and evaluate only the final response(s). This mode is efficient for high-throughput preference labeling because it does not require real-time interaction—annotators process pre-logged conversations in batches.
Interactive-chat annotation: Annotators engage directly with the AI character in real-time, conducting turn-by-turn conversations based on the character description. After each turn, they evaluate the character's response(s) before proceeding to the next turn. This mode is more expensive but provides higher-quality signal because annotators can probe the model's behavior dynamically—asking follow-up questions, testing boundaries, and observing how the model maintains character consistency across multiple turns. This mode is particularly important for character steerability assessment, where mild adversarial probing reveals whether the model truly adheres to character instructions or reverts to default behaviors under pressure.

Annotation tasks (performed in either mode):

Structured quality questions: Annotators answer specific questions about response quality. The paper notes that "throughout development, the specific questions have evolved based on online feedback, new capabilities, and observed model failures," but two core questions are consistently included: (i) whether the response is a false refusal (the model declines a benign request that it should have fulfilled), and (ii) whether the response is a templated response (a repetitive pattern or phrase that occurs across multiple responses or conversations, indicating the model has overfit to a particular phrasing style rather than generating contextually appropriate content). These two questions target the most frequently observed failure modes.
Pairwise preference ranking: Annotators compare two alternative responses generated for the same conversation context and judge which is more engaging. This produces the training data for the Bradley-Terry preference models (Section 2.3.1). The two alternatives are generated either by sampling from different policy models in a pre-defined pool (enabling cross-model comparisons) or by using expert-designed chain-of-thought rewriting prompts to produce high-quality reference responses. To prevent the preference model from learning superficial correlations rather than genuine quality differences, the generated pairwise comparisons are filtered based on constraints such as response length difference and emoji count difference. If the two responses differ too dramatically on these surface features, the pair is excluded from training. This filtering step ensures that annotators are comparing responses that are roughly matched on surface form, forcing them to focus on substance (coherence, engagement value, character adherence) rather than choosing the longer or more emoji-rich response by default.
Response rewriting: When annotators judge a response to be low quality, they rewrite it according to predefined guidelines. These rewritten responses become part of the SFT training data, providing the model with positive examples of what a good response should look like in the exact context where it previously produced a poor response. This is a form of targeted correction: rather than only telling the model "this response was bad" (which the preference model provides), it also shows the model "here is what good looks like."

Character steerability annotation (a specialized workflow): In addition to the standard interactive-chat annotation focused on engagement, the team conducts a separate interactive-chat workflow specifically for character adherence. Here, annotators are explicitly instructed to mildly challenge the model—asking questions or making statements that test whether the model follows the provided character traits or instructions. The character instructions are highlighted in the annotation interface to ensure annotators are aware of what the model is supposed to do. At each turn, two alternative responses are presented, and annotators tag any response that violates the character description. If both alternatives fail to adhere, annotators provide rewritten responses that align with the character. This workflow is the primary mechanism by which the instruction violation rate improved from 26.6% (V2) to 5.8% (V8)—a 78% relative reduction—as reported in Table 6 and Section 3.2.2.

Annotation agreement and multi-review protocols: The paper's experiment in Section 3.4.1 (Table 9) reveals a critical insight about annotation quality. When three independent annotators evaluate the same preference pair, they do not always agree—engagement is inherently subjective. The paper compares three approaches to handling this disagreement: (a) Multi-Review: With Agreement, which keeps only data points where all three annotators unanimously agree on which response is better; (b) Single-Review: All, which keeps all annotations including conflicting labels for the same data point; and (c) Single-Review: Random, which keeps each data point once with a label randomly sampled from the three annotators. The finding is nuanced: for evaluation (measuring whether a trained model improves over a baseline), multi-review consensus evaluation is essential—single-review evaluation sets contain too much label noise to distinguish trained models from untrained baselines. However, for training, single-review data (with conflicting labels) can still produce strong performance on multi-review evaluation, suggesting that "the model can often distill robust preference patterns by aggregating noisy signals from diverse perspectives." This implies that the annotation budget can be allocated efficiently: use multiple annotators for evaluation sets (where precision matters) but single annotators for training sets (where volume matters and the model can average out noise).

3.4.4 Reward Model Architecture and Training: Preference Models and User Signal Models

The reward modeling system (Section 2.3) is the linchpin of the entire optimization pipeline—it converts the non-differentiable engagement objective into differentiable signals that can guide gradient-based policy updates. The system consists of two families of models: preference models (the primary optimization signal, trained on annotator pairwise comparisons) and user signal models (auxiliary signals, trained on production user behaviors).

3.4.4.1 Preference Models: Pointwise and Pairwise Bradley-Terry Reward Models

The preference models learn to predict human annotators' pairwise judgments of which response is more engaging. The paper trains two complementary architectures—pointwise and pairwise—both initialized from Llama 3.1 70B weights and trained on the consolidated preference datasets.

Input representation. Both models take the same structured input, denoted $x$ :

$x = [\text{System Prompt}, \text{Character Instructions}, \text{Conversation History}]$

where $x$ is the concatenation of three textual components: the global system prompt defining the model's role and behavioral guidelines, the character-specific instructions that define the persona the model should embody (name, traits, tone, and behavioral rules), and the full conversation history up to the point where the response being evaluated was generated. This means the reward model sees exactly the same context the policy model saw when generating the response—it evaluates the response in context, not in isolation.

Pointwise model. The pointwise model independently scores a single response given the context. It produces a scalar reward $r_\theta(x, y)$ for response $y$ in context $x$ . The preference between two responses $y_c$ (chosen) and $y_r$ (rejected) is determined by comparing their scalar rewards. The training loss is:

$\mathcal{L}_{\text{pointwise}} = -\log\bigl(\sigma(r_{\theta}(x, y_c) - r_{\theta}(x, y_r))\bigr)$

where $\sigma(z) = 1/(1 + e^{-z})$ is the sigmoid function, $r_\theta(x, y_c)$ is the scalar reward for the chosen response, and $r_\theta(x, y_r)$ is the scalar reward for the rejected response.

What it computes: the negative log-likelihood that the chosen response receives a higher reward than the rejected response, under the Bradley-Terry model of paired comparisons. The sigmoid converts the difference in rewards into a probability $P(y_c \succ y_r) = \sigma(r_\theta(x, y_c) - r_\theta(x, y_r))$ ; the loss penalizes deviations from the annotator's judgment that $y_c$ should be preferred. Minimizing this loss pushes the reward difference $r_\theta(x, y_c) - r_\theta(x, y_r)$ toward large positive values, increasing the model's confidence that the chosen response is better.

Why this form: Bradley-Terry is the standard probabilistic model for pairwise comparison data because it defines a consistent likelihood function over the space of possible rankings. The log-sigmoid loss is exactly the negative log-likelihood under the assumption that the probability of choosing $y_c$ over $y_r$ follows $\sigma(r_c - r_r)$ . This is preferred over a margin-based loss (e.g., hinge loss) because it produces calibrated probabilities that can be interpreted as confidence scores and because it provides a smooth gradient everywhere, avoiding the zero-gradient regions that margin losses produce when the margin is satisfied. The independent scoring architecture (pointwise) is efficient during RL training because only one forward pass is needed per response during reward computation, and because the scalar rewards can be directly used as advantages or reward signals in RL algorithms like GRPO.

Pairwise model. The pairwise model jointly encodes both responses and directly classifies which is superior. Instead of producing separate scores, it takes as input the context $x$ and both responses $y_0$ and $y_1$ (with the superior response randomly assigned to either position to prevent position bias) and outputs a scalar logit $s_\theta(x, y_0, y_1)$ . The training loss is:

$\mathcal{L}_{\text{pairwise}} = -\bigl[t \log \sigma(s_\theta(x, y_0, y_1)) + (1 - t) \log(1 - \sigma(s_\theta(x, y_0, y_1)))\bigr]$

$t = \begin{cases} 1 & \text{if } y_0 \succ y_1 \\ 0 & \text{if } y_1 \succ y_0 \end{cases}$

where $t$ is the binary label indicating which response is preferred (1 if $y_0$ is better, 0 if $y_1$ is better), and $s_\theta(x, y_0, y_1)$ is the model's raw logit for the pair.

What it computes: standard binary cross-entropy on the paired comparison task. When $t=1$ (meaning $y_0$ is the chosen response), the loss is $-\log \sigma(s_\theta)$ , pushing the model to output a large positive logit. When $t=0$ (meaning $y_1$ is chosen), the loss is $-\log(1 - \sigma(s_\theta))$ , pushing the model to output a large negative logit. The sigmoid converts the logit to a probability that the first response is preferred.

Why this form over pointwise: the pairwise model sees both responses simultaneously and can learn to attend to the specific differences between them—for example, noticing that response A has a more appropriate tone for the character while response B contains a factual error. This joint encoding can capture interaction effects that independent scoring misses. However, the pairwise model is more expensive to use during RL training because it requires encoding two responses per comparison (doubling inference cost if both are needed) and does not produce a scalar reward signal that can be directly plugged into advantage estimation. This is why the paper uses the pointwise model for RL training (where per-response efficiency matters) but uses both models during offline evaluation (where the pairwise model's potentially finer-grained discrimination provides a complementary signal to guard against pointwise-specific reward hacking, as discussed in Section 3.1.2).

Training details. Both models are initialized from Llama 3.1 70B weights and trained on the consolidated preference datasets. The paper reports iterative retraining: each new data batch is incorporated into the training set, producing updated reward models (denoted RM_YYMMDD, where the date indicates the latest batch included). Table 8 in Section 3.4 shows the progressive improvement in reward model accuracy as more data is accumulated, with the aggregated evaluation set accuracy improving from 0.652 (RM_240923) to 0.746 (RM_241229) for static data and from 0.573 to 0.650 for interactive data. Specific training hyperparameters (learning rate, batch size, optimizer, number of epochs) are not provided in the paper for the reward models.

Dual-model evaluation to mitigate reward hacking. During offline evaluation, the paper calculates win-rates using both pointwise and pairwise models on the same prompt set. This dual-model approach serves as a safeguard: if the policy overfits to the pointwise model's specific biases (e.g., learning to exploit features that the pointwise model overweights), the pairwise model—which was trained with a different architecture and thus has different inductive biases—should show lower or divergent win-rates. The divergence between the two models' win-rates serves as an early warning signal of reward hacking, as demonstrated by the V12 failure analysis in Section 3.1.2, where the RM User win-rate (based on user-traffic-trained preference models) spiked to 70.7% while the RM Internal win-rate (based on internal-traffic-trained models) dropped to 43.7%.

3.4.4.2 User Signal Models: Predicting Production User Behaviors

The user signal models (Section 2.3.2) are a separate family of binary classifiers trained to predict specific user behaviors from production logs. These behaviors—thumbs-up, conversation continuation, response regeneration, emoji reactions, and others—are abundant (millions occur daily) and directly reflect user satisfaction, making them attractive as auxiliary training signals.

Training formulation. For each user signal $i$ , a binary classifier $u_\theta$ is trained to predict whether the signal occurred given the context $x$ (same structured input as the preference models: system prompt, character instructions, conversation history) and the model response $y$ . The loss is:

$\mathcal{L}_{\text{signal}_i} = -\bigl[s \log \sigma(u_\theta(x, y)) + (1 - s) \log(1 - \sigma(u_\theta(x, y)))\bigr]$

$s = \begin{cases} 1 & \text{if signal}_i \text{ occurs} \\ 0 & \text{otherwise} \end{cases}$

where $s$ is the binary label derived from production logs, and $\sigma(u_\theta(x, y))$ is the model's predicted probability that the signal will occur.

What it computes: standard binary cross-entropy between the model's predicted signal probability and the observed binary outcome. The model learns to map conversation context and response content to the probability of specific user behaviors.

Why this form: binary cross-entropy is the appropriate loss for probabilistic binary classification. The model outputs a probability via the sigmoid, which is well-calibrated for expected-frequency prediction.

Model scale. For most user signals, the paper experimented with initializing $u_\theta$ from both Llama 3.1 8B and 70B, but primarily used the smaller 8B model because "it is parameter-efficient yet sufficient to fit the signal data." This is a practical design choice: the user signal models need to be accurate enough to provide useful ranking signals for rejection sampling but do not need the full capacity of a 70B model, and the 8B model is much cheaper to train and run inference on during data processing.

Signals explored and ultimate usage. Table 2 (referred to as Table 7 in the text due to a numbering inconsistency—the paper's Section 3.3.4 is labeled Table 7 but discusses failure modes, while Section 2.3.2 describes the user signals table) lists all the user signals the team experimented with during development. However, the paper explicitly states that they ultimately used only $p(\text{continue})$ (the probability that the user continues the conversation after this response) and $p(\text{thumb up})$ (the probability that the user gives a thumbs-up reaction) as signals for rejection sampling data selection, because these two demonstrated "consistent and reliable performance." All other user signal models were explored but not used for optimization.

Why user signal models are not used for direct RL optimization. Section 3.5.5 provides a detailed analysis of why user signals, despite their abundance and direct connection to user satisfaction, are unsuitable for direct RL training. The paper catalogs four specific confounds:

Delayed feedback: Users skip thumbs-up on early clarifying responses but react to the final concrete answer. This causes the $p(\text{thumb up})$ model to favor verbose, immediate-answer responses over useful clarification questions, penalizing a genuinely helpful conversational behavior.
Ending bias: Users typically thumbs-up at conversation end, where flattery ("thank you," "have a good night") is common. This reproduces the sycophancy problem documented in ChatGPT, where models learn to be excessively agreeable and complimentary.
Inconsistent positive/negative ratios across job types: The ratio of positive to negative signals varies dramatically across different conversation categories (e.g., "Role-playing – Romantic" vs. "Image generation"). This enables reward models to partially identify the conversation type from context features and predict signals based on the base rate for that type, shortcutting actual response quality assessment.
Confounding context: The user signal models over-index on prior-turn sentiment or satisfaction signals (e.g., if the user was happy in previous turns, predict high probability for the current turn regardless of response quality) rather than focusing on the current response's content.

These confounds explain a critical design decision: user signal models are used only as additional scores in the rejection sampling ranking (Section 2.4.1, Algorithm 1), where they supplement the preference model scores in selecting which candidate responses to include in the SFT dataset. They are explicitly NOT used as reward signals in RL training because their confounds make them susceptible to reward hacking—the policy would learn to exploit the predictors' blind spots rather than genuinely improving user experience. This decision was further validated by the V12 failure analysis: "Given these limitations and failure of the V12 experiment, we use user signal models as additional scores in rejection sampling ranking rather than for direct RL optimization."

3.4.5 Rejection Sampling: Constructing High-Quality SFT Data from Production Prompts

The rejection sampling pipeline (Section 2.4.1, Algorithm 1) is the mechanism for converting curated production prompts and reward model scores into a high-quality supervised fine-tuning dataset. The core idea is simple but powerful: for each prompt in the curated set, generate multiple candidate responses from a pool of strong policy models, score them with the reward model, keep only the highest-scoring response IF it exceeds a quality threshold, and use these (prompt, response) pairs as SFT training examples.

Algorithm 1: Rejection Sampling

The paper presents the algorithm formally. I will walk through it step by step:

Inputs:

$\mathcal{D}_{\text{prompt}}$ : A set of prompts drawn from the curated production traffic (output of the data curation pipeline in Section 2.2.1).
$\{\mathcal{M}_1, \ldots, \mathcal{M}_L\}$ : A family of candidate LLMs/policies from which to generate responses. The paper notes that although only 70B models are deployed in production (for inference efficiency), they also develop 405B models in parallel using the same CharacterFlywheel process and include them in the candidate pool. The larger models can produce higher-quality responses, which the 70B model can then learn from during SFT.
$r$ : A reward model (typically the pointwise preference model from Section 2.3.1).

Procedure:

For each prompt $X_i$ in $\mathcal{D}_{\text{prompt}}$ :

Model selection: Find one candidate model $\mathcal{M}_l$ most suitable for $X_i$ . The paper does not specify the selection mechanism in detail, but it likely involves either routing based on prompt characteristics (e.g., certain model versions are better at certain conversation types) or simply cycling through the candidate pool to ensure diversity.
Response generation: Generate $k$ candidate responses for $X_i$ using the selected model $\mathcal{M}_l$ , resulting in $\{Y_{i,1}, \ldots, Y_{i,k}\}$ . The paper does not specify the exact value of $k$ used in practice, but based on the scale of the operation, it is likely a small number (perhaps 4–16) given that this must be done for every prompt in a large curated set and across multiple iterations.
Reward scoring: Use the reward model to compute a scalar score for each candidate response: $r_{\text{max}} = \max_{j=1,\ldots,k} r(X_i, Y_{i,j})$ with $j^* = \arg\max_{j=1,\ldots,k} r(X_i, Y_{i,j})$ identifying which candidate achieved the highest score.
Threshold-based filtering: If $r_{\text{max}} \geq \tau$ for some pre-specified threshold $\tau > 0$ , keep the highest-scoring pair $(X_i, Y_{i, j^*})$ and add it to the rejection sampling dataset $\mathcal{D}_{\text{RS}}$ . If no candidate exceeds the threshold, the prompt is skipped entirely—no training example is created from it.

Output: The rejection sampling training dataset $\mathcal{D}_{\text{RS}}$ , consisting of (prompt, highest-scoring response) pairs that all exceed the quality threshold.

Off-policy to near-policy bridging. The paper explicitly acknowledges that rejection sampling is fundamentally an off-policy process—the responses are generated by models in the candidate pool, not by the current model being trained. However, the team strives to keep the dataset as up-to-date as possible: "we strive to keep the dataset as up-to-date as possible with inference-time outputs, thereby approximating an on-policy setting, as it has been shown to enhance RL performance." This is achieved by reconstructing $\mathcal{D}_{\text{RS}}$ with each new model update, leveraging the most recent user traffic $\mathcal{D}_{\text{prompt}}$ . The idea is that even though the responses are generated off-policy, the prompts come from the distribution induced by the latest deployed model, keeping the training data distributionally close to what the current model will encounter in production.

Practical significance. The rejection sampling dataset is one component of the SFT training mixture described in Section 2.4.2. It provides high-quality, reward-model-approved examples covering the full diversity of real user prompts. Because the threshold $\tau$ filters out low-quality responses, the dataset only contains examples where the model can observe "what a good response looks like" in realistic contexts. This is superior to training only on human-written responses (which may not cover the full prompt distribution) or on unfiltered model outputs (which would include many low-quality examples that could degrade performance).

Why rejection sampling over other data selection methods: rejection sampling provides a principled way to convert a reward model's scores into a binary accept/reject decision with a quality guarantee (only responses above threshold are included). This is simpler and more robust than alternatives like weighting all responses by their reward scores (which would give partial credit to poor responses) or using the reward model directly as a loss function during SFT (which would change the training objective away from standard likelihood maximization). The threshold provides a clean separation: the model sees only "good enough" examples and learns to imitate them, while poor examples are simply excluded.

3.4.6 The Staged Training Recipe: SFT → DPO → RL

The policy optimization follows a fixed three-stage recipe (Section 2.4.2 and 2.4.3), each stage serving a distinct purpose in the overall improvement strategy.

3.4.6.1 Stage 1: Supervised Fine-Tuning (SFT)

SFT establishes the baseline policy that subsequent stages refine. The training dataset is a mixture of six components, carefully balanced:

Rejection sampling (RJS) data from periodically updated internal interactive chats: High-quality responses generated by strong models (including 405B variants) on prompts from internal annotator interactions, filtered through the reward model threshold in Algorithm 1.
Rejection sampling (RJS) data from periodically updated user traffic: The same process applied to curated production prompts, providing broad coverage of real user interactions.
Internal safety data: Examples designed to teach the model appropriate refusal behavior—when to refuse unsafe requests and, critically, when not to falsely refuse benign requests. This addresses the false refusal failure mode that the paper monitors extensively (Section 3.3.1, Figure 10).
Capability and tool-calling data: Examples for image generation (both explicit user requests and implicit autonomous triggering) and search functionality, teaching the model when and how to invoke these tools.
Ad-hoc internal and user data for failure modes: Targeted examples addressing specific observed model weaknesses, collected and added reactively as new failure patterns emerge in production or evaluation. This is the mechanism by which the "self-correcting" behavior described in Section 3.3 operates—when a version exhibits elevated false refusals or preachy tone, the next iteration's SFT data includes counterexamples targeting those specific failures.
Llama 3.1 post-training SFT data: Data from the original Llama 3.1 training pipeline (grattafiori2024llama), included to maintain competitive performance on community benchmarks and general knowledge tasks. This prevents catastrophic forgetting of the base model's utility capabilities while optimizing for social engagement.

Data mixture ratio tuning. The paper states that "the data mixture ratio is carefully tuned to ensure optimal performance," but does not provide the specific ratios or the tuning methodology. This is a significant practical detail that affects reproducibility—the balance between engagement-focused data (components 1–2, 5), safety data (3), capability data (4), and general-purpose data (6) determines how much the model's behavior shifts toward social engagement versus retaining utility. The tuning likely involves offline evaluation on community benchmarks (to ensure minimal regression) and production metrics (to ensure engagement improvement).

3.4.6.2 Stage 2: Direct Preference Optimization (DPO)

After SFT, the checkpoint undergoes DPO on a small set of preference data. DPO (Direct Preference Optimization) is an alignment technique that directly optimizes the policy to prefer chosen responses over rejected ones without training a separate reward model, using a loss function that implicitly represents the reward as a function of the policy's log-probability ratio relative to a reference model.

The DPO dataset includes: internal safety preference data (pairs where the chosen response correctly handles a safety-sensitive prompt and the rejected response fails), image generation preference data (pairs where the chosen response appropriately triggers or describes image generation and the rejected response does not), and Llama 3.1 preference data.

Why DPO is used as a "small patch" rather than the primary optimization: The paper explicitly acknowledges the off-policy nature of DPO (tang2024understanding)—it uses a fixed preference dataset collected before training rather than generating on-policy responses during training. The paper states that "despite the off-policy nature of DPO, we observe that treating DPO as a small patch for urgent safety and style fixes remains effective in production scenarios without over-complicating the overall training process." This is a pragmatic design choice: SFT establishes the broad behavior, RL (the next stage) handles engagement optimization at scale, and DPO serves as a lightweight mechanism to inject targeted corrections (safety edge cases, style adjustments) without the complexity and cost of full online RL for these specific fixes.

3.4.6.3 Stage 3: Online Reinforcement Learning

The final and most engagement-focused stage is online RL, where the model generates its own responses, receives reward scores (from the preference model), and updates its policy to increase expected reward. The paper experimented with two RL loss formulations and ultimately adopted one based on empirical A/B test results.

Single-turn optimization formulation. A critical design choice that simplifies the RL setup: rather than simulating full multi-turn conversations (which would require modeling user responses and maintaining trajectory-level credit assignment), the RL training uses static prompts—fixed partial conversation histories sampled from production traffic—and optimizes only the model's final response for that prompt. The paper acknowledges that "this setup avoids the complexity in simulating full conversation, it might compromise the on-policy property." However, when combined with near-policy prompts and tight model iteration loops, this semi-online approach "remains effective for optimizing engagement."

Online DPO loss (initial approach). Online DPO (qi2024online) extends standard DPO by using self-generated responses rather than a fixed offline dataset. The model generates candidate responses for each prompt during training, the reward model scores them to identify chosen and rejected pairs, and the policy is updated to increase the probability of chosen responses relative to rejected ones. The paper does not provide the exact loss equation for this variant but references the standard online DPO formulation.

GRPO variants with importance sampling (final approach). The paper switched from online DPO to a GRPO (Group Relative Policy Optimization) variant (shao2024deepseekmath) with importance sampling corrections (wu2025llamarl). The loss is:

$\mathcal{L}_{\text{GRPO}} = \mathbb{E}_{x \sim \pi_{\text{gen}}} \left[ \frac{\pi_{\theta_{\text{old}}}(x)}{\pi_{\text{gen}}(x)} \min\!\left( \frac{\pi_{\theta}(x)}{\pi_{\theta_{\text{old}}}(x)} A_t, \operatorname{clip}\!\left(\frac{\pi_{\theta}(x)}{\pi_{\theta_{\text{old}}}(x)}, 1-\epsilon, 1+\epsilon\right) A_t \right) \right] - \beta D_{\mathrm{KL}}(\pi_{\theta} \| \pi_{\text{ref}})$

where $\pi_{\theta}$ is the current policy being optimized, $\pi_{\text{gen}}$ is the behavior policy used for data collection (the policy that generated the training responses), $A_t$ is the estimated advantage for a particular response (reflecting how much better or worse it is compared to a baseline), $\epsilon$ is the clipping threshold that prevents excessively large policy updates, and $\pi_{\text{ref}}$ is the reference policy maintained as an exponential moving average of the initial and intermediate checkpoints.

What it computes: a clipped surrogate objective with importance sampling correction, followed by a KL divergence penalty. The first term (the expectation over prompts and the policy ratio) is the standard PPO-style clipped objective: for each response, it computes the ratio $\pi_\theta(x) / \pi_{\theta_{\text{old}}}(x)$ (how much more or less likely the current policy makes that response compared to the old policy), multiplies by the advantage $A_t$ (reward signal indicating response quality), and clips the ratio to stay within $[1-\epsilon, 1+\epsilon]$ of the old policy. The $\min$ operation ensures the objective is a lower bound—it prevents the policy from taking excessively large steps when the advantage is positive and the ratio is large. The importance sampling ratio $\pi_{\theta_{\text{old}}}(x) / \pi_{\text{gen}}(x)$ corrects for the fact that the training data was generated by the behavior policy $\pi_{\text{gen}}$ (which may differ from $\pi_{\theta_{\text{old}}}$ due to distributed training asynchronicity), ensuring unbiased gradient estimates. The second term $-\beta D_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{ref}})$ penalizes the policy for diverging too far from the reference policy, which serves as a regularizer preventing catastrophic forgetting and maintaining basic capabilities.

Why this form over online DPO: the paper reports (Section 3.5.3) that a model trained with GRPO achieved a +1.52% Engagement Breadth Metric Lift over a model trained with Online DPO loss from the same starting checkpoint and training data, in a direct A/B test. The paper hypothesizes this is "likely due to its ability to exploit reward scores from all generated responses, which provide a more fine-grained supervision signal." Unlike online DPO, which reduces all responses to binary chosen/rejected pairs, GRPO can use the continuous reward scores to compute advantages that distinguish between multiple response quality levels—a response that scored 0.8 is treated as better than one that scored 0.3, not just as "better than the worst one."

Reference policy maintenance. The reference policy $\pi_{\text{ref}}$ is maintained as an exponential moving average (EMA) of the initial checkpoint and intermediate checkpoints during RL training. This means it slowly tracks the policy's evolution, providing a moving baseline that prevents the policy from drifting too far in any single update while still allowing gradual movement over many steps.

Design choices more influential than the exact RL loss. The paper notes that "several other design choices can be more influential than the exact RL loss to use within CharacterFlywheel development cycles," pointing to prompt selection (near-policy vs. off-policy, Section 3.5.2), variance-based downsampling (Section 3.5.4), and reward model quality as being at least as important as the choice between DPO and GRPO. This is consistent with the broader literature on RLHF, where data quality and reward model accuracy often dominate algorithmic choices.

3.4.7 Prompt Selection for RL: Near-Policy Data and Variance-Based Downsampling

The effectiveness of RL training depends critically on which prompts are used to generate training responses. The paper explores two dimensions of prompt selection, each motivated by the landscape-climbing metaphor.

Near-policy vs. off-policy prompts (Section 3.5.2). The paper conducted a controlled experiment comparing two RL prompt sets from the same starting checkpoint: (a) prompts from the latest deployed model's production traffic (near-policy), and (b) prompts from earlier model versions' traffic (off-policy). In a direct A/B test, the near-policy set achieved +10.6% Engagement Depth Metric Lift and +1.6% Engagement Breadth Metric Lift compared to the off-policy counterpart. This is a substantial difference, and the paper interprets it through the landscape-climbing metaphor: "we obtain the most effective gradients when training samples sufficiently estimate the contour near the current policy, enabling continual hill-climbing within the policy space." If prompts come from an old policy's distribution, they lie in a different region of the engagement landscape, and the reward model's gradient estimates at those points may not point in the direction that would improve the current policy.

Why this matters practically: this finding implies that each RL training cycle should use the most recently deployed model's traffic as its prompt source, not a historical corpus. This creates a tight coupling between deployment and training—each new deployment not only validates the previous cycle's improvements but also generates the data for the next cycle's RL training.

Variance-based downsampling (Section 3.5.4). Not all prompts are equally useful for RL training. A standard heuristic in post-training is to focus on "hard" prompts where the current model performs poorly, as these provide the most learning signal (yu2025rip). The standard approach is to select prompts with the lowest average reward model scores (indicating the model's responses are low quality on those prompts). However, the paper found this approach unreliable in their setting because "preference RMs do not regularize score magnitudes across prompts, so scores often reflect stylistic factors (e.g., length, conversation turns) rather than difficulty." For instance, longer-turn conversations systematically receive lower RM scores regardless of response quality, causing certain conversation types (Roleplay, Romantic) to be over-represented by 4× when sampling based on average RM scores.

The paper's solution is a variance-based strategy: instead of selecting prompts with low mean RM scores, select prompts with high RM score variance across multiple responses. The intuition is that "difficult prompts induce wider spreads in response quality," because on easy prompts, the model consistently produces good (or consistently produces bad) responses, resulting in low score variance. On genuinely difficult prompts, some responses happen to be good while others are poor, creating high variance that the RL training can exploit—the model can learn to shift its output distribution toward the higher-scoring responses.

What this achieves: variance-based downsampling provides a more robust difficulty signal that is less confounded by stylistic score miscalibration. It automatically upweights prompts where the model's performance is inconsistent (and thus improvable through policy updates) and downweights prompts where performance is uniformly good or uniformly poor (where policy updates would have little effect).

3.4.8 Stylistic Artifact Mitigation: Preventing Shallow Optimization

A persistent risk in reward-model-guided optimization is that the policy learns to exploit surface-level features that correlate with high reward scores but do not reflect genuine quality improvements. The paper addresses this through an active artifact-mitigation process (Section 2.4.4) that monitors and controls stylistic patterns throughout training.

Artifact features. An artifact feature is defined as a function of the conversation history and response that returns either a binary indicator (e.g., whether the response contains the phrase "I feel like...") or a real-valued measurement (e.g., the number of emojis in the response). Most features in practice depend only on the response itself, not the preceding context.

Monitoring in preference data. The team compares feature prevalence (for binary features) and feature distributions (for real-valued features) between chosen and rejected responses in the preference training data. If a specific stylistic pattern is significantly more common in chosen responses than in rejected ones, it may be spuriously correlated with annotator preferences—annotators might prefer emoji-rich responses or longer responses not because they are genuinely more engaging but because they appear more effortful or personable at first glance. Identifying these correlations allows the team to adjust annotation guidelines (e.g., instructing annotators to focus on substance rather than style) or to apply corrective filtering to the preference dataset.

Monitoring in rejection sampling data. Similarly, the team compares accepted (above-threshold) versus rejected (below-threshold) candidate responses in the rejection sampling pipeline. If artifacts disproportionately drive acceptance decisions—for example, if the reward model systematically assigns higher scores to responses containing emojis regardless of content quality—this monitoring catches the bias so that corrective action can be taken.

Monitoring during training. The team tracks model generations from checkpoints after SFT, DPO, and RL to determine whether particular training stages induce significant shifts in stylistic features. If RL training causes a sudden spike in emoji usage (as actually occurred with V11-V12, noted in Section 3.3.2 and Figure 10), this monitoring enables rapid detection and intervention in subsequent iterations.

The V12 emoji spike as a case study. Section 3.3.2 documents that "Contains Emoji exhibits the most dramatic variation (238.5% relative change), with notable spikes at V11-V12." This spike coincided with the V12 engagement degradation described in Section 3.1.2. The subsequent versions (V13-V15) successfully moderated emoji frequency by adjusting annotation guidelines and data composition—a concrete example of the self-correcting feedback loop in action.

The emoji reduction experiment (Section 3.5.6). The paper provides direct experimental evidence that conversation history, not just reward model bias, can drive artifact amplification. In an emoji reduction experiment, the team removed all emojis from the reward model's input during scoring (effectively "debiasing" the RM with respect to emoji usage). Despite this, the average emoji count in model responses still increased from 0.2 to 0.48 over 120 RL steps. This demonstrates that "models can inherit and amplify biases directly from conversation history"—the training prompts themselves contained emojis, and the autoregressive policy model mimicked this pattern, amplifying it over successive generations even though the RM was blind to emojis. This finding motivated the implementation of prompt pre-processing and the broader bias monitoring and mitigation framework described in Section 2.4.4.

3.4.9 Offline Evaluation: Five-Pronged Gating Before Deployment

Before any model checkpoint reaches production users, it must pass through a comprehensive offline evaluation framework (Section 2.5.1) comprising five categories of assessment. This gating process is essential because once a model is deployed, regressions in quality, safety, or engagement affect millions of users and take at least a week to detect through A/B tests.

1. Community Benchmarks. The model is evaluated on a suite of standard LLM benchmarks listed in Table 3: MMLU (general knowledge), GSM8K (grade-school math), MATH (competition math), HumanEval (code generation), MBPP (code generation), ARC Challenge (reasoning), GPQA (graduate-level QA), and IFEval (instruction following). The intention is explicitly NOT to achieve state-of-the-art results but to "ensure robust performance on factual and utility-seeking questions" and to detect catastrophic forgetting of base capabilities. Table 5 and Figure 9 show that CharacterFlywheel V7 maintains competitive performance across most benchmarks—MMLU at 79.5% (vs. 83.6% for the Llama 3 70B baseline), GSM8K at 92.3% (vs. 95.1%), and ARC Challenge at 93.1% (vs. 94.8%)—with the largest regressions in MATH (50.5% vs. 68.0%) and MBPP (66.6% vs. 86.0%). The paper accepts these trade-offs as the cost of optimizing for engagement over utility.

2. Human Comparison. To assess whether a new model genuinely outperforms the previous version on engagement quality, the team conducts side-by-side human comparisons using the interactive-chat annotation procedure. Annotators engage with both models on the same characters and conversation contexts, seeing responses from both at each turn. To ensure the conversation history does not favor either model, the annotator randomly selects which model's response to continue the conversation with for the next turn, regardless of which response was preferred. This randomization prevents the conversation from drifting into regions where one model has an advantage. The resulting win-rate—the fraction of turns where annotators prefer the new model's response—must exceed 50% (the neutral threshold) for the model to advance. Figure 6 (right panel) shows that pre-launch versions consistently achieved human win-rates of 50.2%–52.5% against their immediate predecessors, and human win-rates against GPT-4o improved from 37.4% (V3) to 46.2% (V7).

3. Reward Model Win-rate. The trained preference models (both pointwise and pairwise) are used to compute win-rates on held-out evaluation prompt sets that are kept completely separate from training prompts. This serves two purposes: (a) it provides an automated, scalable signal of whether the new model is improving according to the surrogate objective, and (b) the divergence between pointwise and pairwise win-rates, and between RM Internal (trained on internal traffic) and RM User (trained on user traffic) win-rates, serves as an early warning of reward hacking. The V12 failure was detected in part because RM User win-rate spiked to 70.7% while RM Internal dropped to 43.7%—a divergence that signaled the model was overfitting to the user-traffic-trained RM at the expense of generalization.

4. Custom Production Metrics. The team developed a suite of ad-hoc metrics evaluated using LLM-as-a-judge or rule-based methods on curated traffic prompts (separate from all training prompts). Table 4 summarizes the most significant metrics, which are tracked across all 15 versions in Figure 10: false refusal rate (both on internal and user traffic), response formatting characteristics (average response length, contains list percentage, contains emoji percentage), tone and sentiment quality (preachy tone, positive sentiment, cooperative ratio, non-preachy rate), and quality/failure modes (instruction violation rate, wall-of-text failure). These metrics are iteratively refined—"adding and removing metrics based on efficiency considerations"—as new failure modes are discovered and old ones are resolved.

5. Safety and Privacy Evaluation (Section 2.6). In addition to the quality metrics, models undergo automated safety classifier evaluation at multiple stages and manual red-teaming review before deployment. Only models that pass both automated and manual safety reviews are deployed. The safety framework operates on a "fail-closed" principle: if any required safety rule fails at the character creation or interaction stage, the system automatically rejects the character or response. This is described more fully in Section 3.4.12 below.

3.4.10 Online Evaluation: A/B Testing with Fieller Confidence Intervals

The ultimate validation of any model improvement is whether it produces statistically significant lifts in real user engagement metrics during controlled online A/B tests (Section 2.5.2). The paper formalizes the engagement metrics and the statistical methodology with unusual precision.

A/B test design. For each model update or promising training recipe change, eligible users are randomly assigned to either a test arm (receiving the updated model) or a control arm (receiving the current baseline). The randomization is independent and adheres to platform constraints; typically 10% of traffic goes to each arm to balance engineering velocity, risk, and statistical power. Metrics are assessed over a one-week readout window using consistent inclusion criteria and cumulative exposure logging to define the analysis population.

Engagement breadth metric. This measures the proportion of the user's evaluation periods in which they exhibit any engagement with the AI character. Formally, let $i$ index users in group $g \in \{\text{test}, \text{control}\}$ and let $d \in \mathcal{D}$ index evaluation periods (e.g., days) in the one-week readout window. For each user-period pair, observe a binary engagement indicator $Y_{i,d} \in \{0, 1\}$ . The average engagement for user $i$ over the window is:

$\bar{Y}_i = \frac{1}{|\mathcal{D}|} \sum_{d \in \mathcal{D}} Y_{i,d}$

The group-level engagement breadth estimand and its empirical estimator are:

$\mu_g^{\text{breadth}} = \mathbb{E}[\bar{Y}_i \mid i \in g], \quad \hat{\mu}_g^{\text{breadth}} = \frac{1}{n_g} \sum_{i=1}^{n_g} \bar{Y}_i$

What it computes: the expected fraction of evaluation periods (e.g., days in the week) in which a user in group $g$ engages with the AI, estimated as the simple average of each user's per-period engagement rate. A higher breadth means users engage more consistently over the week rather than trying once and abandoning.

Engagement depth metric. This measures the intensity of engagement among users who engage at all. Let $S_i$ denote a nonnegative aggregate engagement measure for user $i$ over the readout window (e.g., total number of messages sent, total conversation turns, total time spent), and define $A_i = \mathbb{I}(S_i > 0)$ as an indicator of any engagement. The group-level engagement depth estimand conditions on positive engagement:

$\mu_g^{\text{depth}} = \mathbb{E}[S_i \mid i \in g, A_i = 1], \quad \hat{\mu}_g^{\text{depth}} = \frac{\sum_{i=1}^{n_g} S_i}{\sum_{i=1}^{n_g} A_i}$

What it computes: the expected total engagement for users who engage at all, estimated as the ratio of total engagement sum to total number of engaged users in group $g$ . A higher depth means that engaged users have longer, more involved conversations.

Percentage lift and confidence intervals. For both metrics, the percentage lift of the test group over the control group is:

$\widehat{\text{Lift}}(\%) = 100 \times \left( \frac{\hat{\mu}_{\text{test}}}{\hat{\mu}_{\text{control}}} - 1 \right)$

Because this is a ratio of means, standard symmetric normal confidence intervals can be unreliable—the denominator's uncertainty matters, and the distribution may be skewed. The paper uses Fieller's Theorem (fieller1954some) to construct confidence intervals. Fieller's method provides an exact (under normality) or near-exact confidence interval for a ratio by solving for the set of plausible ratios $r$ satisfying:

$\frac{(\hat{\mu}_T - r \hat{\mu}_C)^2}{\sigma_T^2 + r^2 \sigma_C^2} \leq z^2$

where $\hat{\mu}_T$ and $\hat{\mu}_C$ are the test and control group means, $\sigma_T^2$ and $\sigma_C^2$ are their estimated variances (computed as $\sigma_g^2 = S_g^2 / n_g$ where $S_g^2$ is the sample variance of user-level outcomes), and $z = z_{1-\alpha/2}$ is the standard normal critical value (e.g., 1.96 for 95% confidence). Solving this quadratic inequality for $r$ yields the closed-form bounds provided in Appendix 6, Equation 17, which produce naturally asymmetric intervals that account for denominator uncertainty.

Why Fieller over delta method or bootstrap: the delta method approximates the variance of a ratio using a first-order Taylor expansion, which can be inaccurate when the denominator's coefficient of variation is large. The bootstrap can handle skewness but is computationally expensive at Meta's scale (millions of users, hundreds of metrics). Fieller's method provides a computationally cheap, theoretically motivated alternative that produces valid asymmetric intervals under mild normality assumptions.

Statistical significance. A model update is declared to have a statistically significant positive lift if the null lift (0%) is not contained within the Fieller confidence interval. The paper reports lifts with green markers for significant positive, red for significant negative, and gray for non-significant results (Figure 8, top panel).

3.4.11 Empirical Guardrails: The V12 Failure and the 65% RM Win-Rate Threshold

The V12 deployment failure (Section 3.1.2) is the paper's central empirical finding about safe optimization practices, and it directly motivates the conservative thresholds that govern subsequent iterations.

What happened with V12. After several successful versions (V8-V11) that demonstrated positive engagement lifts, V12 was trained using more aggressive optimization against the reward model trained on user traffic. The results were stark: engagement breadth showed only +0.05% lift (effectively zero) and engagement depth showed -2.9% lift (a meaningful regression). Meanwhile, the RM User win-rate (the reward model's preference for V12 responses over the baseline on user traffic evaluation prompts) spiked to 70.7%, dramatically higher than the typical 50-65% range of successful versions. Simultaneously, the RM Internal win-rate (the reward model trained on internal traffic) dropped to 43.7%, indicating that from the internal annotators' perspective, V12 was actually worse than the baseline.

Interpretation through the landscape-climbing metaphor. The paper interprets this as the policy having been pushed into a region of the reward landscape where the reward model's predictions were unreliable—"analogous to climbing beyond the reliable contours of our optimization map." The reward model, trained on a finite dataset of user-traffic-derived preferences, had high confidence (steep gradient) in certain directions that correlated with high reward scores in the training distribution but did not generalize to genuine user satisfaction. Aggressive RL optimization exploited these spurious correlations, driving the RM win-rate artificially high while degrading the true objective.

The established guardrail. Based on this failure analysis and empirical observation across successful and failed deployments, the paper established a safer operating threshold: "RM win rates should remain below 65%, with 60% being the ideal target for sustainable optimization." This threshold ensures the policy exploits the reward signal while maintaining sufficient margin from the unreliable regions of the learned reward landscape. If RM win rates approach or exceed 65%, the optimization is considered too aggressive and the training recipe is adjusted (e.g., earlier stopping, reduced RL steps, stronger KL regularization).

Monitoring multiple RM signals. The paper now monitors both RM Internal and RM User win-rates as complementary signals. Significant divergence between them—as occurred with V12 (70.7% User vs. 43.7% Internal)—is treated as a red flag even if individual win-rates are within range, because divergence indicates the policy is overfitting to one data distribution at the expense of generalization.

Subsequent validation. Versions V13-V15, trained under these guardrails, demonstrated restored positive engagement trends, validating that conservative optimization is both safer and more effective in the long run than aggressive reward hacking.

3.4.12 Safety and Privacy: Layered, Fail-Closed Protection

The safety and privacy framework (Section 2.6) operates at multiple stages of the development and deployment pipeline, guided by principles inherited from Llama 3.1 (grattafiori2024llama) with the primary objective of "minimizing safety violations and reducing false refusals."

Layered evaluation. Safety classifiers (automated models) are applied at five distinct stages: (1) during character auto-generation (when users create new AI characters, safety classifiers reject unsafe character descriptions before they are published), (2) during character updates (if a user modifies a character's description or instructions, classifiers re-check for safety), (3) during model updates (new model versions are screened before deployment), (4) during user online interaction (real-time classification of both user prompts and model responses, with unsafe content automatically rejected), and (5) during user traffic sampling for the data curation pipeline (Section 2.2.1), ensuring that no unsafe content enters the training data. Manual human review is triggered for uncertain classifications, reported characters, and ambiguous cases that the automated classifiers cannot resolve with high confidence.

Fail-closed design. The paper describes a "fail-closed" architecture at two levels. At the character creation level: "if any required safety rule fails, the system automatically rejects the creation, preventing unsafe characters from being published. Ambiguous cases are escalated to manual review for a final decision." At the interaction level: "the system enforces safety by rejecting unsafe prompts and model responses automatically, ensuring all interactions comply with established safety standards." For model updates: "only those models that successfully pass both automated evaluation and redteaming review are deployed to production."

Privacy protections. In addition to safety, the data curation pipeline applies "privacy rules, including model-based and rule-based checks on high-risk identifiers" to ensure no personally identifiable information (names, phone numbers, addresses, email addresses, etc.) is included in downstream processing—including annotation, reward model training, SFT, and RL. This is a hard requirement: any data point that triggers a privacy rule is excluded entirely from all subsequent stages.

Why this matters for the optimization loop. The safety and privacy filters are not just compliance measures—they directly affect the optimization process. If safety classifiers are too aggressive (rejecting benign content as unsafe), the model learns from a censored data distribution and may become overly conservative, leading to the elevated false refusal rates observed in versions V5-V6 (Figure 10, top row). If safety classifiers are too permissive, unsafe content enters the training data and may be reproduced by the model. The "self-correcting" behavior documented in Section 3.3—where false refusal spikes are resolved within a few iterations—relies on the monitoring system detecting these imbalances and adjusting the data mixture or annotation guidelines in subsequent cycles.

3.4.13 Image Generation: Explicit and Implicit Tool-Calling for Engagement

Image generation capability (Section 2.7) is integrated into the CharacterFlywheel pipeline as an agentic tool-calling task that contributes significantly to engagement. The paper reports that explicit image generation (user-requested) in V9 achieved +1.7% Engagement Breadth Metric Lift, and implicit image generation (model-initiated) in V10 contributed an additional +2.1% lift.

Two generation scenarios:

Explicit Generation: The user explicitly prompts the model to create an image, functioning similarly to standard multimodal chatbots. The model must recognize the request, generate an appropriate image prompt for the downstream text-to-image (T2I) model, and invoke the tool.
Implicit Generation: A novel mechanism where the LLM autonomously decides to trigger image generation when it determines that visual content will enrich the conversation—for example, generating an illustration of a scene being described in a role-playing conversation without the user asking for one. This requires the model to assess conversational context, infer when an image would be engaging, and generate a relevant prompt.

Training formulation. The paper frames this as an agentic tool-calling task (schick2023toolformerlanguagemodelsteach): the LLM learns both when to trigger generation and what generation parameters (specifically, the image prompt) to provide. The downstream T2I model is treated as a standalone service called via the tool interface—the CharacterFlywheel model does not generate images directly.

Annotation challenges. Given the subjectivity of implicit image generation (when is an image appropriate? What kind of image enhances engagement?), data annotation is particularly challenging. The paper enforces multi-review annotation protocols to retain only high-consensus data, requiring annotator agreement on two dimensions: (1) the appropriateness of triggering generation at the current conversational turn, and (2) the quality of the image prompt in capturing the full conversational context including history.

Integration into the training pipeline. Preference data collected through this annotation process are used to train preference models that understand image-generation quality, and these preference models inform the construction of SFT and DPO datasets for the image generation capability. This means the image generation component follows the same pre-herding/herding cycle as the text-only engagement optimization—annotator preferences train reward models, reward models guide data selection, and SFT/DPO/RL train the policy.

3.4.14 Summary of Key Design Choices and Their Justifications

Preference models as primary reward signal, user signal models as auxiliary. Preference models provide controlled, fine-grained annotations with known quality (through multi-review agreement tracking). User signal models are abundant but deeply confounded (Section 3.5.5), making them suitable only for rejection sampling ranking, not RL optimization. This hierarchical approach prevents the reward hacking that naive user-signal optimization would cause.

Iterative reward model retraining with consistent annotation guidelines. Each new data batch is added to the reward model's training set without discarding old data, and annotation guidelines are kept stable. This enables progressive accuracy improvement (Table 8) without catastrophic forgetting or inter-batch conflicts, as validated by the steady improvement on the aggregated evaluation set.

Staged SFT → DPO → RL training. SFT establishes a broad baseline using mixed data sources, DPO provides targeted safety and style patches, and RL optimizes for engagement using online data. This separation allows each stage to focus on what it does best without interference.

Near-policy prompts for RL. Training on prompts from the latest deployed model's traffic provides gradient estimates that are locally accurate around the current policy, enabling effective hill-climbing. Off-policy prompts provide stale gradients that may point in wrong directions (Section 3.5.2, +10.6% depth lift advantage for near-policy).

Variance-based prompt downsampling. Selecting prompts with high RM score variance provides a more robust difficulty signal than selecting prompts with low mean scores, because the latter is confounded by stylistic score miscalibration across conversation types (Section 3.5.4).

GRPO over online DPO for RL. GRPO exploits continuous reward scores from all generated responses, providing more fine-grained supervision than binary chosen/rejected pairs. In an A/B test, GRPO achieved +1.52% breadth lift over online DPO (Section 3.5.3).

RM win-rate cap at 65%. The V12 failure demonstrated that aggressive RM optimization pushes the policy into unreliable regions of the reward landscape, producing high RM scores but degraded true engagement. The 65% cap (with 60% as ideal target) ensures optimization stays within the reliable region.

Multi-review annotation for evaluation, single-review for training. Multi-review consensus provides stable ground truth for measuring model progress (Section 3.4.1), while single-review data (even with conflicting labels) is sufficient for training because the model can average out annotator noise. This allows efficient allocation of annotation budget.

Dual pointwise/pairwise reward model evaluation. Computing win-rates under both architectures provides complementary signals that help detect reward hacking—if the policy overfits to pointwise-specific biases, the pairwise model should show divergent win-rates.

Comprehensive offline evaluation gating. Five categories of evaluation (community benchmarks, human comparison, RM win-rates, custom production metrics, safety review) ensure that no model reaches users without multi-dimensional validation, preventing the catastrophic regressions that would damage user trust at scale.

Stylistic artifact mitigation throughout training. Monitoring response characteristics (length, emoji usage, list formatting, tone) in both training data and model outputs prevents shallow optimization on surface features, with the emoji reduction experiment (Section 3.5.6) demonstrating that this monitoring must extend to training prompts, not just reward model inputs.

Character steerability annotation as a separate workflow. Mildly adversarial interactive-chat annotation focused specifically on character adherence complements the engagement-focused annotation, achieving a 78% reduction in instruction violations without requiring a separate optimization objective.

4. Key Insights and Innovations

Innovation 1: The Engagement Landscape as a Conceptual Model for Non-Differentiable Optimization

The paper's most distinctive intellectual contribution is reframing production-scale optimization of LLMs for subjective social objectives as iterative landscape climbing with local surrogate models. This is not just a metaphor—it is a diagnostic framework that explains why specific engineering choices work, predicts which failures will occur, and prescribes the guardrails needed to prevent them.

What the field did before. Prior work on RLHF (ouyang2022training; bai2022training) developed the mechanics of training reward models on pairwise preferences and optimizing policies against them, but treated this as a single optimization step: collect preferences, train a reward model, optimize once. Iterative approaches existed in the literature (yuan2024self; rosset2024direct) but were studied in controlled settings with differentiable objectives and static data distributions. No prior work provided a conceptual model for why iterative optimization of non-differentiable objectives at production scale requires fundamentally different design choices than single-step optimization, or why certain configurations (e.g., near-policy prompts, variance-based downsampling, conservative RM win-rate caps) are necessary conditions for success rather than optional improvements.

What the landscape-climbing model explains. The model posits that the true engagement objective defines an unknown, non-differentiable landscape over policy space. Each deployment provides sparse, noisy samples of this landscape around the current policy's position. The reward model interpolates these samples into a differentiable local surrogate—but this surrogate is only reliable within a bounded region around the training data distribution. The critical insight: the optimization step size must be constrained to remain within this reliable region. Step too aggressively and the policy moves into regions where the surrogate's gradients point in wrong directions—exactly the V12 failure mode, where RM win-rates hit 70.7% while engagement degraded (Section 3.1.2, Figure 8).

This framework unifies several of the paper's empirical findings that would otherwise appear as disconnected engineering heuristics:

Near-policy prompts are essential (Section 3.5.2, +10.6% depth lift over off-policy) because they sample the landscape close to the current policy, where the surrogate is most accurate. Off-policy prompts sample distant regions where the surrogate may be miscalibrated.
Variance-based prompt selection works better than mean-based (Section 3.5.4) because high-variance regions are where the landscape has genuine structure to learn from, whereas low-mean regions may simply reflect systematic score miscalibration (e.g., longer conversations receiving lower scores regardless of quality).
The 65% RM win-rate cap is necessary (Section 3.1.2) because it prevents the optimizer from stepping beyond the region where the surrogate's contours are trustworthy. This is not an arbitrary threshold—it is the empirical boundary where reward model confidence diverges from calibration, as demonstrated by the V12 failure where 70.7% RM win-rate coincided with negative engagement lift.
Multi-review annotation is necessary for evaluation but optional for training (Section 3.4.1, Table 9) because evaluation must establish the true landscape heights (requiring low-noise consensus measurements), while training can use noisy samples (the surrogate model averages them out).

Significance beyond performance gains. This reframing is fundamental, not incremental. It converts a collection of engineering techniques into a coherent diagnostic model with predictive power. It tells practitioners how to reason about failures—when a model version regresses, the model directs attention to reward model calibration near the current policy, distribution shift in prompts, or step size exceeding the reliable region. It is analogous to the role that the bias-variance tradeoff plays in supervised learning: not a specific algorithm, but a framework for understanding why algorithms succeed or fail. For the social AI domain specifically, where objectives are inherently subjective and signals are noisy, such a framework is arguably more valuable than any single new training technique would be.

Innovation 2: Empirically Characterizing When and Why Reward Models Break Under Production Optimization

The paper provides the first detailed production-scale case study of reward model over-optimization as the central failure mode in iterative LLM improvement for subjective objectives. While reward hacking is a known theoretical concern in RLHF (bai2022training), the paper makes three distinctive contributions that move from theoretical concern to operational diagnostic.

First, it establishes that the failure is non-monotonic and detectable. The V12 degradation was not a gradual erosion of quality—it was characterized by a specific signature: RM win-rate on user traffic spiking to 70.7% (far above the 50–65% range of successful versions) while RM win-rate on internal traffic simultaneously dropped to 43.7% (Figure 8, middle panel). This divergence pattern—one reward signal rising while another falls—is the operational signature of overfitting to a specific data distribution. Prior work had demonstrated reward over-optimization in controlled settings, but had not characterized this divergence signature as a deployable monitoring tool at production scale.

Second, it quantifies a safe operating threshold. The paper's 65% RM win-rate cap (with 60% as ideal target) is an empirical boundary derived from comparing successful and failed deployments across 15 model versions—not a theoretical bound, but a data-driven guardrail validated by the subsequent recovery in V13-V15. This is conceptually analogous to how the learning rate in gradient descent must be bounded to prevent divergence, but applied to the qualitatively different problem of reward model trust region size. The threshold is specific to this reward model, annotation protocol, and optimization algorithm—but the methodology for establishing such a threshold (monitoring divergence between complementary reward signals, relating win-rates to true engagement outcomes, identifying the inflection point where correlation breaks) is generalizable.

Third, it demonstrates that reward model architecture matters for robustness. The dual pointwise/pairwise evaluation protocol is not just an evaluation detail—it is a defense mechanism. Because the pointwise and pairwise models have different inductive biases (the pointwise model scores responses independently; the pairwise model attends to differences between them), they are unlikely to overfit in identical ways. Divergence between their win-rates signals that the policy is exploiting architecture-specific biases. This is a practical instantiation of ensemble-based reward modeling for robustness, adapted to the constraint that both models must be trainable from the same annotation data.

Significance beyond this paper. This innovation is incremental in its theoretical machinery (Bradley-Terry models, win-rate computation) but is fundamental as a diagnostic methodology for any team deploying RLHF-optimized models in production. The core lesson—monitor multiple complementary reward signals, establish empirical safety thresholds, and treat divergence between signals as a red flag—is directly transferable to other domains and model families. The paper provides the first concrete evidence that this monitoring regime is not merely precautionary but detects real failures that would otherwise degrade user experience at scale.

Innovation 3: Disentangling the Signal Confounds That Make Naive User-Feedback Optimization Dangerous

The paper's systematic cataloging of why user behavioral signals cannot be used directly for RL optimization (Section 3.5.5) is a significant negative result that reshapes how practitioners should think about training data for social AI. The finding is not that user signals are useless—they provide value as auxiliary scores in rejection sampling—but that direct optimization against them produces degenerate behaviors that masquerade as improvements.

What makes this distinctive. Prior work on learning from user feedback (xu2023learning; xu2023improving; don2024naturally) typically treats user signals as noisy-but-valid proxies for satisfaction and focuses on methods for denoising them or combining them with other signals. The CharacterFlywheel analysis reveals something stronger: these signals are not merely noisy but are systematically confounded in ways that make them anti-correlated with genuine quality under optimization pressure. The four confounds identified are:

Delayed feedback causes clarification-seeking responses (which are engaging conversational behavior) to be penalized.
Ending bias causes sycophantic flattery to be rewarded, reproducing a known ChatGPT failure mode.
Job-type-dependent base rates enable reward models to shortcut by detecting conversation category rather than assessing quality.
Prior-turn sentiment confounding causes models to ignore current response quality in favor of momentum effects.

Each of these can be individually mitigated; together, they create a situation where optimizing against user-signal-based reward models reliably produces responses that score highly on the surrogate but degrade true engagement—precisely the V12 failure pattern.

Why this is more than an engineering observation. This finding reframes the relationship between user signals and preference annotations from "user signals are a cheaper, noisier alternative to annotations" to "user signals and annotations encode fundamentally different information, and conflating them is dangerous." Annotations capture judgments of quality conditional on context; user signals capture behavioral responses that are influenced by dozens of factors beyond response quality (conversation phase, user mood, task type, prior expectations). Using user signals as if they were noisy quality labels is a category error that the paper's analysis exposes.

The implicit design principle. The paper does not state it explicitly, but the finding implies a design principle: signals that correlate with the objective but are causally downstream of factors other than model quality cannot be used for direct optimization; they can only be used for filtering or ranking where over-optimization is structurally limited (as in rejection sampling, where the threshold $\tau$ provides a hard bound on how much the signal drives selection). This principle generalizes beyond social AI to any domain where user behaviors are multiply-determined—recommendation systems, dialogue agents, content generation.

Innovation 4: The Self-Correcting Flywheel as a Stability Property of Comprehensive Monitoring

The paper demonstrates that a properly monitored iterative optimization process exhibits emergent self-correction: temporary degradations in individual quality metrics (false refusals, preachy tone, emoji overuse, instruction violations) are resolved within subsequent iterations without explicit corrective optimization steps. This is not a designed feature but a property that emerges from the interaction between comprehensive metric tracking and the feedback loop, and it is arguably the most important validation that the overall framework works.

The evidence. Figure 10 tracks twelve response characteristics across all 15 model versions. The dynamics are striking: false refusals spike at V5-V6 (exceeding 30% on internal traffic) but decline to approximately 20% by V15; contains emoji spikes dramatically at V11-V12 (238.5% relative increase from V1) but normalizes in V13-V15; cooperative ratio dips at V5-V6 but recovers to over 80% by V15. The paper explicitly notes: "These fluctuations and recoveries demonstrate that the iterative process can absorb and correct over-cautious safety tuning—when a version becomes too conservative, subsequent iterations recalibrate based on false refusal signals, restoring appropriate balance."

What makes this distinctive. Prior work on iterative optimization typically focuses on monotonic improvement—each iteration should be better than the last. The CharacterFlywheel data shows that in multi-objective optimization with competing desiderata (engagement vs. safety vs. style vs. steerability), temporary regressions in individual metrics are inevitable and acceptable if the monitoring framework catches them and the feedback loop corrects them. This reframes the goal from "never regress on any metric" (which is impossible when optimizing for subjective engagement while maintaining safety) to "regress temporarily on bounded metrics in ways that the system can detect and reverse."

The mechanism. The self-correction emerges from three interacting components: (1) the annotation pipeline continuously collects labels on specific failure modes (false refusals, instruction violations, templated responses), meaning that when a failure mode spikes, the next iteration's training data naturally contains more labeled examples of that failure; (2) the SFT data mixture includes "ad-hoc internal and user data for failure modes" (Section 2.4.2, component 5), which means the team can deliberately inject corrective examples when monitoring detects a problem; and (3) the offline evaluation gates prevent deployment if regressions exceed thresholds, creating a hard stop for catastrophic failures while allowing minor oscillations during development.

Significance. This finding is fundamental rather than incremental because it validates that iterative optimization in the face of subjective, multi-dimensional objectives is not merely possible but stable—the process converges toward improved quality across dimensions when comprehensive monitoring is in place, even though individual optimization steps may temporarily worsen specific metrics. This is the core validation of the entire CharacterFlywheel methodology and the answer to the implicit question: "Can you really optimize for something as vague as engagingness without the whole system falling apart?" The answer, conditional on the monitoring infrastructure described, is yes—but only because the monitoring infrastructure exists.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper's primary evaluation dataset is Meta's proprietary production user traffic across Instagram, WhatsApp, Messenger, and the Web, specifically the curated prompts sampled through the pipeline described in Section 2.2.1. For community benchmarks, the paper uses standard publicly available evaluation sets including MMLU, GSM8K, MATH, HumanEval, MBPP, ARC Challenge, GPQA, and IFEval (Table 3). Character steerability is evaluated on interactive chat sessions where internal users and domain experts conduct mildly adversarial conversations. All evaluation prompt sets are kept completely separate from any training prompts.
Base model(s). All CharacterFlywheel models are fine-tuned from Llama 3.1 70B (grattafiori2024llama) weights. The paper also develops 405B model variants in parallel using the same CharacterFlywheel process, but these larger models are only used as candidate response generators in the rejection sampling pipeline (Algorithm 1)—only 70B models are deployed in production for inference efficiency. The choice of Llama 3.1 70B as the base reflects a balance between capability and inference cost at Meta's scale.
Metrics. The paper evaluates models across three fundamentally different metric categories. Online engagement metrics (Section 2.5.2): engagement breadth (the expected fraction of evaluation periods where a user exhibits any engagement with the AI, computed as $\hat{\mu}_g^{\text{breadth}} = \frac{1}{n_g}\sum_{i=1}^{n_g} \bar{Y}_i$ where $\bar{Y}_i$ is user $i$ 's average daily engagement rate over the one-week readout window) and engagement depth (the expected total engagement among users who engage at all, computed as $\hat{\mu}_g^{\text{depth}} = \sum_{i=1}^{n_g} S_i / \sum_{i=1}^{n_g} A_i$ where $S_i$ is aggregate engagement and $A_i$ indicates any engagement). Offline quality metrics (Section 2.5.1, Table 4, Figure 10): human win-rates in side-by-side comparisons, reward model win-rates (pointwise and pairwise, on both internal and user traffic evaluation sets), community benchmark scores (Table 3), and custom production metrics tracked via LLM-as-a-judge or rule-based methods—false refusal rate (on both internal and user traffic), response formatting characteristics (average response length, contains list percentage, contains emoji percentage), tone and sentiment quality (preachy tone, positive sentiment, cooperative ratio, non-preachy rate), and quality/failure modes (instruction violation rate, wall-of-text failure). Character steerability metrics (Section 3.2.2, Table 6): IFEval score for general instruction following and instruction violation rate for character-specific adherence, measured via LLM-as-a-judge on interactive chat sessions.
Baselines. The paper's online evaluation compares each new model version against the immediately preceding production deployment in A/B tests—a sequential baseline where each version serves as the control for the next. For offline evaluation, the paper compares against the previous CharacterFlywheel version (human win-rates and RM win-rates, Figure 6 right panel), against GPT-4o (human win-rates, Figure 6 left panel), and against the original Llama 3.1 70B checkpoint as well as other Llama 3 family models and commercial systems (community benchmarks, Table 5). The FLOPs-matched or parameter-matched comparisons against larger models that appear in comparable papers (e.g., the ~14× larger model comparison in the reference example) are not present in this work—the paper does not conduct a formal pretraining-vs-inference compute tradeoff analysis.
Generation budget / compute accounting. The paper does not report generation budgets in terms of token counts, FLOPs, or number of samples generated per prompt. Unlike the reference example paper where generation budget is the universal unit of test-time compute (N generations per prompt, with cost models for beam search and lookahead), CharacterFlywheel measures efficiency purely through the development cycle count (15 versions over 15 months) and online engagement outcomes. The compute cost of individual training stages (SFT, DPO, RL) and inference costs during deployment are not quantified in the paper. This is a notable gap: the reader cannot assess the computational cost of iterating through 15 development cycles versus alternatives.
Cross-validation / statistical protocol. For online A/B tests (Section 2.5.2), users are randomly assigned to test and control arms (typically 10% of traffic each), with metrics assessed over a one-week readout window. The percentage lift is computed as $\widehat{\text{Lift}}(\%) = 100 \times (\hat{\mu}_{\text{test}} / \hat{\mu}_{\text{control}} - 1)$ and confidence intervals are constructed using Fieller's Theorem (fieller1954some) rather than standard normal approximations, because the lift is a ratio of means where the denominator's uncertainty must be accounted for (Appendix 6, Equations 15–18). The Fieller method produces naturally asymmetric confidence intervals that are more reliable when the coefficient of variation of the denominator is non-negligible. Statistical significance is declared when the null lift (0%) is not contained within the Fieller confidence interval. The paper does not report a cross-validation protocol for strategy selection (unlike the reference example, which uses two-fold cross-validation within difficulty bins to select compute-optimal strategies)—model versions are evaluated directly against the production baseline through A/B testing without a held-out validation fold for hyperparameter selection that is described in the paper. For offline reward model evaluation (Section 3.4, Table 8), performance is reported on specific data batches and an aggregated evaluation set, but the paper does not specify whether these evaluation sets are held-out from training or whether results are averaged across folds.

Main Quantitative Results

Pre-Launch Development and Validation (V1–V7)

The pre-launch phase (January–July 2024) established that offline quality improvements translate to online engagement gains, providing the foundational validation for the entire CharacterFlywheel framework.

Human win-rate progression against GPT-4o (Figure 6, left panel): V3 achieved 37.4% win-rate against GPT-4o, V4 reached 41.8%, V6 reached 44.4%, and V7 reached 46.2%—a steady 8.8 percentage point improvement over four versions (a 23.5% relative increase from V3 to V7). This demonstrates that engagement-focused optimization was producing responses that human evaluators increasingly preferred over a strong commercial baseline, even though CharacterFlywheel was not optimized for the same utility-oriented objectives as GPT-4o.

Human and reward model win-rates against previous versions (Figure 6, right panel): Both signals consistently exceeded the 50% neutral threshold. Human win-rates ranged from 50.2% to 52.5%, while reward model win-rates ranged from 53.6% to 57.6%. The reward model consistently showed larger win-rates than human evaluators—a gap of approximately 3–5 percentage points—suggesting that the reward model was somewhat more sensitive to the improvements being made than human annotators, or that the reward model was partially overfitting to features that correlated with but did not perfectly capture human preferences. The paper does not analyze this gap explicitly, but the later V12 failure analysis (Section 3.1.2) demonstrates the danger of trusting inflated RM win-rates without human verification.

Pre-launch A/B test engagement validation (Figure 7): Small-scale A/B experiments on V2, V3, and V4 showed consistent positive lift across both engagement breadth and depth metrics when compared to their respective previous versions. The paper notes that "despite limited statistical power in these early tests—with some confidence intervals including zero due to small sample sizes—the directional consistency provided early validation that our offline optimization approach aligned with online engagement objectives." These were conducted on a fixed set of characters with random online users, not the full production deployment. The exact lift magnitudes are not reported numerically in the text—Figure 7 displays them visually with Fieller confidence intervals, but the specific percentage values are only readable from the figure (which is not fully described in the text).

Post-Launch Iterative Improvement (V8–V15)

The post-launch phase (August 2024–April 2025) provides the paper's central quantitative evidence that the CharacterFlywheel process produces sustained engagement improvements at production scale. Figure 8 presents the full trajectory across three panels.

A/B test engagement lifts by version (Figure 8, top panel): Of the eight deployed versions, seven demonstrated positive lift, with notable successes and one clear failure:

V8 (deployment date not specified): The paper does not report V8's specific lift numbers in the text. Figure 8 top panel shows V8 with positive breadth and depth lifts but the exact values cannot be precisely extracted from the prose description.
V9 (+1.7% Engagement Breadth Metric Lift attributed primarily to explicit image generation; Section 3.5.1): The paper states that "we started public feature of explicit image generation in V9, achieving +1.7% Engagement Breadth Metric Lift over text-only baselines."
V10 (+2.1% Engagement Breadth Metric Lift over V9 attributed to implicit image generation, alongside other improvements; Section 3.5.1): "In V10, implicit image generation became the primary engagement driver, contributing to an additional +2.1% Engagement Breadth Metric Lift over V9 (alongside other improvements)."
V11 (+4.47% breadth, +18.2% depth): The paper explicitly states "notable successes including V11 (+4.47% breadth, +18.2% depth)." This represents the largest depth improvement of any reported version, suggesting that V11 made the model substantially more engaging for users who chose to interact with it.
V12 (+0.05% breadth, -2.9% depth): The only version with negative depth lift and effectively zero breadth lift. The paper treats this as the central failure case for the entire framework and uses it to establish the 65% RM win-rate guardrail (discussed in detail below).
V13: Specific lift numbers are not reported in the text. Figure 8 top panel shows V13 returning to positive lifts in both metrics, validating the corrective adjustments made after V12.
V14 (+8.8% breadth, +11.2% depth): The paper explicitly states "notable successes including V14 (+8.8% breadth, +11.2% depth)." This represents the largest breadth improvement across all versions, indicating V14 was particularly effective at getting users to engage more consistently over time.
V15: Specific lift numbers are not reported. Figure 8 top panel shows V15 continuing the positive trend.

Statistical significance: Green markers in Figure 8 top panel indicate statistically significant positive results (the null lift of 0% is not contained within the Fieller confidence interval), red indicates significant negative, and gray indicates non-significant. The paper does not enumerate which specific versions achieved significance for which metrics, but the visual evidence in Figure 8 suggests that most positive lifts, particularly the larger ones (V11, V14), were statistically significant, while V12's negative depth lift was also significant.

Reward model win-rate trajectory (Figure 8, middle panel): The two reward model signals—RM Internal (trained on internal annotator traffic) and RM User (trained on user traffic)—tracked closely for most successful versions, typically in the 50–60% range. The critical exception is V12: RM User spiked to 70.7% (far above the typical range) while RM Internal dropped to 43.7%, producing a divergence of 27 percentage points. The paper interprets this divergence as the operational signature of reward overfitting: the policy learned to exploit the user-traffic-trained RM's specific biases, achieving high scores on that surrogate while actually degrading in quality as judged by the independent internal-traffic-trained RM (and, crucially, by real user engagement metrics). No other version shows divergence of this magnitude.

Cumulative engagement growth (Figure 8, bottom panel): The paper reports a "sustained upward trend of engagement" over the post-launch period, with cumulative engagement growth showing a clear positive trajectory. However, the paper explicitly notes that this cumulative growth is "not entirely attributable to model updates"—other factors such as product features, user base growth, and seasonal effects may contribute. The engagement baseline is offset to 1 at the start.

Key empirical finding from the post-launch trajectory: The relationship between RM win-rate and true engagement is non-monotonic. Moderate RM win-rates (50–65%) are associated with positive engagement lifts, while aggressive RM win-rates (above 65%, as in V12's 70.7%) are associated with degraded engagement. This empirically derived threshold forms the basis of the paper's primary guardrail: "RM win rates should remain below 65%, with 60% being the ideal target for sustainable optimization" (Section 3.1.2).

Community Benchmarks and Steerability

Community benchmark performance (Table 5, Figure 9): CharacterFlywheel V7, fine-tuned from Llama 3.1 70B, achieves competitive but not state-of-the-art performance across standard benchmarks. The explicit goal is to ensure the model does not catastrophically forget base capabilities while being optimized for social engagement.

Table 5 reports: MMLU 79.5% (vs. 83.6% Llama 3.1 70B baseline, a 4.1 percentage point regression), GSM8K 92.3% (vs. 95.1%, 2.8 point regression), MATH 50.5% (vs. 68.0%, 17.5 point regression—the largest relative decline), HumanEval 77.4% (vs. 80.5%, 3.1 point regression), MBPP 66.6% (vs. 86.0%, 19.4 point regression), ARC Challenge 93.1% (vs. 94.8%, 1.7 point regression), GPQA 39.3% (vs. 46.7%, 7.4 point regression), and IFEval 84.8% (vs. 87.5%, 2.7 point regression).

The pattern is consistent: engagement optimization produces modest regressions across most benchmarks, with the largest regressions in mathematics (MATH) and code generation (MBPP)—domains that are most distant from social conversation. The paper explicitly accepts these trade-offs: "Our intention is not to achieve state-of-the-art results, but rather to ensure robust performance on factual and utility-seeking questions" (Section 2.5.1). Compared to commercial models, V7 is competitive with Claude 3 Haiku and GPT-3.5 Turbo on most benchmarks, but substantially behind GPT-4o and Claude 3.5 Sonnet—which is expected given the 70B parameter scale and the engagement-focused optimization.

Benchmark progression across pre-launch versions (Figure 9): Most benchmarks remained "stable or improved across versions, with particularly notable gains in IFEval (climbing from approximately 75% in V2 to 84.8% in V7) and ARC Challenge (rising to 93.1%)." The stability validates that "engagement-focused optimization did not catastrophically degrade general capabilities." The IFEval improvement is particularly notable because it aligns with the character steerability objective—IFEval measures general instruction following, which is related to the character adherence that CharacterFlywheel explicitly optimizes through the steerability annotation workflow.

Character steerability improvement (Table 6, Section 3.2.2): Instruction violations decreased from 26.6% (V2) to 5.8% (V8)—a 78% relative reduction—with monotonic or near-monotonic improvement across versions. The trajectory: V2 26.6%, V3 22.6%, V4 17.9%, V5 22.2% (a temporary regression that was subsequently corrected), V6 13.7%, V7 7.5%, V8 5.8%. The V5 regression (from 17.9% to 22.2%) is an example of the "self-correcting" behavior documented in Section 3.3—a temporary degradation that the monitoring system detected and the next iterations corrected. The paper notes that this improvement "emerged naturally from our annotation process—while annotators primarily focused on engagement quality, they also labeled instruction violations and edited responses when necessary to better align with character instructions." This demonstrates that optimizing for engagement and instruction-following need not be in conflict when the annotation process captures both signals.

Preference Modeling Results

Iterative reward model accuracy improvement (Table 8): The paper reports pointwise preference model accuracy on both static chat preference data and interactive chat preference data across five reward model versions (RM_240923 through RM_241229). Each version incorporates an additional batch of training data.

On the aggregated evaluation set ("All Data until 241229"), static data accuracy improved from 0.652 (RM_240923) to 0.746 (RM_241229), a 9.4 percentage point absolute improvement (14.4% relative error reduction). Interactive data accuracy improved from 0.573 to 0.650, a 7.7 point improvement (18.0% relative error reduction). This demonstrates that the iterative data accumulation strategy effectively captures more preference signal over time without catastrophic forgetting of earlier patterns.

Zero-shot generalization (Table 8, off-diagonal entries): Models achieve approximately 55–60% accuracy on future unseen batches for static data and 50–55% for interactive data. The paper describes interactive data performance as "near-random," attributing this to "a higher degree of distribution shift or task complexity compared to static preferences." This finding implies that reward models must be regularly retrained on recent data to remain calibrated—they do not generalize well to preference patterns that emerge in later time periods.

Limited backward transfer (Table 8): Newer models trained on additional data do not substantially improve performance on earlier data batches. For example, RM_241229 achieves 0.742 on Batch_240923 static data, comparable to RM_240923's original 0.725. The paper interprets this as evidence that "each batch captures relatively distinct preference patterns, and newer data does not substantially refine the model's understanding of earlier distributions." This is attributed to the deliberate maintenance of consistent annotation guidelines—preference patterns stay stable enough that earlier batches don't benefit from later data, but not so different that inter-batch conflicts cause forgetting.

Response Characteristics Trajectory

Response characteristics across 15 versions (Figure 10, Section 3.3): The paper tracks twelve metrics measured via LLM-as-a-judge and rule-based methods, revealing complex dynamics across development cycles.

False refusal rates (Figure 10, top row): On internal traffic, false refusals show "considerable variation across versions, with peaks at V5-V6 exceeding 30%, but demonstrate overall 25.5% improvement, settling around 20% by V15." On user traffic, false refusals drop "from over 20% to under 5%, despite temporary increases in mid-development versions." The paper interprets the V5-V6 spike as "over-cautious safety tuning" that was corrected when subsequent iterations recalibrated based on false refusal signals. The large gap between internal and user traffic false refusal rates (internal traffic consistently shows higher rates) suggests that either the sampling procedure for internal evaluation over-represents challenging prompts, or the annotation criteria for identifying false refusals differ between internal annotators and how the metric is computed on user traffic.

Response formatting (Figure 10, second row): Average response length remains "relatively stable throughout development, ranging from approximately 50-65 tokens, with controlled variations reflecting different optimization priorities." Contains List percentage shows "moderate fluctuation (13.7% relative change)." Contains Emoji exhibits the most dramatic variation—"238.5% relative change" from V1 to V15, with "notable spikes at V11-V12." The paper explicitly connects the V12 emoji spike to the engagement degradation: "when V12's aggressive emoji usage became apparent through both this metric and engagement degradation, we adjusted annotation guidelines and data composition in V13-V15, successfully moderating emoji frequency to more appropriate levels."

Tone and sentiment (Figure 10, third row): Preachy tone decreased 30.9% (from approximately 25% to under 18%), positive sentiment increased 33.2% (from approximately 45% to 60%), cooperative ratio improved 78.2% (from under 60% to over 80%), and non-preachy rate reached 92.6%. The cooperative ratio shows temporary dips at V5-V6 that were recovered, reinforcing the self-correcting pattern. The paper notes that the simultaneous increase in positive sentiment and decrease in preachy tone addresses "a common complaint in AI assistants that adopt condescending or judgmental tones."

Quality and failure modes (Figure 10, bottom row): Instruction violation decreased 39.9% (from approximately 27% to 16%) when measured on the broader evaluation set (distinct from the steerability-specific measurement in Table 6). Wall-of-text failure decreased 58.2% (from over 10% to under 5%).

The self-correcting dynamics as a key finding: The paper's analysis of Figure 10 emphasizes that "the dynamics reveal that temporary metric degradations—inevitable in any optimization process—are reliably resolved within a few iterations when comprehensive monitoring is maintained." This is presented as evidence that "properly monitored iterative optimization is a stable and effective approach for improving complex, multi-objective systems like social conversation." However, it's worth noting that the paper does not quantify the "recovery time" (how many iterations it takes for a spike to resolve) or demonstrate that the recovery mechanism is automatic rather than driven by manual intervention in the data mixture and annotation guidelines.

Analysis Experiments: On-Policy vs. Off-Policy and DPO vs. GRPO

On-policy vs. off-policy RL prompts (Section 3.5.2): A controlled A/B test comparing two models initialized from the same checkpoint but trained with different prompt sets showed that near-policy prompts (from the latest model's traffic) achieved +10.6% Engagement Depth Metric Lift and +1.6% Engagement Breadth Metric Lift over off-policy prompts (from earlier model versions). The paper describes this as a direct A/B test without specifying the exact model versions or sample sizes, but the magnitude of the depth lift difference (10.6 percentage points) is substantial—comparable to the lifts reported for entire new model versions (e.g., V14's +11.2% depth). This result is the empirical foundation for the paper's emphasis on maintaining near-policy data throughout the RL training process.

Online DPO vs. GRPO (Section 3.5.3): An A/B test comparing two models initialized from the same checkpoint and training data, trained with either Online DPO loss or GRPO loss, showed that GRPO achieved +1.52% Engagement Breadth Metric Lift over Online DPO. The paper does not report the depth lift for this comparison. The paper hypothesizes that GRPO's advantage comes from "its ability to exploit reward scores from all generated responses, which provide a more fine-grained supervision signal" than Online DPO's binary chosen/rejected pairs. This result motivated the team's switch from Online DPO to GRPO during the product cycle.

Image Generation Engagement Impact

Explicit image generation (Section 3.5.1): V9 achieved +1.7% Engagement Breadth Metric Lift attributed to explicit image generation over text-only baselines, validated through a 7-day A/B test. The specific baseline (which text-only model version) is not stated.

Implicit image generation (Section 3.5.1): V10 contributed an additional +2.1% Engagement Breadth Metric Lift over V9, attributed primarily to implicit image generation alongside other improvements. The paper notes this "highlights the value of autonomous image generation that enriches conversations without requiring explicit user prompts." The total image-generation-related engagement gains from V9 and V10 (approximately 3.8% breadth lift cumulative, though note that V10's lift is measured over V9 and may include non-image-generation improvements) represent a meaningful fraction of the total engagement gains across the post-launch period.

Ablation Studies and Robustness Checks

Annotation agreement impact on preference modeling (Section 3.4.1, Table 9): The paper compares models trained on three data variants—Multi-Review (With Agreement, containing only unanimous annotations), Single-Review (All, containing all annotations including conflicting labels), and Single-Review (Random, containing one randomly sampled label per data point)—and evaluates them on both Single-Review (Random) and Multi-Review (With Agreement) evaluation sets. The pointwise RM trained on Multi-Review data achieves 64.83% accuracy on the Multi-Review evaluation set (vs. 60.81% for the untrained baseline, a +4.02 point improvement), but only 62.82% on the Single-Review (Random) evaluation set (vs. 62.75% for the baseline, a negligible +0.07 point improvement). The pairwise RM shows a similar pattern. The critical finding: multi-review evaluation is necessary for reliably measuring model improvement because single-review evaluation sets contain too much label noise to distinguish trained models from baselines. However, models trained on single-review data (with conflicting labels) still achieve strong performance on multi-review evaluation, indicating that "the model can often distill robust preference patterns by aggregating noisy signals from diverse perspectives."

User signal model confound analysis (Section 3.5.5): Not a formal ablation with quantitative comparisons, but a systematic analysis of why user signal models are unsuitable for direct RL optimization that serves as a de facto ablation study. The paper catalogs four confounds (delayed feedback, ending bias, inconsistent positive/negative ratios across job types, and confounding context from prior turns) and reports that the V12 failure reinforced the decision to use user signal models only for rejection sampling ranking, not RL. The paper also reports a small-scale study finding that p(continue) and p(thumb up) models "have high correlation with our preference reward model, showing similar win-rates between models and an upward scoring trend from CH7-12." The exact correlation coefficients or quantitative metrics are not reported.

Emoji reduction experiment (Section 3.5.6): The paper verifies that biases can be inherited from conversation history rather than from the reward model. Despite removing all emojis from reward model inputs during scoring, average emoji count in model responses still increased from 0.2 to 0.48 over 120 RL steps. This demonstrates that the autoregressive policy model mimics and amplifies stylistic patterns present in the training prompts. The experiment motivated the implementation of prompt pre-processing and the broader bias monitoring framework in Section 2.4.4.

Variance-based vs. mean-based prompt selection (Section 3.5.4): The paper reports that mean-based prompt selection (selecting prompts with the lowest average RM scores, the standard heuristic for identifying "hard" prompts) is unreliable because RM scores are not calibrated across prompts—longer-turn conversations receive systematically lower scores regardless of response quality, causing 4× over-representation of Roleplay and Romantic prompts. The paper's variance-based alternative (selecting prompts with high RM score variance across multiple responses) is presented as a more robust difficulty signal. However, this is not a controlled ablation study with A/B test results comparing models trained with the two selection strategies—it is presented as a design rationale rather than an empirically validated finding.

SFT data mixture components (Section 2.4.2): The paper describes six components of the SFT training dataset and states that "the data mixture ratio is carefully tuned to ensure optimal performance," but does not report an ablation study showing the sensitivity of results to different mixing ratios, or the specific ratios used. This is a significant missing ablation—it is unclear how much each data component contributes to the final model quality, and whether the tuning process is critical or the model is robust to mixture variations.

Reference model maintenance in GRPO (Section 2.4.3): The reference policy $\pi_{\text{ref}}$ is maintained as an exponential moving average of the initial and intermediate checkpoints. The paper does not report the decay rate or ablate different reference model strategies (e.g., fixed initial checkpoint vs. EMA of recent checkpoints vs. no reference), leaving the sensitivity of results to this hyperparameter unknown.

Multi-turn vs. single-turn RL formulation (Section 2.4.3): The paper adopts a single-turn optimization formulation (optimizing only the final response given a static conversation history) to avoid the complexity of simulating full conversations. No ablation is reported comparing this approach to multi-turn optimization, leaving open the question of whether full-conversation optimization would yield additional engagement gains or introduce new failure modes.

Critical Assessment

The paper's central claim is that systematic, iterative optimization of LLMs for subjective social engagement is feasible at production scale when embedded within a comprehensive monitoring framework. The evidence for this claim comes primarily from the post-launch A/B test trajectory (Figure 8), which shows 7 of 8 deployed versions producing positive engagement lifts, with cumulative engagement trending upward over nine months. This is substantial production-scale evidence that would be difficult to fake or cherry-pick. However, several aspects of the experimental design limit the strength of the conclusions that can be drawn, and some of the paper's supporting claims are supported more weakly than the headline results suggest.

What is genuinely demonstrated. The paper convincingly shows that iterative refinement of a 70B Llama 3.1 model on production social chat data, using human preference annotations to train reward models and then applying SFT, DPO, and RL, can produce models that achieve higher engagement in A/B tests than their immediate predecessors. The 7-of-8 success rate, with lifts as high as +8.8% breadth and +19.4% depth, represents real improvement—these are not borderline or noise-level effects. The V12 failure alongside the successful versions is particularly informative because it demonstrates that the process can detect and recover from failures, providing evidence for the self-correcting property the paper claims. The steerability improvement (26.6% to 5.8% violation rate) is dramatic and well-supported by the LLM-as-a-judge evaluation on adversarial interactive chats, though whether this metric correlates with real user experience of character consistency is not directly validated.

What is not demonstrated or is demonstrated only weakly.

1. Attribution of engagement gains to specific methodological components. The paper attributes engagement improvements to the CharacterFlywheel process as a whole, but provides very limited evidence about which specific components contribute to the gains. The two controlled component-level experiments—near-policy vs. off-policy prompts (+10.6% depth lift) and GRPO vs. Online DPO (+1.52% breadth lift)—are useful but sparse. No ablations are reported for: the SFT data mixture ratios, the contribution of the 405B models in rejection sampling, the impact of the stylistic artifact mitigation process on engagement (vs. just on monitored metrics), the value of the dual pointwise/pairwise RM evaluation (beyond its diagnostic role in detecting V12), or the specific contribution of the multi-review annotation protocol to final model quality. This makes it difficult for other teams to know which aspects of CharacterFlywheel are essential to replicate and which are incidental.

2. The 65% RM win-rate threshold is empirically derived but its generalizability is unknown. The threshold is based on a single failure case (V12) and the observed range of successful versions (50–65%). With only one clear failure and only 15 total versions, the threshold could be specific to this reward model architecture, this annotation protocol, this base model, and this engagement metric. A different reward model trained with different data might have a different reliable operating range. The paper acknowledges this implicitly by framing the 65% cap as an empirical finding rather than a theoretical bound, but does not discuss the uncertainty around the threshold or whether it has been validated on variants of the training pipeline.

3. Cumulative engagement growth may be confounded. The paper explicitly notes that cumulative engagement growth is "not entirely attributable to model updates" (Figure 8 bottom panel, caption), and acknowledges that other factors—product features, user base growth, seasonal effects—may contribute. This is an honest disclosure, but it means the cumulative trajectory should not be interpreted as evidence of model improvement compounding over time. The per-version A/B tests (which control for temporal confounds through randomization) are the clean evidence; the cumulative trajectory is suggestive but not causal.

4. The community benchmark comparisons are against a weak baseline (the original Llama 3.1 70B), not against an alternative training recipe that preserves benchmark performance while improving engagement. The paper shows that CharacterFlywheel causes regressions of 4–19 percentage points on math and coding benchmarks (Table 5) and accepts these as the cost of engagement optimization. But it does not compare against an alternative approach—for example, a simpler data mixture that combines engagement data with general-purpose data at different ratios—to determine whether these regressions are necessary or whether the specific CharacterFlywheel recipe is more destructive to base capabilities than it needs to be. A multi-objective optimization framing that explicitly trades off benchmark retention against engagement gains would be more convincing than the current presentation, which essentially says "we optimized for engagement and benchmarks dropped somewhat, which is fine."

5. The human win-rate comparisons have methodological gaps. The human side-by-side evaluations (Section 2.5.1) randomly select which model's response continues the conversation for the next turn, which is good practice for preventing conversation drift from biasing the comparison. However, the paper does not report the number of annotators, the number of evaluation turns, inter-annotator agreement rates for the human comparison task, or confidence intervals on the win-rates. The RM win-rates are reported with much more precision, creating an asymmetry in the evidential weight given to automated vs. human evaluation.

6. User signal model evaluation is entirely qualitative. Section 3.5.5 catalogs confounds in user signal models through qualitative description and the V12 failure, but does not provide quantitative metrics on user signal model accuracy, calibration, or the specific correlations with preference model scores that are mentioned ("high correlation" is stated but the correlation coefficient is not reported). The decision to use only p(continue) and p(thumb up) for rejection sampling is reasonable given the confounds, but the evidence for even these two signals' reliability is primarily anecdotal ("consistent and reliable performance").

7. The scale of the compute investment is not quantified. The paper describes 15 development cycles over 15 months but does not report the computational cost of each cycle—the GPU-hours for SFT, DPO, and RL training; the inference cost of generating rejection sampling data from 405B models; the annotation cost for preference collection; or the total cost relative to alternative approaches (e.g., a single large-scale training run rather than 15 iterative cycles). This makes it impossible to assess the efficiency of the CharacterFlywheel approach compared to alternatives, or to determine whether the engagement gains justify the compute and human annotation investment.

8. The "self-correcting" property is described but its mechanism is not isolated. The paper observes that spikes in undesirable metrics (false refusals, preachy tone, emoji overuse) are resolved within a few iterations, and attributes this to the monitoring framework and feedback loop. However, it is unclear whether this resolution is automatic (the data pipeline naturally includes corrective examples because annotated data reflects the spike) or manual (the team notices the spike and deliberately adjusts annotation guidelines or data mixture). If the resolution requires manual intervention, the "self-correcting" label is somewhat misleading—the process is human-guided correction, not autonomous stabilization. The paper states that the team "adjusted annotation guidelines and data composition in V13-V15" to moderate the V12 emoji spike, suggesting manual intervention was at least partially responsible.

9. Single model family, single deployment context. All results are for Llama 3.1 70B deployed across Meta's ecosystem with this specific product design (user-created AI characters, social chat focus). The paper does not provide evidence that the CharacterFlywheel methodology generalizes to different base model families, different model scales, different product contexts, or different definitions of engagement. This is not a flaw per se—production papers are inherently about specific systems—but it limits the strength of claims about the approach's general applicability.

10. The claimed annotation agreement insight (Section 3.4.1) confounds training data quality and evaluation data quality. The experiment varies both the training data composition (Multi-Review vs. Single-Review) and the evaluation set (Multi-Review vs. Single-Review), but does not fully cross the design—the paper reports only the four (training, evaluation) combinations shown in Table 9. It would be informative to see whether a model trained on Single-Review data and evaluated on Single-Review data outperforms the same model evaluated on Multi-Review data, as this would disentangle whether the evaluation set or the training data is driving the observed differences. The current results support the paper's conclusion that multi-review evaluation is necessary, but the conclusion about training data is less clean because the model trained on Single-Review data still performs well on Multi-Review evaluation.

What experiments would strengthen the paper. (1) An ablation study removing individual SFT data components (rejection sampling data, safety data, capability data, Llama 3.1 data, failure mode data) to measure each component's contribution to engagement and benchmark retention. (2) A comparison against a simpler iterative approach—for example, SFT-only iterations without RL—to quantify the marginal value of the full pre-herding/herding cycle. (3) A FLOPs-matched or compute-cost comparison: given a fixed compute budget, does 15 iterative cycles produce more engagement gain than one large-scale training run? (4) A replication on a different base model family to test generalizability. (5) Quantitative characterization of the relationship between RM win-rate and true engagement lift across all 15 versions (not just the V12 failure case) to provide a more robust empirical basis for the 65% threshold, ideally with a calibration curve showing where the correlation breaks down. (6) A held-out validation fold for hyperparameter and recipe selection within each development cycle, to ensure that the recipe chosen for each iteration is not overfit to the test set (the A/B test population).

Overall assessment. The paper provides compelling real-world evidence that iterative engagement optimization for social LLMs is feasible and produces meaningful gains, with the V12 failure serving as an informative counterexample that validates the monitoring framework's diagnostic value. The core claim—that systematic, monitored iteration works—is supported by the A/B test trajectory. However, the paper's more specific empirical claims (the 65% threshold, the necessity of near-policy prompts, the superiority of GRPO over Online DPO, the precise attribution of gains to image generation) rest on thinner evidence—single A/B tests without reported sample sizes, qualitative rather than quantitative analysis of user signal confounds, and a single failure case driving the primary guardrail. The paper is best read as a production engineering report that demonstrates feasibility and identifies key failure modes, rather than as a controlled scientific study that isolates causal mechanisms. Its primary value to the research community is in the methodology—the overall architecture, the monitoring framework, the diagnostic signals for detecting reward overfitting—rather than in the specific numerical findings, which are likely specific to this deployment context.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted For in the Efficiency Gains

The assumption or constraint. The entire CharacterFlywheel framework relies on the ability to detect and respond to model quality changes—reward model win-rate divergence, metric spikes, failure mode regressions—through comprehensive offline and online evaluation. However, the paper's headline engagement improvements (e.g., V14's +8.8% breadth, V11's +18.2% depth) are reported as per-version A/B test lifts without any accounting for the cost of the monitoring infrastructure that enables these gains. The paper does not quantify: (1) the compute cost of training and maintaining the reward models (pointwise and pairwise, retrained on accumulating data), (2) the inference cost of running reward model evaluation on held-out prompts for every candidate checkpoint, (3) the annotation cost of the continuous preference collection pipeline (multi-review annotation for evaluation sets, single-review for training), (4) the human evaluation cost of side-by-side comparisons before each deployment, or (5) the opportunity cost of running 7-day A/B tests (during which some fraction of users receive potentially suboptimal models) before full rollout.

The paper explicitly acknowledges this gap only indirectly. Section 2.5.1 describes the offline evaluation framework but does not attach costs. Section 3.1.2 describes the V12 failure as "instructive" and notes that subsequent versions adhered to conservative RM win-rate constraints—but the cost of detecting V12 (the A/B test that revealed -2.9% depth degradation, affecting actual users in the test arm for a week) is not factored into any efficiency calculation. The paper states that typically 10% of traffic goes to each arm, meaning that each failed experiment exposes a non-trivial user population to a degraded model for the full readout window.

The consequence. A practitioner deciding whether to adopt CharacterFlywheel cannot assess its total cost of ownership. The per-version engagement lifts are gross improvements—they do not net out the annotation budget, the reward model training compute, the evaluation inference cost, or the A/B test overhead. If the annotation pipeline costs as much as model training, the effective return on investment is halved. If failed versions like V12 impose engagement costs on the test arm before being detected, the net cumulative engagement gain across all users (including those in failed test arms) is lower than the sum of successful version lifts would suggest. The paper's claim of "sustained upward trend of engagement" (Figure 8, bottom panel) is cumulative across all users, but the per-version A/B test lifts are measured on the test arm only—the overall system-wide engagement improvement, accounting for users exposed to regressions during A/B tests and the delay between model development and full deployment, is not reported.

What evidence exists in the paper. The paper provides no quantitative cost analysis. The number of annotators, the volume of annotated data per iteration, the GPU-hours for reward model training, the inference cost of evaluating candidate checkpoints, and the total A/B test traffic exposure are not reported. The paper does not compare the CharacterFlywheel total cost (15 iterations over 15 months, with full annotation and evaluation pipelines) to a counterfactual approach—for example, a single large-scale training run using all the data accumulated over 15 months, or fewer iterations with larger data batches per iteration. Without such a comparison, the paper cannot claim that the iterative flywheel approach is more cost-effective than alternatives; it can only claim that it produces engagement improvements, which a simpler approach might also achieve at lower total cost.

Mitigation status. The paper does not address this limitation, propose a cost model, or suggest future work on cost-efficiency analysis. This is a significant omission for a paper that positions itself as providing "practical guidance for teams building engagement-focused AI" (Section 4).

The 65% RM Win-Rate Threshold Rests on a Single Failure Case

The assumption or constraint. The paper's central operational guardrail—"RM win rates should remain below 65%, with 60% being the ideal target for sustainable optimization" (Section 3.1.2)—is derived from the V12 failure where RM User win-rate hit 70.7% while engagement depth declined by -2.9%. This is a single data point. The paper does not provide a calibration curve showing how RM win-rate relates to true engagement lift across all 15 versions, does not report whether any successful versions approached but stayed below 65% (validating the threshold as a boundary rather than a coincidence), and does not test whether the threshold generalizes to different reward model architectures, annotation protocols, or RL algorithms.

The paper implicitly acknowledges the uncertainty by framing the threshold as "empirical" and "based on empirical observations across successful and failed deployments," but the only failed deployment discussed in detail is V12. The paper does not report whether earlier pre-launch versions ever exhibited elevated RM win-rates or whether any other post-launch versions approached the boundary. The middle panel of Figure 8 shows RM win-rates for V8-V15, and V12 is the only version where RM User exceeds approximately 62%—suggesting that the 65% threshold is derived from a single excursion, not from observing degradation at multiple threshold crossings.

The consequence. The threshold may be specific to this particular reward model, this particular base model (Llama 3.1 70B), this particular annotation protocol, and this particular definition of engagement. A different team training a different reward model on different preference data might have a higher or lower safe operating range. If the true boundary is higher (e.g., 75%), the 65% cap unnecessarily constrains optimization, leaving engagement gains on the table. If the true boundary is lower (e.g., 55%), the 65% cap provides false reassurance and future versions may degrade despite being "within range." The paper frames the threshold as a key actionable finding for practitioners, but the evidence base is too thin to support prescribing a specific number with confidence.

What evidence exists in the paper. Figure 8 (middle panel) shows the RM win-rate trajectory for versions V8-V15. V12's RM User spike to 70.7% is the clear outlier. Successful versions (V8-V11, V13-V15) appear to have RM User win-rates in roughly the 50-62% range, and RM Internal win-rates in roughly the 48-55% range, but the exact values for each version are not tabulated—they must be estimated from the figure. The paper reports V8-V15 RM win-rate data only visually; no numerical table is provided. The pre-launch versions (V1-V7) RM win-rates are shown in Figure 6 (right panel) in a different format (win-rates against the immediately previous version, not against a fixed baseline), making direct comparison difficult.

Mitigation status. The paper partially addresses this by monitoring both RM Internal and RM User win-rates and treating divergence between them as an additional red flag. The V12 failure was characterized not only by high RM User but also by low RM Internal (43.7%), creating a divergence of 27 percentage points that was not observed in any successful version. This dual-signal approach provides some robustness—even if the absolute threshold is miscalibrated, the divergence pattern may still detect overfitting. However, the paper does not propose a divergence-based threshold (e.g., "divergence exceeding X percentage points") as an alternative to or complement of the absolute cap, nor does it analyze whether any successful versions exhibited moderate divergence that resolved without degradation. The paper suggests that "developing principled methods for detecting reward hacking" remains an open challenge (Section 4), acknowledging that the current approach is heuristic rather than theoretically grounded.

The Self-Correcting Property Is Observed But Its Mechanism Is Not Isolated

The assumption or constraint. The paper presents the "self-correcting behavior" of the iterative optimization process as a key finding: "temporary degradations in individual metrics (such as elevated false refusals, preachy tone, or emoji overuse) are reliably resolved within subsequent iterations through our comprehensive feedback loop" (Section 4). This is illustrated in Figure 10, where spikes in false refusals (V5-V6), emoji usage (V11-V12), preachy tone (V5-V6), and instruction violations (V5) are all observed to decline in later versions.

However, the paper does not isolate whether the resolution mechanism is automatic (an emergent property of the data pipeline and training process) or manual (the team observes the spike and deliberately intervenes). The paper states that for the V12 emoji spike, "we adjusted annotation guidelines and data composition in V13-V15, successfully moderating emoji frequency" (Section 3.3.2). This describes a manual intervention—the team noticed the problem and changed the process. For the V5-V6 false refusal spike, the paper states that "when a version becomes too conservative, subsequent iterations recalibrate based on false refusal signals, restoring appropriate balance" (Section 3.3.1)—language that suggests an automatic mechanism, but without specifying whether the recalibration required changes to the safety data mixture, the annotation guidelines, or the reward model training.

The consequence. If the self-correction requires manual intervention, the process is not "self-correcting" in any autonomous sense—it is human-guided correction enabled by comprehensive monitoring. This distinction matters for practitioners: deploying CharacterFlywheel requires not just the technical infrastructure but the ongoing attention of a team that can interpret metric spikes, diagnose root causes, and adjust training recipes. The paper's framing suggests that the system stabilizes itself; the evidence suggests that the stabilization is human-driven. A team that deployed the same infrastructure but lacked the expertise to diagnose metric spikes and adjust data mixtures might experience persistent degradation rather than recovery.

Furthermore, the "recovery time" is not quantified. If a failure mode spikes at version N and takes until version N+2 or N+3 to resolve, users are exposed to the degraded behavior for weeks or months. The paper does not report how many iterations it takes for a spike to return to baseline, or whether some metrics (e.g., false refusals) recover faster than others (e.g., emoji overuse). This matters for risk assessment: a monitoring system that detects problems but takes several iterations to fix them exposes users to persistent quality issues in the interim.

What evidence exists in the paper. Figure 10 provides the time-series evidence for metric fluctuations and recoveries across 15 versions. The V5-V6 false refusal spike appears to resolve by V7-V8 (1-2 iterations). The V5 cooperative ratio dip resolves by V6-V7 (1-2 iterations). The V11-V12 emoji spike resolves by V13-V14 (1-2 iterations). The V5 instruction violation spike (Table 6: 22.2% at V5, down from 17.9% at V4) resolves by V6 (13.7%). These recovery timelines suggest that the feedback loop operates relatively quickly—within 1-2 iterations—but the paper provides no analysis of why some spikes occur, whether earlier detection is possible, or what specific interventions (if any) were applied in each case.

Mitigation status. The paper does not explicitly address the distinction between automatic and manual correction, nor does it propose methods for reducing recovery time. The offline evaluation gates (Section 2.5.1) prevent deployment of models that fail safety or quality thresholds, which would catch catastrophic regressions before they reach users, but the paper does not specify whether the metric spikes observed in Figure 10 would have triggered these gates or whether they represent acceptable oscillations within the operating range. The paper's conclusion that "properly monitored iterative optimization is a stable and effective approach" (Section 3.3) is supported by the observed recoveries but overstates the autonomy of the stabilization mechanism.

The Paper Provides No Compute Cost Accounting, Making Efficiency Impossible to Assess

The assumption or constraint. CharacterFlywheel describes 15 complete development cycles over 15 months, each involving: (1) data collection from production deployments and internal annotators, (2) reward model retraining on accumulating preference data (pointwise and pairwise models, both initialized from Llama 3.1 70B), (3) rejection sampling data generation using a pool of models including 405B variants, (4) SFT on a multi-component data mixture, (5) DPO on preference data, (6) RL using GRPO with online data generation, and (7) comprehensive offline evaluation (community benchmarks, human comparisons, RM win-rates, custom metrics, safety review) followed by a 7-day A/B test. None of these steps has its computational cost reported.

The paper does not provide: GPU-hours for SFT, DPO, or RL training; the number of GPUs used for training; the inference cost of generating rejection sampling data from 405B models; the inference cost of RM evaluation on candidate checkpoints; the volume of training data (number of examples) at each stage; the training duration (wall-clock time) per stage; or the total compute investment across all 15 cycles.

The consequence. A practitioner cannot assess whether the CharacterFlywheel approach is compute-efficient compared to alternatives. For example, given the same total compute budget, would a single large-scale training run incorporating all the preference data accumulated over 15 months produce comparable or better engagement gains? Would fewer iterations with larger data batches per iteration be more efficient? Would a different architecture (e.g., directly training a larger model once rather than iteratively refining a 70B model) achieve similar engagement at lower total cost? The paper's claim that CharacterFlywheel provides "a template for developing and monitoring complex AI systems" (Section 4) is undermined by the absence of cost data—a template without resource requirements is incomplete.

The compute cost is particularly relevant because the paper reports that only 70B models are deployed in production "for better inference efficiency," but the training pipeline uses 405B models for rejection sampling data generation. This means each iteration incurs the inference cost of running 405B models on production prompts—a cost that is entirely absent from the deployed system's operational budget but is part of the development budget. The tradeoff between this development cost and the resulting engagement improvement is not characterized.

What evidence exists in the paper. No compute cost data is reported. The paper mentions that 405B models are "develop[ed] in parallel using the same CharacterFlywheel process" (Section 2.4.1) and used for rejection sampling, but does not report the inference cost. The scale of the deployment—"serving millions of users" (Section 1)—is mentioned, but this describes the product scale, not the training scale. The paper does not report training dataset sizes, number of RL steps, batch sizes, or any other information that would allow even a rough estimate of compute requirements.

Mitigation status. The paper does not acknowledge this as a limitation or suggest future work on cost characterization. This is a significant gap for a paper that positions itself as providing practical guidance for production deployments. The reference example paper (on compute-optimal test-time scaling) explicitly accounts for generation budgets and provides FLOPs-matched comparisons; CharacterFlywheel does neither. While the objectives differ (engagement optimization vs. accuracy optimization), the absence of any cost model limits the paper's utility as an engineering reference.

Hard Problems (Fundamental Capability Gaps) Are Not Addressed, and the Boundary Is Not Characterized

The assumption or constraint. The CharacterFlywheel framework optimizes for engagement within the capability envelope of the base model (Llama 3.1 70B). It does not expand that envelope into fundamentally new capabilities—it makes the model more engaging at what it can already do, but does not enable it to do things it fundamentally cannot. The paper acknowledges this implicitly through the benchmark regression results (Table 5: MATH drops from 68.0% to 50.5%, MBPP from 86.0% to 66.6%), which show that engagement optimization trades off against certain reasoning and coding capabilities, but does not frame this as a capability boundary.

More importantly, the paper does not characterize what types of social interactions remain outside the model's capability range—conversations requiring deep factual knowledge, multi-turn coherent planning, long-term memory, or nuanced emotional intelligence that exceeds the base model's training distribution. The engagement metrics measure whether users interact more (breadth) and longer (depth) with the AI characters, but do not measure whether the interactions are substantively better in ways that require capabilities the base model lacks. The paper's improvements could be entirely attributable to making the model better at the types of conversations it could already handle (more engaging tone, fewer refusals, better character adherence) rather than expanding the range of conversations it can participate in.

The consequence. If the base model has a fundamental ceiling on certain types of social intelligence—for example, it cannot maintain coherent multi-session relationships, cannot remember user-specific details across conversations, or cannot engage in complex collaborative storytelling—the CharacterFlywheel process will not overcome these ceilings. Each iteration climbs the local engagement landscape but cannot create new peaks that were not reachable from the starting point. The paper's landscape-climbing metaphor (Section 2.1, Figure 2) implicitly assumes the landscape is connected and that iterative local steps can reach the global optimum; if the base model's capability boundaries create disconnected regions of the engagement landscape, iterative refinement from a single starting point cannot cross the gap.

The benchmark regressions provide indirect evidence of this limitation: the model loses capability on tasks that are far from social conversation (math, coding) while improving on tasks close to social conversation (instruction following, conversational tone). This suggests that the optimization process reshapes the model's capabilities within a fixed total capacity envelope—gains in social engagement come at the cost of other capabilities, and there may be hard limits on how much engagement can improve without either expanding total capacity (through larger models or more pretraining) or sacrificing unacceptable levels of utility.

What evidence exists in the paper. The paper does not directly investigate capability boundaries. Table 5 shows benchmark regressions but does not relate them to the types of social interactions affected. The engagement metrics (breadth and depth) are aggregate behavioral measures that do not decompose by conversation type, difficulty, or required capability. The paper reports that V12's RM User win-rate spiked to 70.7% while engagement degraded, but does not analyze whether the degradation was concentrated in particular conversation types (e.g., complex role-playing scenarios) where the reward model's training data was sparse or unreliable. The user signal model confound analysis (Section 3.5.5) identifies biases in user feedback signals but does not characterize the types of interactions where the model fundamentally cannot improve.

Mitigation status. The paper does not address capability boundaries, does not analyze which types of social interactions benefit most vs. least from iterative refinement, and does not propose methods for detecting or expanding capability ceilings. The paper's conclusion acknowledges that "several challenges remain, including developing principled methods for detecting reward hacking, improving multi-turn optimization formulations, and better understanding of generalization in preference modeling" (Section 4), but does not identify capability boundaries as an open challenge. The landscape-climbing metaphor, while useful for conceptualizing local optimization, may obscure the possibility that the true engagement landscape has discontinuities or inaccessible regions that iterative refinement cannot traverse.

Findings Are Specific to a Single Model Family, Single Product Context, and Single Engagement Definition

The assumption or constraint. All results in the paper are from fine-tuning Llama 3.1 70B for deployment across Meta's specific social chat product (user-created AI characters on Instagram, WhatsApp, Messenger, and the Web). The engagement metrics are defined by this product's specific interaction patterns (conversation-based, turn-by-turn, with explicit character personas). The preference annotations are collected from annotators trained on this product's specific quality criteria. The user signal models are trained on this product's specific behavioral signals (thumbs-up, continue, etc.). The safety and privacy frameworks are inherited from Meta's specific compliance requirements.

The paper does not provide evidence that the CharacterFlywheel methodology generalizes to: (1) different base model families (e.g., non-Llama architectures, different parameter scales), (2) different product contexts (e.g., voice-based assistants, group chat, customer service bots), (3) different definitions of engagement (e.g., task completion rate, user satisfaction surveys, long-term retention), (4) different cultural contexts with different conversational norms and engagement patterns, or (5) different annotation workforces with different demographic compositions and quality standards.

The consequence. A practitioner building a social AI product in a different context—with a different base model, different user interaction patterns, different engagement objectives, or different annotation resources—cannot assume that the specific findings of this paper will transfer. The 65% RM win-rate threshold, the superiority of GRPO over Online DPO, the effectiveness of variance-based prompt selection, and the value of dual pointwise/pairwise RM evaluation may all be specific to the interaction between Llama 3.1 70B's inductive biases, Meta's engagement metric definitions, and the specific annotation protocol. The paper's methodology—the overall architecture of the flywheel, the monitoring framework, the staged SFT→DPO→RL training—is more likely to generalize than the specific numerical findings, but even the methodology may depend on infrastructure (MultiRay for diversity sampling, DRAMA-1B for embeddings, the specific safety classifier architecture) that is not universally available.

The paper's positioning as providing "scientific rigor" (Section 3.6) and "practical guidance for teams building engagement-focused AI" (Section 4) implies a level of generality that the single-context evidence does not fully support. The reference example paper (on compute-optimal test-time scaling) also uses a single model family (PaLM 2-S*) and single benchmark (MATH), but the compute-optimal framework is explicitly designed to be general and the paper provides a methodology for computing optimal strategies that other teams can apply to their own models and benchmarks. CharacterFlywheel provides a production narrative and a set of empirical findings, but no portable methodology that a different team could directly apply without adapting the entire infrastructure to their own context.

What evidence exists in the paper. The paper provides no evidence of generalizability—no replication on a different base model, no evaluation on a different product or engagement definition, no comparison across annotation workforces. The paper's language about generalizability is largely aspirational: "We release this technical report to contribute to scientific progress in social LLM development and to provide practical guidance for teams building engagement-focused AI" (Section 4). This is a reasonable goal for a production report, but the paper does not claim to have tested generalizability and provides no evidence that the specific findings transfer.

Mitigation status. The paper does not address the generalizability limitation or propose replication studies. The paper's value is primarily as a detailed case study—a description of what one team did, what worked, and what failed—rather than as a validated general methodology. This is not inherently a flaw (production papers are necessarily about specific systems), but the paper's rhetoric sometimes conflates "this worked for us" with "this will work for you." The key empirical findings (the 65% threshold, the near-policy advantage, the GRPO advantage) are presented as discoveries rather than as context-dependent observations, and the paper would be strengthened by explicitly characterizing the scope conditions under which these findings are expected to hold.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around LLM alignment from a paradigm where the objective is assumed to be well-defined and static (e.g., "helpfulness, harmlessness, honesty") to one where the objective itself is a moving target—non-differentiable, subjective, and only fully measurable through aggregate user behavior in production. Before CharacterFlywheel, the field had developed sophisticated techniques for optimizing against learned reward models (RLHF, DPO, iterative self-play), but these techniques were validated primarily on tasks with verifiable ground truth or on utility-focused benchmarks where "better" has an objective definition. The paper demonstrates that the same techniques can be applied to inherently subjective social objectives, but only if they are embedded within a substantially more complex framework that monitors for the specific failure modes that subjective optimization introduces.

This is not a paradigm shift in the sense of introducing a new algorithm or theoretical framework—the individual components (Bradley-Terry preference models, SFT, DPO, GRPO, rejection sampling) are well-established. Rather, it is a methodological reframing that changes what the field considers necessary for production-scale optimization of subjective objectives. The paper's core message is that you cannot simply take an off-the-shelf RLHF pipeline, point it at user engagement signals, and expect it to work. You need: (1) multiple complementary reward signals that can detect when any single signal is being gamed, (2) conservative optimization thresholds that prevent the policy from exploiting reward model miscalibration, (3) continuous monitoring across dozens of quality dimensions to catch regressions that engagement metrics alone would miss, and (4) a tight iteration loop where monitoring feeds directly back into data curation and annotation guidelines.

If this message is absorbed by the field, it will change how practitioners build social AI systems. The default approach of "collect some preference data, train a reward model, run RLHF once, and deploy" will be recognized as insufficient—not because the techniques are wrong, but because single-cycle optimization without ongoing monitoring inevitably produces the V12 failure pattern: reward model win-rates climb while true quality degrades, and without complementary signals, no one knows until users complain or engagement metrics decline. The paper makes this failure mode operationally detectable through the divergence between RM Internal and RM User win-rates—a specific, actionable diagnostic that any team can implement.

Reconciling contradictions in prior work. The paper implicitly resolves a tension in the literature on learning from user feedback. Prior work had shown that user behavioral signals (thumbs-up, response regeneration, conversation continuation) could be used to improve conversational models (xu2023learning; xu2023improving; don2024naturally), but other work had documented the sycophancy and reward hacking that emerge when models optimize directly for user approval (openai2024sycophancy). The CharacterFlywheel analysis in Section 3.5.5 provides a unified explanation: user signals are not merely noisy—they are systematically confounded in ways that make them anti-correlated with genuine quality under optimization pressure. Delayed feedback penalizes clarification; ending bias rewards flattery; job-type-dependent base rates enable shortcutting; prior-turn sentiment confounds current-turn assessment. These confounds explain both why user signals can be useful in controlled settings (where optimization pressure is limited, as in rejection sampling) and why they produce degenerate behavior under aggressive optimization (as in RL). The resolution is not to abandon user signals but to constrain their role to settings where over-optimization is structurally bounded—specifically, as auxiliary scores in rejection sampling ranking, where the threshold $\tau$ provides a hard ceiling on how much the signal can influence data selection.

Research directions that become more attractive. The paper's findings make reward model robustness the central research priority for social AI—more important than developing new RL algorithms, new model architectures, or new annotation protocols. The V12 failure demonstrates that the bottleneck is not the optimizer's ability to climb the reward landscape but the reward landscape's fidelity to the true objective. Research on out-of-distribution detection for reward models, ensemble-based reward uncertainty estimation, and methods for detecting when a policy has moved into an unreliable region of the reward landscape all become directly relevant to production social AI in ways that were previously theoretical. The paper's 65% RM win-rate threshold is a crude empirical guardrail; principled methods for dynamically estimating the trustworthy range of a reward model would replace it.

Research directions that become less attractive. The paper's finding that GRPO outperformed Online DPO by a modest +1.52% engagement breadth lift, and that "several other design choices can be more influential than the exact RL loss to use" (Section 3.5.3), suggests that the field's intense focus on algorithmic innovation in RLHF loss functions may be misallocated. Prompt selection (near-policy vs. off-policy: +10.6% depth lift), reward model quality, and monitoring infrastructure appear to matter more than the choice between PPO, DPO, and GRPO variants. This is not to say that algorithmic research is unimportant, but that for production social AI specifically, the bottleneck is data quality and monitoring rigor, not optimizer sophistication. A team that invests in better preference annotation protocols, more frequent reward model retraining, and comprehensive metric tracking will likely outperform a team that implements the latest RL algorithm but neglects these infrastructure components.

Follow-Up Research This Work Enables

1. A calibration curve mapping reward model win-rate to true engagement lift across reward model architectures, base model families, and annotation protocols. The paper's 65% RM win-rate threshold is derived from a single failure case (V12) and the observed range of successful versions using one specific reward model architecture (Llama 3.1 70B pointwise/pairwise) on one specific base model. A systematic study would train multiple reward model variants (different initialization scales, different training data volumes, different annotation quality levels) on the same underlying preference data, then optimize policies against each reward model to varying RM win-rate targets, and measure the resulting true engagement lifts through A/B tests. This would produce a family of calibration curves showing where the correlation between RM win-rate and true engagement breaks down for each variant. The key outcome would be a methodology for estimating the safe operating range of a reward model before deployment, using only offline metrics (e.g., ensemble disagreement, prediction uncertainty on held-out data, or divergence between pointwise and pairwise scores). Without this, every team deploying RLHF for subjective objectives must discover their own V12-equivalent failure through trial and error.

2. Online detection of reward overfitting without waiting for A/B test results. The V12 failure was detected through a 7-day A/B test that exposed real users to a degraded model. The paper shows that the RM Internal / RM User divergence was visible in offline evaluation—RM User hit 70.7% while RM Internal dropped to 43.7%—but does not report whether this divergence was detected before deployment or only after A/B test results arrived. A concrete follow-up would train reward models specifically designed to detect when the policy has moved into an unreliable region: for example, an ensemble of reward models trained on different data splits or different annotation subsets, where increasing disagreement signals that the policy is in a region of the reward landscape where individual models' predictions are not trustworthy. The experiment would compare the detection lag of ensemble-based methods against the RM Internal/RM User divergence signal that CharacterFlywheel used, measuring how many RL steps before engagement degradation each method would have flagged the V12 failure. A successful method would enable early stopping of RL training before deployment, eliminating the need for A/B tests to catch overfitting.

3. A controlled comparison of single-cycle RLHF versus iterative flywheel refinement at fixed total annotation budget. The paper describes 15 complete development cycles but does not compare this approach against a counterfactual: given the same total annotation budget spread over 15 months, what would happen if a single RLHF cycle consumed all the data at once? This could be studied retrospectively using the CharacterFlywheel data: take the full preference dataset accumulated across all 15 cycles, train a single reward model on it, run one large-scale SFT+RL training from the Llama 3.1 70B checkpoint, and evaluate the resulting model's engagement against the V15 model in an A/B test. If the single-cycle model achieves comparable or better engagement, it would suggest that the iterative flywheel's value is not in the iteration per se but in the adaptive data collection—the ability to target annotation effort at the current model's weaknesses. If the flywheel substantially outperforms, it would validate the paper's core claim that iterative refinement with near-policy data is necessary for optimal results. This experiment is feasible because the preference data already exists; it requires only the compute for a single large training run and an A/B test.

4. Stress-testing whether the "self-correcting" property is automatic or requires human intervention. The paper observes that spikes in false refusals, preachy tone, emoji overuse, and instruction violations resolve within 1-2 iterations, but does not isolate the mechanism. A controlled experiment would run two parallel development tracks from the same checkpoint: one where the data pipeline and training process remain fixed (no manual adjustment of annotation guidelines, data mixture, or safety thresholds), and one where the team is allowed to intervene as they did in the paper (adjusting annotation guidelines, modifying SFT data composition, tuning safety classifier thresholds). If the fixed-pipeline track also exhibits metric recovery, it would demonstrate that the self-correction is an emergent property of the annotation-and-retraining loop—annotators naturally label more examples of a spiking failure mode, and the model corrects through standard training. If only the intervention track recovers, it would demonstrate that the self-correction requires human diagnosis and recipe adjustment, which has significant implications for the operational cost and expertise required to maintain a CharacterFlywheel-style system. The paper's description of the V12 emoji recovery ("we adjusted annotation guidelines and data composition") suggests manual intervention, but the paper does not test the counterfactual.

5. Extending the engagement landscape metaphor to multi-turn conversation optimization. The paper adopts a single-turn RL formulation—optimizing the model's response given a static conversation history—to avoid the complexity of simulating full conversations. This is acknowledged as a simplification that "might compromise the on-policy property." A follow-up could investigate whether multi-turn RL (where the model generates a full conversation trajectory and receives reward at the trajectory level) yields engagement gains beyond single-turn optimization. The key challenge is credit assignment: in a 10-turn conversation, how do you determine which turns contributed to the user continuing versus abandoning? The paper's user signal models—particularly $p(\text{continue})$ , which predicts conversation continuation—provide a natural per-turn reward signal that could be used for multi-turn credit assignment. However, Section 3.5.5 documents the confounds that make user signals unreliable for optimization. A concrete experiment would compare single-turn RL (the current approach) against multi-turn RL using either trajectory-level preference annotations (where annotators judge entire conversations, not individual responses) or learned value functions that estimate expected future engagement from any point in a conversation. The outcome would clarify whether the single-turn simplification is a necessary pragmatic compromise or whether it leaves engagement gains on the table.

6. A FLOPs-matched comparison between iterative refinement and scaling model size for social engagement. The paper shows that a 70B model refined through 15 CharacterFlywheel cycles achieves substantial engagement gains, but does not compare this against simply training a larger model once. The reference example paper (on compute-optimal test-time scaling) conducted a FLOPs-matched comparison showing that a smaller model with additional test-time compute can outperform a ~14× larger model on certain difficulty tiers. A similar analysis for CharacterFlywheel would compare the total compute cost of 15 cycles (including all SFT, DPO, RL training runs; all rejection sampling data generation from 405B models; all reward model retraining; all evaluation inference) against the compute cost of pretraining and post-training a larger model from scratch, and ask: at fixed total FLOPs, which produces a more engaging model? The paper reports that 405B models are used for rejection sampling but only 70B models are deployed—suggesting that 405B models are considered too expensive for inference but acceptable for offline data generation. A systematic analysis of this compute tradeoff would provide practitioners with concrete guidance on whether to invest in iterative refinement infrastructure or in larger one-time training runs.

Practical Applications and Downstream Use Cases

1. Production chatbots and social AI products with user-defined personas. The most direct application of CharacterFlywheel is to any production system where users interact with AI characters or personas and the goal is sustained, engaging conversation rather than task completion. This includes the paper's own deployment context (Meta's AI Studio across Instagram, WhatsApp, Messenger) but also extends to independent platforms like Character.ai, Chai, and Replika, as well as enterprise applications like brand ambassador chatbots, interactive storytelling systems, and social companionship services. The paper's specific finding that character steerability improved from 26.6% to 5.8% instruction violation rate—a 78% relative reduction—while engagement simultaneously improved (V14: +8.8% breadth, +11.2% depth) demonstrates that persona adherence and engagement are not in tension; they can be jointly optimized through a unified annotation and training pipeline. A team deploying a persona-based chatbot today could adopt the paper's dual annotation workflow (engagement-focused annotation for quality, separate mildly-adversarial annotation for character adherence) as a concrete, immediately applicable practice.

2. Continuous improvement of deployed conversational models without full redeployment cycles. The paper's 15-iteration trajectory over 15 months demonstrates a cadence of approximately one improvement cycle per month. Each cycle includes data collection, reward model retraining, policy optimization, offline evaluation, and A/B testing. For teams operating conversational AI products at scale, this provides a benchmark timeline for what a sustainable improvement velocity looks like: roughly monthly deployments with engagement lifts in the +1% to +8% breadth range and +1% to +19% depth range per deployment, punctuated by occasional regressions (V12) that require diagnostic analysis and procedure adjustment. The cumulative engagement growth over nine months of post-launch deployment (Figure 8, bottom panel) suggests that this velocity can be maintained over extended periods without diminishing returns, though the paper's caveat that cumulative growth is "not entirely attributable to model updates" means that the per-version lifts are a more reliable metric for planning than the cumulative trajectory.

3. Data annotation pipeline design for subjective quality dimensions. The paper's finding that single-review training data is sufficient for reward model training while multi-review evaluation is necessary for measuring progress (Table 9: +4.02 point improvement on Multi-Review evaluation vs. negligible +0.07 point improvement on Single-Review evaluation) has direct implications for annotation budget allocation. For any project optimizing a subjective quality dimension (conversational engagement, creativity, humor, empathy, writing quality), this finding suggests a concrete annotation strategy: use single annotators for the high-volume training data (where the model can average out noise) and triple-annotator consensus for the evaluation sets used to select model checkpoints and measure progress. If the annotation budget for a project is, say, $100,000, the CharacterFlywheel result suggests allocating perhaps$ 70,000 to single-review training annotations (maximizing volume and coverage) and $30,000 to multi-review evaluation annotations (ensuring reliable measurement), rather than splitting the budget evenly. The paper does not explicitly recommend this ratio, but it is a natural operational conclusion from the Table 9 results.

4. Safety-tuned conversational AI where refusal calibration is critical. The paper's experience with false refusal rates—spiking above 30% in versions V5-V6 before settling around 20% on internal traffic and under 5% on user traffic by V15—provides a case study in managing the safety-engagement tradeoff during iterative optimization. The mechanism by which false refusals were corrected (the monitoring system detecting the spike, subsequent iterations recalibrating based on false refusal signals in the annotation data and safety data mixture adjustments) is directly applicable to any system where safety classifiers can become overly conservative during fine-tuning. The key operational insight: safety and engagement optimization must be interleaved, not sequential. If safety tuning is done once and then engagement optimization proceeds independently, there is no mechanism to correct safety drift. CharacterFlywheel's continuous monitoring across both safety and engagement metrics, with both feeding into the next iteration's data mixture, ensures that safety-engagement balance is maintained dynamically rather than established once and then gradually eroded.