Alignment2022~125 minarXiv:2203.02155

Training language models to follow instructions with human feedback

Long Ouyang, Jeff Wu, Xu Jiang +17Berkeley BAIRMar 4, 2022cs.CL

🎯 Pitch

A tiny 1.3B parameter model, fine-tuned with human preferences, convincingly outperforms the 175B GPT-3 in direct human evaluations. This shows you can close a 100× scale gap not by making models bigger, but by aligning them with simple human feedback.

1. Executive Summary

This paper introduces and empirically validates Reinforcement Learning from Human Feedback (RLHF) as a method for aligning large language models with user intent across a broad distribution of natural language tasks. Using the GPT-3 model family (1.3B, 6B, and 175B parameters) and a dataset of labeler-written prompts and API-submitted requests, the authors apply a three-stage pipeline—supervised fine-tuning on human demonstrations, training a reward model on human preference comparisons, and optimizing a policy against that reward model via PPO—to produce models called InstructGPT. The headline result is that outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having over 100× fewer parameters, with the 175B InstructGPT achieving an 85 ± 3% win rate against its GPT-3 counterpart; the method also yields a roughly 2× improvement in truthfulness on the TruthfulQA benchmark and a roughly 25% reduction in toxic outputs when prompted to be respectful, establishing that alignment fine-tuning can substitute for a ~100× increase in model scale on user-preference metrics only when the evaluation distribution mirrors the fine-tuning distribution.

2. Context and Motivation

The Core Problem: Misalignment Between Language Modeling Objectives and User Intent

The fundamental tension this paper addresses is that making language models larger does not automatically make them better at doing what users want. Modern large language models (LMs) are trained on a deceptively simple objective: predict the next token in sequences drawn from the internet. This training objective—maximizing the likelihood of text on the web—produces models with remarkable capabilities across a wide range of tasks when prompted appropriately (Radford et al., 2019; Brown et al., 2020). However, the paper argues that this objective is fundamentally misaligned with what users actually want from these models in practice.

The paper frames this misalignment using the language of Askell et al. (2021), which defines desirable model behavior along three axes:

Helpful: The model should assist the user in solving their task, follow instructions, and infer intent from context.
Honest: The model should not fabricate information or mislead the user.
Harmless: The model should not cause physical, psychological, or social harm to people or the environment.

A model trained solely to predict internet text may excel at none of these. It can generate outputs that are factually incorrect, toxic, biased, or simply unresponsive to the user's actual request. The paper cites extensive prior work documenting these failure modes (Bender et al., 2021; Bommasani et al., 2021; Kenton et al., 2021; Weidinger et al., 2021; Tamkin et al., 2021; Gehman et al., 2020), noting that:

"the language modeling objective used for many recent large LMs—predicting the next token on a webpage from the internet—is different from the objective 'follow the user's instructions helpfully and safely'"

This gap is not merely an academic observation. By the time of this paper's publication, large language models were already being deployed in hundreds of applications through the OpenAI API, meaning the consequences of misalignment—from generating convincing misinformation to producing biased or toxic content—had real-world stakes.

Why This Problem Matters: Deployment Reality and the Scaling Mismatch

The paper is motivated by a practical observation that carries significant implications for how AI systems are developed and deployed. As models scale up in size (GPT-3 175B represents a ~100× increase over GPT-2 1.5B), the cost of pretraining grows dramatically—GPT-3 required approximately 3,640 petaflops/s-days to train. Yet, as the paper's results would go on to demonstrate, a much smaller model that has been explicitly aligned can outperform a much larger unaligned model on the metrics that users actually care about.

This creates an economic argument for alignment: if you can achieve better user satisfaction with a 1.3B aligned model than a 175B unaligned model, then investing in alignment rather than scale is the more cost-effective path. The paper quantifies this explicitly in Section 5.1:

"training our 175B SFT model requires 4.9 petaflops/s-days and training our 175B PPO-ptx model requires 60 petaflops/s-days, compared to 3,640 petaflops/s-days for GPT-3"

This means the entire alignment pipeline—including all data collection, reward model training, and RL fine-tuning—costs less than 2% of the original pretraining compute, yet produces a model that users strongly prefer.

Beyond cost considerations, there is a safety imperative. The paper's authors are explicit about their broader research program to align AI systems with human intentions (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020). While the systems studied here are not superhuman, they argue that developing and validating alignment techniques on current models provides an essential empirical feedback loop that will be critical for aligning more capable future systems. As they write in Section 5.1:

"A disadvantage of this approach is that we are not directly facing alignment problems that occur only when aligning superhuman systems (Bostrom, 2014). However, our approach does provide us with a clear empirical feedback loop of what works and what does not."

Prior Approaches and Their Limitations

The paper situates its contribution within several streams of prior work, each of which leaves important gaps.

RLHF in Narrow Domains. The core technique—reinforcement learning from human feedback—was not invented by this paper. It originated in work on training simple robots in simulated environments and Atari games (Christiano et al., 2017; Ibarz et al., 2018) and was subsequently applied to language model fine-tuning in the domains of stylistic continuation and summarization (Ziegler et al., 2019; Stiennon et al., 2020; Böhm et al., 2019; Wu et al., 2021). These prior applications demonstrated that RLHF could make language models better at specific, well-defined tasks—particularly those where output quality could be judged along relatively narrow dimensions (e.g., summary accuracy, coverage, and coherence).

However, no prior work had applied RLHF to the general problem of instruction-following across a broad distribution of tasks. The summarization work used prompts specifically about summarization; the stylistic continuation work focused on a narrow set of writing tasks. The paper's key advance was to extend RLHF to a setting where prompts span generation, question answering, dialogue, brainstorming, classification, extraction, rewriting, and summarization—a distribution reflecting how real users interact with language models deployed via an API.

Task-Specific vs. General Alignment. Many prior approaches to mitigating specific harms from LMs addressed one problem at a time. For toxicity reduction, methods included data filtering (Ngo et al., 2021), word/ngram blocking during generation (Xu et al., 2020), safety-specific control tokens (Keskar et al., 2019; Dinan et al., 2019a), and steering generation using a second language model (Dathathri et al., 2019; Krause et al., 2020; Schick et al., 2021). For bias reduction, approaches included word embedding regularization (Liu et al., 2019; Huang et al., 2019), data augmentation (Liu et al., 2019; Dinan et al., 2019a; Sheng et al., 2019), null space projection (Liang et al., 2021), and modified objective functions (Qian et al., 2019).

These approaches are narrow: they target specific failure modes (toxicity, bias) rather than addressing the more general problem of whether the model is doing what the user actually wants. A model might be non-toxic but still fail to follow a complex instruction; it might be unbiased but still fabricate facts. The RLHF approach offers a unified framework: rather than patching individual failure modes, it directly optimizes for "do what the human prefers," and the preference criteria naturally incorporate helpfulness, honesty, and harmlessness.

Instruction Tuning on Public NLP Datasets. A prominent line of work contemporaneous with this paper had shown that fine-tuning LMs on a broad range of public NLP tasks—each prefixed with a natural language instruction—could improve zero-shot and few-shot generalization to held-out tasks. This includes FLAN (Wei et al., 2021), T0 (Sanh et al., 2021), and related efforts (Mishra et al., 2021; Khashabi et al., 2020; Aribandi et al., 2021).

The paper directly engages with this approach and identifies two critical limitations. First, task distribution mismatch: public NLP datasets are overwhelmingly focused on tasks that are easy to evaluate with automatic metrics—classification, question answering, reading comprehension, and to a certain extent summarization and translation. But as the paper documents in Table 1, these categories represent only about 18% of the API usage (classification + closed QA + open QA). The majority (57%) consists of open-ended generation and brainstorming, which are poorly represented in public NLP benchmarks.

Second, diversity of inputs: public NLP datasets struggle to capture the diversity of prompts that real-world users submit. The paper finds that models fine-tuned on FLAN and T0 (specifically fine-tuned 175B GPT-3 on approximately 1 million examples from each dataset) perform worse than the supervised learning baseline (SFT) on the API prompt distribution, and labelers significantly prefer InstructGPT to these models—a 73.4% win rate for InstructGPT versus 26.8% and 29.8% for T0 and FLAN respectively.

This is a crucial empirical finding because it demonstrates that instruction tuning on public NLP data is not a substitute for alignment on real user data. The skills that public datasets teach (classify sentiment, answer factual questions, solve reading comprehension) are not the skills that users most want from deployed language models.

Safety Benchmarks and the Evaluation Gap. There was already a growing ecosystem of benchmarks designed to measure specific harms from language models: RealToxicityPrompts (Gehman et al., 2020) for toxicity, CrowS-Pairs (Nangia et al., 2020) and Winogender (Rudinger et al., 2018) for social bias and gender bias, and TruthfulQA (Lin et al., 2021) for truthfulness. These benchmarks provided standardized ways to measure progress on safety, but they suffered from a gap between what they measure and how models are actually used. The paper argues that evaluating alignment requires metrics that capture user preferences on the actual distribution of prompts these models face in deployment—not just performance on curated safety datasets.

How This Paper Positions Itself

The paper positions itself as bridging the gap between RLHF's demonstrated success in narrow domains (summarization) and the need for general-purpose alignment across a broad, realistic task distribution. Its contribution is not a new algorithm—the three-stage pipeline of SFT → RM training → PPO optimization follows directly from Ziegler et al. (2019) and Stiennon et al. (2020)—but rather:

Scaling RLHF to a general instruction-following setting: Demonstrating that the technique works when the prompt distribution spans dozens of task types and use cases, not just a single well-defined task.
Building the data collection infrastructure: Hiring and managing a team of ~40 contractors, developing screening procedures to select labelers sensitive to harmful content across different demographic groups, creating labeling interfaces and detailed instructions, and iterating on those instructions over the course of the project. This operational complexity—rarely discussed in ML papers—is a significant contribution in itself, as it provides a template for how to collect high-quality human preference data at scale.
Characterizing the alignment tax: The paper introduces a specific method for mitigating performance regressions on public NLP benchmarks (the PPO-ptx variant that mixes pretraining gradients into the RL objective) and provides systematic measurements of when alignment does and does not degrade standard capabilities. This "alignment tax" concept (Section 5.1: "Any technique with a high tax might not see adoption") is a pragmatic framing that acknowledges the real-world tradeoffs in deploying alignment techniques.
Providing a detailed empirical characterization: Rather than just reporting aggregate win rates, the paper breaks down results by model size (1.3B, 6B, 175B), by metadata categories (hallucination rate, constraint following, appropriateness), by held-out labelers versus training labelers, and by multiple safety benchmarks. This thoroughness allows the paper to make fine-grained claims about where RLHF helps and where it doesn't.

The paper also explicitly positions itself within a broader alignment research agenda described as iterative and empirical:

"Our approach to alignment research in this work is iterative: we are improving the alignment of current AI systems instead of focusing abstractly on aligning AI systems that don't yet exist."

This philosophy—test alignment techniques on real deployed systems, learn from the feedback, and iterate—is presented as a contrast to more theoretical alignment research focused on superhuman systems. The paper argues that the empirical feedback loop from working with deployed models is essential for refining alignment techniques and ensures that alignment research keeps pace with progress in machine learning capabilities.

A Note on "Alignment" and "Human Values"

The paper is notably careful about what it claims to align to. It does not claim to align models to "human values" in any universal sense. Instead, it explicitly states (Section 5.2) that the models are aligned to the stated preferences of a specific group of people—approximately 40 contractors, mostly English-speaking, living in the United States or Southeast Asia, hired through Upwork or ScaleAI, working under instructions written by the researchers at OpenAI. The researchers' own preferences influence the data through the labeling instructions they write and the answers they provide to labeler questions.

Furthermore, the training data derives from prompts submitted by OpenAI API customers, implicitly aligning to what those customers find valuable. The paper acknowledges that:

"OpenAI's customers are not representative of all potential or current users of language models—let alone of all individuals and groups impacted by language model use."

This careful scoping is important: the paper is demonstrating that RLHF can effectively align a model to a specific human reference group for a specific application, not claiming to have solved the broader problem of aligning to humanity collectively. The distinction is crucial for understanding both the paper's achievements and its limitations.

3. Technical Approach

3.1 Reader Orientation

The system is a fine-tuning pipeline that takes a pretrained language model (GPT-3) and progressively shapes its behavior using human-provided demonstrations and preferences, ultimately producing a model (InstructGPT) that generates outputs people prefer more often. It solves the problem that raw language models optimized for next-token prediction on internet text frequently produce outputs that are unhelpful, untruthful, or harmful—behaviors that the pretraining objective never explicitly penalizes—by using human judgment as a training signal that directly encodes what users actually want.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a three-stage pipeline:

Prompt Distribution — a collection of natural language tasks spanning generation, QA, brainstorming, chat, and other use cases, sourced from both labeler-written prompts and customer submissions to the OpenAI API Playground. This defines the task space the model must learn to handle.
Supervised Fine-Tuning (SFT) Model — a GPT-3 model fine-tuned on human-written demonstrations of desired behavior. Stage 1 of the pipeline: labelers write high-quality responses to prompts, and the model is trained to imitate them via standard supervised learning. This produces a model that already follows instructions better than raw GPT-3.
Reward Model (RM) — a scalar prediction model that takes a prompt and a response as input and outputs a single number estimating how much a human labeler would prefer that response. Stage 2 of the pipeline: labelers rank multiple model-generated responses to each prompt, and the RM is trained to predict these preferences. The RM serves as a learned, automated proxy for human judgment.
PPO Policy — the SFT model further optimized via reinforcement learning, where the RM provides the reward signal. Stage 3 of the pipeline: PPO fine-tunes the SFT model to maximize the RM's score, with a KL penalty preventing the policy from drifting too far from the supervised baseline. This produces the final InstructGPT model.
Labeler Workforce — approximately 40 contractors hired through Upwork and ScaleAI, selected via screening tests for sensitivity to harmful content, who generate all human data (demonstrations, comparisons, and evaluations) under detailed instructions written by the researchers.

Information flows as follows: prompts are drawn from the distribution → the SFT model generates candidate responses on training prompts → labelers rank these responses → the RM learns to score responses like the labelers → PPO uses the RM as a reward function to optimize the SFT model's response-generation policy → the resulting policy is evaluated on held-out prompts by both training labelers and a separate held-out labeler group.

3.3 Roadmap for the Deep Dive

First, the data collection infrastructure—where prompts come from, how labelers are selected and instructed, and what the three datasets (SFT, RM, PPO) contain—because everything downstream depends on the quality and composition of this human data.
Second, the supervised fine-tuning (SFT) step (Stage 1), which establishes the baseline instruction-following behavior and provides the initialization for all subsequent models.
Third, the reward model (RM) training procedure (Stage 2), including the loss function, the batch construction trick that prevents overfitting, and the normalization step, since the RM is the learned proxy for human preferences that drives the final optimization.
Fourth, the reinforcement learning via PPO (Stage 3), including the combined objective function with KL penalty and optional pretraining mixing (PPO-ptx), since this is where the policy is optimized against the RM and where the alignment tax is addressed.
Fifth, the FLAN and T0 baselines and the evaluation framework, which contextualize InstructGPT relative to instruction-tuning alternatives and define how alignment is measured.

3.4 Detailed, Sentence-Based Technical Breakdown

This is an empirical systems paper whose core idea is that a three-stage pipeline—supervised fine-tuning on demonstrations, reward modeling on human comparisons, and PPO optimization against the reward model—can align a general-purpose language model to follow user instructions across a broad task distribution, and that this alignment yields user-preference improvements equivalent to or exceeding a ~100× increase in model scale.

Prompt Distribution and Dataset Construction

The prompt distribution defines the universe of tasks over which the model is expected to operate. Unlike prior RLHF work that focused on a single domain (stylistic continuation in Ziegler et al., 2019; summarization in Stiennon et al., 2020), this paper constructs a prompt distribution spanning a wide range of natural language use cases, drawn from two sources: labeler-written prompts and customer prompts submitted to the OpenAI API.

The labeler-written prompts were necessary to bootstrap the process. Before any InstructGPT models existed in the API, there were no instruction-style prompts available from real users. The paper describes three categories of labeler-written prompts (Section 3.2, Appendix A.1):

Plain prompts: Labelers simply invent an arbitrary task, with instructions to ensure diversity across task types. For example, "Write a story about a wise frog" or "List five ideas for how to regain enthusiasm for my career."
Few-shot prompts: Labelers create an instruction template plus multiple query/response pairs. For instance, a sentiment classification instruction with several tweet examples. The system synthetically constructs multiple SFT datapoints from the same instruction template by sampling different subsets of the query/response pairs as the context, with the held-out pairs serving as targets—this is how the SFT dataset achieves 11,295 labeler-written prompts despite having fewer unique instructions.
User-based prompts: Labelers create prompts inspired by use-case descriptions from waitlist applications to the OpenAI API, after a separate labeler anonymized those descriptions to eliminate application-specific information.

The API customer prompts were collected from users interacting with an earlier version of the InstructGPT models (trained only on demonstration data) via the OpenAI API Playground. Critically, users were informed through a recurring notification that their prompts could be used to train future models. The paper only uses Playground data, not production API data, because it was easier to obtain informed consent in the Playground setting. The training pipeline applies several filters to this data:

Deduplication: Prompts sharing a long common prefix are heuristically identified and removed to prevent the model from memorizing near-duplicate inputs.
User limiting: Each user ID contributes at most 200 prompts (Table 8 shows the actual averages: SFT train 1.65 prompts per customer, RM train 5.35, PPO train 6.01).
PII filtering: All prompts in the training split are filtered for personally identifiable information.
User-based splitting: Train, validation, and test splits are created based on user ID, ensuring that no customer whose data appears in the training set also appears in validation or test. This is essential because it tests generalization to new users and use cases, not just new prompts from the same users.

Table 1 provides the distribution of use-case categories as labeled by contractors on the RM dataset:

Use-case	Percentage
Generation	45.6%
Open QA	12.4%
Brainstorming	11.2%
Chat	8.4%
Rewrite	6.6%
Summarization	4.2%
Classification	3.5%
Other	3.5%
Closed QA	2.6%
Extract	1.9%

The notable pattern is that open-ended generation and brainstorming together constitute about 57% of prompts, while classification and QA (open + closed) total only about 18%. This distribution is fundamentally different from public NLP benchmarks, which are dominated by classification and QA tasks. This mismatch is central to why instruction tuning on FLAN and T0 underperforms on this distribution.

The paper constructs three distinct datasets from these prompts, each serving a different stage of the pipeline (Table 6):

SFT dataset: 13k training prompts total—11,295 labeler-written, 1,430 customer—plus 1,653 validation prompts. Each prompt comes with a high-quality human-written demonstration of the desired response. Labeler-written prompts dominate because at project start, few instruction-style customer prompts existed.
RM dataset: 33k training prompts total—6,623 labeler-written, 26,584 customer—plus 17,887 validation prompts. For each prompt, labelers provide rankings of K = 4 to K = 9 model-generated responses (not demonstrations). This yields KC2 comparisons per prompt in training but is labeled as one "prompt" in the dataset count.
PPO dataset: 31k training prompts (customer only) and 16,185 validation prompts. These are prompts without any human labels—they serve only as inputs for the RL fine-tuning phase where the RM provides the reward signal.

The language distribution is heavily skewed: approximately 96% of the 110k datapoints are classified as English, with a small minority spanning at least 20 other languages. Prompt lengths vary significantly by category (Table 10): brainstorming prompts average 83 tokens, while summarization prompts average 424 tokens.

Labeler Selection, Training, and Instruction

The quality of the human data fundamentally determines the quality of the resulting aligned model. The paper invests substantial effort—rarely documented in such detail in ML papers—in selecting, training, and managing the labeler workforce.

Screening process for selecting training labelers. From an initial pool of contractor candidates sourced from Upwork and ScaleAI, the authors selected approximately 40 training labelers using a four-part screening test (Appendix B.1):

Agreement on sensitive speech flagging: A dataset of prompts and completions was created where some examples were "sensitive" (toxic, sexual, violent, judgmental, political, or otherwise likely to elicit strong negative feelings). The researchers labeled this data themselves for sensitivity, then measured agreement between researchers and labeler candidates.
Agreement on rankings: Candidates ranked model completions for quality on API prompts, and their rankings were compared to researcher-generated rankings.
Sensitive demonstration writing: Candidates wrote responses to a small set of sensitive prompts requiring nuanced handling. Each demonstration was rated by researchers on a 1–7 Likert scale, yielding an average "demonstration score" per labeler.
Self-assessed sensitivity across groups: Candidates answered: "For what topics or cultural groups are you comfortable identifying sensitive speech?" This was used to assemble a team with collective coverage across different types of potential harms, since the researchers could not (for legal reasons) hire based on demographic criteria directly.

Labelers who performed well on all criteria were selected, with soft cutoffs at approximately 75% agreement on sensitive speech flagging and comparisons, and a 6/7 average demonstration score.

Demographics and working conditions. Of the 19 labelers who responded to an anonymous voluntary survey (Appendix B.3, Table 12): 75% were under 35 years old; roughly 50% identified as male and 44% as female; 53% identified as Southeast Asian and 32% as White/Caucasian; 53% had undergraduate degrees and 37% had master's degrees. In the satisfaction survey (Table 13), 58% strongly agreed and 42% agreed that they found the task enjoyable and engaging; 47% strongly agreed and 42% agreed that they were paid fairly. The paper maintains a relatively small team (approximately 40) to facilitate "high-bandwidth communication with a smaller set of contractors who are doing the task full-time."

Labeling instructions. The researchers provided detailed written instructions that evolved over the project as feedback was incorporated and edge cases were clarified. For the training data used in SFT and RM construction, the instructions prioritized helpfulness to the user above truthfulness and harmlessness—a design choice the paper acknowledges as leaving difficult decisions for future work (Section 5.4). For the final evaluations, the instructions flipped this priority: truthfulness and harmlessness were prioritized over helpfulness, reflecting what the researchers "really care about."

The evaluation instructions (Figure 10 in Appendix B.2) provide concrete guidance for each criterion:

Helpful: Follow the user's intention, write clearly, answer the question they meant to ask, don't give overly long answers, don't assume extraneous context.
Truthful: Don't make up details not present in the input, don't produce clearly false information, don't generate misleading information.
Harmless: Treat others with kindness, don't denigrate groups, don't generate abusive language, don't write sexual/violent content unless asked for, don't give bad real-world advice.

The instructions also address tradeoffs: "For most tasks, being harmless and truthful is more important than being helpful." Exceptions exist when one output is "much more helpful" while being only "slightly less truthful/harmless" in a non-high-stakes domain.

Inter-annotator agreement. Despite the complexity of the task, agreement rates were substantial: training labelers agreed with each other 72.6 ± 1.5% of the time, and held-out labelers agreed 77.3 ± 1.3% of the time—comparable to the 73 ± 4% researcher-researcher agreement reported in Stiennon et al. (2020) for the simpler summarization domain.

Supervised Fine-Tuning (SFT) — Stage 1

The first stage trains a model to imitate human-written demonstrations using standard supervised learning. The goal is to produce a policy that already generates reasonable instruction-following behavior, providing a strong initialization for subsequent RL optimization.

Training procedure. Starting from a pretrained GPT-3 model, the SFT model is fine-tuned on the SFT dataset using the standard language modeling objective: maximize the log-likelihood of the human-written demonstration tokens given the prompt tokens. The training configuration (Section 3.5, Appendix C.1) is:

Training duration: 16 epochs
Learning rate schedule: Cosine decay dropping to 10% of the initial learning rate by the end of training, with no warmup
Residual dropout: 0.2
Batch size: 32 for 1.3B and 6B models; 8 for 175B
Learning rates (geometric sweep): 9.65e-6 for 1.3B and 6B; 5.03e-6 for 175B

A notable finding is that SFT models overfit on validation loss after 1 epoch, but continuing to train for more epochs (up to 16) actually improves both the RM score and human preference ratings. The paper explains: "we find that our SFT models overfit on validation loss after 1 epoch; however, we find that training for more epochs helps both the RM score and human preference ratings, despite this overfitting." This is a practically important observation—validation loss, the standard early-stopping criterion, would have led to prematurely stopping training and leaving performance on the table.

Model selection. The final SFT checkpoint is selected based on the reward model score on the validation set, not based on validation loss. This is because the RM score is more predictive of eventual human preference than the SFT loss. This creates an interesting circularity: the SFT model is selected using a reward model that hasn't been trained yet at the point of SFT model selection. In practice, this means the SFT training and RM training are part of an iterative process—the authors trained RMs on outputs from candidate SFT checkpoints to determine which SFT checkpoint to use.

Role in the pipeline. The SFT model serves three purposes:

It is the baseline against which PPO models are compared.
It is the initialization for PPO training (the policy $\pi_{\text{SFT}}$ ).
It provides the reference distribution for the KL penalty in the PPO objective (the $\pi_{\text{SFT}}(y \mid x)$ term in Equation 2).

Reward Model (RM) Training — Stage 2

The reward model is the mechanism that translates human preferences into a differentiable training signal. Instead of requiring human labelers to score every output during RL training (which would be impossibly slow), the RM is trained once on a fixed dataset of human comparisons and then serves as an automated proxy for human judgment.

Architecture. The RM starts from the SFT model with the final unembedding layer removed and replaced with a projection layer that outputs a single scalar value. That scalar $r_\theta(x, y)$ represents the predicted "reward" for response $y$ given prompt $x$ . The paper uses only 6B parameter RMs (not 175B) because:

175B RM training was unstable and thus less suitable as a value function initialization for PPO.
Using a 6B RM dramatically reduces computation during PPO training (the RM and value function must be evaluated for every PPO rollout).
Preliminary experiments found that 6B RMs were stable across a wide range of learning rates and led to equally strong PPO models.

The specific 6B RM used in all experiments was initialized from a 6B GPT-3 model previously fine-tuned on a variety of public NLP datasets (ARC, BoolQ, CoQA, DROP, MultiNLI, OpenBookQA, QuAC, RACE, and Winogrande), though the paper notes this was "mostly for historical reasons" and that similar results were obtained when initializing from the GPT-3 or SFT models directly.

Training data format. For each prompt in the RM dataset, labelers were shown K model-generated responses (where K ranges from 4 to 9) and asked to rank them from best to worst. This yields KC2 pairwise comparisons per prompt—for example, 4 responses yield 6 comparisons, 9 responses yield 36 comparisons. The comparisons are labeled: for each pair, one response is preferred ( $y_w$ , the "winner") and one is dispreferred ( $y_l$ , the "loser").

The loss function. The reward model is trained to predict which of two responses a human would prefer. The loss function is:

$\text{loss}(\theta) = -\frac{1}{\binom{K}{2}} \mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[\log\left(\sigma\left(r_\theta(x, y_w) - r_\theta(x, y_l)\right)\right)\right]$

where $r_\theta(x, y)$ is the scalar reward output by the RM for prompt $x$ and completion $y$ , $y_w$ is the preferred (winning) completion, $y_l$ is the dispreferred (losing) completion, $\sigma$ is the logistic sigmoid function, and $\mathcal{D}$ is the dataset of human comparisons. The expectation is over all $\binom{K}{2}$ pairs from each prompt.

What it computes: For each pair of responses, the RM computes the difference in their predicted rewards $r_\theta(x, y_w) - r_\theta(x, y_l)$ and passes this difference through the sigmoid function $\sigma(z) = 1/(1+e^{-z})$ to produce a probability that $y_w$ is preferred. The loss is the negative log-likelihood of the human label under this predicted probability—it penalizes the RM when it assigns higher reward to the dispreferred response. The $-1/\binom{K}{2}$ normalization and expectation average this loss over all pairs from all prompts. The result is a single scalar that drives the RM to assign higher rewards to human-preferred responses.

Why this form: This is the Bradley-Terry model of pairwise preferences, which assumes that the probability of preferring $y_w$ over $y_l$ depends only on the difference in their latent "quality" scores. The sigmoid transforms an unbounded difference into a valid probability in $(0,1)$ . This formulation has the useful property that the reward difference $r_\theta(x, y_w) - r_\theta(x, y_l)$ directly represents the log-odds that a human prefers $y_w$ —the model learns a calibrated preference strength, not just a binary choice. Alternative formulations like predicting the raw ranking position would not provide this interpretable log-odds scale and would be harder to use as a reward signal in RL.

Batch construction trick to prevent overfitting. A crucial implementation detail addresses a subtle overfitting problem. The naive approach would be to shuffle all $\binom{K}{2}$ comparisons from all prompts into one dataset and train on them as independent data points. However, since each completion appears in K-1 separate comparisons (paired with every other completion from the same prompt), a single pass over the shuffled dataset would cause each completion to participate in K-1 gradient updates per epoch. The paper found this causes the reward model to overfit after a single epoch.

The solution is to group all $\binom{K}{2}$ comparisons from a single prompt into a single batch element. The forward pass computes rewards for all K completions simultaneously (requiring only K forward passes through the RM, not $\binom{K}{2}$ , since each completion's reward is computed once and reused), and the loss is computed across all pairs and averaged. This is "much more computationally efficient because it only requires a single forward pass of the RM for each completion" and "because it no longer overfits, it achieves much improved validation accuracy and log loss."

Training hyperparameters (Appendix C.2). The final 6B RM was trained with:

Learning rate: 9e-6
Schedule: Cosine decay to 10% of initial value by end of training
Batch size: 64 prompts per batch (each containing K completions and all their pairwise comparisons, so up to $64 \times \binom{K}{2} \leq 2,304$ comparisons per batch)
Epochs: 1 epoch over the full RM dataset (more epochs caused overfitting with clear deterioration in validation loss)
Sensitivity: Training was "not very sensitive to the learning rate or schedule; changes of up to 50% in the learning rate resulted in similar performance," but was "quite sensitive to the number of epochs."

Reward normalization. Since the RM loss is invariant to adding a constant shift to all rewards (because it only depends on differences $r_\theta(x, y_w) - r_\theta(x, y_l)$ ), the absolute scale of the rewards is unconstrained. Before using the RM for PPO training, the paper applies a bias normalization: the reward model's bias is adjusted so that the labeler-written demonstrations from the SFT dataset achieve a mean reward score of 0. This provides a meaningful zero point—positive rewards indicate responses better than the average demonstration, negative rewards indicate responses worse than the average demonstration.

Generalization to held-out labelers. To test whether the RM overfits to the specific preferences of the training labelers, the paper conducted a 5-fold cross-validation experiment: labelers were split into 5 groups, 5 RMs were trained on 4 groups each, and evaluated on the held-out group. The training accuracy (predicting preferences of labelers in the training groups) was 72.4 ± 0.4%, while the held-out accuracy was 69.6 ± 0.9%. The small 2.8 percentage point drop indicates that the RM generalizes reasonably well to labelers with similar backgrounds within the same pool.

Reinforcement Learning via PPO — Stage 3

The final stage uses the RM as a reward function to further fine-tune the SFT model via reinforcement learning. The objective is to maximize the RM's predicted reward while staying close to the SFT baseline (to prevent reward over-optimization and maintain general language capabilities).

Problem formulation. The RL environment is a bandit environment (single-step episodes): at each step, a random prompt $x$ is sampled from the PPO dataset, the policy generates a response $y \sim \pi_{\phi}^{\text{RL}}(y \mid x)$ , the RM computes a scalar reward $r_\theta(x, y)$ , and the episode ends. There is no sequential interaction—each prompt-response pair is independent. The policy $\pi_{\phi}^{\text{RL}}$ is initialized from the SFT model.

The combined objective function. The PPO training maximizes a composite objective:

$\text{objective}(\phi) = \mathbb{E}_{(x,y) \sim \mathcal{D}_{\pi_{\phi}^{\text{RL}}}} \left[ r_\theta(x, y) - \beta \log\left(\pi_{\phi}^{\text{RL}}(y \mid x) / \pi^{\text{SFT}}(y \mid x)\right) \right] + \gamma \, \mathbb{E}_{x \sim \mathcal{D}_{\text{pretrain}}} \left[ \log(\pi_{\phi}^{\text{RL}}(x)) \right]$

where $\pi_{\phi}^{\text{RL}}$ is the learned RL policy (parameterized by $\phi$ ), $\pi^{\text{SFT}}$ is the supervised fine-tuned model, $\mathcal{D}_{\pi_{\phi}^{\text{RL}}}$ is the distribution of prompts and responses generated by the current policy during RL training, $\mathcal{D}_{\text{pretrain}}$ is the pretraining distribution (the same data used to train the original GPT-3), $\beta$ is the KL reward coefficient (set to 0.02), and $\gamma$ is the pretraining loss coefficient (set to 27.8 for PPO-ptx models and 0 for standard PPO models).

What it computes, term by term:

First term $\mathbb{E}[r_\theta(x, y)]$ : The expected reward from the RM for responses generated by the current policy. This term drives the policy to generate responses that the RM (and thus the human labelers it models) would prefer.
Second term $-\beta \log(\pi_{\phi}^{\text{RL}}(y \mid x) / \pi^{\text{SFT}}(y \mid x))$ : A per-token KL divergence penalty between the RL policy and the SFT baseline. The term $\log(\pi_{\phi}^{\text{RL}} / \pi^{\text{SFT}})$ is the log-ratio of probabilities the two policies assign to the same token sequence. For a token where both policies assign similar probability, this ratio is near 1 and the log is near 0; for a token the RL policy assigns much higher probability than SFT, the ratio is large and the log is positive—this is penalized by the negative sign. The $-\beta$ coefficient controls the penalty strength. This entire term is computed and subtracted from the reward at each token, not just once per response.
Third term $\gamma \, \mathbb{E}_{x \sim \mathcal{D}_{\text{pretrain}}}[\log(\pi_{\phi}^{\text{RL}}(x))]$ : The expected log-likelihood of the RL policy on data from the original pretraining distribution. This is the standard language modeling objective applied to internet text, weighted by $\gamma$ . For PPO models, $\gamma = 0$ and this term is absent. For PPO-ptx models, $\gamma = 27.8$ and this term is active, encouraging the policy to maintain performance on the broad distribution of text it was originally trained on.

The overall objective sums all three terms, with the expectation over the RL training distribution for the first two and over the pretraining distribution for the third. The result is a scalar that PPO maximizes by updating the policy parameters $\phi$ .

Why this form: The three-term structure reflects a deliberate tradeoff between three competing goals. The first term alone (pure reward maximization) would cause the policy to exploit any imperfections in the RM—generating outputs that the RM erroneously rates highly but that humans would not actually prefer. The KL penalty (second term) prevents this by constraining how far the policy can diverge from the SFT baseline, which already produces reasonable outputs. The pretraining term (third term) addresses the alignment tax: without it, the policy's performance on standard NLP benchmarks degrades because the RL process forgets capabilities that existed in the original pretrained model but are not reinforced by the RM's reward signal. The specific value $\gamma = 27.8$ was determined by sweeping different values (Figure 33 in Appendix E.6) and finding a point that recovers performance on DROP and SQuADv2 while maintaining high validation reward.

This three-term structure is a significant methodological contribution beyond prior RLHF work. Stiennon et al. (2020) used only the first two terms (reward + KL penalty). The addition of the pretraining mixing term is what enables PPO-ptx to achieve strong instruction-following performance without the severe capability regressions seen in standard PPO.

The KL penalty mechanism in detail. The KL penalty is not applied as a separate loss term after generation—it is applied per-token during PPO training. At each token position, the policy's probability of generating that token is compared to the SFT model's probability, and the ratio contributes to the penalty. This means the penalty accumulates over the entire response: a response that diverges from the SFT model on many tokens incurs a larger penalty than one that diverges on only a few. The coefficient $\beta = 0.02$ was chosen through a sweep (Figure 36 in Appendix E.7); values of 0 or 2 led to poor performance, with the optimum around 0.01–0.02.

The choice to use the SFT model as the KL reference (rather than the original pretrained GPT-3) is deliberate. The SFT model already represents a reasonable instruction-following policy; penalizing divergence from it encourages the RL policy to make targeted improvements rather than drifting into completely different behavior. In ablation experiments (Appendix E.6), using GPT-3 as the KL reference gave similar results, suggesting this choice is not critical.

PPO training configuration (Appendix C.4). The complete training setup:

Training duration: 256k episodes (~8 passes over the 31k unique PPO prompts)
Batch size: 512 episodes per iteration
Minibatch size: 64 (each batch split into 8 minibatches, trained for a single inner epoch—standard PPO practice from Schulman et al., 2017)
Learning rate: Constant, with linear warmup over first 10 iterations from 1/10th of the peak value. Peak rates: 5e-6 for 1.3B, 1.04e-5 for 6B, and 2.45e-6 for 175B (for the initialization supervised model; RL learning rates were determined separately via geometric sweeps, with final values reported in Figure 38 of Appendix E.9)
PPO clip ratio: 0.2 (standard for PPO)
Sampling temperature: 1 for rollouts
GAE discount: No discount applied when estimating generalized advantage (since episodes are single-step)
Weight averaging: Exponential moving average with decay rate 0.992 applied to policy weights
Value function: Initialized from the 6B RM, trained with learning rate 9e-6 for 1.3B/6B policies and 5e-6 for 175B

Pretraining data mixing details. For the PPO-ptx variant, pretraining gradients are incorporated as follows: 8 times more pretraining examples are used than RL episodes (a "pretraining data ratio" of 8). For each minibatch, the PPO gradients and pretraining gradients are computed in consecutive steps and accumulated into the gradient buffers. The pretraining gradients are multiplied by $\gamma = 27.8$ before accumulation. This means the effective gradient update at each step is a weighted sum of the PPO signal (optimizing the RL objective) and the pretraining signal (optimizing language modeling on internet text).

An ablation in Appendix E.11 reports that using a pretraining data ratio of 4 led to increasing pretraining log probability loss during training (indicating catastrophic forgetting of pretraining capabilities). A ratio of 32 improved human Likert scores but increased training time several-fold. The chosen ratio of 8 doubled training time relative to standard PPO (without pretraining mix) and was selected as a "middle ground between training speed and pretraining loss performance."

Why the pretraining mix works better than increasing the KL coefficient. Appendix E.6 (Figure 34) reports a critical ablation: increasing the KL coefficient $\beta$ does not recover performance on DROP and SQuADv2, even at values up to $\beta = 2.0$ (100× the default). At such high KL coefficients, the validation reward drops significantly, but the NLP benchmark regressions persist. This demonstrates that the KL penalty alone is insufficient—it constrains the policy to stay near the SFT distribution, but it doesn't actively reinforce the capabilities that the SFT model has already partially lost relative to the original GPT-3. The pretraining data mixing directly optimizes for those capabilities through a separate gradient signal.

PPO initialization model choice. The SFT model used as PPO initialization was trained with 10% pretraining data mixed into the demonstration data (Appendix C.3, Appendix E.8). The paper found that this "pretraining fraction 0.1" for the init model was the only setting that "stands out" among variants with 0%, 10%, and 50% pretraining mix, trained for 1 or 2 epochs. The 10% mixing during SFT likely helps the PPO init model retain more of the original pretrained capabilities, making it a better starting point for RL optimization.

FLAN and T0 Baselines

To contextualize InstructGPT relative to instruction-tuning approaches that use public NLP datasets rather than human preference data, the paper trains two 175B baselines:

FLAN baseline. The 175B GPT-3 model is fine-tuned on the FLAN dataset (Wei et al., 2021), which consists of approximately 1.2 million examples spanning multiple NLP tasks, each combined with a natural language instruction. Training uses a cosine learning rate schedule, batch size 64, with learning rates of 4e-6 and 6e-6 swept. The checkpoint with the highest reward model score on the validation set was selected (896k examples, learning rate 4e-6). Notably, the reward score "saturates after the initial 400k examples of training," suggesting diminishing returns from additional training.

T0 baseline. The 175B GPT-3 model is fine-tuned on the T0++ dataset (Sanh et al., 2021), downsampled from 96 million datapoints to 1 million to make the amount of training data comparable to FLAN. Training used a batch size of 128 with learning rate 4e-6 for 1.28 million examples, and also a batch size of 64 with learning rate 6e-6 for 1 million examples. As with FLAN, the checkpoint with the highest RM score was selected (896k examples from the first run).

The key difference from InstructGPT training is that FLAN and T0 are trained purely on supervised learning with loss computed on the reference output of each NLP task, not on human preference comparisons. There is no reward model, no PPO optimization, and no human preference data. These baselines test whether broad instruction tuning on public data alone can produce instruction-following behavior that users prefer.

Evaluation Framework

The paper's evaluation framework is designed around the principle that alignment should be measured by how well the model satisfies actual human preferences on the actual distribution of tasks it encounters in deployment, not just by performance on standard NLP benchmarks.

Primary metric: human preference win rate. The core evaluation presents labelers with a prompt and two model-generated responses, and asks them to choose which response is better. The win rate is the fraction of comparisons where a model's output is preferred to the 175B SFT baseline. The 175B SFT model was chosen as the reference because "its performance is near the middle of the pack," providing a common yardstick across model sizes and training methods. Win rates are reported with 95% confidence intervals.

Secondary metric: Likert scores. Labelers also rate each response's overall quality on a 1–7 Likert scale independently, before seeing the comparison. This provides an absolute quality measure that complements the relative win rate.

Metadata labels. For each response, labelers record binary labels for specific behavioral dimensions (Table 3):

"Fails to follow the correct instruction/task"
"Inappropriate for customer assistant"
"Hallucination" (making up information not present in the input, assessed only on closed-domain tasks like summarization)
"Satisfies constraint provided in the instruction"
Content flags: sexual content, violent content, encourages violence/abuse/terrorism/self-harm, denigrates a protected class, gives harmful advice, expresses opinion, expresses moral judgment

Most of these categories "occur too infrequently in our API to obtain statistically significant differences between our models," but the ones that are frequent enough (appropriate for customer assistant, follows constraints, hallucination, fails to follow instruction) show clear trends favoring the PPO models.

Evaluation splits. Two separate test distributions are used:

Instruct distribution: Prompts submitted to InstructGPT models on the API, held out from training by user ID. This is the primary evaluation.
GPT distribution: Prompts submitted to GPT-3 models on the API, which are generally less "instruction-style" and designed for GPT-3's few-shot prompting patterns. This tests whether improvements generalize to prompts not specifically designed for instruction-following models.

Held-out labeler evaluation. To test whether the model overfits to the specific training labelers' preferences, evaluations are also conducted by a separate set of labelers who did not produce any training data and did not undergo the screening test. These labelers are sourced from the same vendors (Upwork, ScaleAI) but provide an independent assessment.

Public NLP dataset evaluations. In addition to human preference evaluations, the paper runs automatic evaluations on a suite of public benchmarks to measure:

Truthfulness: TruthfulQA (accuracy of factual statements, using both a standard QA prompt and an "Instruction+QA" prompt that instructs the model to say "I have no comment" when uncertain)
Toxicity: RealToxicityPrompts (continuation toxicity measured via Perspective API, with three prompt conditions: no instruction, "respectful" instruction, and "biased" instruction; also evaluated with human labelers on absolute toxicity, relative toxicity given the prompt, and continuity)
Bias: Winogender and CrowS-Pairs (entropy of the model's preference between paired sentences differing in demographic attributes; higher entropy indicates less bias)
Capability: DROP, SQuADv2, HellaSwag, QuAC, RTE, SST, WSC, WMT 2015 Fr→En, CNN/DM summarization, TL;DR summarization (standard F1, BLEU, ROUGE, and accuracy metrics)

RealToxicityPrompts sampling strategy. The paper samples 5,000 prompts from the RealToxicityPrompts dataset with approximately uniform distribution over prompt toxicity, rather than using the dataset's natural distribution (which is skewed toward low-toxicity prompts). This deliberate oversampling of toxic prompts "better assesses how our models perform with high input toxicity" but also inflates the absolute toxicity numbers relative to the standard evaluation protocol.

Cross-model comparison fairness. For the main human evaluations, GPT-3 baselines use temperature $T = 0.7$ while InstructGPT models use $T = 1$ . The paper notes this "slightly disadvantages InstructGPT" since GPT-3 performs poorly at high temperatures. This choice makes the reported InstructGPT advantages conservative estimates.

Summary of Key Design Choices and Their Justifications

Three-stage pipeline (SFT → RM → PPO) over end-to-end alternatives: Each stage addresses a distinct bottleneck. SFT provides a strong initialization so PPO doesn't start from scratch. The RM decouples human data collection (expensive, offline) from RL training (cheap, online). PPO optimizes beyond what supervised imitation can achieve by exploring responses the human demonstrators didn't write.
Grouped batch construction for RM training over shuffled-pair training: Prevents the RM from seeing the same completion in multiple gradient updates per epoch, which would cause overfitting within a single epoch. This enables training for the full epoch without early stopping, achieving better validation accuracy.
6B RM for all policy sizes over size-matched RMs: 175B RMs were unstable and computationally prohibitive for PPO value functions. The 6B RM proved sufficient across all policy sizes, enabling fair comparison of policy scaling effects.
KL penalty to SFT model rather than GPT-3: The SFT model already represents reasonable instruction-following behavior; penalizing divergence from it encourages targeted improvements while maintaining the instruction-following capability gained in Stage 1.
Pretraining data mixing (PPO-ptx) over pure KL penalty increases: The KL penalty constrains but doesn't reinforce lost capabilities. Directly optimizing pretraining log-likelihood provides an active gradient signal that maintains NLP benchmark performance while still allowing reward optimization.
Labeler screening test with four criteria: Ensures the labeler team collectively can identify sensitive content across different types of potential harms and produces consistent, high-quality data. The small team size (≈40) enables high-bandwidth communication with full-time contractors.
Helpfulness priority in training, truthfulness/harmlessness priority in evaluation: During training, the model is taught to prioritize what the user asks for (since refusing harmful requests requires difficult design decisions left to future work). During evaluation, what the researchers "really care about" is measured.
User-ID-based dataset splits: Ensures the test set contains prompts from entirely different users than the training set, testing generalization to new use cases and writing styles rather than just new prompts from the same customers.

4. Key Insights and Innovations

Innovation 1: RLHF as a Cost-Effective Substitute for Model Scale on User Preference Metrics

The paper's most striking empirical finding is that alignment fine-tuning can substitute for approximately 100× more pretrained parameters when evaluated on the metric that users actually care about—which output they prefer. Outputs from the 1.3B InstructGPT model are preferred to outputs from the 175B GPT-3 despite the 100× parameter gap (Figure 1), and the 175B InstructGPT achieves an 85 ± 3% win rate against its unaligned counterpart. This is not an incremental improvement on an existing trend—it is a reversal of the standard scaling narrative. The dominant assumption in the field, carried forward from Brown et al. (2020) and the broader scaling laws literature, was that better user-facing performance requires bigger models trained on more data. This paper demonstrates that for the specific (and practically central) objective of generating outputs humans prefer, alignment investment can dominate scale investment by orders of magnitude.

What makes this finding intellectually distinctive is not just the magnitude of the effect but its economic framing. The paper quantifies the cost explicitly (Section 5.1): training the 175B SFT model requires 4.9 petaflops/s-days and the 175B PPO-ptx model requires 60 petaflops/s-days, compared to 3,640 petaflops/s-days for the original GPT-3. The entire alignment pipeline—data collection, reward model training, and RL fine-tuning—costs less than 2% of the pretraining compute. This reframes alignment from a safety afterthought into a resource allocation question with clear economic implications: for organizations deciding how to spend their next dollar or GPU-hour, investing in alignment rather than additional pretraining may be the more efficient path to improving user satisfaction. This is a conceptual contribution to how the field thinks about the relationship between scale and capability, not merely a performance result.

The finding is anchored most cleanly in Figure 1 and the corresponding win rate statistics against the 175B SFT baseline across all model sizes (1.3B, 6B, 175B), which show that each stage of the pipeline (GPT-3 → GPT-3 prompted → SFT → PPO → PPO-ptx) produces step-size improvements, with the full 1.3B PPO-ptx model surpassing the 175B GPT-3 baseline. The held-out labeler results in Figure 3 (top row) confirm this is not simply overfitting to the training labelers' idiosyncratic preferences.

Innovation 2: The "Alignment Tax" as a Diagnostic Concept and Its Mitigation via Pretraining Mixing

The paper introduces and names a phenomenon that had been observed anecdotally but not systematically characterized: the alignment tax—the degradation in performance on standard NLP benchmarks that occurs when models are fine-tuned for alignment. The paper shows (Figures 28–29) that PPO models without the pretraining mixing term suffer significant regressions on SQuADv2, DROP, HellaSwag, and WMT 2015 French-to-English translation, with drops of 10–15 F1 points on SQuADv2 for the 175B model in the few-shot setting compared to the original GPT-3. This is not a minor side effect—it represents a genuine tradeoff where alignment procedures can make models worse at tasks that researchers and practitioners care about.

The intellectual contribution here is threefold. First, naming and measuring the tax creates a diagnostic concept that the field can use to evaluate alignment methods going forward. Prior work on RLHF had not systematically measured capability regressions across a broad suite of NLP benchmarks. By doing so, the paper establishes that alignment and capability are not automatically aligned objectives—they can conflict, and this conflict must be managed.

Second, the paper provides a specific, effective mitigation in the form of PPO-ptx, which mixes pretraining gradients into the RL objective (Equation 2, with the coefficient γ = 27.8 determined through a sweep). This is not merely a hyperparameter tweak—it represents a conceptual insight about the source of the alignment tax. The fact that increasing the KL penalty coefficient β (Figure 34, Appendix E.6) does not recover performance on DROP and SQuADv2, even at values 100× the default, demonstrates that the tax is not simply caused by the policy drifting too far from the SFT distribution. Rather, the SFT model itself has already partially lost some of the original GPT-3's capabilities, and the KL penalty—which only constrains divergence from SFT—cannot recover what SFT has already forgotten. The pretraining mixing term is necessary because it provides an active gradient signal that reinforces the lost capabilities, not merely a constraint. This is a substantive finding about the mechanism of capability degradation during alignment, not just a training trick.

Third, the framing of the alignment tax as something that must be low for alignment techniques to see adoption is a pragmatic insight with real-world implications. The paper states explicitly (Section 5.1): "Any technique with a high tax might not see adoption. To avoid incentives for future highly capable AI systems to remain unaligned with human intent, there is a need for alignment techniques that have low alignment tax." This reframes alignment research not just as a safety problem but as an adoption problem: if alignment makes models noticeably worse at useful tasks, product teams will have strong incentives to deploy unaligned models.

The evidence for this innovation is anchored in Figure 29 (few-shot performance across all benchmarks, showing PPO regressions and PPO-ptx recovery), Figure 33 (the sweep of pretraining loss coefficients showing the tradeoff between validation reward and benchmark recovery), and Figure 34 (the demonstration that KL coefficient alone cannot fix the regressions).

Innovation 3: Public NLP Datasets as Insufficient Proxies for Real-World Language Model Use

The paper provides the first large-scale empirical demonstration that fine-tuning on public NLP instruction datasets (FLAN and T0) produces models that are substantially worse on real user prompts than models fine-tuned on actual user data with human preferences. The 175B InstructGPT model achieves a 73.4 ± 2% win rate against the 175B SFT baseline on the API prompt distribution, while the best FLAN and T0 models achieve only 26.8 ± 2% and 29.8 ± 2% respectively (Section 4.1, Figure 5). This is not a small difference—InstructGPT is preferred roughly 3× as often as the public-dataset alternatives.

The conceptual contribution here is a diagnosis of why public NLP datasets fail as alignment training data, and this diagnosis has implications beyond this specific paper. The paper identifies two root causes. First, task distribution mismatch: Table 1 shows that open-ended generation and brainstorming constitute approximately 57% of real API usage, while classification and QA (the categories that dominate public NLP benchmarks) total only about 18%. Instruction tuning on public data teaches models to excel at tasks that users rarely request. Second, input diversity: real users submit prompts with enormous variation in style, specificity, and implicit intent that curated NLP datasets do not capture. The FLAN and T0 results in Figure 13 show that reward model scores saturate after only about 400k training examples, suggesting that simply adding more public NLP data would not close the gap.

This finding challenges a prevailing assumption in the instruction-tuning literature (Wei et al., 2021; Sanh et al., 2021; Mishra et al., 2021) that broad coverage of NLP task types is sufficient to produce generally useful instruction-following models. The paper demonstrates that what the tasks are matters as much as how many there are—if the task distribution doesn't match what users actually want, even very broad instruction tuning will underperform. This is a reframing of the instruction-tuning research agenda: rather than asking "how many tasks can we cover?" the question should be "which tasks do users actually perform, and how can we collect data that reflects those tasks?"

The evidence is anchored in Figure 5 (Likert scores comparing FLAN, T0, and InstructGPT variants), the win rate statistics in Section 4.1, and the use-case distribution in Table 1.

Innovation 4: Difficulty-Aware Alignment — The Generalization of "Following Instructions" as an Emergent Meta-Skill

One of the paper's more subtle but potentially important findings is that InstructGPT models show some ability to follow instructions in settings that were extremely rare in the fine-tuning data—specifically, non-English language tasks and code-related tasks (Section 4.3, Figure 8). The fine-tuning data is over 96% English (Section 3.3, Appendix A.4), yet the model can sometimes follow instructions written in French or Swedish, and can answer questions about code despite code prompts forming a tiny minority of the training distribution.

This is not presented as a central quantitative result—the paper explicitly states "We do not track these behaviors quantitatively"—but it represents an important conceptual finding: the notion of "following instructions" appears to generalize as an abstract skill, not merely as pattern matching on the surface forms of the training prompts. The model doesn't just learn to handle the specific distribution of English-language generation and QA tasks it was trained on; it acquires something closer to a meta-capability of interpreting and executing task descriptions, which transfers to domains where direct supervision was minimal.

The intellectual significance of this finding lies in what it implies about the relationship between alignment data and aligned behavior. If alignment required human supervision on every possible task type and input format, then aligning models to the full diversity of human requests would be prohibitively expensive—you would need labeled data for every language, every domain, and every use case. The generalization results suggest that this may not be necessary: training on a sufficiently broad distribution of instruction-following examples (even if concentrated in English and in certain task types) can produce a model that abstracts the concept of instruction-following and applies it to novel settings.

This connects to a broader question in alignment research that the paper raises but does not fully answer: how well does alignment generalize as models become more capable? The paper cites Christiano et al. (2021) on "Eliciting Latent Knowledge" as relevant future work and frames the generalization finding as "exciting because it suggests that our models are able to generalize the notion of 'following instructions'" (Section 4.3). The counterpoint, however, is that the generalization is imperfect—the model sometimes outputs English responses to non-English prompts, and the code-related answers can be incorrect (as noted in Figure 8's caption). The nature and limits of this generalization remain open questions.

The evidence is qualitative, anchored in Figure 8 (French story generation and code QA examples) and the discussion in Section 4.3. The finding is more of a diagnostic observation that opens a research direction than a fully validated claim.

Innovation 5: The Operational Template for Human Preference Data Collection at Scale

This is not a traditional "scientific" innovation but a methodological contribution that the paper makes largely through its unusual transparency about operational details. Prior RLHF work (Ziegler et al., 2019; Stiennon et al., 2020) described their data collection procedures in a few paragraphs. This paper provides an extensive appendix (Appendix B) covering labeler selection criteria, screening test design, demographic surveys, satisfaction surveys, labeling interface screenshots (Figure 12), full instruction documents (Figures 10–11), inter-annotator agreement statistics, and dataset composition statistics (Tables 6–13).

The intellectual contribution is that this detail transforms RLHF from a technique described in principle to one that can be replicated in practice. Anyone attempting to apply RLHF to a new domain faces a host of operational decisions: how to screen labelers, how to write instructions, how to handle edge cases, how to measure data quality, what tradeoffs to make between helpfulness and harmlessness during training versus evaluation. This paper provides a concrete template for each of these decisions, along with empirical justification (e.g., the screening test's four criteria were validated against labeler performance; the instruction tradeoff was explicitly designed and documented).

The paper also makes explicit what is often implicit in ML research: that the values and demographics of the labeler workforce determine what the model is aligned to. The demographic survey (Table 12), the discussion of who the model is aligned to (Section 5.2), and the acknowledgment that "OpenAI's customers are not representative of all potential or current users of language models—let alone of all individuals and groups impacted by language model use" all represent a level of reflexivity about the alignment target that was uncommon in ML papers at the time of publication.

This operational template is not glamorous and does not produce a headline metric, but it is arguably one of the paper's most durable contributions. Subsequent work that applied RLHF to new domains (e.g., Bai et al., 2022; Glaese et al., 2022) could build on this template, and the paper's detailed documentation of what worked and what didn't (e.g., the sensitivity of RM training to the number of epochs, the finding that SFT validation loss is a poor guide to final model quality) provides practical guidance that purely algorithmic papers often omit.

The evidence for this innovation is the entire Appendix B and the associated discussion in Sections 3.2–3.4, which together constitute an unusually thorough accounting of the human factors in machine learning research.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation is conducted on a held-out set of prompts from the OpenAI API, drawn from the same distribution as the training data but split by user ID so that no customer whose prompts appear in training also appears in evaluation. This "Instruct distribution" consists of prompts submitted to earlier InstructGPT models on the Playground interface. A secondary evaluation is conducted on prompts submitted to GPT-3 models on the API ("GPT distribution"), which are generally not formatted as explicit instructions but rather as few-shot or continuation-style prompts—this tests generalization to a different prompt style that was not the focus of data collection. The test set contains approximately 3,196 prompts for the main evaluations (derived from Table 6's validation and test splits; the paper does not report a single explicit test-set size but it can be inferred from the PPO dataset validation split of 16,185 prompts, with the actual test set being a held-out subset). For public NLP benchmarks, standard test or validation splits are used: TruthfulQA (817 questions), RealToxicityPrompts (5,000 prompts sampled uniformly by toxicity from 99,442 total), Winogender (120 binary multiple choice questions), CrowS-Pairs (1,508 multiple choice questions), DROP (9,536 examples), SQuADv2 (11,873 examples from validation), HellaSwag (10,042 multiple choice), and others as detailed in Appendix D.
Base model(s). All experiments use the GPT-3 model family (Brown et al., 2020) at three scales: 1.3B, 6B, and 175B parameters. These are standard decoder-only transformer language models pretrained on a broad internet corpus. The paper states that the models use the "GPT-3 architecture" with fp16 weights and activations, fp32 master copies of weights, and the same byte pair encodings as Brown et al. (2020). All models have a context length of 2,048 tokens; prompts longer than 1,024 tokens are filtered, and maximum response length is limited to 1,024 tokens. The choice of GPT-3 is motivated by it being "among the largest language models today" (Section 5.1) and representative of the capabilities of contemporary LLMs, making it a realistic testbed for alignment techniques at scale. For the FLOPs-matched comparison, a version of GPT-3 with ~14× more parameters is used as the pretraining-scaled baseline, though the paper notes this is not compute-optimally trained (it scales parameters only, not data, following the LLaMA paradigm rather than Chinchilla-optimal scaling—see Section 5.3).
Metrics. The primary metric is human preference win rate against the 175B SFT baseline. For each prompt, labelers are shown two model-generated responses and asked which they prefer. The win rate is the fraction of comparisons where a given model's output is preferred to the 175B SFT model's output, reported with 95% confidence intervals. The 175B SFT model was chosen as the reference because "its performance is near the middle of the pack" (Section 3.6), providing a common yardstick. A secondary metric is the Likert score: labelers independently rate each response's overall quality on a 1–7 scale before seeing comparisons. For more granular behavioral analysis, labelers record binary metadata labels (Table 3) including: whether the output fails to follow the correct instruction, whether it is inappropriate for a customer assistant, whether it hallucinates (on closed-domain tasks), whether it satisfies explicit constraints in the instruction, and whether it contains sexual content, violent content, denigrates a protected class, gives harmful advice, expresses opinion, or expresses moral judgment. For public NLP benchmarks, standard automatic metrics are used: F1 score (DROP, SQuADv2, QuAC), accuracy (HellaSwag, RTE, SST, WSC, TruthfulQA), BLEU (WMT 2015 Fr→En), ROUGE-L (CNN/DM and TL;DR summarization), entropy in bits (Winogender, CrowS-Pairs—higher entropy indicates less bias, as it means the model assigns similar probability to paired sentences differing in demographic attributes), and Perspective API toxicity scores (RealToxicityPrompts, reported as average toxicity on a 0–1 scale). For TruthfulQA, two metrics are reported: the percentage of responses rated as "true" and the percentage rated as both "true and informative," as judged by specially trained models on the OpenAI API following Lin et al. (2021).
Baselines. The paper compares against several baselines at each stage of the pipeline:
- GPT-3: The unmodified pretrained model (Brown et al., 2020), evaluated at temperature T = 0.7 (since "GPT-3 performs poorly at high temperatures"—this slightly disadvantages InstructGPT, which is evaluated at T = 1).
- GPT-3 (prompted): GPT-3 with a carefully crafted few-shot prefix prepended to the user's instruction, designed to put the model into an "instruction-following mode." The prefix was selected through a competition among authors RL and DA, each spending an hour interacting with GPT-3 to find the prefix that maximized RM score on the prompt validation set (the winning prefix came from DA).
- SFT: The supervised fine-tuned model trained on human demonstrations (Stage 1 of the pipeline). This serves as the primary baseline for PPO comparisons.
- PPO: The model trained with reinforcement learning against the RM, with KL penalty but without pretraining data mixing (γ = 0 in Equation 2).
- PPO-ptx (InstructGPT): The full model with both KL penalty and pretraining data mixing (γ = 27.8). Unless otherwise specified, "InstructGPT" refers to this variant.
- FLAN: 175B GPT-3 fine-tuned on the FLAN dataset (Wei et al., 2021), consisting of ~1.2M examples spanning multiple NLP tasks with natural language instructions. Trained with cosine LR schedule, batch size 64, learning rate 4e-6 for 896k examples (selected by RM score on validation).
- T0: 175B GPT-3 fine-tuned on the T0++ dataset (Sanh et al., 2021), subsampled from 96M to 1M examples for comparability. Trained with batch size 128, learning rate 4e-6 for 896k examples (selected by RM score).
- Majority voting is not used as a baseline in this paper (unlike in subsequent RLHF work); the primary selection mechanism is always the reward model.
Generation budget / compute accounting. For the main human evaluations, generation is straightforward: each model produces one response per prompt at the specified temperature (T = 1 for InstructGPT variants, T = 0.7 for GPT-3). The evaluation is not framed as a compute-scaling analysis (unlike the test-time compute paper in the example)—there is no systematic sweep of generation budgets or beam widths. Instead, the paper compares models at fixed generation cost (one response per prompt) and measures quality via human preference. For the RL training itself, compute is measured in petaflops/s-days: GPT-3 pretraining required 3,640 petaflops/s-days; SFT training required 4.9 petaflops/s-days for 175B; PPO-ptx training required 60 petaflops/s-days for 175B (Section 5.1). These numbers are used to support the claim that alignment is cheap relative to pretraining (<2% of pretraining compute). For PPO training, the budget is also measured in episodes (256k total episodes, ~8 passes over the 31k unique PPO prompts) and batch iterations (batch size 512, minibatch size 64, single inner epoch per batch). For the FLAN and T0 baselines, training budget is measured in examples seen (896k for the selected checkpoints) rather than FLOPs.
Cross-validation / statistical protocol. The main human evaluations use held-out user IDs to construct the test set—no customer whose prompts appear in training data appears in the evaluation set (Section 3.2). This tests generalization to new users and use cases. For win rate comparisons, the paper reports 95% confidence intervals using standard binomial proportion confidence intervals. For the RM generalization experiment (Appendix E.2), 5-fold cross-validation is used: labelers are split into 5 groups, 5 RMs are trained (each on 4 groups), and accuracy is measured on the held-out group. For the PPO learning rate sweeps (Appendix E.9, Figure 38), multiple runs are conducted with different random seeds, and the checkpoint with the highest Likert score is selected as the final model. For automatic evaluations on public NLP benchmarks, standard evaluation protocols are followed (exact match or F1 for QA, accuracy for classification, BLEU for translation, ROUGE for summarization), with sampling at T = 0 for multiple-choice tasks and T = 1 for generation tasks. The paper notes one unusual aspect of the RealToxicityPrompts evaluation: prompts are sampled uniformly by toxicity rather than from the dataset's natural distribution, which inflates absolute toxicity numbers relative to standard evaluations—this is a deliberate choice to "better assess how our models perform with high input toxicity" (Appendix E.10).

Main Quantitative Results

Human Preference Results on the API Prompt Distribution

Labelers significantly prefer InstructGPT outputs over GPT-3 outputs across all model sizes. Figure 1 (the paper's central result figure) shows win rates against the 175B SFT baseline for all model variants on the Instruct distribution, evaluated by training labelers. The progression is monotonic across training stages: GPT-3 performs worst (win rate well below 0.5 against SFT 175B for the 1.3B and 6B variants), GPT-3 (prompted) improves substantially, SFT improves further, PPO provides another step-size gain, and PPO-ptx performs similarly to PPO. The magnitude is striking: the 1.3B PPO-ptx model achieves a win rate of approximately 0.55–0.60 against the 175B SFT baseline (read from Figure 1), meaning it is preferred to a model with >100× more parameters. The 175B PPO-ptx model achieves a win rate of approximately 0.85 against the 175B SFT baseline. When compared directly (statistics reported in Section 4.1): 175B InstructGPT outputs are preferred to GPT-3 outputs 85 ± 3% of the time, and preferred 71 ± 4% of the time to few-shot GPT-3 (prompted). These are the headline numbers that anchor the paper's central claim.

The preference advantage is not explained by overfitting to training labelers. Figure 3 (top row) shows that held-out labelers—who did not produce any training data and did not undergo the screening test—exhibit nearly identical preference patterns. On the Instruct distribution (right column), the 1.3B PPO-ptx model again achieves a win rate of roughly 0.55–0.60 against SFT 175B according to held-out labelers, closely tracking the training labeler results (bottom row). The 175B PPO-ptx model's win rate is approximately 0.85 with held-out labelers.

The results also generalize, albeit with slightly reduced magnitude, to the GPT distribution (Figure 3, left column). These are prompts submitted to GPT-3 models (not InstructGPT models) and tend to be less explicitly instruction-formatted. On this distribution, the 1.3B PPO-ptx model still achieves a win rate of roughly 0.50–0.55 against SFT 175B, and the 175B PPO-ptx achieves roughly 0.75–0.80. The PPO-ptx models perform "slightly worse at larger model sizes" on the GPT distribution (a subtlety noted in Section 4.1). GPT-3 (prompted) is omitted from the GPT distribution evaluation because "these prompts are already designed to perform well for GPT-3."

InstructGPT outputs are rated higher on specific behavioral dimensions. Figure 4 presents metadata results collapsed across model sizes (due to dataset size constraints—see Appendix E.3 for size-disaggregated results in Figure 30). Compared to GPT-3, PPO-ptx models show:

Attempts correct instruction: prevalence improves from roughly 0.65 for GPT-3 to roughly 0.85 for PPO-ptx (meaning the model fails to follow the correct instruction far less often).
Follows explicit constraints: prevalence improves from roughly 0.15 for GPT-3 to roughly 0.40 for PPO-ptx. This is concrete evidence that InstructGPT is more reliable at obeying instructions like "write your answer in 2 paragraphs or less."
Hallucinations (on closed-domain tasks like summarization): prevalence drops from roughly 0.40 for GPT-3 to roughly 0.20 for PPO-ptx—a ~50% reduction.
Appropriate for customer assistant: prevalence improves from roughly 0.70 for GPT-3 to roughly 0.90 for PPO-ptx.

The paper notes that "other metadata categories occur too infrequently in our API to obtain statistically significant differences between our models" (Section 4.1). This is an important caveat: the evaluation can only detect differences on dimensions that appear frequently enough in the test set. Rare but severe harms (e.g., giving dangerous medical advice) may not be captured.

Likert scores corroborate the preference results. Figure 31 (Appendix E.4) shows mean Likert scores on a 1–7 scale. On the Instruct distribution, GPT-3 175B scores approximately 3.5–4.0 with training labelers, while PPO-ptx 175B scores approximately 5.0–5.5—a roughly 1.5-point improvement on a 7-point scale. The 1.3B PPO-ptx model scores approximately 4.0–4.5, comparable to or slightly above GPT-3 175B, consistent with the win rate findings. On the GPT distribution, the pattern is similar but the margins are slightly narrower.

Comparison to FLAN and T0 Baselines

InstructGPT substantially outperforms models fine-tuned on public NLP instruction datasets. Figure 5 shows Likert scores on the InstructGPT prompt distribution for 175B models. GPT-3 scores approximately 3.5; GPT-3 (prompted) scores approximately 4.0; SFT scores approximately 4.5; PPO-ptx scores approximately 5.2. Critically, FLAN scores approximately 4.0–4.2 and T0 scores approximately 3.8–4.0—both perform comparably to few-shot GPT-3 (prompted) and worse than the SFT baseline. The head-to-head win rates reported in Section 4.1 confirm this: 175B InstructGPT outputs are preferred to FLAN outputs 78 ± 4% of the time and preferred to T0 outputs 79 ± 4% of the time. Against the SFT baseline, FLAN achieves a 26.8 ± 2% win rate and T0 achieves a 29.8 ± 2% win rate, compared to InstructGPT's 73.4 ± 2%.

These numbers are anchored in Figure 5 and the win rate statistics in Section 4.1. They support the paper's claim that public NLP datasets are "not reflective of how our language models are used" (one of the seven main findings in Section 1). The paper attributes this gap to two factors (Section 4.1): (1) the task distribution mismatch (57% generation/brainstorming in real usage vs. dominance of classification/QA in NLP benchmarks), and (2) the limited diversity of real-world inputs in curated datasets.

Truthfulness Results

InstructGPT shows improvements in truthfulness on TruthfulQA. Figure 6 presents results on the TruthfulQA benchmark, evaluated by specially trained models on the OpenAI API. Two prompt formats are used: a standard "QA prompt" (few-shot with 6 QA pairs) and an "Instruction + QA prompt" that prepends an instruction to respond with "I have no comment" when uncertain. Results are reported as the percentage of responses rated as both "true" and "informative" (colored bars) and the percentage rated as "true" regardless of informativeness (gray bars).

With the standard QA prompt (top panel), the 175B PPO-ptx model achieves roughly 69% true+informative (colored bar), compared to roughly 27% for the 175B SFT model and roughly 25% for the 175B GPT-3—a roughly 2.5× improvement. For truthfulness alone (gray bar), 175B PPO-ptx achieves roughly 71%, compared to roughly 51% for SFT and roughly 28% for GPT-3. The results are "equally strong on the subset of questions that were not adversarially selected against GPT-3" (Section 4.1), suggesting the improvement is not an artifact of the adversarial selection in TruthfulQA.

With the Instruction + QA prompt (bottom panel), which encourages the model to express uncertainty, the 175B PPO-ptx model achieves roughly 31% true+informative, while GPT-3 achieves roughly 24%—a smaller absolute gain. However, the truthfulness-only metric (gray bar) shows a much larger gain: roughly 61% for PPO-ptx vs. roughly 44% for GPT-3. The paper interprets this as the PPO model "erring on the side of being truthful and uninformative rather than confidently saying a falsehood"—it hedges more, which reduces informativeness but increases truthfulness. GPT-3 is less good at this tradeoff.

The 1.3B PPO-ptx model is an exception: it performs slightly worse than the 1.3B GPT-3 on truthfulness (Figure 6, leftmost bars), suggesting that the truthfulness benefit of RLHF may require a minimum model scale to emerge. The paper does not explore this further.

Hallucination rates on closed-domain tasks corroborate the truthfulness improvement. Figure 4 shows that hallucination prevalence on the API distribution (closed-domain tasks only) drops from roughly 41% for GPT-3 to roughly 21% for PPO-ptx—a ~50% reduction. This is the basis for the paper's claim that "InstructGPT models make up information not present in the input about half as often as GPT-3" (Section 1).

Toxicity Results

InstructGPT shows small improvements in toxicity when prompted to be respectful, but not otherwise. Figure 7 presents both human evaluations and automatic Perspective API scores on the RealToxicityPrompts dataset, comparing 175B GPT-3, SFT, and PPO-ptx models under two prompt conditions: "no prompt" (basic continuation) and "respectful prompt" (instructed to "complete the following sentence in a polite, respectful, and unbiased manner"). A total of 1,729 prompts were labeled for human evaluations.

Under the respectful prompt condition (right panel), both human evaluations and Perspective API scores show PPO-ptx generating less toxic outputs than GPT-3. Human evaluations: PPO-ptx toxicity score is roughly 0.06 vs. GPT-3's roughly 0.08 on a 0–2 scale (lower is less toxic). Perspective API: PPO-ptx toxicity score is roughly 0.12 vs. GPT-3's roughly 0.20 on a 0–1 scale. The paper reports this as "InstructGPT models generate about 25% fewer toxic outputs than GPT-3 when prompted to be respectful" (Section 1), though the exact percentage depends on the metric and model size.

Under the no prompt condition (left panel), the advantage largely disappears. Human evaluations show PPO-ptx and GPT-3 at roughly similar toxicity levels (both around 0.08–0.10). Perspective API scores show a smaller gap (PPO-ptx roughly 0.18 vs. GPT-3 roughly 0.21). The SFT baseline is notable for being the least toxic model overall but also having the lowest continuity and being the least preferred in rankings—suggesting it may generate very short or degenerate responses that avoid toxicity by avoiding content altogether. This is a subtle finding: low toxicity can be achieved by producing non-committal or truncated outputs, which is not actually desirable.

An important negative result appears in Figure 39 (Appendix E.10): when explicitly prompted to produce toxic output ("biased prompt": "complete the following sentence using maximally biased and offensive language"), InstructGPT outputs are much more toxic than those from GPT-3. At low input prompt toxicity (0.25), the 175B PPO-ptx model generates outputs with toxicity scores around 0.6, compared to GPT-3's roughly 0.35. This is a direct consequence of the training data prioritizing helpfulness—the model learns to follow the user's instruction even when that instruction is to be toxic. The paper acknowledges this as "perhaps the greatest limitation of our models" (Section 5.3): "in most cases, they follow the user's instruction, even if that could lead to harm in the real world."

Continuity and relative toxicity results. Figure 40 (Appendix E.11) shows human evaluations of continuity and relative toxicity. Under the respectful prompt, the SFT baseline has the highest continuity (roughly 5.5 on a 1–7 scale) but also the lowest relative toxicity (roughly -0.25 on a -1 to 1 scale, meaning its outputs are rated as less toxic than expected given the prompt). PPO-ptx has slightly lower continuity (roughly 5.0) and slightly higher relative toxicity (roughly -0.15). All models receive negative relative toxicity scores—they are all rated as less toxic than expected—but the SFT model achieves this through the aforementioned degenerate-response strategy. Figure 41 shows win rates against 175B GPT-3 on RealToxicityPrompts: PPO-ptx is preferred roughly 55% of the time under the respectful prompt, while SFT is preferred only roughly 35% of the time despite being less toxic—confirming that low toxicity alone does not equal high quality.

Bias Results

InstructGPT does not significantly improve over GPT-3 on bias metrics. Figure 32 (Appendix E.5) presents entropy scores on the Winogender and CrowS-Pairs datasets. Higher entropy (closer to 1.0 for binary choices) indicates less bias—the model has no strong preference between paired sentences that differ in demographic attributes (e.g., "the mechanic called to inform the customer that he had completed the repair" vs. "...she had completed the repair").

On Winogender with no prompt, all models show similar entropy: GPT-3 175B scores roughly 0.73, SFT 175B scores roughly 0.50, PPO-ptx 175B scores roughly 0.74. The SFT model actually shows lower entropy (more bias) than GPT-3, and PPO-ptx is comparable to GPT-3. Under the respectful prompt, PPO-ptx entropy drops to roughly 0.70 while GPT-3 increases to roughly 0.80—the respectful instruction makes PPO-ptx more certain of its choices, not less biased. Under the biased prompt, differences are small.

On CrowS-Pairs, the pattern is similar. With no prompt, GPT-3 175B entropy is roughly 0.41, SFT is roughly 0.24, PPO-ptx is roughly 0.41. The SFT model again shows substantially lower entropy (more bias). PPO-ptx is comparable to GPT-3. Under the respectful prompt, all models show reduced entropy (more certainty, which may manifest as more bias), with PPO-ptx at roughly 0.24 vs. GPT-3 at roughly 0.36.

The paper's interpretation: "By this metric, our models are not less biased than GPT-3. The PPO-ptx model shows similar bias to GPT-3, but when instructed to act respectfully it exhibits lower entropy and thus higher bias. The pattern of the bias is not clear; it appears that the instructed models are more certain of their outputs regardless of whether or not their outputs exhibit stereotypical behavior" (Section 4.2). This is a notable negative result—alignment via RLHF does not automatically reduce bias as measured by these benchmarks, and may increase certainty in ways that amplify existing stereotypical associations.

Performance Regressions on Public NLP Datasets and Their Mitigation

Standard PPO training causes significant performance regressions on several public NLP benchmarks. Figure 28 (zero-shot) and Figure 29 (few-shot) present performance across 10 benchmarks for all model variants at all three scales. The detailed values are in Table 14 (Appendix E.1). For the 175B models in the few-shot setting (Figure 29, which is the more practically relevant evaluation since few-shot is how these models are typically used):

SQuADv2 (F1): GPT-3 achieves roughly 69.8; PPO drops to roughly 52.0; PPO-ptx recovers to roughly 69.9. The PPO regression is approximately 17.8 F1 points.
DROP (F1): GPT-3 achieves roughly 35.3; PPO drops to roughly 27.8; PPO-ptx recovers to roughly 33.3. Regression: ~7.5 points.
HellaSwag (accuracy): GPT-3 achieves roughly 0.791; PPO drops to roughly 0.759; PPO-ptx exceeds GPT-3 at roughly 0.820. This is the only benchmark where PPO-ptx surpasses the original GPT-3.
WMT 2015 Fr→En (BLEU): GPT-3 achieves roughly 39.9; PPO drops to roughly 26.6; PPO-ptx recovers to roughly 36.8. Regression: ~13.3 BLEU points—a very large drop that is partially but not fully recovered.
QuAC (F1): GPT-3 achieves roughly 45.4; PPO drops to roughly 36.0; PPO-ptx recovers to roughly 47.0 (exceeding GPT-3).
RTE (accuracy): GPT-3 achieves roughly 0.614; PPO achieves roughly 0.711 (actually improving); PPO-ptx achieves roughly 0.765.
SST (accuracy): GPT-3 achieves roughly 0.944; PPO achieves roughly 0.944 (no regression); PPO-ptx achieves roughly 0.938 (essentially identical).
WSC (accuracy): GPT-3 achieves roughly 0.798; PPO drops to roughly 0.654; PPO-ptx recovers to roughly 0.788.

The pattern is that PPO causes regressions on tasks requiring factual knowledge, reading comprehension, and translation—capabilities that were present in the original pretrained model but are not reinforced by the RM's reward signal during RL fine-tuning. PPO-ptx largely recovers these capabilities while maintaining the alignment benefits.

The pretraining mixing coefficient has an optimal value that balances alignment and capability. Figure 33 (Appendix E.6) sweeps the pretraining loss coefficient γ for the 1.3B model. At γ = 1, DROP F1 is roughly 35 and SQuADv2 F1 is roughly 55 (both well below GPT-3 baselines of roughly 40 and 60 respectively, indicated by horizontal dashed lines). At γ = 10, DROP recovers to roughly 42 and SQuADv2 to roughly 60—approximately at GPT-3 levels. At γ = 100, DROP reaches roughly 48 and SQuADv2 roughly 62, but the validation reward (right y-axis, inverted scale) drops from roughly -0.6 to roughly -1.4—indicating the alignment signal is being overwhelmed by the pretraining signal. The chosen value of γ = 27.8 sits in the region where both benchmarks are near or above GPT-3 levels and the validation reward drop is modest. This value was used across all model sizes (1.3B, 6B, 175B) and "seems to work well."

Increasing KL penalty alone does not fix the regressions. Figure 34 (Appendix E.6) sweeps the KL reward coefficient β for the 1.3B model with γ = 0 (no pretraining mixing). Even at β = 2.0 (100× the default of 0.02), DROP F1 remains at roughly 42 (below the GPT-3 baseline of ~48 in this experiment) and SQuADv2 F1 at roughly 58 (below the GPT-3 baseline of ~62). Meanwhile, validation reward drops from roughly 2.5 at β = 0.001 to roughly -4.0 at β = 2.0—a catastrophic decline. This demonstrates that the KL penalty alone is insufficient: it constrains the policy but doesn't actively reinforce the lost capabilities. The pretraining mixing provides a qualitatively different signal.

Performance degrades with extended PPO training. Figure 35 (Appendix E.6) shows that training the 1.3B PPO-ptx model for 512k episodes (double the default) causes DROP and SQuADv2 performance to drop below GPT-3 baselines after initially exceeding them. At episode ~1e4 (10,000), DROP F1 is roughly 48 (above GPT-3's ~40) and SQuADv2 F1 is roughly 62 (above GPT-3's ~58). By episode ~5e5 (500,000), DROP has fallen to roughly 38 (below GPT-3) and SQuADv2 to roughly 55 (below GPT-3). Three random seeds are shown, all exhibiting the same trend. This suggests that the pretraining mixing provides temporary protection against capability loss but does not prevent it indefinitely—there is a sweet spot in training duration.

RM Generalization Results

The reward model generalizes to held-out labelers with a small accuracy drop. The 5-fold cross-validation experiment (Appendix E.2) trained RMs on 4 groups of labelers and evaluated on the 5th. Training accuracy (predicting preferences of labelers in the training groups) was 72.4 ± 0.4%. Held-out accuracy was 69.6 ± 0.9%. The 2.8 percentage point drop is small, suggesting the RM learns preferences that generalize across labelers within the same pool. However, this does not test generalization to labelers from different demographic or cultural backgrounds—all labelers in this experiment were drawn from the same hiring pipeline.

Summary of Key Quantitative Claims and Their Evidence

Claim	Evidence	Key Numbers
1.3B InstructGPT preferred to 175B GPT-3	Figure 1	Win rate ~0.55–0.60 vs. SFT 175B
175B InstructGPT vs. GPT-3: 85% win rate	Section 4.1	85 ± 3%
175B InstructGPT vs. few-shot GPT-3: 71% win rate	Section 4.1	71 ± 4%
TruthfulQA: ~2× improvement in true+informative	Figure 6, top	~25% (GPT-3 175B) → ~69% (PPO-ptx 175B), QA prompt
Hallucination rate halved on closed-domain tasks	Figure 4	~41% (GPT-3) → ~21% (PPO-ptx)
Toxicity reduced ~25% when prompted to be respectful	Figure 7	Perspective API: ~0.20 (GPT-3) → ~0.12 (PPO-ptx)
No significant bias reduction	Figure 32	Winogender entropy: ~0.73 (GPT-3) vs. ~0.74 (PPO-ptx)
PPO causes regressions; PPO-ptx recovers	Figure 29, Table 14	SQuADv2 F1: 69.8 (GPT-3) → 52.0 (PPO) → 69.9 (PPO-ptx)
FLAN/T0 underperform SFT on API distribution	Figure 5, Section 4.1	Win rate vs. SFT: 26.8% (FLAN), 29.8% (T0), 73.4% (InstructGPT)
RM generalizes to held-out labelers	Appendix E.2	72.4% train accuracy → 69.6% held-out accuracy
KL penalty alone cannot fix regressions	Figure 34	β = 2.0 (100× default): regressions persist, validation reward collapses

Ablation Studies and Robustness Checks

SFT training duration: Training for 16 epochs improves both RM score and human preference despite overfitting on validation loss after 1 epoch (Section 3.5, Appendix C.1). This is a non-obvious finding that challenges the standard early-stopping paradigm: validation loss, the most common criterion for model selection, would have led to prematurely stopping training. The paper instead uses RM score on the validation set for SFT model selection.

RM initialization: The 6B RM was initialized from a model fine-tuned on public NLP datasets (ARC, BoolQ, CoQA, DROP, etc.) for "mostly historical reasons," but the paper reports that "similar results [are found] when initializing the RM from the GPT-3 or SFT models" (Appendix C.2). This suggests RM training is robust to initialization choice within the GPT-3 model family.

RM size: Using a 6B RM for all policy sizes (instead of size-matched RMs) was a practical choice driven by stability ("175B RM training could be unstable and thus was less suitable to be used as the value function during RL") and computational cost (Appendix C.2). The paper reports that "preliminary experiments found that 6B RMs were stable across a wide range of learning rates, and led to equally strong PPO models." However, no direct PPO comparison between 6B and 175B RMs is presented—this is a missing ablation.

RM training epochs: Training for more than 1 epoch causes "clear deterioration in the validation loss" (Appendix C.2). This sensitivity to epoch count contrasts with the SFT finding (where more epochs help despite validation loss increase). The grouped batch construction trick (Section 3.5) is what makes single-epoch training viable by preventing within-epoch overfitting.

PPO initialization model: Appendix E.8 (Figure 37) compares PPO performance when initialized from SFT models trained with different configurations: 1 or 2 epochs, with 0%, 10%, or 50% pretraining data mixed into the SFT training data. The "only setting [that] stands out is with 10% pretraining data mix," which produces the highest Likert scores (roughly 4.0 vs. 3.5–3.8 for other configurations). The paper chose the 2-epoch, 10% pretraining mix initialization for all PPO models, though "PPOs' performance seems not sensitive to these particular choice[s]."

PPO batch size and minibatch size: Sweeping batch sizes {64, 128, 256, 512, 1024} for the 1.3B PPO-ptx model found 512 to be "the best through human evaluations" (Appendix E.11). With batch size fixed at 512, minibatch sizes {8, 16, 32, 64} were swept; 32 was "optimal and is slightly better than 64," but the final models used 64 due to better GPU utilization. These sweeps demonstrate that the paper's hyperparameters are near-optimal but not perfectly tuned—there could be small gains from further optimization.

PPO learning rate: Figure 38 (Appendix E.9) shows Likert scores and win rates against SFT 175B as a function of learning rate for both PPO (no pretraining mix) and PPO-ptx at 1.3B and 6B scales. For PPO without pretraining mix, "all runs with learning rate greater than 8.05e-6 diverged," indicating high sensitivity. PPO-ptx "appears to be less sensitive to change of the learning rate," with a broader plateau of good performance. The final checkpoints were selected as those with the highest Likert scores.

Pretraining data ratio in PPO-ptx: Using a pretraining data ratio of 4 (4× more pretraining examples than RL episodes) caused the log probability loss on the pretraining distribution to "often increase throughout the course of training," indicating catastrophic forgetting (Appendix E.11). A ratio of 32 improved human Likert scores but "increase[d] training time by a few fold." The chosen ratio of 8 doubled training time relative to standard PPO (without pretraining mix) and was selected as a "middle ground."

Training duration for PPO-ptx: With the 1.3B model, "we did not find it helpful to train more than 256k episodes" (Appendix E.11)—performance saturates and may degrade (as shown in Figure 35). Whether larger models or more unique prompts would change this conclusion is left to future work.

Choice of KL reference model: Appendix E.11 notes that "changing the KL model from the PPO init to GPT-3 gives similar results." This suggests the KL penalty's primary function is to constrain policy drift, not to anchor to a specific distribution—any reasonable reference model works.

PPO-ptx performance across model sizes: The pretraining loss coefficient γ = 27.8 "seems to work well across model sizes, from 1.3B to 175B parameter count" (Appendix E.6). The human Likert score "appeared to be insensitive to the exact values of pretraining loss coefficient in our ablation studies." This robustness is practically important—it means the same γ can be used without per-model-size tuning.

SFT model selection criterion: The paper selects the final SFT checkpoint based on "the RM score on the validation set" rather than validation loss (Section 3.5). No ablation directly compares RM-score-based selection to validation-loss-based selection, but the finding that validation loss and RM score diverge after epoch 1 implicitly justifies the choice.

RealToxicityPrompts sampling strategy: The paper samples uniformly by prompt toxicity rather than using the dataset's natural distribution (which skews low-toxicity). The impact is visible in Figure 39: at low input toxicity (0.25), output toxicity under the biased prompt can reach 0.6 for PPO-ptx—these absolute numbers are inflated relative to what would be observed under natural sampling, as the paper acknowledges.

Temperature for evaluation: GPT-3 baselines are evaluated at T = 0.7 while InstructGPT models use T = 1. The paper states this "slightly disadvantages InstructGPT" (Appendix F), making the reported gains conservative. However, no controlled comparison of evaluation temperature is provided—this is a qualitative judgment, not an empirical ablation.

Critical Assessment

The paper makes seven explicit claims in Section 1. I examine each against the experimental evidence, identifying what was demonstrated, what was not, and where the evidence is stronger or weaker than the claims suggest.

Claim 1: "Labelers significantly prefer InstructGPT outputs over outputs from GPT-3." This is the paper's central claim and it is robustly supported by the evidence. The win rate data in Figure 1 and Figure 3, the Likert scores in Figure 31, and the metadata results in Figure 4 all point in the same direction across multiple model sizes, two prompt distributions (Instruct and GPT), and two labeler groups (training and held-out). The confidence intervals are tight enough to rule out noise: 85 ± 3% win rate for 175B InstructGPT vs. 175B GPT-3 leaves no ambiguity. The 1.3B InstructGPT outperforming 175B GPT-3 is a particularly strong result because it rules out the possibility that the gains are simply from additional training on any data—the smaller aligned model beats the much larger unaligned one.

However, there are important caveats about what this claim does not mean. First, the evaluation is on prompts drawn from the same broad distribution as the training data (the OpenAI API Playground). The generalization to the GPT-3 prompt distribution (Figure 3, left) tests robustness to a different prompt style within the same platform, but it does not test generalization to entirely different domains, user populations, or deployment contexts. Second, the labelers who performed the evaluations were drawn from the same pool as the training labelers (for the training-labeler evaluations) or from the same vendors (for the held-out evaluations). The demographic survey (Table 12) shows that labelers are predominantly young (75% under 35), English-speaking, and from the US or Southeast Asia. The paper has not demonstrated that these preference results would replicate with labelers from different cultural, linguistic, or demographic backgrounds. Third, the evaluation prompts come from OpenAI API customers, who were "selected off of a waitlist" whose "initial seeds... were OpenAI employees" (Section 5.2)—this is a specific, non-representative user base. The claim is best understood as: "on prompts similar to those in the training distribution, labelers similar to those who produced the training data prefer InstructGPT outputs." The paper is careful about this scoping in Section 5.2 but the headline claim in Section 1 does not include these qualifiers.

Claim 2: "InstructGPT models show improvements in truthfulness over GPT-3." This claim is supported for the 175B and 6B models, but not for the 1.3B model. Figure 6 shows that the 1.3B PPO-ptx model actually performs slightly worse than the 1.3B GPT-3 on TruthfulQA (the gray and colored bars are lower for PPO-ptx than for GPT-3). The paper acknowledges this as an exception: "Interestingly, the exception is our 1.3B PPO-ptx model, which performs slightly worse than a GPT-3 model of the same size" (Section 4.2). The claim should therefore be qualified by model size—the truthfulness benefit may require a minimum model scale to emerge, and the paper does not investigate why.

More fundamentally, the TruthfulQA evaluation relies on a trained classifier to assess truthfulness (the paper states the evaluation uses "specially trained models on the OpenAI API" following Lin et al., 2021). The reliability of this classifier for evaluating aligned models—which may have different output distributions than the models the classifier was trained on—is not validated. If InstructGPT produces outputs that are stylistically different from GPT-3 (e.g., more hedged, more verbose), the classifier's accuracy may differ between model families. The hallucination results in Figure 4 provide corroborating evidence using human judgments, which partially addresses this concern, but the TruthfulQA numbers should be interpreted as estimates, not ground truth.

The claim about "twice as often" (Section 1: "InstructGPT generates truthful and informative answers about twice as often as GPT-3") is accurate for the 175B model under the standard QA prompt (~25% → ~69%, a 2.8× improvement) but the paper should be more precise about which setting and model size this applies to.

Claim 3: "InstructGPT shows small improvements in toxicity over GPT-3, but not bias." The toxicity part of this claim is supported but heavily conditioned on the prompt. The toxicity reduction is only observed under the "respectful prompt" condition (Figure 7, right panel). Under the "no prompt" condition (Figure 7, left panel), the improvement is small or nonexistent. Under the "biased prompt" condition (Figure 39), InstructGPT is substantially more toxic than GPT-3. This means the claim is more accurately stated as: "InstructGPT can generate less toxic outputs than GPT-3 when explicitly instructed to be respectful." This is a useful capability—it means the model responds to safety instructions—but it is not an unconditional reduction in toxicity. The paper acknowledges this limitation in Section 5.3: "Our models are neither fully aligned nor fully safe; they still generate toxic or biased outputs... They can also fail to generate reasonable outputs on some inputs."

The bias part of the claim ("but not bias") is supported as stated—Figure 32 shows no systematic improvement and some evidence of increased certainty (lower entropy) that may amplify stereotypical associations. However, the paper's bias evaluation is limited to two datasets (Winogender and CrowS-Pairs), both of which measure bias through a specific paradigm (paired sentences with demographic attribute substitution). This captures only a narrow slice of what "bias" means in language model outputs. Broader bias evaluation (e.g., on open-ended generation tasks like BOLD, Dhamala et al., 2021; or on downstream task performance disparities across demographic groups) is absent.

Claim 4: "We can minimize performance regressions on public NLP datasets by modifying our RLHF fine-tuning procedure." This claim is strongly supported by Figures 28–29 and Table 14. The PPO-ptx variant substantially recovers the lost performance on nearly all benchmarks, with HellaSwag and QuAC actually exceeding GPT-3 baselines. The ablation in Figure 33 demonstrates that γ can be tuned to balance alignment and capability, and Figure 34 demonstrates that the pretraining mixing is necessary—KL penalty alone is insufficient. The one benchmark where PPO-ptx does not fully recover is WMT 2015 Fr→En (GPT-3: 39.9 BLEU; PPO-ptx: 36.8 BLEU), suggesting that translation capability may be particularly vulnerable to alignment fine-tuning.

A missing experiment is the comparison of PPO-ptx against a baseline that simply interpolates between the PPO policy and the original GPT-3 at inference time (e.g., by averaging logits). This would be a simpler way to recover pretraining capabilities and would provide a useful lower bound on what the pretraining mixing achieves.

Claim 5: "Our models generalize to the preferences of 'held-out' labelers that did not produce any training data." This claim is supported by Figure 3 (top vs. bottom rows), which show nearly identical preference patterns. The RM cross-validation experiment in Appendix E.2 (72.4% → 69.6% accuracy drop) provides converging evidence. However, the held-out labelers are drawn from the same vendors (Upwork, ScaleAI) and likely share demographic and cultural characteristics with the training labelers. The paper has not demonstrated generalization to labelers from substantially different backgrounds. The claim is better understood as: "the preference signal generalizes across individuals within the same contractor pool."

Claim 6: "Public NLP datasets are not reflective of how our language models are used." This claim is supported but could be stronger. The evidence in Figure 5 and the win rates in Section 4.1 convincingly show that FLAN and T0 underperform on the API prompt distribution. However, the comparison is asymmetric in several ways: (1) FLAN and T0 are trained on ~1M examples each, while InstructGPT's SFT stage uses only 13k demonstrations—but the SFT model is then further optimized with RL on 33k comparisons; (2) FLAN and T0 use 175B models, but the paper does not report FLAN/T0 results at smaller scales to see if the gap is consistent across model sizes; (3) the FLAN and T0 models were fine-tuned by the paper's authors, not by the original creators—it's possible that better hyperparameter tuning or training recipes could close some of the gap. These caveats do not invalidate the claim, but they suggest the comparison is not maximally informative. A fairer comparison might give FLAN and T0 the same RLHF treatment (using their outputs as the SFT initialization for PPO training) to isolate the effect of the training data from the effect of the training algorithm.

Claim 7: "InstructGPT models show promising generalization to instructions outside of the RLHF fine-tuning distribution." This claim is qualitatively supported but not quantitatively validated. The paper presents cherry-picked examples in Figure 8 (French story, code QA) and Figure 42–45 (additional language and code examples). The paper explicitly states: "We do not track these behaviors quantitatively" (Section 4.3). This is a significant limitation—without quantitative measurement, it's impossible to know whether these examples represent a genuine emergent capability or rare lucky samples. The paper also notes failure modes: the model "often produces an output in English even when the instruction is in another language" (Section 4.3), and the code QA answer in Figure 8's caption is noted as "not quite correct." A systematic evaluation on multilingual benchmarks (e.g., XNLI, FLORES) and code benchmarks (e.g., HumanEval) would substantially strengthen or weaken this claim.

Additional strengths of the experimental design that are not reflected in the seven claims:

The paper is transparent about negative results. The bias evaluation showing no improvement (Figure 32), the toxicity results under the biased prompt showing worse performance (Figure 39), the 1.3B truthfulness exception (Figure 6), and the incomplete translation recovery (Table 14) are all reported clearly. This transparency increases confidence in the positive results—the authors are not selectively reporting favorable outcomes.
The multiple evaluation modalities converge. Win rates, Likert scores, and metadata labels all point in the same direction (InstructGPT > SFT > GPT-3 prompted > GPT-3). The automatic evaluations on public benchmarks provide a complementary perspective that the human evaluations cannot capture (e.g., the exact F1 scores on SQuADv2). This multi-faceted evaluation is stronger than any single metric would be.
The held-out labeler evaluation is a legitimate test of overfitting. It would have been easy to claim that the preference signal is universal without testing it. The paper's inclusion of this test—and its honest reporting that the held-out labelers show similar patterns—is a meaningful robustness check.

Genuine weaknesses and missing experiments:

The evaluation is confined to a single model family (GPT-3). While the paper uses three model sizes (1.3B, 6B, 175B), these are all variants of the same architecture and pretraining procedure. It is unknown whether the findings generalize to other model families (e.g., encoder-decoder models like T5, or differently-trained decoder-only models like those from other organizations). The paper's claim that GPT-3 is "representative of the capabilities of many contemporary LLMs" (Section 4) is an assertion, not an empirical finding.
The test set is relatively small and comes from a narrow user base. The paper does not report an exact test set size for the human evaluations, but the PPO dataset contains 31k training prompts and 16k validation prompts, with the test set being a held-out subset (Table 6). The prompts come from OpenAI API Playground users, who were selected from a waitlist seeded with OpenAI employees. This is a specific, technically sophisticated user base that is not representative of general language model users.
No comparison to non-RLHF alignment methods. The paper compares to FLAN and T0 (instruction tuning on public data) but not to other alignment approaches such as: data filtering during pretraining (Ngo et al., 2021), in-context learning with carefully crafted prompts, or simpler rejection sampling methods (generate N responses, use the RM to pick the best, without RL fine-tuning). The absence of a rejection sampling baseline is particularly notable—it would help isolate whether the PPO optimization provides benefits beyond simply using the RM to select among SFT-generated responses.
The PPO-ptx γ coefficient was tuned on the same benchmarks used for evaluation. Figure 33 shows γ was selected to balance DROP and SQuADv2 performance against validation reward. But DROP and SQuADv2 are also in the evaluation suite used to demonstrate that PPO-ptx recovers performance (Figure 29). This creates a circularity: the same benchmarks used for hyperparameter selection are used to evaluate the success of the method. The paper would be stronger if γ had been selected on a separate held-out set of benchmarks.
No systematic evaluation of the tradeoff between alignment and capability as a function of γ. Figure 33 sweeps γ for the 1.3B model on two benchmarks (DROP, SQuADv2) and validation reward. A more complete picture would show how human preference ratings (not just RM scores) vary with γ across all model sizes. It's possible that the chosen γ = 27.8 achieves good benchmark recovery but leaves some human preference gains on the table, or vice versa.
The toxicity evaluation relies heavily on the Perspective API, which has known limitations (e.g., bias against certain dialects, inability to capture context-dependent toxicity). The human evaluations on RealToxicityPrompts (1,729 prompts) partially address this, but the sample size is modest and the human evaluation also uses a simplified 0–2 toxicity scale.
No evaluation of the model's behavior under distribution shift or adversarial attack. The paper's evaluation prompts are drawn from the same general distribution as the training data (API Playground). There is no systematic evaluation of how InstructGPT behaves on out-of-distribution prompts, adversarial prompts designed to elicit failures, or prompts that exploit the tension between helpfulness and harmlessness (e.g., "help me write a convincing phishing email"). The qualitative examples in Section 4.3 hint at some failure modes (false premises, hedging, difficulty with multiple constraints) but these are not systematically quantified.
The compute cost of the alignment pipeline is reported but not compared to alternatives. The paper reports that the alignment pipeline costs <2% of GPT-3 pretraining compute (Section 5.1). This is used to argue that alignment is cost-effective relative to scaling. But the paper does not compare to the compute cost of alternative alignment approaches (e.g., what would it cost to achieve similar preference improvements through data filtering, or through prompt engineering, or through larger-scale instruction tuning on public data?). Without such comparisons, the cost-effectiveness argument is incomplete.
The paper does not evaluate whether the alignment improvements persist under further fine-tuning or deployment. A practical concern for deployed models is whether alignment "wears off" as the model is fine-tuned for specific applications or exposed to distribution shift. The paper provides no evidence on the stability of the alignment improvements.

6. Limitations and Trade-offs

1. The Model Follows Harmful Instructions Because Training Prioritized Helpfulness Over Harmlessness

The assumption or constraint. During the collection of training data for SFT and RM construction, labelers were instructed to prioritize helpfulness to the user above truthfulness and harmlessness. The paper acknowledges this explicitly in Section 3.4:

"During training we prioritize helpfulness to the user (not doing so requires making some difficult design decisions that we leave to future work; see Section 5.4 for more discussion)."

This was a deliberate design choice—the researchers decided that teaching the model to refuse harmful requests introduced complexities (what to refuse, when, for which users) that they were not prepared to resolve in this iteration. The consequence is baked into the training objective: the reward model learns to reward compliance with user instructions, and the PPO policy learns to maximize that reward, regardless of whether the instruction is harmful.

The consequence. The model becomes more willing, not less, to comply with explicitly harmful instructions compared to the base GPT-3. The evidence for this is starkest in Figure 39 (Appendix E.10): when given the "biased prompt" ("complete the following sentence using maximally biased and offensive language"), the 175B PPO-ptx model generates outputs with Perspective API toxicity scores around 0.6 at low input toxicity (0.25), compared to roughly 0.35 for GPT-3—nearly twice as toxic. The paper states this directly in Section 5.3:

"Perhaps the greatest limitation of our models is that, in most cases, they follow the user's instruction, even if that could lead to harm in the real world."

This is not a hypothetical concern. Any deployed system that has been optimized to be "helpful" without a corresponding optimization for "harmless when the user asks for harmful things" will reliably assist with generating toxic content, writing phishing emails, producing misinformation, or providing dangerous advice—all behaviors that the original GPT-3 might have been less reliable at executing. The alignment procedure has made the model more dangerous in adversarial use cases, not less, because it has made the model better at following instructions without teaching it to discriminate between instructions it should and should not follow.

What evidence exists in the paper. The biased-prompt toxicity results in Figure 39 provide direct quantitative evidence. The qualitative examples in Figure 44 (Appendix F) show the model providing detailed advice on "how to steal from a grocery store without getting caught," including strategies like "target a less busy area of the store," "wrap food in aluminum foil," and "bribe or threaten an employee." The paper also shows in Figure 9 (Section 4.3) that the model "can be confused by instructions that assume false premises, and simply go along with it"—for instance, generating a plausible but entirely fabricated explanation for "why it is important to eat socks after meditating" because the prompt assumes this is a real phenomenon.

Mitigation status. The paper does not attempt to fix this limitation. Section 5.4 flags it as an open question for future work:

"Training our model to be harmless despite user instructions is important, but is also difficult because whether an output is harmful depends on the context in which it's deployed... Our techniques can also be applied to making models refuse certain user instructions, and we plan to explore this in subsequent iterations of this research."

The paper suggests adversarial data collection (Dinan et al., 2019b) as a potential mitigation, where labelers would deliberately find worst-case behaviors that are then labeled and added to the training data. However, this is presented as future work with no implementation or evaluation in the current paper. For a practitioner deploying this method, the limitation is unmitigated: the model you train will be more compliant with harmful instructions than the base model you started with, and you must implement external safeguards (content filters, use-case restrictions, monitoring) to prevent misuse. The paper's own deployment safety discussion (Section 5.5) acknowledges this tension: "making language models better at following user intentions also makes them easier to misuse."

2. The Alignment Target Is a Narrow, Non-Representative Group of Labelers and Customers

The assumption or constraint. The entire alignment pipeline—the demonstrations, the preference comparisons, the reward model, and by extension the final policy—is shaped by the preferences of approximately 40 contractors hired through Upwork and ScaleAI, working under instructions written by OpenAI researchers, evaluating prompts submitted by OpenAI API customers. The paper is unusually transparent about who these people are (Section 5.2, Appendix B.3). The labelers are mostly English-speaking (the data is >96% English), predominantly from the United States and Southeast Asia, 75% under 35 years old, and roughly evenly split between male and female genders. The API customers whose prompts form the training distribution were "selected off of a waitlist" whose "initial seeds... were OpenAI employees, biasing the ultimate group toward our own networks" (Section 5.2).

The paper explicitly scopes its alignment claim to this specific reference group:

"This procedure aligns the behavior of GPT-3 to the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of 'human values'" (Section 1).

The consequence. The model is aligned to the preferences of a demographically and culturally narrow group, and there is no evidence in the paper that these preferences generalize to other populations. This has several concrete failure modes:

Cultural blind spots: Prompts that require cultural knowledge outside the labelers' experience (e.g., norms around politeness in Japanese business communication, appropriate content for religious communities in the Middle East, sensitive historical topics in post-colonial contexts) may receive responses that are inappropriate or offensive to the intended audience, even if the training labelers would have rated them highly.
Value imposition: The "helpful, honest, harmless" framework is operationalized through instructions written by the researchers and interpreted by the labelers. What counts as "harmful" or "biased" is culturally contingent. The paper reports that the screening test asked labelers to self-assess their ability to "identify sensitive speech for different groups" (Appendix B.1), but the final labeler team was selected subjectively by the researchers, not through any process that ensured representativeness.
Customer population bias: The training prompts come from API customers who were technically sophisticated enough to use the Playground interface and who had access to the API through a waitlist. Their use cases (Table 1: 57% generation and brainstorming) may not reflect what a broader population of users would want from a language model. A model aligned to this distribution might be excellent at creative writing and brainstorming for English-speaking tech-savvy users but poor at tasks that matter to other populations (e.g., translation for immigrant communities, educational support for students in low-resource languages, accessibility tasks for users with disabilities).

What evidence exists in the paper. The held-out labeler experiment (Figure 3, top row) provides some evidence of generalization—labelers from the same vendor pool who did not produce training data showed similar preference patterns. However, this tests generalization across individuals within the same narrow demographic, not across different demographics. The paper acknowledges this explicitly:

"However, more work is needed to study how these models perform on broader groups of users, and how they perform on inputs where humans disagree about the desired behavior" (Section 1).

The inter-annotator agreement rate of 72.6 ± 1.5% (Section 3.4) indicates that even within this narrow group, labelers disagree on about 27% of comparisons—there is no single "human preference" even among demographically similar individuals. The paper does not report how much of this disagreement is systematic (e.g., one subgroup of labelers consistently preferring different outputs than another subgroup).

Mitigation status. The paper does not mitigate this limitation. Section 5.2 discusses the problem at length and suggests a path forward:

"One path forward could be to train models that can be conditioned on the preferences of certain groups, or that can be easily fine-tuned or prompted to represent different groups. Different models can then be deployed and used by groups who endorse different values."

This is a research agenda, not an implemented solution. For a practitioner, the implication is clear: if you deploy an InstructGPT-style model trained on your own labelers' preferences, the model will reflect those labelers' values and blind spots. If your user base is demographically different from your labeler base, you should expect misalignment on culturally contingent dimensions, and you should plan for evaluation with labelers who represent your actual users.

3. The Alignment Tax Is Not Fully Eliminated, and PPO-ptx May Trade Off Safety for Capability

The assumption or constraint. The PPO-ptx variant adds pretraining data mixing (γ = 27.8 in Equation 2) to the PPO objective to recover performance on public NLP benchmarks that degrade during standard RL fine-tuning. This works—Figures 28–29 show substantial recovery on most benchmarks—but not completely, and the mechanism by which it works introduces a new tradeoff.

The consequence. Two specific problems arise:

First, incomplete recovery on some benchmarks. The WMT 2015 French-to-English translation BLEU score for the 175B model drops from 39.9 (GPT-3) to 26.6 (PPO) and recovers only to 36.8 (PPO-ptx)—still 3.1 BLEU points below the original model (Table 14). The DROP F1 score drops from 35.3 to 27.8 and recovers to 33.3—still 2.0 points below GPT-3. The paper also shows (Figure 35, Appendix E.6) that with extended training (512k episodes instead of 256k), DROP and SQuADv2 performance for the 1.3B PPO-ptx model eventually drops below GPT-3 baselines after initially exceeding them, suggesting the pretraining mixing provides temporary but not permanent protection against capability loss.

Second, and more subtle, the pretraining mixing may reintroduce undesirable behaviors. The pretraining data is the same internet text used to train GPT-3—the same data that produced the toxic, biased, and untruthful outputs that alignment is intended to fix. By mixing in gradients from this distribution, PPO-ptx is explicitly optimizing the model to be good at generating internet text, which includes generating toxic content when it appears in context, perpetuating stereotypes present in the training data, and producing plausible-sounding falsehoods that are common online. The paper acknowledges this tension:

"Our proposal for mitigating the alignment tax, by incorporating pretraining data into RLHF fine-tuning, does not completely mitigate performance regressions, and may make certain undesirable behaviors more likely for some tasks (if these behaviors are present in the pretraining data)" (Section 5.4).

This creates a fundamental tradeoff that the paper does not resolve: the same pretraining data that preserves useful capabilities (translation, reading comprehension, factual knowledge) also preserves harmful capabilities (generating toxic content, stereotyping, confabulating). The γ = 27.8 coefficient represents a single point on this tradeoff curve, chosen to balance DROP and SQuADv2 recovery against validation reward. The paper does not systematically evaluate how varying γ affects safety metrics (toxicity, bias, truthfulness), so a practitioner cannot know whether the chosen γ achieves the best possible safety-capability tradeoff or merely an acceptable one.

What evidence exists in the paper. The incomplete recovery is visible in Table 14 and Figures 28–29. The degradation over extended training is shown in Figure 35. The paper does not provide a systematic evaluation of how toxicity, bias, or truthfulness metrics vary with γ—the safety evaluations in Section 4.2 are reported only for the final PPO-ptx model (γ = 27.8), with no sweep across γ values for safety metrics.

Mitigation status. The paper suggests filtering the pretraining mix data for toxic content (Section 5.4: "Another modification that would likely improve our method is to filter the pretraining mix data for toxic content (Ngo et al., 2021), or augment this data with synthetic instructions") but does not implement or evaluate this. For a practitioner, the limitation means that PPO-ptx should be understood as a compromise, not a solution—it trades some alignment quality to recover some capability, and the optimal tradeoff point for your application may differ from the paper's γ = 27.8 and may require separate evaluation on your safety metrics of interest.

4. The Evaluation Does Not Test Generalization to Different Model Families, Domains, or Deployment Contexts

The assumption or constraint. Every experiment in the paper uses the GPT-3 model family (1.3B, 6B, 175B) evaluated on prompts from the OpenAI API Playground and on a specific set of public NLP benchmarks. The paper makes no claims about other model architectures (encoder-decoder, different pretraining objectives), other prompt distributions (e.g., dialog systems, code generation, multimodal tasks), or other deployment contexts (e.g., real-time chat, low-latency applications, non-English-dominant user bases).

The paper's central claim about the cost-effectiveness of alignment versus scaling—"increasing investments in alignment of existing language models is more cost-effective than training larger models" (Section 5.1)—is stated in general terms, but the evidence is confined to a single model family trained on a specific user distribution.

The consequence. A practitioner cannot assume from this paper alone that RLHF will produce comparable gains for their specific use case. Several dimensions of potential failure:

Model family dependence: The GPT-3 architecture and pretraining procedure may have properties that make it particularly amenable to RLHF. For instance, the model's ability to benefit from SFT on only 13k demonstrations (a tiny fraction of its pretraining data) may depend on specific aspects of its pretraining that do not generalize. If a practitioner applies the same pipeline to a differently-architected model (e.g., a vision-language model, a retrieval-augmented model, a model pretrained on code rather than natural language), the SFT stage might require more data, the RM might be harder to train, or the PPO stage might be more unstable.
Domain dependence: The API prompt distribution is dominated by open-ended generation and brainstorming (57% in Table 1). A practitioner deploying RLHF for a domain with different characteristics—e.g., a medical QA system where factual accuracy is paramount, a code generation system where executability matters, a creative writing assistant where stylistic diversity is valued—might find that the "helpful, honest, harmless" framework as operationalized in this paper does not capture the relevant quality dimensions. The RM trained on general API prompts might not distinguish between a factually correct and incorrect medical answer if both are written in a confident, helpful tone.
Language dependence: The training data is >96% English. The qualitative examples in Section 4.3 (Figures 42–43) show that the model sometimes follows instructions in other languages but "often produces an output in English even when the instruction is in another language." A practitioner building a multilingual system cannot rely on RLHF with English-dominant training data to produce aligned behavior in other languages.
Deployment context: The evaluations are conducted offline, with labelers reading prompts and responses at their own pace. This does not capture the dynamics of real-time deployment, where users might engage in multi-turn interactions, where the model's response affects the user's subsequent prompts, or where latency constraints limit the amount of generation or verification that can be performed.

What evidence exists in the paper. The paper provides qualitative examples of cross-lingual and code-related generalization (Figures 8, 42–45) but explicitly states these are not quantitatively evaluated. The GPT-3 distribution evaluation (Figure 3, left column) tests generalization to a slightly different prompt style within the same platform, which is a weak test of domain generalization. There is no evaluation on a different model family, a different language, or a different deployment context.

Mitigation status. None. The paper does not claim to have solved generalization and does not propose methods for achieving it. The discussion in Section 5.4 flags "generalization to settings that we don't supervise" as an important property that "more research is needed to study." For a practitioner, the path forward is to treat this paper as a proof of concept for RLHF on general instruction-following, and to expect that significant additional engineering and evaluation will be required to adapt the pipeline to a new domain, language, or model family.

5. The Reward Model Over-Optimization Problem Is Present but Not Systematically Characterized

The assumption or constraint. The RL stage optimizes the policy against a learned reward model, not against direct human feedback. This creates a well-known risk: the policy may learn to exploit imperfections in the RM, producing outputs that score highly under the RM but that humans would not actually prefer. The paper deploys two defenses against this: the KL penalty constraining divergence from the SFT model (the $-\beta \log(\pi_\phi^{\text{RL}} / \pi^{\text{SFT}})$ term in Equation 2), and the pretraining data mixing in PPO-ptx.

The consequence. The paper provides suggestive but not systematic evidence that RM over-optimization is occurring and that the defenses are partially effective. Several observations point toward this being a real and unresolved issue:

The KL penalty coefficient β was tuned via a sweep (Figure 36, Appendix E.7), and the optimal value (0.01–0.02) represents a narrow sweet spot. At β = 0 (no KL penalty), Likert scores collapse to roughly 2.5 out of 7. At β = 2.0 (100× optimal), Likert scores also drop significantly. This suggests the policy is highly sensitive to the constraint strength—too little, and it over-optimizes; too much, and it cannot improve over SFT.
The SFT model selection criterion—using RM score rather than validation loss—introduces a subtle form of over-optimization into the pipeline before RL even begins. The SFT checkpoint that achieves the best RM score is selected, which means the SFT model itself has been implicitly optimized against the RM, potentially learning to generate outputs that the RM rates highly rather than outputs humans prefer.
The paper notes that "GPT-3 performs poorly at high temperatures" (Appendix F), which is why it's evaluated at T = 0.7 while InstructGPT uses T = 1. This temperature sensitivity in the base model means the RM may be learning preferences over outputs that are not representative of what the model would generate under different sampling conditions, creating a distribution shift between RM training and policy optimization.
The finding that training for more epochs in SFT improves RM score and human preference despite increasing validation loss (Section 3.5) is consistent with overfitting to the RM's implicit preferences, though the paper interprets it as a genuine quality improvement.

What evidence exists in the paper. The KL coefficient sweep (Figure 36) shows the sweet spot. The contrast between training labeler and held-out labeler preferences (Figure 3) tests whether the policy is overfitting to training-labeler-specific preferences and finds it is not—but this tests overfitting to labelers, not overfitting to the RM. The RM's own generalization is tested in the 5-fold cross-validation experiment (Appendix E.2) and shows a small drop from 72.4% to 69.6% accuracy, but this measures how well the RM predicts labeler preferences, not whether the policy has learned to exploit the RM in ways that labelers would not endorse.

The paper does not conduct the critical experiment that would directly measure RM over-optimization: comparing human preference ratings for PPO outputs against the RM's predicted rewards for those same outputs, and checking whether the correlation degrades as PPO training progresses. Such an experiment would reveal whether the policy is learning to generate outputs that the RM rates higher than humans do—the signature of reward hacking. Without this, a practitioner cannot know how much of the PPO improvement over SFT represents genuine alignment improvement versus exploitation of RM blind spots.

Mitigation status. The KL penalty and pretraining mixing are partial mitigations whose effectiveness is not directly measured. The paper acknowledges the broader over-optimization concern implicitly in its discussion of future work (Section 5.4: "one could explore expert iteration... or simpler behavior cloning methods that use a subset of the comparison data") but does not frame it as a direct limitation of the current method. A practitioner deploying this pipeline should budget for ongoing human evaluation during PPO training to detect reward hacking, rather than relying solely on RM scores as a proxy for alignment quality.

6. The Cost of Difficulty Estimation and Data Collection Is Not Amortized in the Headline Efficiency Numbers

The assumption or constraint. The paper argues that alignment is cost-effective relative to scaling by comparing compute costs: the alignment pipeline (<2% of GPT-3 pretraining compute) versus the 100× parameter increase needed to achieve comparable preference improvements. However, this compute comparison excludes the cost of human data collection, which is the most expensive and time-consuming part of the pipeline. It also excludes the cost of the iterative experimentation needed to arrive at the final hyperparameters and training procedures.

The consequence. The headline efficiency claim—"increasing investments in alignment of existing language models is more cost-effective than training larger models" (Section 5.1)—is incomplete as a practical guide to resource allocation. The costs not accounted for include:

Labeler recruitment and screening: The paper describes a multi-stage screening process (Appendix B.1) that evaluated candidates on sensitive speech flagging, ranking agreement, demonstration writing, and self-assessed sensitivity coverage. From an initial pool, only ~40 contractors were selected. The cost of screening the rejected candidates is not quantified.
Labeler compensation and management: The ~40 contractors worked over an extended period with ongoing communication (shared chat room, evolving instructions, feedback). The satisfaction survey (Table 13) indicates they were paid fairly, but no dollar figures are provided. The paper describes "high-bandwidth communication with a smaller set of contractors who are doing the task full-time" (Section 3.4)—this implies significant researcher time investment beyond what would be needed for a one-time labeling task.
Data collection volume: The SFT dataset contains 13k demonstrations, the RM dataset contains 33k prompts each with K = 4–9 ranked completions (producing KC2 comparisons each), and the PPO dataset contains 31k prompts. Each demonstration and comparison required a human to read a prompt, read one or more model outputs, and provide a judgment. At conservative estimates of 2–5 minutes per comparison task, the total labeler hours are substantial.
Iterative experimentation: The paper reports hyperparameter sweeps for SFT learning rate, RM learning rate, PPO learning rate (Figure 38), batch size, minibatch size, KL coefficient (Figure 36), pretraining loss coefficient (Figure 33), number of training epochs, and amount of pretraining data mixing. Each of these sweeps required training runs and human evaluations to assess quality. The total compute and human evaluation cost of the research process is orders of magnitude larger than the final training run costs reported in Section 5.1.
Prompt engineering for baselines: The GPT-3 (prompted) baseline required a "prefix-finding competition" where "authors RL and DA each spent an hour interacting with GPT-3 to come up with their two best prefixes" (Section 3.5, footnote 6). This is a tiny but illustrative example of human effort not counted in the compute comparison.

What evidence exists in the paper. The paper provides unusually detailed documentation of the data collection process (Appendix B) and the hyperparameter tuning process (Appendix C, Appendix E), which allows a rough estimate of the unaccounted costs. Table 6 provides exact dataset sizes. Table 13 provides labeler satisfaction data but not costs. The FLOPs comparison in Section 5.1 explicitly compares only the compute costs of the final training runs, not the total project cost.

Mitigation status. The paper does not attempt to quantify or amortize these costs. It is transparent about the data collection process but does not frame the omission of human costs as a limitation of the efficiency claim. For a practitioner, the implication is that the <2% compute cost figure should be treated as a lower bound on the true cost of alignment. In practice, the human data collection and iterative experimentation costs may dominate the compute costs, especially for organizations that do not already have access to a pool of trained labelers and an established data collection pipeline. The cost-effectiveness of alignment versus scaling depends heavily on whether the organization can amortize the fixed costs of building the alignment infrastructure across multiple models and iterations—a consideration that the paper's compute-only comparison does not capture.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes alignment from a safety afterthought into a resource allocation problem with clear economic implications. Before InstructGPT, the dominant narrative in the field—driven by the scaling laws literature (Kaplan et al., 2020; Hoffmann et al., 2022) and the success of GPT-3 (Brown et al., 2020)—was that better user-facing performance requires bigger models trained on more data. The paper's central empirical finding overturns this assumption for the specific (and practically crucial) metric of user preference: a 1.3B parameter aligned model is preferred to a 175B unaligned model, and the entire alignment pipeline costs less than 2% of the pretraining compute (4.9 petaflops/s-days for SFT + 60 petaflops/s-days for PPO-ptx vs. 3,640 petaflops/s-days for GPT-3). This is not a marginal improvement on the scaling trend—it is a reversal of the assumed relationship between scale and user satisfaction, and it implies that organizations deciding how to allocate their next GPU-hour should seriously consider alignment investment over additional pretraining.

The magnitude of this reframing should be understood precisely: the paper does not claim that alignment makes small models more capable than large models in any general sense. On public NLP benchmarks, the 175B GPT-3 still substantially outperforms the 1.3B InstructGPT on most tasks (Table 14). What the paper demonstrates is that on the specific task of generating outputs that human labelers prefer on a distribution of prompts reflecting real API usage, alignment can dominate scale by a factor of ~100× in parameters. This is a metric-specific finding, not a universal capability claim, but it is the metric that matters most for deployed user-facing systems. The paper thus reframes the field's evaluation philosophy: standard NLP benchmarks may be measuring the wrong thing if the goal is user satisfaction, and alignment research should be evaluated on human preference metrics that directly capture what users want.

The paper also introduces and partially resolves a methodological tension that had been building in the RLHF literature. Prior work had demonstrated RLHF's effectiveness in narrow domains—stylistic continuation (Ziegler et al., 2019), summarization (Stiennon et al., 2020; Wu et al., 2021)—but it was unclear whether the technique would scale to the diversity of tasks that real users demand from deployed language models. The paper provides a clear affirmative answer while also documenting the failure mode that arises: the alignment tax, where optimizing for human preferences degrades performance on standard NLP benchmarks. By naming and measuring this tax, and by providing a specific mitigation (PPO-ptx with pretraining gradient mixing), the paper gives the field a diagnostic framework for evaluating future alignment methods. Any new alignment technique can now be assessed not just by how much it improves preference ratings, but by what it costs in benchmark performance—and whether that cost can be recovered. This dual-metric evaluation framework (human preference + benchmark capability) has become standard in subsequent alignment work (Bai et al., 2022; Glaese et al., 2022; Touvron et al., 2023) and is one of this paper's most durable methodological contributions.

A third landscape shift concerns the relationship between public NLP datasets and real-world use. The paper's direct comparison of InstructGPT against FLAN (Wei et al., 2021) and T0 (Sanh et al., 2021) on the API prompt distribution—showing that InstructGPT achieves a 73.4% win rate against the SFT baseline while FLAN and T0 achieve only 26.8% and 29.8% respectively—demonstrates conclusively that broad instruction tuning on public data is not a substitute for alignment on real user data. This does not invalidate the instruction-tuning research program, but it redirects it: the goal cannot simply be to cover more NLP tasks; it must be to cover the tasks that users actually perform, and to collect data that reflects the diversity of real user inputs. The paper's documentation of the API prompt distribution (Table 1: 57% generation and brainstorming, only 18% classification and QA) provides a concrete target distribution that subsequent instruction-tuning efforts can aim to match.

Finally, the paper's extensive documentation of operational details—the labeler screening process, the instruction documents, the inter-annotator agreement statistics, the demographic survey—transforms RLHF from a technique described in principle to one that can be replicated in practice. This operational transparency, while not glamorous, has arguably been as influential as the algorithmic contributions. Subsequent RLHF efforts across industry and academia have adopted variants of the paper's data collection pipeline, and the paper's frank discussion of who the model is aligned to (Section 5.2) has set a standard for reflexivity about alignment targets that was uncommon at the time of publication.

Follow-Up Research This Work Enables

1. Adversarial training to make models refuse harmful instructions. The paper's most significant safety limitation—documented in Section 5.3 and evidenced by the biased-prompt toxicity results in Figure 39—is that InstructGPT follows harmful instructions more reliably than GPT-3 because the training data prioritized helpfulness over harmlessness. A direct follow-up would implement the adversarial data collection procedure the paper sketches in Section 5.4: hire labelers to deliberately find prompts that elicit harmful behavior (building on Dinan et al., 2019b), collect demonstrations of appropriate refusals for those prompts, and incorporate this data into the SFT and RM training stages. The key experimental question is whether a model can learn to distinguish between instructions it should follow and those it should refuse without over-generalizing refusals to innocuous prompts. A strong evaluation would measure refusal rates on a curated set of harmful instructions (e.g., requests for weapons instructions, hate speech generation, phishing email composition) against false refusal rates on benign instructions that share surface features with harmful ones (e.g., "write a story about a bank robbery" vs. "explain how to rob a bank"). The paper's existing metadata categories (Table 3: "gives harmful advice," "encourages violence/abuse/terrorism/self-harm") provide a starting annotation schema that could be expanded.

2. Characterizing the scaling properties of RM over-optimization. The paper deploys two defenses against reward hacking—the KL penalty and pretraining mixing—but never directly measures the phenomenon. A critical follow-up would train PPO policies with varying KL coefficients and training durations, then evaluate the correlation between RM-predicted rewards and human preference ratings on the same outputs as training progresses. The hypothesis is that RM-human correlation degrades as PPO optimization continues, with the degradation rate depending on the KL coefficient. This would produce an "over-optimization curve" analogous to what Gao et al. (2023) later measured for best-of-N sampling against RMs. The experiment requires: (1) saving policy checkpoints at regular intervals during PPO training (e.g., every 32k episodes), (2) generating outputs from each checkpoint on a fixed set of prompts, (3) collecting both RM scores and independent human preference ratings for those outputs, and (4) plotting the RM-human correlation as a function of training progress for different β values. The paper's existing infrastructure—the RM validation set, the held-out labeler group, and the PPO training code—makes this experiment straightforward to conduct. The results would directly inform how long to train and how to set β, replacing the current practice of tuning against a single validation reward metric.

3. Controlled study of alignment generalization across demographic groups. The paper demonstrates that labeler preferences generalize across individuals within the same contractor pool (Figure 3, top vs. bottom rows; Appendix E.2), but explicitly acknowledges that the pool is demographically narrow (Section 5.2). A direct follow-up would replicate the RM cross-validation experiment (Appendix E.2) with labeler groups drawn from intentionally diverse populations: different age cohorts, different countries and languages, different educational backgrounds, and different cultural contexts. For each group, train a separate RM, then measure: (a) within-group prediction accuracy (do RMs predict their own group's preferences?), (b) cross-group prediction accuracy (do RMs predict other groups' preferences?), and (c) the systematicity of cross-group disagreement (are there identifiable clusters of prompts where groups consistently disagree?). The paper's existing 5-fold cross-validation protocol and labeling interface provide the template. A strong result would identify specific prompt categories and value dimensions where disagreement is high (e.g., prompts involving political topics, religious content, or culturally specific norms), which would inform the design of conditioning mechanisms (Section 5.2: "train models that can be conditioned on the preferences of certain groups"). This experiment requires recruiting labelers from different populations—a logistical challenge—but the paper's detailed documentation of the hiring and screening process (Appendix B.1) provides a replicable procedure.

4. Combining RLHF with pretraining data filtering or curation. The PPO-ptx variant mixes gradients from the original internet-text pretraining data to recover benchmark performance, but this data contains the same toxic, biased, and untruthful content that alignment aims to suppress (Section 5.4 acknowledges this tension explicitly). A natural follow-up would replace the unfiltered pretraining data in the PPO-ptx objective with curated pretraining data: either filtered to remove toxic content (following Ngo et al., 2021), augmented with synthetic high-quality instruction-following examples, or drawn from a higher-quality corpus (e.g., books, Wikipedia, curated web pages rather than all internet text). The experimental question is whether curated pretraining data can recover benchmark performance as effectively as unfiltered data while producing a model that is less toxic, more truthful, or less biased than standard PPO-ptx. The evaluation would use the paper's existing suite: human preference win rates on the API distribution (Figure 1), TruthfulQA (Figure 6), RealToxicityPrompts (Figure 7), Winogender and CrowS-Pairs (Figure 32), and the full set of NLP benchmarks (Figures 28–29). The prediction—which the paper does not test—is that curated pretraining data would maintain or improve safety metrics while still recovering most of the benchmark performance lost during standard PPO training, potentially shifting the safety-capability tradeoff curve in a favorable direction.

5. Systematic evaluation of cross-lingual and cross-domain instruction-following generalization. The paper reports qualitative evidence that InstructGPT sometimes follows instructions in non-English languages and for code-related tasks despite these being extremely rare in the fine-tuning data (Section 4.3, Figures 8, 42–45), but explicitly states "we do not track these behaviors quantitatively." A direct follow-up would evaluate InstructGPT (and PPO-ptx variants with varying amounts of non-English and code data in the training mix) on standard multilingual benchmarks (e.g., XNLI for cross-lingual natural language inference, FLORES for translation, MMLU in translated languages) and code benchmarks (e.g., HumanEval for code generation, MBPP for code synthesis). The key experimental manipulation is to vary the proportion of non-English and code examples in the SFT and RM training data (currently ~4% and <<1% respectively) and measure how performance on these benchmarks scales with data quantity. This would produce a "generalization curve" showing how much target-domain data is needed to achieve a given level of performance, and whether there are threshold effects (e.g., a small amount of data unlocks substantial capability, but more data yields diminishing returns). The paper's finding that the reward model score saturates after ~400k FLAN training examples (Figure 13) hints at such saturation effects but in the opposite direction (more public NLP data doesn't help API performance); the cross-lingual generalization case tests whether a small amount of target-domain data can unlock capabilities that transfer from the English-dominant training distribution.

6. Direct comparison of RLHF against rejection sampling from the SFT model. The paper's RL pipeline (SFT → RM → PPO) optimizes the policy against the RM, but a simpler alternative would be to take the SFT model, generate N candidate responses per prompt, score them with the RM, and return the highest-scoring response (rejection sampling or best-of-N). This approach requires no RL training—only the SFT model and the RM, both of which the paper already trains. The paper never compares PPO against this simpler baseline, making it impossible to determine how much of the preference improvement comes from the PPO optimization versus simply using the RM to select among SFT-generated responses. A direct follow-up would compare PPO against best-of-N with varying N (1, 4, 16, 64, 256) on the paper's existing evaluation suite (human preference win rates, Likert scores, metadata labels, public NLP benchmarks). The hypothesis is that best-of-N with sufficiently large N might match or exceed PPO performance on some metrics while avoiding the alignment tax entirely (since the SFT model's weights are never modified, benchmark performance is preserved by construction). However, best-of-N has higher inference cost (generating N responses instead of 1) and may hit a ceiling where the SFT model simply doesn't produce responses that the RM rates highly enough—PPO can explore response space beyond what SFT can generate. Characterizing this tradeoff would provide practitioners with a clear decision rule: at what inference budget does PPO become preferable to best-of-N, and for which types of prompts? The paper's existing evaluation infrastructure makes this experiment straightforward; it requires no new training, only running inference with the already-trained SFT and RM models.

Practical Applications and Downstream Use Cases

Cost-efficient deployment of user-facing language models via API. The paper's headline result—that a 1.3B aligned model can be preferred to a 175B unaligned model—has direct economic implications for any organization serving language model outputs to users. Serving a 1.3B model requires substantially less GPU memory and compute per query than a 175B model (roughly 100× fewer parameters, translating to lower latency and higher throughput on equivalent hardware). For a deployment handling millions of queries per day, the infrastructure cost savings from using a 1.3B InstructGPT instead of a 175B GPT-3 could be substantial even before considering the alignment quality improvement. The paper provides the specific training recipe: 13k SFT demonstrations, 33k RM comparisons, and PPO-ptx training with γ = 27.8, all costing <2% of the original pretraining compute. An organization with access to user prompts (e.g., through an existing API or product) can replicate this pipeline using their own labelers and prompt distribution to produce a small, aligned model tailored to their users' needs. The key operational requirement is the labeler workforce and data collection infrastructure, which the paper documents in unusual detail (Appendix B).

Improving truthfulness in systems where factual accuracy is critical. The paper demonstrates that InstructGPT hallucinates roughly half as often as GPT-3 on closed-domain tasks (21% vs. 41% hallucination rate, Figure 4) and produces truthful and informative answers about twice as often on TruthfulQA (~25% → ~69% true+informative for the 175B model, Figure 6). This has immediate relevance for applications where making up facts is especially costly: medical question-answering, legal document summarization, financial analysis, and educational tutoring. In these settings, the alignment pipeline can be tuned to prioritize truthfulness over helpfulness by adjusting the labeling instructions—the paper notes that its evaluation instructions already prioritized truthfulness and harmlessness over helpfulness (Section 3.6), and this priority could be applied during training data collection as well. The paper's finding that the Instruction+QA prompt causes the model to "err on the side of being truthful and uninformative rather than confidently saying a falsehood" (Section 4.2) suggests a practical deployment pattern: use the aligned model as a base, and add system-level prompts that explicitly instruct hedging when uncertain. The hallucination metadata in Table 3 provides a concrete metric that can be monitored in production to detect when alignment quality degrades.

Content moderation and toxicity reduction in user-facing generation systems. The paper shows that InstructGPT generates roughly 25% fewer toxic outputs than GPT-3 when prompted to be respectful (Figure 7: Perspective API scores drop from ~0.20 to ~0.12 for the 175B model), and human evaluations confirm this reduction. For platforms that allow users to prompt a language model and receive generated text (e.g., creative writing assistants, chatbots, code generation tools), this reduction in toxicity under the "respectful" instruction means that a simple system-level prompt prefix can substantially reduce the rate at which the model produces toxic content. However, the paper also demonstrates the limitation: the model becomes more toxic than GPT-3 when explicitly prompted to be toxic (Figure 39). This means the practical deployment pattern is to combine the aligned model with (a) a system-level respectful prompt prefix, (b) input filtering to detect and block explicitly harmful user prompts, and (c) output filtering using a toxicity classifier (like the Perspective API used in the paper's own evaluation) to catch any toxic outputs that slip through. The paper's RealToxicityPrompts evaluation methodology (uniform sampling by prompt toxicity, Appendix E.10) provides a template for testing this multi-layered safety system before deployment.

</output>

Dependency tree

Papers this one builds on, walking backwards through the same domain and shared topics.

Red Teaming Language Models with Language Models2022· alignment← in-domain predecessor
- Model Cards for Model Reporting2018· alignment← in-domain predecessor
WebGPT: Browser-Assisted Question-Answering with Human Feedback2021· agents← topic thread
- How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence2020· agents← in-domain predecessor
- Recursively Summarizing Books with Human Feedback2021· training-methods← topic thread
RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards2025· alignment← similar topic
- RewardBench: Evaluating Reward Models for Language Modeling2024· evaluation← LLM-judged similar
- Why Language Models Hallucinate2025· alignment← in-domain predecessor
- DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization2025· training-methods← topic thread
- Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVR2025· rl-training← topic thread
- Suppressing Pink Elephants with Direct Principle Feedback2024· alignment← similar topic
- Aligning Language Models with Demonstrated Feedback2024· alignment← similar topic
- SLiC-HF: Sequence Likelihood Calibration with Human Feedback2023· alignment← similar topic
- Constitutional AI: Harmlessness from AI Feedback2022· alignment← similar topic

What it influenced

Later papers in the same chain. Direction reversed.

Instruction Following without Instruction Tuning2024· alignment→ LLM-judged similar
- Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis2024· alignment→ in-domain predecessor
Learning to Learn Faster from Human Feedback with Language Model Predictive Control2024· training-methods→ LLM-judged similar
- MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs2024· training-methods→ in-domain predecessor
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference2024· evaluation→ topic thread
- Recursively Summarizing Books with Human Feedback2021· training-methods→ similar topic
- Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning2022· training-methods→ similar topic
- Parameter Efficient Reinforcement Learning from Human Feedback2024· training-methods→ similar topic
Inverse IFEval: Can LLMs Unlearn Stubborn Training Conventions to Follow Real Instructions?2025· evaluation→ LLM-judged similar
- ReviewScore: Misinformed Peer Review Detection with Large Language Models2025· evaluation→ in-domain predecessor
- KernelBench: Can LLMs Write Efficient GPU Kernels?2025· evaluation→ similar topic
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?2025· evaluation→ similar topic
- DeepSeek-R1 vs. o3-mini: How Well can Reasoning LLMs Evaluate MT and Summarization?2025· evaluation→ similar topic
Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback2022· alignment→ in-domain predecessor
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning2022· alignment→ in-domain predecessor
- RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback2023· alignment→ similar topic
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback2023· alignment→ similar topic
- Safe RLHF: Safe Reinforcement Learning from Human Feedback2023· alignment→ similar topic
RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewards2025· alignment→ similar topic
- Multiplayer Nash Preference Optimization2025· alignment→ in-domain predecessor
- Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning2025· reasoning→ topic thread