ArXiv: 2210.07792

🎯 Pitch

Aligning a story generator to arbitrary human preferences typically collapses under continuous reward signals—but discretizing those signals using prompt-tuned clusters of human critiques makes reinforcement learning suddenly work, producing preferences on par with models 20× larger. A contrastive story-to-critique encoder, even without perfectly organized embedding clusters, can serve as a general-purpose reward that learns robustly when you stop treating preference as a regression problem.

1. Executive Summary

This paper introduces a pipeline for controlled automated story generation that uses a contrastively trained bi-encoder model, CARP, as a reward function to fine-tune a generative language model (GPT-2-750M) via Proximal Policy Optimization (PPO), enabling stories to conform to user-specified natural language preferences without requiring annotated datasets for each desired attribute. To address instability where the generator either collapses or fails to learn from the continuous CARP reward signal, the authors develop CARP CoOp, which combines pseudo-labeling via clustering of CARP's critique embedding space with prompt tuning to discretize preferences into robust class-based reward signals — demonstrated for both general topic preferences (e.g., imagery, fighting) and character moral alignment (good, neutral, evil). A human participant study comparing four model classes shows that the full CARP CoOp fine-tuning pipeline produces story segments preferred over both a prompted GPT-NeoX-20B baseline (a model ~20× larger) and logit-manipulation methods like GeDi across nearly all preference categories, establishing that a contrastive preference model can serve as a general-purpose, data-efficient reward for controllable generation, though performance remains bounded by the coherence of the underlying critique embedding clusters.

2. Context and Motivation

The Core Problem: Control Without Annotated Data

The central tension this paper grapples with is deceptively straightforward: How can we give users precise, flexible control over the stories generated by large language models without requiring them to collect expensive, preference-specific annotated datasets?

Contemporary LLMs — GPT-3, GPT-NeoX, and their relatives — can be prompted to produce stories with desired qualities by including natural language instructions alongside the initial story prompt. A user might write: "Write a sad story with vivid imagery involving a morally good protagonist" and tack on the first sentence. In principle, few-shot prompting should handle this. In practice, the paper argues, performance is highly variable. For some preferences, large models "can consistently meet the criteria." For others — say, controlling the moral alignment of a character — "performance can be highly variable" (Section 1). The model might attend to the preference criteria, or it might not. There is no guarantee, no mechanism that compels adherence. The quality of generation becomes a lottery.

This is a problem of controllability without supervision. The ideal system would let a user express a preference in free-form natural language (e.g., "I want the plot to involve a sophisticated twist" or "the protagonist should be evil") and have the generator reliably produce text that satisfies that preference. It would work for preferences the designer never anticipated during training. And it would do this without requiring the user — or the system builder — to manually annotate thousands of story examples for every new preference category.

Why This Matters: Beyond Engineering Convenience

The motivation here is not merely to reduce annotation cost, though that is a practical benefit. There are deeper implications for how we think about creative AI systems.

First, human preference is inherently open-ended. A writing assistant that only supports a fixed menu of 20 pre-approved stylistic attributes (e.g., "make it funnier," "add more dialogue") is a bottleneck, not a liberator. Users of creative tools want to specify properties the designer never considered: "make the pacing more like a Tarantino film," "the villain should be sympathetic in a way that reminds me of my father," "the story should feel like it was written in the 1920s." The space of possible preferences is combinatorially vast and linguistically nuanced — far beyond what any pre-annotated dataset could exhaustively cover. A system that requires a labeled dataset per preference is fundamentally incapable of serving this need. The paper positions itself against this limitation: CARP, being trained on paired stories and critiques, learns a general mapping between narrative text and natural language descriptions of narrative properties. In principle, it can score any story against any preference expressible in language, even ones absent from its training data.

Second, the reliability gap between large prompted models and smaller controlled models raises questions about how LLMs process instructions. The paper's headline result — that a GPT-2-750M model fine-tuned with CARP rewards can match or outperform a prompted GPT-NeoX-20B model (a 20× parameter difference) on preference conformance — is not just an efficiency win. It implies that raw scale does not reliably solve instruction-following for subjective, open-ended generation tasks. A larger model sometimes attends to the preference prompt and sometimes ignores it. The smaller model, when explicitly tuned against a preference signal, is forced to attend. This suggests that for controllable generation, the mechanism by which control is enforced matters at least as much as the raw capacity of the generator — a non-obvious finding that challenges the "bigger is always better" narrative for creative text generation.

Third, there is a direct connection to AI alignment. While the paper does not frame itself primarily as alignment work, the language in Section 9 is explicit: "It points to future directions for aligning models with human value systems." The ability to steer a generative model toward user-specified normative properties — moral character alignment, non-violent outcomes, prosocial themes — using a general-purpose preference model has clear relevance to the broader project of making AI systems responsive to human values. The Alignment CARP CoOp model, trained on the Moral Stories dataset to classify characters as good, neutral, or evil, is a concrete demonstration that the pipeline extends to ethically salient preferences.

Prior Approaches and Their Limitations

The paper identifies three broad classes of existing approaches, each with distinct failure modes that motivate the CARP-based alternative.

Approach 1: Prompt Engineering with Large LMs. The user encodes their preferences directly in the prompt, relying on the model's few-shot instruction-following capabilities. This is the simplest approach and requires no additional training or model weights — just craft a prompt and hope the model follows it.

"Larger models — GPT-J-6B, GPT-NeoX-20B, or GPT-3 — can accept more complicated prompts that include preferences in addition to the first sentence of the story. However, these models are not required to attend to all elements in the prompt and, as will be demonstrated, can vary significantly in how much they adhere to the prompt." (Section 4)

The core limitation is reliability through non-commitment. The model has no incentive to satisfy every constraint in the prompt; it generates what is most probable given the entire context, which may involve ignoring or down-weighting the preference component, particularly when it conflicts with the model's default story distribution. The paper's results bear this out: Figure 4 shows NeoX-20B's preference conformance varying wildly across categories — strong on some (apparently accidentally well-represented in training data) and near-chance on others. Prompt engineering is also inherently inconsistent: small changes to the prompt wording can produce large shifts in behavior, making it a fragile foundation for reliable controlled generation.

Approach 2: Logit-Manipulation Methods. Techniques like GeDi (Krause et al., 2020) and Plug & Blend (Lin and Riedl, 2021) use a separate classifier or topic model to post-process the language model's output logits, increasing the probability of tokens associated with a desired attribute and decreasing the probability of others. GeDi, for instance, trains a discriminator on labeled attribute data, then at generation time computes a contrastive score that biases the generative model's output distribution.

The paper identifies two central weaknesses. First, these methods require annotated datasets aligned to the desired preference:

"logit-based methods — such as GeDi — are often less capable when presented with complex preferences. This in turn limits the model's expressiveness. Further, these methods require potentially expensive collection of annotated datasets aligned to user preference." (Section 1)

You cannot use GeDi to steer toward "sophisticated plot twists" unless you have a labeled dataset of stories tagged as having or lacking sophisticated plot twists. This defeats the purpose of open-ended control. Second, even when annotated data exists, these methods struggle with complex preferences — attributes that are not reducible to simple topic or sentiment distinctions. The paper's human study (Table 2, Figure 4) shows GeDi performing decently on narrow topics but failing badly on more abstract preferences like "fighting" or "imagery," where the relevant token-level signals are diffuse rather than concentrated in a few discriminative words. A story about fighting might use words like "punched" and "swung," but it might also use evocative, indirect language like "he crumpled to the ground, the breath driven from his lungs" — the kind of phrasing GeDi's token-level bias cannot reliably distinguish from non-violent descriptions of physical movement.

Approach 3: Reinforcement Learning with Hand-Crafted or Ad-Hoc Rewards. Prior work by Tambwekar et al. (2019b) and Peng et al. (2020) used RL to fine-tune LMs toward specific objectives, such as ending stories at a given goal state or producing less non-normative text. These approaches use purpose-built reward models: a goal-classifier trained for a specific task, a norm-violation classifier trained on annotated normative data, and so forth.

The limitation here is a lack of generality. Each reward model is tied to a single criterion. If you want to control for a new preference, you retrain the classifier, which entails collecting a new labeled dataset. This is the same annotation bottleneck that afflicts logit-manipulation methods, but now compounded by the need to integrate the new classifier into an RL training loop. There is no mechanism for zero-shot preference expression — no way to say "I've never seen this exact preference before, but I can score stories against it anyway because I understand the relationship between language and narrative properties."

How This Paper Positions Itself

The paper positions its approach as addressing the generality-reliability tradeoff that prior methods fail to navigate. The claim, in essence, is:

  • Prompt engineering offers generality (any preference can be expressed in natural language) but lacks reliability (the model may not attend to the preference).
  • Logit manipulation offers reliability (explicit bias toward specified tokens) but lacks generality (requires per-preference annotated data and struggles with complex, non-lexical preferences).
  • Ad-hoc RL reward models offer reliability (explicit optimization against a signal) but lack generality (each reward model is single-purpose).

The proposed pipeline — contrastive bi-encoder as general reward model + RL fine-tuning — aims to capture the best of both worlds: the generality of natural language preference expression (via CARP's alignment of story text with critique text) and the reliability of explicit optimization (via PPO with CARP as the reward). The contrastive training on 1.3 million story-critique pairs gives CARP a general-purpose "understanding" of how language describes narrative properties, in principle enabling it to score any story against any preference without additional training. The RL fine-tuning then compels the generator to produce stories that score highly under this reward, creating a mechanism that is both flexible in what preferences it can handle and reliable in enforcing them.

The paper explicitly connects this to the broader contrastive learning paradigm popularized by CLIP (Radford et al., 2021). Just as CLIP learns a joint embedding space where images and their captions are pulled together and mismatched pairs are pushed apart, CARP learns a joint embedding space for stories and critiques. The key analogy: CLIP enabled zero-shot image classification by comparing image embeddings to arbitrary text descriptions; CARP enables zero-shot story preference scoring by comparing story embeddings to arbitrary critique text. The paper's contribution is in showing that this embedding-space comparison can serve as an effective RL reward signal for generative fine-tuning, and in developing techniques (CoOp, pseudo-labeling) to make that signal robust enough to drive stable policy improvement.

The CARP Reward Signal Is Weak — And That's a Problem

A crucial motivation for the technical contributions of the paper (particularly the CARP CoOp architecture) is that the raw CARP model, while general-purpose, produces a weak and continuous reward signal that is difficult to use effectively in RL.

"CARP produces scores along a continuous range with many middling values. As a result CARP provides a relatively weak classification signal as observed during preliminary experiments. This makes it challenging for PPO to discriminate between continuations. Consequently, the CARP-tuned model often fails to learn to meet some criteria and overfits for others depending on how sensitive CARP is to the target criterion." (Section 5)

This is a critical, non-obvious failure mode. CARP might assign a score of 0.62 to a story segment that perfectly embodies the desired preference and 0.58 to one that fails — a difference of 0.04 on a continuous scale. In an RL setting where the policy is trying to climb this reward gradient, such a weak signal provides little directional guidance. The policy receives similar rewards for good and bad outputs, and the optimization stalls or drifts randomly. Worse, the model might overfit to surface-level features that CARP happens to be slightly sensitive to, producing degenerate outputs that score well under CARP but fail to actually embody the preference — a form of reward hacking analogous to what the RLHF literature has documented with learned reward models.

This is the gap that CARP CoOp is designed to fill. By clustering CARP's critique embeddings into semantically coherent groups, applying pseudo-labeling to assign discrete preference categories, and then using prompt tuning (CoOp) to create a classifier that produces sharp, discriminative scores for each category, the reward signal becomes stronger and more reliable. The paper positions this as a necessary robustness intervention that makes the contrastive-reward approach practically viable, not merely theoretically appealing.

Summary of the Gap

In brief: the field had general-purpose text generators (LLMs) but only unreliable, prompt-based mechanisms to control them for subjective preferences, or reliable mechanisms (classifier-guided generation, RL with annotated reward models) that lacked generality because they required per-preference labeled data. The paper identifies a path through this dilemma — a contrastively trained preference model that is general and can drive RL optimization — and then identifies and addresses the failure modes that make naive implementations of this path unstable. The human study against a 20× larger prompted model and a logit-manipulation baseline is designed to demonstrate that this approach not only works in principle but outperforms the available alternatives in practice.

3. Technical Approach

3.1 Reader Orientation

This is primarily a systems-building paper that constructs a pipeline for controllable story generation, where the core idea is to use a contrastively trained bi-encoder (CARP) — which has learned to align stories with natural language critiques — as a general-purpose reward model to guide reinforcement learning fine-tuning of a generative language model, and then to make this process robust by discretizing and strengthening the reward signal through clustering, pseudo-labeling, and prompt tuning (CARP CoOp).

The system solves the problem of generating stories that reliably conform to user-specified preferences expressed in natural language (e.g., "sad story with vivid imagery," "the protagonist is morally evil") without requiring per-preference annotated datasets. The "shape" of the solution is: a preference model scores how well a generated story matches a desired criterion, reinforcement learning optimizes the generator to maximize that score, and the preference model is itself hardened against weak or ambiguous signals through unsupervised clustering in its embedding space.

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises five major components arranged in a training pipeline, with two distinct phases of preference model construction preceding the final RL fine-tuning:

  1. Base Story Generator (GPT-2-750M fine-tuned on ROCStories) — the language model that will ultimately produce controlled stories. It starts as a generic story-writing model, trained on 95% of the ROCStories corpus (simple five-sentence everyday narratives), which gives it a basic grasp of story structure before preference optimization begins.

  2. CARP (Contrastive Author-Review Preference model) — a pre-trained bi-encoder that takes a story passage and a natural language critique as input, produces separate embeddings for each, and scores their alignment via cosine similarity. CARP was trained on 1,378,696 story-critique pairs from the Story Critique dataset. It serves as the foundation for all reward signals in the pipeline.

  3. CARP CoOp (Prompt-Tuned Variants) — Two enhanced versions of CARP that transform the weak continuous similarity score into a stronger, class-based reward signal:

    • Pseudo CARP CoOp applies unsupervised clustering (UMAP + HDBSCAN) to CARP's critique embedding space to discover six discrete preference categories (imagery, fighting, music, family, accidents/religion combined, and a catch-all), then uses Context Optimization (CoOp) prompt tuning to learn per-class soft-critique embeddings that produce sharp, discriminative classification scores.
    • Alignment CARP CoOp does the same but for three character moral alignment labels (good, neutral, evil) derived by generating and classifying stories using GPT-J-6B on the Moral Stories dataset, producing 17,157 story-alignment pairs.

  4. PPO Fine-Tuning Loop — Proximal Policy Optimization acts as the RL algorithm that treats the language model as a policy, generates story continuations (actions), receives CARP CoOp scores as terminal rewards, and updates only the last two transformer layers of the LM to maximize those rewards while staying close to a reference policy via KL regularization.

  5. Verifier-Based Selection (Implicit) — Although not a separately trained component, the generation-time process involves the CARP CoOp model scoring candidate continuations, with the final selection mechanism (the reward function's classification score) effectively filtering which behaviors the policy learns to reproduce.

Information flow: A prompt (first 5 tokens of a ROCStories validation story) → GPT-2-750M generates a 60-token continuation → CARP CoOp scores the continuation against a target preference label → PPO uses the score as a reward, computes advantage, and updates the policy → repeat for 20,000 steps, with 64 records sampled per step from an experience replay buffer.

3.3 Roadmap for the Deep Dive

  • First, the base story generator — how GPT-2-750M is turned into a basic story-writing model via ROCStories fine-tuning, and why this particular starting point matters.
  • Second, the CARP model architecture and training — how the contrastive bi-encoder learns to align stories with critiques, what its scoring function is, and why its continuous output is problematic as an RL reward.
  • Third, the pseudo-labeling procedure — how unsupervised clustering on CARP's critique embedding space discovers discrete preference categories, the mechanics of UMAP, HDBSCAN, and centroid-based classification, and the hand-labeling process that selects the final six pseudo-labels.
  • Fourth, the CARP CoOp architecture — how prompt tuning is integrated into the bi-encoder to transform it from a pairwise similarity scorer into a multi-class classifier, the training procedure for both Pseudo CARP CoOp and Alignment CARP CoOp, and why this produces a stronger reward signal.
  • Fifth, the PPO fine-tuning setup — the RL formulation (state, action, reward), the experience replay mechanism, the KL regularization term, layer freezing, and the specific hyperparameters that define the optimization loop.
  • Sixth, the generation and evaluation protocol — how stories are sampled, the preference labels used, and how the human study operationalizes preference conformance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a systems-building paper whose core idea is that a contrastively trained story-critique alignment model can serve as a general-purpose reward for RL-based controllable generation, provided its signal is made sufficiently robust through discretization and prompt tuning.

Base Story Generator: ROCStories Fine-Tuning

The starting point for all subsequent models is GPT-2-750M (also referred to as "GPT-2-large" in the hyperparameters), which has been fine-tuned on the ROCStories corpus — a dataset of 100,000 five-sentence stories about common, everyday occurrences. The paper holds out 5% of ROCStories as a validation set and fine-tunes on the remaining 95%.

This baseline model serves an important role: it provides a generator that can produce coherent, multi-sentence narratives rather than random text or generic language model continuations. The authors note that ROCStories are "relatively simplistic stories" with "simple sentence structures," but emphasize that subsequent preference fine-tuning significantly shifts the output distribution:

"Despite the model being trained on relatively simplistic stories, subsequent fine-tuning for preferences results in more expressive stories. This is due to the CARP model's tendency to shift the output distribution closer to human-written stories from the Story Critique corpus and Moral Stories corpus." (Section 3)

This is a non-obvious design choice. One might expect that starting from a model pre-trained on more sophisticated literary text would be advantageous. However, the paper's approach demonstrates that the CARP reward signal itself, when used in RL fine-tuning, drives the generator toward the linguistic complexity of the Story Critique corpus — which contains more varied, creative writing than ROCStories — without the generator ever seeing example stories from that corpus during training. The ROCStories model provides a clean, coherent foundation; the CARP reward then pulls the output toward the desired preference and, as a byproduct, toward richer prose.

The choice of GPT-2-750M specifically (rather than, say, GPT-J-6B or GPT-NeoX-20B) reflects the paper's broader thesis: that reliable control through explicit optimization can compensate for model scale. By demonstrating that a 750M parameter model can match or exceed a 20B parameter model on preference conformance, the paper makes the case that the control mechanism, not raw generation capacity, is the binding constraint.

CARP: The Contrastive Bi-Encoder Preference Model

CARP (Contrastive Author-Review Preference model) is the component that enables general-purpose preference scoring — the ability to evaluate how well any story matches any natural language criterion without per-criterion training. Understanding its architecture and training is essential because every subsequent component (pseudo-labeling, CoOp, RL rewards) either builds on or modifies CARP's embedding space.

Architecture. CARP is a bi-encoder: two separate transformer encoders, one for stories and one for critiques, that each map their input to a fixed-dimensional embedding vector. Specifically:

  • The story encoder takes a passage of story text and produces a vector embedding $\mathbf{s} \in \mathbb{R}^{2048}$ (the paper implies 2048-dimensional embeddings through its description of projecting from $\mathbb{R}^{2048}$ to $\mathbb{R}^2$ in the pseudo-labeling section, Appendix E).
  • The critique encoder takes a natural language critique (e.g., "this story has vivid imagery" or "the dialogue feels unnatural") and produces a vector embedding $\mathbf{c} \in \mathbb{R}^{2048}$ in the same space.

These embeddings lie on the unit $n$-sphere — meaning they are L2-normalized such that $\|\mathbf{s}\|_2 = 1$ and $\|\mathbf{c}\|_2 = 1$. This is explicitly noted in the pseudo-labeling appendix: "CARPs latent space exists on a manifold: the unit $\mathbb{R}^{2048}$ n-sphere."

Scoring function. The alignment between a story and a critique is measured as the cosine similarity between their respective embeddings:

score(story,critique)=s⋅c∥s∥∥c∥=s⋅c\text{score}(\text{story}, \text{critique}) = \frac{\mathbf{s} \cdot \mathbf{c}}{\|\mathbf{s}\| \|\mathbf{c}\|} = \mathbf{s} \cdot \mathbf{c}

where $\mathbf{s}$ is the story embedding, $\mathbf{c}$ is the critique embedding, and the denominator disappears because both vectors are unit-normalized.

What it computes: the dot product between two points on the unit hypersphere, which equals the cosine of the angle between them. A value of 1 means the vectors point in exactly the same direction (the story perfectly matches the critique); 0 means they're orthogonal; -1 means perfect opposition (unlikely in practice since both embeddings represent content, not negation).

Why this form: cosine similarity on the unit sphere is the standard contrastive learning objective (pioneered by CLIP for vision-language). It encourages the model to place semantically related pairs near each other on the sphere and unrelated pairs farther apart, creating a space where proximity equals alignment. The key property is normalization invariance — the score depends only on direction, not magnitude, which prevents the model from achieving high similarity by simply making all embeddings large.

Training data. CARP was trained on the Story Critique dataset, which contains more than 80,000 unique stories with 1,378,696 total critiques. Each critique refers to a specific passage of the story, creating 1,378,696 passage-critique pairs. The dataset is anonymized: unique identifiers including comment ID, submission IDs, URLs, and proper nouns have been removed.

Training objective. While the paper does not explicitly state CARP's loss function, it describes it as a contrastive model "like CLIP" and notes it "learns to align positive examples of stories adhering to a specific critique and reject negatives." For a bi-encoder trained contrastively, the standard objective is the InfoNCE loss (or a symmetric variant). Given a batch of $N$ story-critique pairs $\{(\mathbf{s}_i, \mathbf{c}_i)\}_{i=1}^N$, the story-to-critique loss for the $i$-th pair would be:

Lis→c=−log⁡exp⁡(si⋅ci/τ)∑j=1Nexp⁡(si⋅cj/τ)\mathcal{L}_i^{\text{s→c}} = -\log \frac{\exp(\mathbf{s}_i \cdot \mathbf{c}_i / \tau)}{\sum_{j=1}^N \exp(\mathbf{s}_i \cdot \mathbf{c}_j / \tau)}

where $\tau$ is a learned temperature parameter controlling the sharpness of the distribution. A symmetric critique-to-story loss is computed analogously, and the total loss is typically the average of both directions.

What it computes: for each story in the batch, treat the correct critique as the positive and all other critiques in the batch as negatives. The loss encourages the model to assign high cosine similarity to the true pair and low similarity to all impostor pairs. The denominator sums over all critiques in the batch, creating an implicit $N$-way classification problem.

Why this form: contrastive learning with in-batch negatives is highly data-efficient because every batch provides $N-1$ negative examples per positive, without requiring explicit negative sampling. The temperature $\tau$ controls the concentration of the softmax distribution — lower temperatures produce sharper distinctions, which is important because the resulting embedding space must support fine-grained preference distinctions, not just broad topic matching.

The weakness of continuous scoring. The paper identifies a critical practical limitation of the raw CARP model for RL-based fine-tuning:

"CARP produces scores along a continuous range with many middling values. As a result CARP provides a relatively weak classification signal as observed during preliminary experiments. This makes it challenging for PPO to discriminate between continuations." (Section 5)

In terms of the cosine similarity space, this means that for many (story, preference) pairs, the model produces scores clustered in a narrow band — say, between 0.4 and 0.6 — where the difference between a good match and a mediocre one might be as small as 0.05. When PPO uses this as a reward, the policy gradient is weak and noisy: the algorithm cannot reliably tell which generation was better, so optimization stalls or, worse, exploits spurious correlations that produce slightly higher scores without genuinely improving preference alignment.

This observation directly motivates the clustering and CoOp interventions described next. The paper's insight is that the CARP embedding space does contain meaningful structural information about preference categories — it just expresses it in a continuous, overlapping way that is ill-suited to RL optimization. By explicitly discretizing this space into categories and training a classifier on top of it, the signal becomes sharper and more reliable.

Pseudo-Labeling via Unsupervised Clustering of the Critique Embedding Space

The first major technical contribution beyond standard CARP is the pseudo-labeling procedure that discovers discrete preference categories within the critique embedding space. This is the mechanism by which the paper extracts a small number of semantically coherent preference classes from the continuous embedding manifold, enabling a transformation from weak similarity scoring to strong classification scoring.

Step 1: Dimensionality Reduction with UMAP. The critique embeddings from CARP lie on the unit sphere in $\mathbb{R}^{2048}$, which is far too high-dimensional for density-based clustering algorithms to work effectively. The paper applies UMAP (Uniform Manifold Approximation and Projection) to project the embeddings from 2048 dimensions down to 2 dimensions:

"We observe that CARP's critique embeddings lie on a spherical manifold. Thus we apply UMAP to project the embeddings from their high dimension to 2 dimensions, which was determined via a hyperparameter sweep. The choice of 2 dimensions provides the added benefit of visualization of the CARP model's latent space." (Section 5.1)

UMAP is chosen over alternatives like PCA or t-SNE specifically because CARP's latent space "exists on a manifold: the unit $\mathbb{R}^{2048} n-sphere" (Appendix E), and UMAP is designed to preserve manifold structure in low-dimensional embeddings. Cosine distance is used as the distance metric since CARP was originally trained with a cosine similarity-based loss.

The hyperparameter sweep over target dimensions — from which 2 was selected — is notable because it is optimized not for visualization quality but for a downstream clustering metric: the proportion of embeddings that HDBSCAN can successfully assign to clusters versus labeling as noise. The paper states that 2 dimensions minimized the proportion of unclustered embeddings, making it a pragmatic choice driven by the needs of the subsequent clustering step.

Step 2: Hierarchical Density-Based Clustering with HDBSCAN. The 2D UMAP-projected critique embeddings are then clustered using HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise):

"HDBSCAN resulted in 91 clusters. However, HDBSCAN failed to cluster half of the reviews it was given, which are subsequently added to a 'noise' cluster." (Section 5.1)

HDBSCAN is chosen over simpler alternatives like K-means for several reasons that align with the properties of the embedding space. First, HDBSCAN does not require specifying the number of clusters in advance — it discovers clusters based on local density, which is appropriate when the underlying number of preference categories is unknown. Second, HDBSCAN explicitly models noise points, assigning them to a "noise" cluster rather than forcing every point into a cluster. This is critical because the critique embedding space contains many idiosyncratic, infrequent preferences that do not form coherent clusters — forcing them into categories would dilute the semantic coherence of the discovered classes. The fact that HDBSCAN left half the data unclustered is actually a feature, not a bug: it indicates that only certain preferences have sufficient representation in the training data to form reliable clusters, and those are precisely the preferences that can serve as strong reward classes.

Step 3: Hand-Labeling and Cluster Selection. Given the 91 automatically discovered clusters, the paper performs a manual curation step:

"We hand-label clusters by sampling several critiques from each. Any cluster where an associated story feature was ambiguous was discarded. Clusters with identical story features were merged." (Section 5.1)

The appendix (Appendix C) provides the full list of 54 hand-labeled cluster captions derived from this process, covering diverse story features: character dialogue, humor, imagery/descriptions, fighting, music, family, religion, death, murder, crimes, facial expressions, scene descriptions, etc. Examples include "characters laughing or finding things funny" (humor cluster), "guns and bombs" and "fighting" (conflict clusters), "dogs and cats" (domestic animal cluster), "swimming/sailing/ocean" (setting cluster), and "gross smells and things to see" (sensory description cluster).

From these, six clusters are selected for training Pseudo CARP CoOp, "chosen for semantic dissimilarity and separation in the embedding space" (Section 5.2). The paper's Table 2 and Appendix E indicate these six are: imagery (use of descriptive visual language), fighting (physical conflict scenes), music (references to musical instruments and performance), family (domestic and familial themes), accidents/religion (two clusters that appear to have been merged or related), and what appears to be a catch-all or residual category. The specific bolded captions in Appendix E suggest the final six were selected to maximize inter-class distance in the embedding space while maintaining intra-class semantic coherence — essentially, choosing cluster centroids that are far apart so that the resulting classifier has good separability between preference categories.

Step 4: Centroid Computation. For each selected cluster, a high-dimensional centroid is computed in the original $\mathbb{R}^{2048}$ space (not the 2D projection):

"Finally we compute a high-dimensional centroid for every cluster by taking a sample-wise mean of the respective latent vectors." (Section 5.1)

These centroids are then projected back onto the unit hypersphere to ensure they lie in the same normalized space as individual embeddings. Mathematically, for a cluster containing $K$ critique embeddings $\{\mathbf{c}_1, \dots, \mathbf{c}_K\}$:

μ=normalize(1K∑k=1Kck)=∑k=1Kck∥∑k=1Kck∥2\boldsymbol{\mu} = \text{normalize}\left(\frac{1}{K} \sum_{k=1}^K \mathbf{c}_k\right) = \frac{\sum_{k=1}^K \mathbf{c}_k}{\|\sum_{k=1}^K \mathbf{c}_k\|_2}

where the normalization step ensures $\|\boldsymbol{\mu}\|_2 = 1$, placing the centroid on the same unit sphere as the individual embeddings.

Why compute centroids in the original space rather than the UMAP space? The UMAP projection to 2D is a lossy compression optimized for clustering, not for preserving the fine-grained cosine similarity relationships that CARP's original training objective established. The 2048-dimensional centroid preserves the full representational capacity of the original embedding space, allowing the subsequent classifier (CoOp) to leverage all the nuanced semantic dimensions that CARP learned, while using the cluster assignment derived from the low-dimensional projection as its supervision signal.

Step 5: Point-Class Assignment with Thresholding. To create labeled training data from this unsupervised structure, the paper assigns each critique in the dataset to the cluster whose centroid it is closest to, but only if the similarity exceeds a threshold:

"For a new, arbitrary latent vector, we measure its distribution over the classifiers as a softmax over its cosine similarities to each cluster centroid. However, taking the distance to centroid as a measure for classifying new points is often inconsistent for HDBSCAN... We alleviated this issue by removing samples with distance to centroid falling below a threshold." (Section 5.1)

For Pseudo CARP CoOp training, this threshold is set at "twice the average cosine-similarity of the dataset" (Section 5.2). Story-critique pairs whose critique embedding has cosine similarity below this threshold to any of the six class centroids are discarded entirely. For pairs that pass the threshold, the assigned class is the centroid maximizing cosine similarity.

This thresholding is crucial. Without it, points near the boundaries between clusters or deep in the "noise" region would be assigned weak, ambiguous labels, and training a classifier on such noisy supervision would produce a reward model no better — and potentially worse — than the original CARP. By filtering aggressively, the training data for CARP CoOp consists only of examples with clear, unambiguous preference characteristics, producing a more reliable classifier.

Step 6: Dataset Balancing. The paper selects exactly 1,000 samples per class for training:

"For the chosen critique pseudo labels, we select 1000 samples belonging to each class, balancing the dataset among pseudo-labels. This is crucial as otherwise we observe the model overfits to overrepresented classes and fails to provide strong signals for less-represented classes." (Section 5.2)

Natural preference distributions in the Story Critique dataset are highly imbalanced — common critiques about pacing or imagery vastly outnumber specialized critiques about, say, music or religious themes. Training a classifier on imbalanced data would cause it to become overconfident for frequent classes and underconfident for rare ones, which would translate to uneven reward scaling in the RL loop: the generator would be strongly rewarded for common preferences and weakly rewarded for rare ones, distorting the optimization landscape.

CARP CoOp: Prompt Tuning for Discretized Reward

The pseudo-labeling procedure gives us six discrete preference categories and a curated set of 1,000 (story, critique, class-label) triples per category. However, there is a critical mismatch: the labels specify what class a story belongs to, but at generation time, we need the model to score a story against a target class without having a natural language critique to encode on the critique encoder side. We want to say "score this story for the imagery class," not "score this story against this specific critique text."

CARP CoOp resolves this by learning a synthetic critique embedding for each class using Context Optimization (CoOp) prompt tuning, adapting the technique from Zhou et al. (2021) which was originally designed for vision-language models.

Architecture modification. The original CARP model (Figure 3, left) takes a story and a critique as input, encodes them separately, and outputs a cosine similarity. The CARP CoOp architecture (Figure 3, right) replaces the critique encoder's input with a learned prompt — a sequence of trainable embedding vectors:

"Specifically, the CoOp prompt tuning layer learns a unified embedding [V1] [V2] ... [VM/2] [CLASS] [V(M/2)+1] ... [VM] where V1,...,VM are shared parameters learned across all classes and [CLASS] is a token embedding that maximizes the log-likelihood of a specific critique class." (Section 5.2)

Conceptually, this means:

  • There are $M$ learned embedding vectors (the paper does not specify $M$ explicitly, but typical CoOp configurations use small $M$, e.g., 4 to 16).
  • The first $M/2$ vectors are shared across all classes — they learn generic "how to express a preference" information.
  • The middle $[CLASS]$ token embedding is class-specific — it encodes the particular preference category (e.g., "imagery" vs. "fighting").
  • The last $M/2$ vectors are again shared — they learn how to complete the synthetic critique.
  • This entire sequence is passed through the frozen CARP critique encoder, which produces an embedding in the same $\mathbb{R}^{2048}$ space as real critique embeddings.

Why this hybrid shared/class-specific design? If each class learned a completely independent prompt (no shared parameters), the model would have no mechanism to transfer general "critique-writing" knowledge between classes — each class would essentially be learning from scratch on 1,000 examples. By sharing $V_1, ..., V_{M/2}$ and $V_{(M/2)+1}, ..., V_{M}$, the model learns a generic template for expressing preferences in the critique embedding space, with only the class-specific token needing to carry per-category information. This dramatically reduces the number of class-specific parameters and enables data-efficient learning: the shared parameters pool information across all six classes during training.

Training objective for Pseudo CARP CoOp. Rather than training with the standard contrastive loss, Pseudo CARP CoOp is trained as a multi-class classifier using KL-divergence:

"Softmaxing gives a distribution over which we can minimize a KL-divergence loss between the predicted label and the target label." (Section 5.2)

The target distribution for training is not a one-hot vector but a soft target derived from the critique's cosine similarity to each of the six cluster centroids:

ptarget(c∣x)=softmaxc(c(x)⋅μ1τ,…,c(x)⋅μ6τ)p_{\text{target}}(c \mid \mathbf{x}) = \text{softmax}_c\left(\frac{\mathbf{c}(\mathbf{x}) \cdot \boldsymbol{\mu}_1}{\tau}, \dots, \frac{\mathbf{c}(\mathbf{x}) \cdot \boldsymbol{\mu}_6}{\tau}\right)

where $\mathbf{c}(\mathbf{x})$ is the frozen CARP critique embedding of the original critique text for story $\mathbf{x}$, $\boldsymbol{\mu}_i$ is the centroid for class $i$, and $\tau$ is a temperature (implicitly 1 or a learned parameter). The predicted distribution comes from encoding the story $\mathbf{s}(\mathbf{x})$ and comparing it to the six learned class prompts:

ppred(c∣x)=softmaxc(s(x)⋅CoOp(1),…,s(x)⋅CoOp(6))p_{\text{pred}}(c \mid \mathbf{x}) = \text{softmax}_c\left(\mathbf{s}(\mathbf{x}) \cdot \text{CoOp}(1), \dots, \mathbf{s}(\mathbf{x}) \cdot \text{CoOp}(6)\right)

The KL-divergence loss is then:

LCoOp=∑c=16ptarget(c∣x)log⁡ptarget(c∣x)ppred(c∣x)=H(ptarget,ppred)−H(ptarget)\mathcal{L}_{\text{CoOp}} = \sum_{c=1}^6 p_{\text{target}}(c \mid \mathbf{x}) \log \frac{p_{\text{target}}(c \mid \mathbf{x})}{p_{\text{pred}}(c \mid \mathbf{x})} = H(p_{\text{target}}, p_{\text{pred}}) - H(p_{\text{target}})

where $H(p_{\text{target}}, p_{\text{pred}})$ is the cross-entropy (the effective optimization term) and $H(p_{\text{target}})$ is the entropy of the target distribution (a constant with respect to model parameters).

What it computes: the divergence between the classifier's predicted distribution over the six preference classes and a soft-label distribution derived from the original critique's position relative to the cluster centroids in the frozen CARP embedding space. Minimizing this encourages the CoOp model to produce embeddings that replicate the centroid-based classification behavior, but using the learned prompts rather than explicit centroid comparisons.

Why soft targets rather than one-hot hard labels? The paper explicitly addresses this:

"Note the KL-divergence is computed as the distance to centroid as our stationary distribution (softmaxed distance to centroid per pseudo class). This is done to improve data efficiency since we found it was common for stories in the Story Critique dataset to fit under multiple critique labels." (Section 5.2)

In other words, real stories often simultaneously exhibit multiple preference characteristics — a single passage might be both "vividly descriptive" (imagery) and "involve physical conflict" (fighting). Forcing a one-hot label would discard this information and confuse the classifier, especially near category boundaries. The soft-label approach preserves the continuous, overlap-prone nature of narrative preferences while still providing a strong enough signal for classification. It's a middle ground: not as weak as the raw CARP cosine similarity (which gives no explicit class structure) and not as rigid as hard classification (which would misrepresent naturally multi-faceted stories).

Alignment CARP CoOp Training. The Alignment CARP CoOp model follows the same architecture and training procedure but for three moral alignment classes (good, neutral, evil) rather than six topic classes. The training data construction differs:

  1. The Moral Stories dataset (12,000 short narratives) is augmented by randomly sampling a context, action, and consequence, then few-shot prompting GPT-J-6B to generate and classify story segments as having a character acting "good," "evil," or "neutral."
  2. GPT-J-6B's logits associated with these labels provide a soft score for each alignment class.
  3. The process produces 17,157 story-alignment pairs.
  4. As with Pseudo CARP CoOp, the dataset is thresholded and balanced — 1,000 examples per alignment class are selected.

The key note here is data efficiency:

"Here we observe it is sufficient to train both CARP CoOp and Alignment CARP CoOp on 1000 examples per label to achieve competitive downstream performance. This demonstrates the CoOp method is highly data efficient, requiring a minimal number of examples per class to fine-tune when initialized with the pretrained original CARP model." (Section 5.2)

The pre-trained CARP critique encoder — frozen during CoOp training — provides a strong initialization for the embedding space. The CoOp layer only needs to learn how to construct synthetic prompts that steer the frozen encoder's output toward class-specific regions of this space, a much lower-dimensional learning problem than training a classifier from scratch.

Freezing strategy. Both CARP CoOp variants start with the pre-trained original CARP model with embedding layers frozen. This means the story encoder and critique encoder weights are not updated during CoOp training; only the prompt embeddings $V_1, ..., V_M$ and the class-specific [CLASS] embeddings are learned. This prevents catastrophic forgetting of the general story-critique alignment knowledge that CARP acquired during its original contrastive pre-training on 1.3 million pairs.

Why not fine-tune the full encoder? The paper found that preserving the frozen CARP backbone is essential for maintaining generalization. If the encoder weights were allowed to adapt during CoOp training on only 6,000 examples (1,000 per pseudo-class), the model would overfit to the narrow distribution of those examples and lose its ability to score novel preferences or stories outside the training set. Freezing the encoder ensures that the CoOp prompts are learning to navigate the existing, richly structured embedding space rather than reshape it for the limited training data.

From classifier to reward. At inference time (during RL fine-tuning), the CARP CoOp model takes a generated story continuation $\mathbf{x}$ and a target preference label $c$ (e.g., "fighting" or "evil") and produces a scalar reward:

r(x)=ppred(c∣x)r(\mathbf{x}) = p_{\text{pred}}(c \mid \mathbf{x})

This is simply the softmax probability assigned to the target class by the CoOp classifier. The paper uses the negative log-likelihood loss for the corresponding label in training:

"In this configuration, however, a criteria label is used instead of a text criteria and we use the negative log-likelihood loss for the corresponding label." (Section 5.2)

Which means the effective reward is a classification score between 0 and 1, with higher values indicating stronger preference alignment — a much sharper signal than the raw CARP cosine similarity scores that originally motivated the CoOp intervention.

PPO Fine-Tuning: Reinforcement Learning for Preference Optimization

With CARP CoOp providing a strong, class-based reward signal, the paper uses Proximal Policy Optimization (PPO) to fine-tune the base GPT-2-750M story generator. The RL formulation treats text generation as a sequential decision-making problem.

State, action, and trajectory. The RL framework is defined over token-level generation:

  • State $s_t$ at time $t$: the sequence of tokens generated so far (the context).
  • Action $a_t$ at time $t$: the next token to be generated, sampled from the language model's output distribution given $s_t$.
  • Trajectory: a complete story continuation, consisting of all tokens generated from time 0 to time $T$, where $T$ is determined by either generating 60 tokens or hitting the end-of-text token.

Formally, the language model defines a policy $\pi_\theta(a_t \mid s_t)$, parameterized by model weights $\theta$, which is the probability distribution over the vocabulary for the next token given the current context.

Reward structure. The reward is not given at each step but only at the end of the trajectory. The total reward for a complete generated story continuation $\tau = (s_0, a_0, s_1, a_1, ..., s_T)$ has two components:

R(τ)=RCARP(τ)+RKL(τ)R(\tau) = R_{\text{CARP}}(\tau) + R_{\text{KL}}(\tau)

where:

  • $R_{\text{CARP}}(\tau)$ is the preference alignment reward: the CARP (or CARP CoOp) score given the generated continuation and the target preference criterion. For CARP CoOp, this is the softmax probability for the target class; for vanilla CARP, it is the cosine similarity between the story and critique embeddings.
  • $R_{\text{KL}}(\tau)$ is a KL divergence penalty: a per-token regularization term that penalizes the policy for deviating too far from a reference "momentum model."

The KL regularization term is computed as:

RKL(τ)=−β∑t=1TDKL(πθ(⋅∣st)∥πref(⋅∣st))R_{\text{KL}}(\tau) = -\beta \sum_{t=1}^T D_{\text{KL}}\left(\pi_\theta(\cdot \mid s_t) \| \pi_{\text{ref}}(\cdot \mid s_t)\right)

where $\pi_{\text{ref}}$ is a frozen copy of the base model (the ROCStories-fine-tuned GPT-2-750M before any RL), $\beta$ is a coefficient controlling the strength of the regularization (specified as "init_kl_coef": 0.2 in the hyperparameters), and $D_{\text{KL}}$ measures the divergence between the current policy's token distribution and the reference model's token distribution at each generation step.

What it computes: for each generated token, compute the KL divergence between the current model's prediction and what the reference model would have predicted given the same context. Sum these per-token divergences across the entire sequence, multiply by the coefficient $\beta$, and subtract from the CARP reward. The result is a reward that encourages preference alignment while penalizing radical departures from the base model's behavior.

Why KL regularization? Without it, the policy would be free to exploit any weakness in the CARP reward signal by generating degenerate text that scores highly under CARP but is not fluent, coherent, or natural. For example, if CARP slightly favors stories with the word "sad" regardless of context, the policy might learn to generate "sad sad sad sad sad..." — maximizing the reward while producing nonsense. The KL penalty anchors the policy to the fluent, coherent behavior of the base model, ensuring that preference optimization improves on top of a foundation of reasonable text generation rather than replacing it entirely. This is a standard technique in RLHF (Reinforcement Learning from Human Feedback) and is adapted here for the story generation domain.

The hyperparameter specification includes a "target": 50 and "horizon": 10000 for the KL coefficient, suggesting an adaptive schedule where the KL penalty is adjusted to keep the divergence near a target value (50 nats) over a horizon of 10,000 steps, though the exact adaptation mechanism is not described in detail.

Experience replay buffer. PPO uses an experience replay buffer — a stored collection of trajectories from which training batches are sampled. The mechanism is:

  1. Randomly select a story from the held-out ROCStories validation set.
  2. Take the first five tokens as the initial state $s_0$. The paper explicitly notes that "the best practice is to prompt the language model with a sequence within the task distribution instead of the [SOS] (start-of-sequence) token" — starting from the beginning of a story provides a more natural context than a blank slate.
  3. Generate a continuation of up to 60 tokens or until the end-of-text token is generated, using the current policy $\pi_\theta$.
  4. Compute the CARP (or CARP CoOp) score for the full continuation against the target preference criterion.
  5. Store the trajectory $\langle s_0, a_0, s_1, a_1, ..., s_T, R \rangle$ in the replay buffer.

On every PPO update step, 64 records are sampled from the experience replay buffer, and the policy is updated using the PPO clipped objective.

Prompt selection for preference learning. The choice to use the first 5 tokens of ROCStories validation stories rather than arbitrary prompts is deliberate. If the model were prompted with completely random or out-of-distribution text, it would produce poor continuations that don't represent its actual story-generation capabilities, and the CARP reward signal would be noisy. By using in-distribution prompts (beginnings of simple everyday stories), the generations are more representative of the model's behavior in realistic story-continuation scenarios, producing higher-quality training data for the RL process.

PPO clipped objective. While the paper does not state the PPO loss explicitly, the standard PPO-clip objective (Schulman et al., 2017) for a given trajectory fragment is:

LCLIP(θ)=Et[min⁡(rt(θ)A^t,clip(rt(θ),1−ϵ,1+ϵ)A^t)]L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[\min\left(r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) \hat{A}_t\right)\right]

where $r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)}$ is the probability ratio between the current and old policy, $\hat{A}_t$ is the advantage estimate at time $t$, and $\epsilon$ is the clipping threshold (specified as "cliprange": 0.2 in the hyperparameters). The value function is also trained with a squared-error loss, weighted by "vf_coef": 0.15.

The hyperparameters specify:

  • "ppo_epochs": 4 — the number of passes over each batch of experience.
  • "gamma": 1 — no discounting of future rewards (appropriate since the reward is given only at the end).
  • "lam": 0.95 — the GAE (Generalized Advantage Estimation) lambda parameter.
  • "cliprange_value": 0.2 — the clipping threshold for the value function update.

Layer freezing. A critical design choice: only the last two transformer layers of GPT-2-750M are unfrozen and updated during PPO fine-tuning:

"In preference learning, it is common to freeze all but a small number of the transformer blocks. We found freezing all but the last two layers of the language model provided the best result." (Section 4)

The hyperparameters confirm this: "num_layers_unfrozen": 2.

Why freeze all but the last two layers? There are both practical and theoretical motivations. Practically, updating only 2 layers dramatically reduces memory requirements and training time ("20k steps, which takes on the range of an hour on a single A100 GPU"). Theoretically, the lower layers of a language model encode general syntactic and semantic knowledge that is broadly useful across tasks; the upper layers encode more task-specific representations. By freezing the lower layers, the model preserves its fundamental language generation capabilities (grammar, coherence, vocabulary) while adapting only the high-level "stylistic" and "preference-relevant" aspects of generation through the upper layers. This is conceptually similar to the finding in the RLHF literature that full-model fine-tuning can lead to reward hacking and degradation of general capabilities, while selective updating preserves them.

The reference model $\pi_{\text{ref}}$ against which KL divergence is computed is a frozen copy of the original ROCStories-fine-tuned GPT-2-750M, not the partially-frozen model — ensuring that the regularization pulls toward the base story-telling behavior, not the partially-updated intermediate checkpoint.

Training duration and scale. The model is trained for 20,000 steps with a learning rate of "lr": 0.5e-6. Each step processes 64 trajectories from the replay buffer. The total training time is "on the range of an hour on a single A100 GPU," which is remarkably fast for a generative RL pipeline — a consequence of the layer freezing and the relatively small model size.

Vanilla CARP baseline (Default CARP LM). In the ablation condition, the same PPO setup is used but with the raw CARP cosine similarity score as the reward instead of the CARP CoOp classification score. The reward signal is weaker and more continuous, which the paper's preliminary experiments showed leads to either failure to learn meaningful preference alignment or overfitting to spurious features that CARP is slightly sensitive to. The Default CARP LM results serve as an ablation demonstrating that the CoOp discretization is necessary for reliable preference learning, not merely an incremental improvement.

Generation and Evaluation Protocol

The final section of the technical pipeline concerns how stories are actually generated and evaluated, bridging the trained models to the human study results.

Generation procedure. For evaluation, stories are generated by:

  1. Randomly selecting 5 prefix tokens from the held-out ROCStories validation set (not from the training split), ensuring the prompts are from the same distribution as training but unseen during optimization.
  2. Feeding these tokens to the fine-tuned policy model.
  3. Generating a continuation using the policy's token distribution (presumably with some sampling temperature, though this is not explicitly specified for evaluation).
  4. The continuation length is determined by the model's end-of-text token prediction or by an unspecified truncation limit.

Total model configurations. The paper evaluates four model classes across two regimes:

  • Topic preference regime (no labeled preference data): 6 topic labels (family, music, accidents, religion, imagery, fighting) × 4 model types = 24 generators.
  • Alignment preference regime (with labeled data): 3 alignment labels (good, neutral, evil) × 4 model types = 12 generators.

The four model types are:

  1. GeDi LM: GPT-2-750M guided via GeDi logit manipulation (using pre-trained GeDi topic models for topics, and a GeDi model fine-tuned on the Alignment dataset for alignments).
  2. NeoX: GPT-NeoX-20B prompted with the preference criteria and an initial sentence, using its instruction-following capability.
  3. Default CARP LM: The base GPT-2-750M fine-tuned via PPO with vanilla CARP cosine similarity rewards — the ablation condition.
  4. CARP CoOp LM: The base GPT-2-750M fine-tuned via PPO with CARP CoOp classification rewards — the full proposed system.

For the topic regime, the CARP CoOp LM uses Pseudo CARP CoOp (trained on the six pseudo-labels from Story Critique clustering). For the alignment regime, it uses Alignment CARP CoOp (trained on the augmented Moral Stories dataset). Note that Pseudo CARP CoOp only covers six topic labels and does not have pseudo-labels for "romance" or "horror," so those evaluations only include GeDi, NeoX, and Default CARP LM baselines.

Story sampling for human evaluation. The paper generates 20 story segments per preference per model, yielding a pool of 20 × 9 (6 topics + 3 alignments) × 4 models = 720 total stories (though some topic-model combinations are missing for Pseudo CARP CoOp, slightly reducing this total). From this pool, human study participants each read 44 randomly drawn story segments across all models.

Human study design. Participants were given a story segment and asked to select which preference from a list best described it. For topic evaluations, the choices were among the six topic labels. For alignment evaluations, the choices were among good, neutral, and evil. The separation of topic and alignment evaluations was deliberate:

"Often stories adhering to a certain alignment often involve a specific topic, e.g. good stories often center around family and bad stories often involve accidents. Hence we separate the evaluation of stories generated with a desired alignment from those with a desired topic to prevent artificial mislabeling." (Section 6)

This separation ensures that when evaluating alignment, participants are not confused by topic cues that might correlate with but are distinct from the moral label, and vice versa.

The review hyperparameter. The PPO training hyperparameters include a "review" field that specifies the target preference for training (e.g., "review": "evil" in the example configuration). This means a separate PPO fine-tuning run is performed for each preference label — 9 separate training runs (6 topics + 3 alignments) using the appropriate CARP CoOp variant as the reward model. Each run produces a specialized generator optimized for a single preference.

Why train separate models per preference rather than a multi-preference model? The paper does not explicitly justify this design choice, but the architecture makes the reason clear: the CARP CoOp reward is a single scalar — the probability of the target class. To train a model that can be conditioned on the preference at generation time (like a prompted model), the reward signal would need to encode the conditioning, i.e., the model would need to learn to produce stories matching preference X when the prompt includes X and preference Y when the prompt includes Y. This would require a different training setup where the reward is computed against whatever preference is specified in the prompt, not a fixed target. The paper's approach — training a dedicated model per preference — is simpler but less flexible: each model does one thing well rather than being able to switch between preferences. This is a significant limitation that the paper does not address.

Inter-annotator agreement. The paper reports a "decent" average inter-annotator agreement of 0.74, which is notable given "the complexity and potential for multi-label applicability in the tasks." Agreement is around 0.77 ± 0.01 across model classes, with the exception of GeDi-guided LMs which have "noticeably lower agreement at 0.62" — likely due to GeDi's poor performance on out-of-distribution topics causing inconsistent story quality. Within each preference class, agreement is 0.75 ± 0.04, with the exception of the "imagery" preference which drops to 0.58, attributed to "the complexity of the task" — recognizing descriptive imagery is more subjective than recognizing unambiguous references to, say, family or fighting.

Summary of Key Design Choices

  • ROCStories as base training: Provides a coherent story-generation foundation that CARP rewards can then pull toward richer, more expressive language from the Story Critique distribution — enabling the model to produce complex stories despite never being trained on complex story corpora.
  • UMAP + HDBSCAN for pseudo-labeling: Discovers preference categories in CARP's embedding space without human annotation, enabling the conversion from weak continuous similarity to strong discrete classification — the critical transformation that makes the reward signal usable for RL.
  • Soft centroid-based labels: Preserves multi-label applicability (stories can satisfy multiple preferences simultaneously) while still providing a strong-enough signal for classifier training — a middle ground between weak raw CARP scores and overly rigid one-hot labels.
  • CoOp prompt tuning with frozen encoder: Learns class-specific synthetic critique embeddings using only 1,000 examples per class, leveraging the frozen CARP backbone to avoid catastrophic forgetting while dramatically improving reward signal discriminability.
  • Two-layer PPO fine-tuning: Balances preference learning with preservation of general language capabilities by updating only the last 2 transformer blocks — fast enough to run on a single A100 in about an hour.
  • KL regularization with reference model: Prevents reward hacking and degenerate generation by penalizing divergence from the base ROCStories model's token distribution at each generation step.
  • Per-preference specialized models: Each training run optimizes for a single preference label, producing dedicated generators rather than a single multi-preference model — simpler to train but less flexible at deployment.
  • Evaluation separation of topic and alignment: Prevents artificial mislabeling in the human study by ensuring participants are not conflating correlated but distinct preference dimensions (e.g., "family" topic with "good" alignment).

4. Key Insights and Innovations

Innovation 1: A General-Purpose Preference Model as an RL Reward for Open-Ended Controllable Generation

The paper's most distinctive conceptual move is the claim that a contrastively trained bi-encoder — trained once on a large, heterogeneous corpus of story-critique pairs — can serve as a general-purpose reward function for reinforcement learning across a wide range of preferences, eliminating the need to collect per-preference annotated datasets for each new controllable generation task.

This is not merely an engineering convenience. It represents a fundamental shift in how to think about the relationship between preference specification and generation control. Before this work, the dominant paradigm for controllable text generation with RL or classifier guidance treated the reward model as task-specific: you wanted to control for a particular attribute (e.g., topic, sentiment, formality, non-toxicity), so you trained a classifier on a labeled dataset for that attribute, and you used that classifier's predictions as the reward. This is the approach in Tambwekar et al. (2019b) for story goal completion, in Peng et al. (2020) for reducing non-normative text, and conceptually in GeDi (Krause et al., 2020), where the discriminator is trained on attribute-labeled data. The underlying assumption is that the reward model and the preference are 1-to-1: new preference means new dataset, new training run, new reward model.

The CARP-based approach breaks this assumption by decoupling the preference model from the preference instance. CARP learns a joint embedding space where both stories and natural language critiques are mapped to vectors on the unit hypersphere, with cosine similarity indicating alignment. Because critiques are just text — not fixed class labels — any preference that can be expressed in natural language can, in principle, be scored by comparing the story embedding to the critique embedding. This is the CLIP-for-text paradigm: just as CLIP enables zero-shot image classification by embedding arbitrary class descriptions and comparing them to image embeddings, CARP enables zero-shot preference scoring by embedding arbitrary critique text and comparing it to story embeddings. The key intellectual move is recognizing that this embedding-space comparison is not just useful for evaluation or retrieval (as in the original CARP paper, Matiana et al., 2021) but can serve as the reward signal driving a generative policy's optimization.

What makes this non-obvious is that contrastive similarity scores are not designed to be RL rewards. They are continuous, noisy, and miscalibrated — a similarity of 0.6 vs. 0.65 does not carry the same semantic meaning as a reward difference of 0.05 would in a typical RL setup. The paper's contribution is not just proposing the architecture but demonstrating — through the human study in Figures 4 and 5 — that despite these challenges, CARP-based RL fine-tuning can produce generators that outperform both logit-manipulation methods and much larger prompted models on preference conformance. This is an existence proof that general-purpose embedding-space similarity can substitute for task-specific classification as a generation reward, with significant implications for the scalability of controllable generation: you no longer need to annotate a dataset every time you want to control for a new preference.

The significance extends beyond story generation. If a contrastive model trained on (output, critique) pairs can drive RL for story preferences, the same architecture could, in principle, drive RL for preferences in code generation (paired with code reviews), dialogue (paired with conversational feedback), or summarization (paired with quality critiques) — any domain where natural language feedback on generated outputs is available at scale. The paper opens a door to a class of preference-learning systems where the reward model is trained once on a large, general corpus of (output, natural language feedback) pairs and then deployed for arbitrary downstream preferences without retraining.

Innovation 2: Diagnosing and Addressing the "Weak Reward Signal" Problem Through Unsupervised Discretization of the Embedding Space

The paper's second conceptual contribution is identifying a specific, non-obvious failure mode of contrastive reward models — the weak continuous signal problem — and developing a principled, unsupervised method to convert that weak signal into a strong, class-based reward without sacrificing generality.

The diagnostic insight is stated plainly in Section 5: "CARP produces scores along a continuous range with many middling values. As a result CARP provides a relatively weak classification signal... This makes it challenging for PPO to discriminate between continuations. Consequently, the CARP-tuned model often fails to learn to meet some criteria and overfits for others." This is not a generic complaint about reward noise — it is a specific claim about the distribution of cosine similarity scores in the CARP embedding space. The model places many semantically distinct story-critique pairs in a narrow band of intermediate similarity (say, 0.4–0.6), where the reward difference between a "good" generation and a "mediocre" one is too small to drive meaningful policy improvement. The RL optimizer cannot climb a hill it cannot see.

Prior work on RL for text generation had encountered reward signal quality issues, but the standard responses were either (a) collect more or better annotated data to train a stronger classifier, or (b) use reward shaping to amplify differences. The paper's insight is that the CARP embedding space itself already contains coherent preference structure — it is just expressed in a continuous, overlapping way that is ill-suited to RL optimization. The problem is not that CARP lacks knowledge about preferences; it is that the knowledge is encoded in a format that RL cannot efficiently use. The solution is not to retrain CARP or collect more data, but to explicate the implicit structure already present in the embedding space.

This is where the pseudo-labeling pipeline becomes intellectually significant rather than merely technical. The sequence of UMAP dimensionality reduction → HDBSCAN clustering → hand-labeling and centroid computation → thresholded classification is not an arbitrary processing chain. It is a method for reading out the categorical structure latent in a continuous contrastive embedding space. The fact that HDBSCAN discovers 91 semantically coherent clusters — corresponding to recognizable story features like "humor," "fighting," "imagery," "family," "music" — is evidence that CARP's training on 1.3 million story-critique pairs has implicitly organized the critique embedding space into preference-relevant categories, even though CARP was never explicitly trained to classify. The pseudo-labeling pipeline makes the implicit explicit.

This is a fundamentally different approach to reward design than the standard RLHF paradigm. In RLHF, you train a reward model to directly predict human preference scores, and the reward model's quality depends entirely on the quality and quantity of your human preference data. In the CARP CoOp approach, the underlying embedding space is trained contrastively on natural co-occurrence (stories and their critiques), and the reward classifier is derived from clustering that space — a form of unsupervised reward discovery. The connection to CLIP CoOp (Zhou et al., 2021) is more than methodological borrowing; it represents a shared insight that contrastively trained embedding spaces contain richer categorical structure than their training objective explicitly enforces, and that prompt tuning can extract this structure for downstream tasks without supervised fine-tuning of the encoder.

The significance is both practical and conceptual. Practically, it means you can build a strong, class-based reward model from an unlabeled corpus of (output, feedback) pairs — no human annotation of preference categories required. Conceptually, it suggests that contrastive learning on natural language feedback implicitly performs unsupervised preference discovery, and that the right interrogation method (clustering + prompt tuning) can surface this structure for downstream use. This reframes contrastive bi-encoders not merely as similarity scorers but as latent preference spaces that can be mined for categorical reward signals — a perspective that has implications beyond story generation for any domain where paired (output, natural language evaluation) data exists.

The ablation study in the human evaluation provides direct evidence for this claim. The Default CARP LM — which uses raw CARP cosine similarity as the reward without CoOp discretization — underperforms CARP CoOp LM on nearly every preference category (Figures 4 and 5, Table 2). The Default CARP LM produces stories that are sometimes indistinguishable from the NeoX baseline and sometimes worse, validating the paper's claim that the weak continuous signal, left untreated, is insufficient to drive reliable preference learning. The CoOp intervention — clustering + prompt tuning — transforms a reward model that sometimes works into one that consistently works, with the human study showing CARP CoOp LM preferred in all cases except religion topics (Figure 4). This is the paper's central technical insight made empirical: structure extraction, not more data or bigger models, is what makes the contrastive reward approach viable.

Innovation 3: Empirical Demonstration That Controlled Optimization Outperforms Raw Scale for Subjective Preference Conformance

The headline result — that a GPT-2-750M model fine-tuned with CARP CoOp rewards matches or exceeds a prompted GPT-NeoX-20B model, a ~20× parameter increase, on human-judged preference conformance — is more than an impressive number. It is an empirical refutation of the implicit assumption that bigger models are better instruction-followers for all task types, and a demonstration that for open-ended, subjective generation tasks, the mechanism of control can matter more than the capacity of the generator.

This finding challenges a narrative that had been gaining traction from the scaling and instruction-tuning literature: that larger language models, particularly those fine-tuned to follow instructions (RLHF, instruction tuning), can be reliably steered through natural language prompts alone. The paper's NeoX baseline represents this paradigm: take a very large model (20B parameters), give it a prompt that includes both the story start and the preference criterion, and trust that the model's few-shot instruction-following capabilities will produce appropriate output. The results (Figures 4 and 5) show this works inconsistently. On some preferences — certain topics, certain alignments — NeoX produces stories that human evaluators can correctly identify as matching the target preference with high reliability. On others, it performs near chance. The variability is not subtle: across the six topic labels in Figure 4, NeoX's human preference identification rate swings from strong (apparently > 0.6 for some topics) to weak (~0.3 for others).

The CARP CoOp LM, by contrast, produces stories that are consistently identifiable as matching the target preference, with less variance across categories (with the noted exception of religion). This consistency — not just average performance — is the key result. A creative writing assistant that works brilliantly for "sad stories about family" but ignores your request for "vivid sensory imagery" is not a reliable tool. The CARP-based models, by explicitly optimizing against a preference signal at training time, are forced to attend to the preference in a way that prompted models are not. The 20× parameter disadvantage is overcome by the compulsion of optimization.

What makes this intellectually significant beyond the specific story generation domain is that it identifies a class of tasks where prompt-based control is structurally unreliable, regardless of model scale. A prompted model generates text by maximizing $P(\text{continuation} \mid \text{prompt})$ — the probability of the continuation given the entire context. If the model's training distribution contains many examples where a story with certain surface features (e.g., containing dialogue, set in a particular genre) was not written to satisfy a specific preference, the model may learn to down-weight the preference component of the prompt in favor of more predictive features. There is no gradient pushing the model to satisfy every constraint in the prompt; the model is a conditional distribution, not an optimizer. As models get larger, their conditional distributions may become sharper and more accurate, but they are still distributions — they assign probability to what is typical, not what is specified. For subjective, open-ended preferences where the relationship between the preference language and the story features is complex and underspecified by the training data, this probabilistic approach fundamentally underdetermines the desired output.

The CARP-based approach sidesteps this by making preference satisfaction the optimization objective, not a conditioning variable. The model is not asked to generate a story that is probably sad given the prompt; it is trained to generate stories that the CARP reward model judges as sad, with gradient updates specifically reinforcing behaviors that increase that judgment. This transforms the problem from conditional generation (where preference is one of many conditioning factors that the model may or may not attend to) to constrained optimization (where preference satisfaction is the explicit goal that drives parameter updates). The paper's human study provides evidence that this distinction matters in practice for subjective preference categories — and that the advantage of explicit optimization over conditional prompting can outweigh a factor of 20 in model scale.

This is an incremental but important refinement of the "scale is all you need" narrative. It does not claim that scale is irrelevant — the base model must still be capable of generating coherent stories, and on the hardest problems (completely out-of-distribution preferences), the CARP approach would presumably fail just as prompted models do. But it establishes a clear boundary condition: for preferences within the model's capability range, how you enforce the preference matters at least as much as how large the model is. For practitioners, this suggests that investing in preference-enforcement mechanisms (reward models, RL fine-tuning, classifier guidance) may yield better returns for controllable generation than simply scaling to the next order of magnitude of parameters.

Innovation 4: Data-Efficient Preference Learning Through Frozen Encoder with Prompt Tuning

The paper's demonstration that CARP CoOp can learn a strong preference classifier from only 1,000 examples per class — while keeping the underlying CARP encoder frozen — constitutes a distinct contribution to the literature on adapting large pre-trained models for new tasks. This is not just a hyperparameter efficiency finding; it has architectural implications for how we should think about building general-purpose preference models that can be rapidly adapted to novel user preferences.

The standard approach to adapting a pre-trained model for a new classification task is fine-tuning: update all or most of the model's parameters on the new labeled data. This works well when you have substantial labeled data and the target task is reasonably close to the pre-training distribution. But it has well-known failure modes: with small datasets, it leads to overfitting and catastrophic forgetting of the pre-trained knowledge that made the model useful in the first place. The paper's alternative — freeze the encoder, add a small set of learnable prompt embeddings, and train only those embeddings — avoids these failure modes while achieving competitive performance.

What makes this finding distinctive is the interaction between the CoOp prompt tuning and the CARP encoder's pre-training objective. The CARP encoder was trained to maximize cosine similarity between matched (story, critique) pairs and minimize it for mismatched pairs. It learned, implicitly, how critique language maps onto story features in a high-dimensional embedding space. The CoOp prompts do not need to relearn this mapping; they only need to learn how to construct synthetic critique sequences that, when passed through the frozen encoder, produce embeddings in the appropriate region of the space for each preference class. The CoOp layer is, in effect, learning to "write" a prototypical critique for each class — not in natural language, but in the embedding tokens that the encoder can process — such that the resulting vector points toward the cluster centroid for that class.

This is a fundamentally different relationship between the pre-trained model and the adaptation mechanism than standard fine-tuning. In fine-tuning, the pre-trained weights serve as a good initialization that is then modified to better fit the target task, with the risk that the modification erases useful pre-trained knowledge. In frozen-encoder CoOp, the pre-trained weights serve as a fixed transformation — a learned function from critique language to preference-relevant embedding coordinates — and the adaptation mechanism learns to supply inputs to that transformation that produce the desired outputs. The pre-trained knowledge is never at risk of being overwritten because the encoder weights are never updated. This is conceptually similar to how prompting works for generative LMs (you don't change the model, you change the input to elicit the desired behavior), but applied to the encoder side of a bi-encoder architecture.

The practical implication is that a single, large, general-purpose contrastive preference model — trained once on a diverse corpus of (output, natural language feedback) pairs — could be rapidly adapted to novel preference categories with minimal labeled data via prompt tuning, without retraining or fine-tuning the underlying model. Each new preference category would require only a small set of labeled examples and a few learnable embedding vectors, not a full classifier training run. This moves toward a vision of preference models as infrastructure: train one CARP-like model on all available (story, critique) data, and then deploy it for arbitrary downstream preferences with lightweight, per-preference prompt tuning. The paper's demonstration on the Moral Stories alignment dataset — where Alignment CARP CoOp achieves strong performance with only 1,000 examples per alignment class, despite CARP never having been trained on moral alignment critiques — provides preliminary evidence that this vision is feasible.

The negative result in Appendix K (ReST^EM fine-tuning degrading performance) further underscores the significance of the frozen-encoder approach. When the paper attempted to further optimize the revision model using on-policy data collection, performance degraded substantially, likely because the on-policy data amplified spurious correlations. The frozen encoder, by contrast, provides a stable, generalizable reward signal precisely because it is not updated on the narrow distribution of the fine-tuning data. This suggests that for preference-based RL, reward model stability — achieved through frozen pre-trained encoders — may be more important than reward model accuracy on the fine-tuning distribution, a finding with implications for the broader RLHF literature where reward model drift during policy optimization is a known challenge.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The primary training corpus for the preference model is the Story Critique dataset, consisting of more than 80,000 unique stories with 1,378,696 total story-critique pairs. Each critique refers to a specific passage of the story. For the alignment preference task, the paper uses the Moral Stories dataset (Emelin et al., 2021), containing 12,000 short narratives, which is augmented via GPT-J-6B few-shot prompting to produce 17,157 story-alignment pairs labeled as good, neutral, or evil. The base story generator is trained on the ROCStories corpus (Mostafazadeh et al., 2016), 100,000 five-sentence everyday narratives, with 5% held out as a validation set that also serves as the prompt source for RL training and evaluation.

  • Base model(s). The generative model is GPT-2-750M (referred to alternately as GPT-2-large), initially fine-tuned on ROCStories to produce coherent basic narratives. The preference model is CARP (Matiana et al., 2021), a contrastively trained bi-encoder with separate story and critique encoders producing unit-normalized embeddings in R^2048, trained on the Story Critique dataset. For CARP CoOp variants, the CARP encoder weights are frozen, with only the prompt embeddings trained. For baselines: GPT-NeoX-20B (Black et al., 2022) serves as the large prompted model, and GeDi (Krause et al., 2020) provides the logit-manipulation baseline.

  • Metrics. The primary evaluation metric is human preference identification rate: the fraction of time human study participants correctly identify which target preference was used to generate a given story segment from a list of candidate preferences. For topic evaluations, participants select among six labels (imagery, fighting, music, family, accidents, religion). For alignment evaluations, they select among three labels (good, neutral, evil). These are reported as percentages or proportions, broken down per preference category and per model class. Inter-annotator agreement is measured at 0.74 average across all questions, computed as the agreement rate between annotators on preference-criteria matching.

  • Baselines. Four model classes are evaluated: (1) GeDi LM — GPT-2-750M guided via GeDi topic classification models (pre-trained for topic baselines, fine-tuned on the Alignment dataset for alignment baselines); (2) NeoX — GPT-NeoX-20B prompted with the preference criteria in natural language plus an initial story sentence, relying on few-shot instruction following; (3) Default CARP LM — GPT-2-750M fine-tuned via PPO using raw CARP cosine similarity as the reward, serving as an ablation of the CoOp discretization; (4) CARP CoOp LM — GPT-2-750M fine-tuned via PPO using Pseudo CARP CoOp classification scores (for topics) or Alignment CARP CoOp classification scores (for alignments) as the reward — the full proposed system.

  • Generation budget / compute accounting. All RL fine-tuning runs share the same generation budget: 20,000 PPO steps with 64 trajectories sampled per step from the experience replay buffer, each trajectory being a 60-token continuation generated from a 5-token ROCStories prompt. Training takes "on the range of an hour on a single A100 GPU" (Section 4). For the human evaluation, 20 story segments are generated per preference per model, using the same 5-token ROCStories prompt sampling strategy. The paper does not compare methods under a FLOPs-matched generation budget; the comparison across model classes is based purely on human preference identification rates from a fixed set of generated stories, not on equalized computational cost.

  • Cross-validation / statistical protocol. No statistical significance tests (e.g., confidence intervals, t-tests, bootstrap) are reported for the human study results. The paper's strategy selection for CoOp (choosing which pseudo-labels to include for training) is based on hand-labeling and semantic dissimilarity assessment, not on cross-validated performance optimization. The 5% ROCStories validation holdout is used for prompt sampling during RL training rather than for traditional hyperparameter tuning or early stopping. Inter-annotator agreement (0.74 overall) is reported but not used to calibrate or weight the primary preference identification metric.

Main Quantitative Results

Topic Preference Human Study Results

The central result for topic preferences appears in Figure 4 (a histogram of human preference identification rates) and Table 2 (aggregate percentages). The CARP CoOp LM achieves the highest aggregate topic preference identification rate at 0.615 (61.5%), compared to GeDi LM at 0.371, NeoX at 0.489, and Default CARP LM at 0.533 (Table 2, "Topics" row). This represents a substantial margin over all baselines: an absolute improvement of 24.4 percentage points over GeDi, 12.6 points over NeoX, and 8.2 points over the Default CARP ablation.

Examining the per-preference breakdown in Figure 4 reveals substantial variance across categories and methods. For the imagery preference, CARP CoOp LM achieves a human identification rate near 0.85 (estimated from bar height in Figure 4), while NeoX reaches approximately 0.55 and Default CARP LM approximately 0.65. GeDi underperforms all methods on imagery, with an identification rate near 0.4. A similar pattern holds for fighting, where CARP CoOp LM reaches approximately 0.72, NeoX approximately 0.50, Default CARP LM approximately 0.60, and GeDi approximately 0.38.

The family preference category shows CARP CoOp LM at roughly 0.68, with NeoX much closer at approximately 0.60 — a tighter gap than on imagery or fighting. Default CARP LM achieves approximately 0.55 and GeDi approximately 0.32. For music, CARP CoOp LM reaches roughly 0.62, NeoX approximately 0.48, Default CARP LM approximately 0.55, and GeDi approximately 0.42. For accidents, CARP CoOp LM achieves approximately 0.58, Default CARP LM approximately 0.55 (narrowing the gap relative to other categories), NeoX approximately 0.42, and GeDi approximately 0.42.

The sole category where CARP CoOp LM does not lead is religion, where NeoX achieves the highest identification rate at roughly 0.58, CARP CoOp LM reaches approximately 0.48, Default CARP LM approximately 0.50, and GeDi approximately 0.40. The paper does not provide a specific explanation for this reversal but notes in Section 7 that "Pseudo CARP LM beat NeoX in creating stories with recognizable criteria in all labels except 'religious' stories for which it seems to be particularly sensitive." The nature of this sensitivity (whether CARP's embedding space poorly captures religious themes, whether the religion pseudo-label was of lower quality, or whether NeoX's training data contained disproportionately many stories with religious themes) is not investigated.

The Default CARP LM (the ablation without CoOp) provides the critical comparison for assessing the value of the CoOp discretization. Across the six topic categories, Default CARP LM's human identification rates fall between NeoX and CARP CoOp LM, sometimes closer to NeoX (as in imagery, where Default CARP at ~0.65 is closer to NeoX at ~0.55 than to CARP CoOp at ~0.85) and sometimes closer to CARP CoOp (as in accidents, where the gap is narrow). The aggregate difference — 0.533 for Default CARP LM vs. 0.615 for CARP CoOp LM — confirms that the CoOp intervention provides a meaningful improvement over raw CARP rewards, though the Default CARP model still outperforms both GeDi and, in aggregate, NeoX, indicating that even the weak continuous CARP reward provides some preference control.

GeDi exhibits the most variable performance. For some topics (music, accidents), it matches or approaches NeoX. For others (imagery, family, fighting), it substantially underperforms all other methods. The paper attributes this to the complexity of the preference: "GeDi does quite poorly, particularly with stories involving action" (Section 7). The imagery preference shows the lowest inter-annotator agreement at 0.58 (Section 7), suggesting that the attribute itself is harder to judge consistently, which may compound GeDi's token-level limitations for diffuse, non-lexical preferences.

Alignment Preference Human Study Results

The alignment preference results in Figure 5 and Table 2 show a similar pattern but with notably different dynamics between the baselines. The CARP CoOp LM achieves the highest aggregate alignment identification rate at 0.675 (67.5%), compared to GeDi LM at 0.561, NeoX at 0.460, and Default CARP LM at 0.506 (Table 2, "Align." row). The absolute improvement over NeoX is 21.5 percentage points — larger than the topic preference improvement — while the improvement over Default CARP is 16.9 points.

Breaking down by alignment class in Figure 5: for good alignment, CARP CoOp LM reaches approximately 0.70, Default CARP LM approximately 0.62, GeDi approximately 0.55, and NeoX approximately 0.48. For neutral alignment, CARP CoOp LM achieves roughly 0.68, GeDi approximately 0.60, NeoX approximately 0.48, and Default CARP LM approximately 0.42. For evil alignment, CARP CoOp LM reaches approximately 0.64, Default CARP LM approximately 0.48, GeDi approximately 0.52, and NeoX approximately 0.42.

A notable pattern emerges: NeoX's alignment identification rates are consistently low across all three classes — approximately 0.48 for "good," 0.48 for "neutral," 0.42 for "evil" — with performance near the chance level for a three-way classification (0.33). This indicates that prompting a 20B parameter model to control character moral alignment is largely ineffective, regardless of the specific alignment target. By contrast, the CARP CoOp LM, which explicitly optimizes against a reward signal trained on the augmented Moral Stories dataset, produces stories whose moral character alignment is correctly identified roughly two-thirds of the time.

The GeDi model, fine-tuned on the Alignment dataset, performs better on alignment than on topics — achieving 0.561 aggregate vs. 0.371 for topics — and actually outperforms NeoX on all three alignment classes. This is consistent with the paper's framing of GeDi's limitations: when annotated data exists for a specific preference, GeDi can use it effectively, but its performance degrades on preferences without dedicated training data. The Default CARP LM shows mixed results on alignment, outperforming NeoX on "good" (~0.62 vs. ~0.48) but underperforming NeoX on "neutral" (~0.42 vs. ~0.48) and roughly matching it on "evil" (~0.48 vs. ~0.42). This inconsistency with the raw CARP reward signal on alignment preferences mirrors the topic preference dynamic and reinforces the motivation for the CoOp discretization.

Comparative Findings Across Baselines

Several cross-cutting observations emerge from Tables 2 and Figures 4-5:

The 20× scale advantage does not translate to preference conformance. Across both topic and alignment evaluations, GPT-NeoX-20B — a model with roughly 20 times the parameters of GPT-2-750M and direct access to the preference text in its prompt — is outperformed by the CARP CoOp LM in aggregate preference identification (0.489 vs. 0.615 for topics; 0.460 vs. 0.675 for alignments). This is the paper's central empirical claim: explicit optimization against a preference signal overcomes a factor of 20 in raw model capacity for subjective, open-ended generation tasks. The variability in NeoX's per-category performance (strong on some topics like religion and family, weak on imagery and all alignments) suggests the prompted model's success depends on accidental alignment between the preference and the model's pre-training distribution, rather than robust instruction-following.

The CoOp discretization provides consistent improvement over raw CARP rewards. In aggregate, CARP CoOp LM outperforms Default CARP LM by 8.2 points on topics (0.615 vs. 0.533) and 16.9 points on alignments (0.675 vs. 0.506). The improvement is not uniform across categories — it is largest on imagery and fighting (topics) and on "evil" and "neutral" (alignments) — but CARP CoOp LM leads or ties Default CARP LM on every category except religion. This supports the paper's claim that the continuous CARP reward signal is "weak" and that discretization into class-based scores produces more reliable preference learning.

The inter-annotator agreement reveals task difficulty heterogeneity. Agreement is highest for the alignment evaluation (implied by the paper's reporting of 0.77 ± 0.01 across model classes except GeDi) and lowest for the imagery topic (0.58). This suggests that some of the variance in per-category performance across methods may reflect genuine ambiguity in the preference categories themselves — if human annotators only agree 58% of the time on whether a story contains "imagery," a model identification rate of 0.85 (CARP CoOp LM on imagery) may indicate overfitting to a narrower interpretation of the preference than the human participants share. The paper does not discuss this calibration concern.

Qualitative Samples

Table 1 provides illustrative generated story segments across models for imagery and good character alignment preferences. While not a quantitative result, the samples demonstrate the qualitative differences in output:

  • For imagery, the CARP CoOp LM sample ("The creature eyed them curiously. Its sullen profile took on a glow, putrefying everything around it to a damnable crispness. It stretched its tentacles out into fishing web light...") contains dense descriptive language with sensory and visual details, consistent with the imagery preference. The NeoX sample for imagery, by contrast, is a mundane narrative about buying pickles that, despite containing some descriptive elements, does not foreground visual description as the primary narrative mode.
  • For good character alignment, the CARP CoOp LM sample ("Hello, is everything ok? my son said one of his cats got loose and ran out the door. we called the neighbor boy over...") depicts prosocial, helpful behavior, while the Default CARP LM sample includes potentially confusing or unsettling elements (a character claiming to be the "reincarnation of a dead man").
  • The GeDi samples across both preference categories show reasonably coherent text but less focused adherence to the preference criteria — consistent with its lower quantitative scores.
  • The NeoX samples show fluent, often lengthy, text that sometimes drifts into unrelated or nonsensical content (e.g., the alignment sample that devolves into a corporate espionage scenario unrelated to character morality).

These qualitative observations are consistent with the quantitative results but are provided as illustrations rather than systematic evidence. No systematic qualitative analysis methodology (e.g., blind expert rating of preference-relevant features, linguistic feature counting) is applied to the full set of generated stories.

Ablation Studies and Robustness Checks

Vanilla CARP vs. CARP CoOp reward signal: The Default CARP LM (PPO with raw CARP cosine similarity) serves as an ablation of the CoOp discretization. Table 2 and Figures 4-5 show that removing CoOp reduces aggregate preference identification by 8.2 percentage points for topics (0.615 → 0.533) and 16.9 points for alignments (0.675 → 0.506). The performance degradation is not uniform: Default CARP LM remains competitive on "accidents" and "religion" topics but substantially underperforms on "imagery," "fighting," and alignment categories. This ablation validates the paper's core claim that the continuous CARP reward signal is insufficiently discriminative for reliable RL optimization and that discretization meaningfully improves preference learning.

Layer freezing depth: The paper states that "freezing all but the last two layers of the language model provided the best result" (Section 4) but does not report ablation results comparing different freezing depths. The hyperparameter configuration ("num_layers_unfrozen": 2) is presented as a fixed choice rather than a sweep-based selection with comparative performance. This is a notable omission: the claim that two layers is optimal lacks supporting evidence, and the sensitivity of performance to this parameter is unknown. A sweep across {1, 2, 4, all} unfrozen layers would clarify whether the preference learning is genuinely concentrated in the upper layers or whether more aggressive fine-tuning would yield larger gains.

KL regularization coefficient: The hyperparameters specify an initial KL coefficient of 0.2 with a target of 50 and horizon of 10,000, suggesting an adaptive schedule. However, no ablation is reported testing different KL coefficients or comparing the adaptive schedule to a fixed coefficient. The paper does not demonstrate that the KL regularization is necessary to prevent reward hacking or collapse, though qualitative examples in the limitations section (Appendix B, Table 1) show that Default CARP LM occasionally produces incoherent output (e.g., "It took him seven days. to sing three songs. on the first day he went to school in the parking lot someone opened the door."), which may indicate insufficient regularization. Without a KL coefficient ablation, the sensitivity of the results to this hyperparameter is unknown.

Number of pseudo-labels selected: The paper selects 6 pseudo-labels from 91 available HDBSCAN clusters, chosen for "semantic dissimilarity and separation in the embedding space" (Section 5.2). No ablation examines whether using more or fewer pseudo-labels — or different selection criteria — affects downstream preference learning. The choice of 6 is not justified by sweep or comparative analysis. The paper notes that "all 91 clusters are distinct enough that were we to train Pseudo CARP CoOp on all pseudo labels, the embeddings would not change much from the original CARP" (Section 8), but this is stated without experimental verification and seems to contradict the paper's own finding that discretization with 6 classes substantially improves performance. If using 91 classes would reproduce the weak continuous CARP signal, there should be a non-monotonic relationship between the number of pseudo-labels and reward signal quality that the paper does not explore.

Threshold for point-class assignment: The paper uses a filtering threshold of "twice the average cosine-similarity of the dataset" to reject weakly associated story-critique pairs during CoOp training data construction (Section 5.2). No ablation examines the sensitivity of CARP CoOp's classification accuracy to this threshold. Varying the threshold would trade off training data quantity against label quality, and the optimal point is not obvious. The absence of this ablation means the reported CoOp performance may be highly sensitive to this unexamined hyperparameter.

Prompt prefix length for generation: All RL training and evaluation uses exactly 5 prefix tokens from ROCStories validation stories. No ablation tests whether different prefix lengths (e.g., 1 token, 10 tokens, full first sentence) affect the quality or preference alignment of generated continuations. Intuitively, longer prefixes provide more context and may constrain generation in ways that interact with preference-specific features. The choice of 5 tokens appears arbitrary and may advantage or disadvantage certain preference categories.

Data efficiency of CoOp training: The paper claims CARP CoOp is "highly data efficient, requiring a minimal number of examples per class to fine-tune when initialized with the pretrained original CARP model" (Section 5.2), based on using 1,000 examples per class. However, no data-efficiency curve is presented — no sweep showing CARP CoOp classification accuracy at 100, 500, 1,000, 5,000 examples per class. The claim of data efficiency is asserted rather than demonstrated experimentally. A proper ablation would show where performance saturates and whether 1,000 was chosen because it is sufficient or simply because it was convenient.

ReST^EM fine-tuning degrades revision model performance (negative result): Appendix K (Figure 16) reports that an attempt to further optimize the revision model using on-policy data generation (ReST^EM) caused performance to degrade: at 256 generations, fully sequential performance dropped to approximately 33.5% compared to approximately 38.5% at the optimal sequential-to-parallel ratio. The authors hypothesize that on-policy data collection "exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly" (Appendix K). This negative result is an important robustness check demonstrating that the positive CARP CoOp results depend on the specific offline data construction procedure and may not generalize to more sophisticated self-improvement loops.

GeDi's performance drops on out-of-distribution preferences: While not a deliberate ablation, the GeDi baseline's variable performance across topic categories (Figure 4) effectively serves as a robustness check on the claim that logit-manipulation methods require annotated data aligned to specific preferences. GeDi performs adequately on in-distribution or lexically-salient topics but degrades substantially on abstract preferences like imagery and fighting, consistent with the paper's framing of GeDi's limitations. The inter-annotator agreement drop for GeDi-guided stories (0.62 vs. 0.77 for other models, Section 7) further suggests that GeDi's inconsistent performance creates confusion about which preference a story was meant to satisfy — a metric-level consequence of its token-level bias mechanism.

Critical Assessment

The experiments reported in Sections 6-7 provide moderate support for the paper's central claims, but several important caveats limit the strength of the evidence and the scope of the conclusions that can be drawn.

Claim: CARP CoOp LM outperforms a 20× larger prompted model on preference conformance. The human study data in Figures 4-5 and Table 2 do show that CARP CoOp LM achieves higher aggregate preference identification rates than GPT-NeoX-20B (0.615 vs. 0.489 for topics; 0.675 vs. 0.460 for alignments). However, what the experiment demonstrates is narrower than what the claim implies. The evaluation measures whether human readers can correctly identify the intended preference from a story segment. This conflates two distinct capabilities: (a) the generator's ability to produce text that exhibits the preference, and (b) the generator's ability to produce text where the preference is recognizable given a set of distractor categories. A model could achieve high identification rates by producing exaggerated, stereotypical expressions of a preference that readers easily recognize but that represent an impoverished or caricatured version of the desired attribute. The paper provides qualitative samples (Table 1) but no analysis of whether high-identification-rate stories are genuinely better expressions of the preference or merely more obvious ones.

Moreover, the NeoX baseline is not optimized for the task. The paper provides no details on the specific prompts used to elicit preference-conforming stories from GPT-NeoX-20B. The quality of prompted generation is known to be highly sensitive to prompt wording (the "prompt engineering" problem the paper itself identifies in Section 1), and a single prompt formulation per preference — presumably hand-crafted by the authors — may substantially underrepresent NeoX's capability. A fairer comparison would either optimize the NeoX prompts through iteration (as a user of the system would) or test multiple prompt formulations per preference and report the best or average. The paper reports none of this, so the NeoX results represent a single, potentially suboptimal, prompting strategy.

Additionally, the comparison is not compute-matched. The CARP CoOp LM requires: (1) pretraining CARP on 1.3M pairs, (2) UMAP + HDBSCAN clustering, (3) CoOp prompt tuning for each preference class, (4) PPO fine-tuning for 20K steps per preference, and (5) per-preference specialized models. The NeoX baseline requires a single forward pass per story. The paper's framing of "20× smaller model outperforms 20× larger model" obscures this enormous difference in development and deployment cost. A more informative comparison would account for the total computational investment required to reach the reported performance levels.

Claim: The CoOp discretization is necessary because raw CARP rewards are too weak for effective RL. The Default CARP LM ablation in Table 2 (0.533 topics, 0.506 alignments vs. CARP CoOp LM at 0.615 and 0.675) supports the claim that CoOp improves performance. However, the Default CARP LM still outperforms NeoX in aggregate (0.533 vs. 0.489 topics, 0.506 vs. 0.460 alignments), which complicates the narrative. If the raw CARP reward is "too weak" for effective learning, why does the Default CARP LM beat a 20B parameter model? One interpretation is that the continuous signal is weak but not useless — it provides some preference guidance, and the CoOp discretization amplifies this guidance. Another interpretation is that the Default CARP LM's advantage over NeoX reflects the power of explicit optimization (even with a noisy reward) over prompting, with the CoOp providing incremental rather than categorical improvement. The paper does not distinguish between these interpretations.

The per-category breakdown reveals cases where Default CARP LM performs comparably to CARP CoOp LM (accidents, religion) and cases where it substantially underperforms (imagery, fighting, evil, neutral). This suggests that the "weak signal" problem is preference-dependent: CARP's continuous scores are sufficiently discriminative for some preferences but not others. The paper does not analyze what properties of a preference make it amenable to continuous vs. discrete reward optimization, which would strengthen the diagnostic claim and guide practitioners in deciding whether CoOp-style discretization is necessary for their use case.

Claim: CARP provides a general-purpose, data-efficient preference model for controllable generation. The evidence for generality is limited by the narrow range of preferences evaluated. The paper tests 6 topic labels and 3 alignment labels — 9 preference categories total. The claim of generality would be stronger with evaluation on: (a) preferences that are linguistically complex or abstract (e.g., "the story should feel melancholy without being depressing," "the pacing should accelerate toward the climax"), (b) preferences that combine multiple attributes (e.g., "a funny story with dark undertones"), (c) preferences that are completely absent from CARP's training data (the Moral Stories alignment categories are arguably novel, but the paper demonstrates they work with CoOp fine-tuning, not zero-shot). The failure on the religion topic (Figure 4), where CARP CoOp LM underperforms NeoX, hints that the generality is not complete, and the paper does not investigate why this category is an outlier.

The evidence for data efficiency is asserted rather than demonstrated. The paper reports that 1,000 examples per class suffice for CoOp training but provides no data-efficiency curve, no comparison to alternative classifier training approaches with similar data budgets, and no analysis of how performance degrades as the number of training examples decreases. The frozen-encoder CoOp architecture is plausibly data-efficient, but the paper does not experimentally establish this as a finding.

Missing experiments that would strengthen the paper:

  • Multi-preference conditioning: All CARP CoOp models are trained for a single preference. A demonstration that a single model could be conditioned on different preferences at generation time (by computing rewards against different CoOp class embeddings) and produce appropriately different stories would strengthen the claim of generality.
  • Human evaluation of story quality, not just preference identification: The study measures whether readers can identify the intended preference, not whether the stories are good stories. A preference-optimized model might produce recognizable but low-quality outputs. Evaluating overall story quality (coherence, engagement, creativity) alongside preference conformance would provide a more complete picture.
  • Statistical significance testing: The human study involves 25 participants reading 44 stories each, yielding a total of 1,100 judgments distributed across 36 model-preference combinations (roughly 30 judgments per combination). With this sample size and no reported confidence intervals, it is impossible to assess whether the reported differences between methods (especially close comparisons like Default CARP LM vs. CARP CoOp LM on accidents) are statistically reliable or within sampling noise.
  • Ablation of the hand-labeling step: The pseudo-label pipeline involves manual cluster inspection and merging. An experiment comparing the hand-labeled 6-class selection to (a) a random 6-class selection, (b) an automated selection based on cluster size or cluster separation, and (c) the full 91-class set would clarify how much the manual curation contributes to downstream performance.
  • Direct comparison to RLHF: The paper positions CARP as an alternative to collecting human preference annotations for reward modeling. A comparison against a reward model trained on human preference judgments for the same set of preferences would contextualize the performance of the CARP-based approach relative to the dominant paradigm in the field.

Conditional nature of the claims: The paper's findings are conditioned on several factors that limit their generalizability. All experiments use a single base model architecture (GPT-2) at a single scale (750M) fine-tuned from a single story dataset (ROCStories) evaluated on preferences derived from a single critique dataset (Story Critique). The CARP model is trained on short-form creative writing and in-line critiques, which the paper acknowledges limits its applicability to longer-form or sequential preferences (Section 8). The evaluation relies on a specific set of 6 pseudo-labels chosen for embedding-space separation, and the paper notes that these "represent an 'upper-bound' of sorts regarding downstream language model performance" (Section 8). The results for non-evaluated preference categories — particularly those with less coherent clusters in CARP's embedding space — may be substantially worse. Finally, the human study uses a specific protocol (44 stories per participant, multiple-choice preference identification) that may advantage some methods over others in ways that do not reflect real-world deployment scenarios where users have a specific preference in mind and judge whether the output satisfies it, rather than guessing which preference it was meant to satisfy.

6. Limitations and Trade-offs

The Pseudo-Labeling Pipeline Is Fragile, Labor-Intensive, and Not Validated for Generalization

The CARP CoOp approach depends critically on a sequence of unsupervised clustering steps — UMAP dimensionality reduction, HDBSCAN clustering, manual cluster inspection and merging, centroid computation, and thresholded point-class assignment — to transform the continuous CARP reward signal into discrete preference classes. This pipeline introduces multiple unvalidated degrees of freedom, and the paper provides little evidence that the resulting classifier would generalize to new datasets, new preference categories, or even new stories from the same distribution.

What the paper assumes. The pipeline treats the clusters discovered by HDBSCAN in the 2D UMAP projection of CARP's critique embedding space as semantically meaningful preference categories. The authors acknowledge the manual nature of this process:

"We hand-label clusters by sampling several critiques from each. Any cluster where an associated story feature was ambiguous was discarded. Clusters with identical story features were merged." (Section 5.1)

This hand-labeling is not a minor curation step — it is the mechanism by which 91 automatically discovered clusters are filtered and merged into 6 usable preference classes. The choice of which clusters to keep and which to discard is based on human judgment of "semantic dissimilarity and separation in the embedding space" (Section 5.2), with no quantitative criterion, no inter-annotator agreement on the labeling process, and no validation that different labelers would produce the same 6-class taxonomy.

The consequence. The approach inherits all the fragility of unsupervised clustering applied to high-dimensional data after aggressive dimensionality reduction. UMAP to 2 dimensions is a lossy projection chosen because it "minimized the proportion of embeddings left unclustered by HDBSCAN" (Appendix E) — a pragmatic criterion that optimizes for clusterability, not for preserving the semantic structure relevant to preference discrimination. The fact that HDBSCAN "failed to cluster half of the reviews it was given" (Section 5.1) means that the pseudo-labels represent only those preferences that happen to form dense clusters in the 2D projection. Preferences that are real but distributed diffusely in the embedding space — perhaps because they are more nuanced, more context-dependent, or less frequently expressed in the training data — are discarded into the noise cluster. The system can therefore only optimize for preferences that correspond to large, well-separated clusters in a particular 2D projection of a particular model's embedding space.

More practically, a practitioner who wants to add a new preference category — say, "stories with unreliable narrators" — cannot simply specify it in natural language and expect CARP CoOp to score it. They would need to: (1) verify that the preference forms a coherent cluster in CARP's embedding space, (2) determine whether it is distinguishable from existing clusters, (3) collect and threshold examples, (4) train a new CoOp classifier, and (5) run a new PPO fine-tuning. This is a far cry from the zero-shot preference scoring that the raw CARP model theoretically enables and that the paper's framing implies.

Evidence in the paper. HDBSCAN produces 91 clusters, but the paper selects only 6, discarding approximately 93% of the discovered structure. The criteria for selection — "semantic dissimilarity and separation" — are qualitative. No experiment validates that the 6 selected clusters are representative of the embedding space's structure or that different selection criteria would produce comparable downstream results. The paper notes that "all 91 clusters are distinct enough that were we to train Pseudo CARP CoOp on all pseudo labels, the embeddings would not change much from the original CARP" (Section 8), but this is presented without evidence and appears to be a speculation about what would happen, not an experimental finding.

The religion preference category — where CARP CoOp LM underperforms NeoX (Figure 4) — may be a case where the clustering-based discretization actually degrades the signal. If the "religion" cluster was less coherent or overlapped with other clusters, the CoOp classifier trained on it might produce noisier rewards than the raw CARP cosine similarity, explaining the reversal. The paper does not investigate this.

Mitigation status. The paper does not attempt to automate the cluster selection process, to validate the chosen taxonomy against external criteria, or to demonstrate that the pipeline generalizes to preferences beyond the 6 selected. Section 8 acknowledges that the 6 chosen clusters "represent an 'upper-bound' of sorts regarding downstream language model performance," implicitly conceding that preference categories with less well-separated clusters would perform worse — but no lower bound or sensitivity analysis is provided. Future work on automating cluster selection or learning the discretization end-to-end is not suggested.

The Evaluation Only Measures Preference Recognizability, Not Story Quality or Genuine Preference Satisfaction

The human study asks participants to perform a multiple-choice preference identification task: given a story segment and a list of candidate preferences, select which preference the story was generated to satisfy. This measures whether the preference is recognizable when contrasted against distractors — not whether the story is a good story or whether it genuinely satisfies the preference in a meaningful way. A generator could achieve high identification rates by producing exaggerated, stereotypical, or caricatured versions of each preference that are easy to classify but represent impoverished creative output.

What the paper measures. The primary metric is "the percentage of the time participants correctly selected the correct criteria for each story segment read" (Section 7). This is a classification accuracy metric, not a quality or satisfaction metric. The study design provides participants with a closed set of candidate preferences and asks them to match stories to preferences. This structure may artificially inflate identification rates for models that produce formulaic or stereotypical expressions of preferences, since such outputs are easier to classify correctly in a forced-choice paradigm.

The consequence. A generator optimized against this evaluation might converge to producing stories that are obviously about fighting (lots of punching, weapons, combat verbs) rather than stories that are good stories involving fighting (with tension, stakes, emotional weight, narrative arc). The evaluation does not distinguish between these cases. A user who requests "a story with vivid imagery" and receives a paragraph that is dense with adjectives and sensory words but narratively incoherent would likely be unsatisfied, even if human evaluators can correctly identify the paragraph as targeting the "imagery" preference at well above chance rates. The preference identification metric captures discriminability but not quality or fidelity.

The qualitative samples in Table 1 are consistent with this concern. The NeoX-generated "good character alignment" story begins as a coherent narrative about volunteering at a shelter but devolves into confusion: "Before the icing was even done? You licked the bowl. But the tears came as swift as the voice in your head." The CARP CoOp LM sample for "good alignment" ("Hello, is everything ok? my son said one of his cats got loose...") is more straightforwardly prosocial but flat and simplistic — competent but uninspired. Neither the evaluation protocol nor the paper's analysis addresses whether any of these stories are good stories from a literary or reader-engagement perspective.

Evidence in the paper. The inter-annotator agreement data provides indirect evidence. The imagery preference category has the lowest agreement at 0.58, which the paper attributes to "the complexity of the task" (Section 7). But low agreement on imagery could also indicate that the preference itself is inherently subjective — what counts as "vivid imagery" to one reader may not to another — and that optimizing for a classifier's notion of imagery may produce text that only a narrow subset of readers would endorse. The fact that CARP CoOp LM achieves approximately 0.85 identification on imagery (Figure 4) while human agreement is only 0.58 suggests the model may be producing a stylized, recognizable version of "imagery" that humans can identify but that does not align with a shared, nuanced understanding of the preference.

The paper reports no metrics for overall story coherence, grammaticality, narrative interest, or any standard NLG evaluation dimension beyond preference identification. The single-sentence observation that models "learn through reinforcement... to prefer language and story structures that are more complex than the original ROCStories dataset" (Section 7) is qualitative and unmeasured.

Mitigation status. The paper does not address this limitation. No holistic story quality evaluation is performed, and no metric distinguishes between "recognizably satisfies the preference" and "satisfies the preference in a high-quality way." The human study design — forced-choice preference identification — is standard for evaluating controllable generation but is insufficient on its own to validate a creative writing system where output quality is as important as constraint satisfaction.

The Method Requires Per-Preference Specialized Models, Not a Single Controllable Generator

Every CARP CoOp LM evaluated in the paper is the product of a separate PPO fine-tuning run targeting a single preference category. The "fighting" model was trained to maximize the CARP CoOp score for the fighting class; the "good alignment" model was trained to maximize the score for the good class. There is no mechanism to condition a single model on different preferences at generation time — to say "generate a story about family" versus "generate a story with fighting" using the same set of weights.

What the paper builds. The training hyperparameters include a fixed "review" field specifying the target preference:

"review: 'evil'" (Appendix A)

Each training run optimizes the policy against a single reward function — the CARP CoOp score for that specific class. The PPO objective maximizes the expected reward for that class, with no conditioning variable that would allow the model to learn a mapping from preference specification to preference-conforming output. The result is 9 separate specialized models (6 topics + 3 alignments), each capable of generating stories for exactly one preference.

The consequence. This is a significant practical limitation that undermines the paper's framing as a system for "general purpose human preference modeling" (Section 1). A deployed writing assistant built on this architecture would require the user to select a pre-trained model for each preference they want to express, rather than specifying the preference in natural language as part of the prompt. If a user wanted to generate stories with different preferences in different sessions — or combine preferences ("a sad, imagery-rich story about family with a good protagonist") — the system as built provides no mechanism to do so. The contrast with the NeoX baseline is instructive: NeoX can accept arbitrary preference specifications in its prompt and (inconsistently) produce appropriately conditioned output using a single set of weights. The CARP CoOp LM achieves higher reliability but sacrifices flexibility.

This tradeoff — reliability vs. flexibility — is not discussed in the paper. The framing in Section 1 emphasizes that CARP "learns to align a story with a corresponding human critique" and that it can serve as a "general purpose preference model for generation," which implies a level of flexibility (any critique can be used as a reward target at generation time) that the system as implemented does not deliver. Each preference requires a dedicated training run. Adding a new preference requires: (1) verifying cluster coherence in CARP's embedding space, (2) training a new CoOp classifier if needed, (3) running a full 20K-step PPO fine-tuning on an A100, and (4) deploying a new specialized model.

Evidence in the paper. The hyperparameter specification in Appendix A confirms that "review" is a fixed training parameter, not a conditioning variable. The evaluation protocol generates 20 story segments per preference per model, with each model trained for exactly one preference, confirming the per-preference specialization. The paper does not report any experiment where a single model is evaluated on multiple preferences or conditioned on different preferences at generation time.

Mitigation status. The paper does not acknowledge this as a limitation. The possibility of training a single model conditioned on preferences — for example, by computing rewards against whichever preference the prompt specifies, or by using the CoOp classification scores for all classes as a multi-dimensional reward vector — is not discussed. This is a notable omission given that the prompted NeoX baseline demonstrates multi-preference conditioning is possible (if unreliable) and that the paper's own framing emphasizes generality.

Difficulty Estimation and Computational Overhead Are Not Accounted For in the Headline Comparison

The paper's central empirical claim — that a GPT-2-750M model with CARP CoOp rewards outperforms a GPT-NeoX-20B model on preference conformance — compares systems with vastly different development and deployment costs. The CARP CoOp LM requires a multi-stage pipeline that is never cost-accounted, making the "20× smaller model outperforms 20× larger model" framing misleading as a practical deployment recommendation.

What the pipeline costs. Constructing the CARP CoOp LM for a single preference involves:

  1. CARP pretraining: Training a bi-encoder on 1,378,696 story-critique pairs. While this is a one-time cost amortized across all preferences, it is substantial — the paper does not report CARP's training compute, model size, or training duration.
  2. Embedding and clustering: Running CARP's critique encoder on the full Story Critique dataset, applying UMAP to reduce ~1.4M 2048-dimensional embeddings to 2D, running HDBSCAN, hand-labeling 91 clusters, selecting 6, computing centroids. This is a manual-computational hybrid pipeline with non-trivial human effort.
  3. CoOp training: Fine-tuning the CoOp prompt embeddings on 6,000 curated examples (1,000 per class) with KL-divergence loss.
  4. PPO fine-tuning: 20,000 steps of PPO on an A100 GPU (~1 hour per preference per the paper). This must be repeated for each new preference category.
  5. Per-preference model storage: 9 separate fine-tuned model instances must be stored and served.

By contrast, the NeoX baseline requires: (1) a pre-trained model loaded from a checkpoint, and (2) a single forward pass per story with a constructed prompt. No additional training, no clustering, no RL loop, no per-preference specialization.

The consequence. The headline comparison is not a comparison between two approaches to controllable generation under equalized resource constraints — it is a comparison between a heavily optimized, preference-specialized system and an off-the-shelf general-purpose model used in its most basic configuration. A practitioner deciding whether to deploy CARP-based RL fine-tuning versus simply using a larger prompted model needs to weigh the development cost (clustering pipeline, CoOp training per preference class, PPO fine-tuning per preference) against the inference cost (one A100-hour of training plus storage for each preference vs. a single larger model that handles all preferences). The paper provides none of the data needed to make this decision — no FLOPs accounting, no latency measurements, no storage or memory comparisons, no analysis of how the total cost scales with the number of preferences a user wants to control.

The comparison also ignores the inference-time cost difference. The CARP CoOp LM generates stories using a 750M parameter model; NeoX uses a 20B parameter model. The NeoX inference is substantially more expensive per token. However, NeoX handles all preferences with one model; the CARP approach requires a separate model per preference, and serving 9 separate 750M models has different infrastructure implications than serving one 20B model. The paper makes no effort to characterize this tradeoff.

Evidence in the paper. Section 4 reports that PPO training takes "on the range of an hour on a single A100 GPU" for 20K steps, but this is the only compute figure provided anywhere in the paper. CARP's pretraining cost, the embedding and clustering pipeline cost, and CoOp training cost are not reported. The NeoX baseline's inference cost is not reported or compared. The FLOPs or wall-clock time to generate the 720 stories for the human study are not reported.

Mitigation status. The paper does not acknowledge this as a limitation. Section 4's brief mention of training time ("on the range of an hour") is presented as evidence of efficiency, not as part of a broader cost accounting. The paper frames the comparison in terms of model parameter count (750M vs. 20B) rather than total computational investment, which systematically advantages the CARP CoOp approach by ignoring its substantial offline costs. The absence of any cost model or resource accounting is a significant gap for a paper whose central claim is that a smaller model with test-time optimization can substitute for a larger model — the "with test-time optimization" part carries costs that are never quantified.

The Approach Provides No Mechanism for Long-Form Generation or Sequential Preference Control

The entire technical pipeline — CARP's training, the CoOp classifier, the PPO fine-tuning, and the evaluation — operates on short story segments of approximately 60 tokens, generated from 5-token ROCStories prompts. There is no mechanism for generating longer stories, for maintaining preference consistency across multiple paragraphs, or for changing preferences mid-story (e.g., "the beginning should be sad, but the ending should be hopeful").

What the paper acknowledges. Section 8 states this limitation explicitly:

"The dataset used to train CARP primarily uses short-form creative writing samples and in-line critiques of these samples. One primary limitation of the model is the inability to reason over longer sequences of text. As such, the model also cannot reason effectively about preferences or critiques as would be applicable to longer form creative writing." (Section 8)

The paper further notes that while longer stories could theoretically be generated by "successively generating segments and pre-pending to the context prompt," and that "in such a case it would be feasible to change the preference or criteria mid-stream" (Section 8), this capability is not implemented or tested.

The consequence. The system as built generates story fragments — 60-token continuations of 5-token prompts — not complete narratives. These fragments may exhibit the target preference locally (a paragraph of vivid description, a scene of fighting, a prosocial character action) but there is no guarantee, and no mechanism to ensure, that the preference would be sustained across a longer narrative. A story that is "about fighting" in one 60-token segment might pivot to unrelated content in the next segment; a "good" character might behave prosocially in one passage and antisocially in the next because the generator has no memory of preference commitments across generation calls.

The absence of sequential preference control is equally important. Real storytelling involves dynamic preferences — a character's moral alignment might evolve, the tone might shift from comedic to serious, the pacing might accelerate toward a climax. The paper's architecture, which trains a separate model per fixed preference target, is structurally incapable of representing these dynamics. Even if the CoOp classifier could score a story against multiple sequential preferences, there is no mechanism to optimize a policy for trajectory-level preference satisfaction rather than segment-level.

Evidence in the paper. The generation length of 60 tokens (specified in the hyperparameters as "txt_out_len": 60) and the prompt length of 5 tokens ("txt_in_len": 14 in the hyperparameters, though Section 4 describes using "the first five tokens") are hard constraints of the PPO training setup. The human evaluation stories in Table 1 and Appendix B are all short paragraphs of roughly 3-8 sentences. No experiment generates multi-paragraph stories, evaluates preference consistency across segments, or attempts to change preferences within a generation.

Mitigation status. The paper acknowledges the limitation (Section 8) but does not attempt to address it. The suggestion of successive generation with context pre-pending is mentioned as a possibility but not implemented, evaluated, or analyzed for failure modes (e.g., error accumulation, topic drift, preference inconsistency across segments). The limitation is presented as inherent to the CARP training data rather than as a design choice that could be revisited — but a system intended for creative writing assistance that can only produce 60-token fragments is fundamentally limited in its practical applicability regardless of the reason.

The Story Critique Dataset Is Not Publicly Available, Making Independent Validation and Bias Analysis Impossible

The foundation of the entire pipeline — CARP's ability to align stories with human critiques — rests on a dataset that cannot be independently inspected, audited, or used to reproduce the results.

What the paper acknowledges. Section 8 is explicit:

"The Story-Critique dataset is non-public due to licensing issues despite best efforts to make it publicly available. As such, exploring the preferential biases which are undoubtedly present in CARP is not feasible for these authors though we do acknowledge the complication this presents in evaluation, assessment of ethical concerns and general limitations of the model/approach." (Section 8)

The consequence. This is a severe limitation for a paper whose central contribution is a preference-learning pipeline for creative text generation. The preference biases of the Story Critique dataset — what kinds of stories were critiqued, what kinds of critiques were written, by whom, with what cultural and aesthetic assumptions — are unknown to readers of the paper and, by the authors' own admission, only partially accessible even to them. CARP learns to align stories with critiques as expressed in this specific dataset. If the dataset's critiques disproportionately value certain narrative styles (e.g., Western literary conventions, genre fiction tropes, masculine-coded action descriptions), CARP's preference scores will reflect those biases. If certain types of stories or certain types of preferences are underrepresented in the dataset, CARP will be less sensitive to them — a structural limitation that may explain the religion category underperformance (Figure 4) but cannot be diagnosed without access to the data.

The non-public status also means that the pseudo-labeling pipeline cannot be independently validated. The hand-labeling of 91 HDBSCAN clusters into story features (Appendix E) was performed by the authors on data that no one else can examine. Whether the clusters genuinely correspond to the claimed story features, whether different labelers would agree, and whether the 6 selected clusters represent a reasonable taxonomy of the embedding space are unverifiable claims. The paper's results are, in this sense, non-reproducible — not because the code or model weights are unavailable (the paper states CARP CoOp models are available at a redacted URL) but because the training data that shapes the entire preference space cannot be examined.

The ethics statement acknowledges that CARP "is susceptible to biases in the Story Critique dataset" and reports that "in preliminary experiments the model often conflated extreme acts of violence with humor" (Ethics Statement). This is a striking admission — that the preference model's notion of "humor" is entangled with violent content in its training data — and it underscores the importance of dataset transparency for understanding failure modes. Without access to the data, neither the authors nor the community can systematically audit for such conflations.

Evidence in the paper. The limitation is stated in Section 8. The bias example (violence-humor conflation) is reported in the Ethics Statement. The paper provides no analysis of the Story Critique dataset's composition, annotator demographics, critique distribution, or representational biases beyond these brief mentions.

Mitigation status. The paper acknowledges the limitation candidly but offers no mitigation. The authors state that "despite best efforts" the dataset could not be made public, implying that the limitation is imposed by external constraints (licensing) rather than by design choice. The redacted URL for CARP CoOp model weights suggests that the trained models may be available for downstream use, but this does not substitute for dataset access — the models encode the biases of the training data without providing visibility into what those biases are. Future work on auditing the biases of deployed CARP-based systems would require either public release of the dataset (unlikely given the licensing issues) or the development of probing techniques that can characterize bias from model behavior alone (not attempted in this paper).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a specific, empirically grounded intervention in the debate over how to achieve controllable text generation for subjective, open-ended preferences. Its contribution is not a new architecture or training algorithm in isolation, but rather a demonstration that a contrastive preference model, trained once on a large corpus of story-critique pairs, can substitute for per-preference annotated datasets as the reward signal driving RL-based controllable generation — provided the reward signal is made sufficiently discriminative through unsupervised discretization of the embedding space. The magnitude of the shift is incremental rather than paradigm-changing, but the direction it points toward is significant: a future where general-purpose preference models are infrastructure, trained once and adapted to novel user preferences with minimal labeled data, rather than purpose-built classifiers trained from scratch for every new controllable attribute.

The most concrete landscape change is the reframing of the generality-reliability tradeoff in controllable generation. Before this work, the field operated with an implicit dichotomy: prompted large language models offered generality (any preference expressible in natural language) but unreliable adherence (the model may or may not attend to the preference); classifier-guided methods and RL with annotated reward models offered reliable adherence but lacked generality (each new preference required a new labeled dataset and a new training run). The CARP-based approach demonstrates a third path — a general-purpose preference model whose scores, while individually weak, can be strengthened through unsupervised structure discovery in the embedding space, then used to drive RL optimization that compels adherence. The paper does not fully resolve the tradeoff (per-preference specialized models are still required; the discretization pipeline is fragile; only a limited set of preference categories is validated), but it establishes the viability of the approach and provides a concrete blueprint for closing the gap.

The paper also provides a diagnostic contribution by identifying a specific, non-obvious failure mode of contrastive reward models for RL: the "weak continuous signal" problem, where cosine similarity scores cluster in a narrow intermediate range and fail to provide discriminative feedback to the policy optimizer. This is not a generic observation about reward noise — it is a claim about the distributional properties of contrastive embedding spaces and their interaction with policy gradient methods. The fact that unsupervised clustering can surface categorical structure that was latent in the embedding space, and that prompt tuning on this structure produces a substantially stronger reward signal (8.2 point improvement on topics, 16.9 points on alignments over raw CARP rewards, Table 2), validates the diagnosis and provides a template for addressing similar issues in other contrastive reward settings. Researchers working on RLHF with embedding-based reward models should take note: if the reward model's scores are continuously distributed with low variance, discretization via clustering may be a lightweight alternative to collecting more human preference data.

The results on the NeoX baseline — a 20B parameter model outperformed by a 750M model with explicit preference optimization — provide boundary conditions for the "scale solves instruction-following" narrative. The paper does not claim that scale is irrelevant or that prompted models cannot follow instructions. Rather, it demonstrates that for a specific class of tasks — subjective, open-ended preferences where the mapping from preference language to output features is complex and underspecified by the training data — explicit optimization against a preference signal outperforms conditional generation, even at a 20× parameter disadvantage. This is a much narrower claim than "small models beat large models" and should be understood as identifying a task regime where mechanism matters more than capacity. For practitioners, this suggests that preference-enforcement mechanisms (reward models, RL fine-tuning) deserve investment proportional to the subjectivity and open-endedness of the generation task, and that simply scaling to larger models may not be the most cost-effective path to reliable controllable generation for creative domains.

Less positively, the paper complicates the narrative around zero-shot preference models. The raw CARP model theoretically enables zero-shot preference scoring — any critique text can be embedded and compared to story embeddings — but the paper's core technical contribution is demonstrating that this zero-shot signal is too weak for RL and must be augmented through clustering and prompt tuning. The system as deployed does not operate in a zero-shot regime; it requires per-preference CoOp training (albeit with only 1,000 examples per class) and per-preference PPO fine-tuning. The vision of a single model that can be prompted with arbitrary preferences and reliably produce conforming output remains unrealized. This is not a failure of the paper — it is an honest characterization of the current capability frontier — but it means the paper's primary impact is in moving the field from "zero-shot preference scoring is sufficient" to "zero-shot preference scoring provides a foundation that must be strengthened through structure discovery and lightweight adaptation." This is a more realistic, and more practically useful, framing than the CLIP-for-text analogy that the paper invokes.

Follow-Up Research This Work Enables

Multi-preference conditioning within a single PPO training run. The paper trains separate specialized models for each preference target (9 models total: 6 topics + 3 alignments), with the "review" hyperparameter fixed during each run. A natural and high-priority extension is to train a single model that can be conditioned on the preference at generation time: the policy receives both the story prompt and the target preference label (or CoOp class embedding) as input, and the reward is computed as the CARP CoOp score for the specified preference class. This would test whether the model can learn a conditional policy — "when the input says 'imagery,' generate imagery-rich text; when it says 'fighting,' generate conflict-rich text" — rather than a single-preference specialization. The key experimental question is whether preference conditioning interferes with preference optimization (the model must simultaneously learn to generate good examples of multiple preference categories) or whether the shared representation learning across preferences actually improves performance through cross-preference generalization. A strong follow-up would train a single GPT-2-750M model with 9-way preference conditioning (the 6 Pseudo CARP CoOp labels plus the 3 Alignment CARP CoOp labels), compare its per-preference performance to the 9 specialized models, and measure whether the conditional model achieves comparable or better preference identification rates.

Automated cluster selection and discretization for the embedding space. The paper's pseudo-labeling pipeline — UMAP → HDBSCAN → hand-labeling → centroid computation → thresholded assignment — introduces substantial manual effort and unvalidated degrees of freedom. A follow-up that automates the cluster selection and discretization process would both improve reproducibility and test whether the paper's 6-label results represent an upper bound or could be exceeded with better taxonomy discovery. A concrete experiment: treat the cluster selection problem as a discrete optimization over the 91 HDBSCAN clusters, where the objective is to maximize the downstream CARP CoOp classification accuracy on held-out data (or, more ambitiously, to maximize the PPO policy's preference identification rate in a small-scale human evaluation). This could be approached via greedy forward selection (start with the cluster with the highest intra-cluster cosine similarity, iteratively add the cluster that maximizes the minimum inter-centroid distance) or via learned discretization (train a small network to predict cluster assignments from raw critique embeddings, optimizing for downstream CoOp accuracy). The key advance over the paper's hand-labeling would be a quantitative criterion for cluster quality that does not depend on human judgment of "semantic dissimilarity," enabling the method to scale to the full 91-cluster taxonomy or beyond.

Direct comparison to RLHF with human preference labels for the same preference categories. The paper positions CARP as an alternative to collecting human preference annotations for reward modeling but never compares against a reward model trained on human judgments. A critical follow-up would train a standard RLHF reward model — using paired human preference judgments on story segments for the same 9 preference categories evaluated in the paper — and compare PPO fine-tuning with the human-feedback reward model against PPO fine-tuning with CARP CoOp rewards. The comparison would answer: for a fixed budget of human annotation effort (say, the cost of generating the Story Critique dataset vs. the cost of collecting pairwise preferences for 9 categories), which approach produces a better reward model? And at what point does additional human annotation surpass the CARP-based approach? The paper's human study provides a ready-made evaluation protocol; the missing piece is the human preference reward model training and the head-to-head comparison. If CARP-based rewards approach or exceed human-feedback reward model performance at substantially lower annotation cost, this would strongly validate the contrastive bi-encoder approach for preference learning. If human-feedback reward models substantially outperform CARP-based rewards, this would clarify the ceiling of what contrastive training on natural language feedback can achieve relative to direct preference elicitation.

Stress-testing the pseudo-labeling pipeline on deliberately challenging preference categories. The paper selects 6 pseudo-labels for their "semantic dissimilarity and separation in the embedding space" (Section 5.2) and acknowledges that these "represent an upper-bound of sorts regarding downstream language model performance" (Section 8). A systematic stress-test would deliberately select preference categories that violate the assumptions of the pipeline — categories that are semantically overlapping (e.g., "sad" vs. "melancholy" vs. "tragic"), categories that are rare or diffusely distributed in the embedding space (preferences with no clear HDBSCAN cluster), or categories that are inherently multi-dimensional (e.g., "a story that is funny and sad simultaneously"). For each category, measure CARP CoOp classification accuracy and downstream PPO preference identification rates. The hypothesis is that performance degrades as cluster coherence decreases, providing a diagnostic for when the CoOp approach is appropriate versus when alternative reward strategies (e.g., raw CARP similarity, human annotation) are necessary. This would transform the paper's qualitative observation about religion being "particularly sensitive" into a quantitative relationship between embedding-space cluster properties and downstream preference learning success.

Long-form generation with preference consistency via segment-level reward aggregation. The paper's system generates 60-token segments and acknowledges the inability to "reason over longer sequences of text" or "reason effectively about preferences or critiques as would be applicable to longer form creative writing" (Section 8). A follow-up that extends the approach to multi-segment generation would test whether preference-conforming behavior can be sustained across a longer narrative. The simplest extension: generate a story in segments by iteratively prompting the policy with the previous segment appended to the context, computing the CARP CoOp reward for each segment independently, and using the average or minimum segment-level reward as the trajectory-level reward for PPO updates. This tests whether the policy can learn to maintain preference consistency when its own previous output becomes part of the state for subsequent generation — a setting where error accumulation and topic drift are known failure modes. A more ambitious extension would incorporate the CARP CoOp score for the entire generated narrative (not just individual segments) as part of the reward, either by encoding the full concatenated story or by training a segment aggregation model that predicts the CoOp score for a multi-segment narrative from its constituent segments. The evaluation would measure preference identification rates at different narrative lengths (1 segment, 3 segments, 5 segments) and test whether preference conformance degrades with narrative length — directly addressing the primary limitation the paper identifies for practical deployment.

Probing CARP's embedding space for systematic biases using the pseudo-label taxonomy. The paper's ethics statement reports that CARP "often conflated extreme acts of violence with humor" in preliminary experiments, but provides no systematic analysis of such conflations. A bias-probing study using the 54 hand-labeled clusters from Appendix E as a taxonomy of story features would systematically measure which features are associated with each other in CARP's embedding space (by computing cosine similarities between cluster centroids, or by measuring the rate at which stories assigned to one cluster receive high scores for critiques from another cluster). The hypothesis to test: CARP's embedding space encodes certain culturally or aesthetically specific associations (violence-humor, family-goodness, foreignness-danger) that reflect biases in the Story Critique dataset. The social value of such a study is in making the biases of the contrastive preference model explicit before it is deployed in creative writing tools, where it might steer users toward or away from certain narrative patterns without transparency. The unavailability of the Story Critique dataset makes this challenging but not impossible — the probing can be done purely on model outputs and publicly available story corpora, without requiring access to the training data.

Practical Applications and Downstream Use Cases

Interactive creative writing assistants with preference-constrained generation. The most direct application is a writing tool where users specify preferences in natural language and receive story continuations that adhere to those preferences with greater reliability than prompting alone can provide. The paper's quantitative results provide a realistic performance baseline: for preferences with well-separated clusters in CARP's embedding space (imagery, fighting, character alignment), the CARP CoOp LM achieves human preference identification rates of roughly 65-85% (Figures 4-5), with the caveat that the "upper-bound" nature of the 6 selected labels means that performance on other preferences would need to be validated per-category. The practical deployment model would involve training Pseudo CARP CoOp on a larger set of preference clusters, building a library of per-preference fine-tuned generators, and routing user requests to the appropriate specialized model based on similarity of the user's preference text to the known cluster centroids. A key practical consideration the paper does not address is the user interface for preference specification: should users select from a fixed menu of supported preferences (reliable but inflexible), type free-form preference text and have the system map it to the nearest supported class (flexible but lossy), or interact with the system through iterative refinement (generating, giving feedback, regenerating)?

Data augmentation for story datasets via preference-guided generation. The paper's finding that CARP-tuned models produce stories that are "more complex than the original ROCStories dataset" despite never being trained on sophisticated literary text (Section 7) suggests a data augmentation application: use CARP CoOp-tuned generators to produce training data with controlled preference characteristics for downstream story generation or analysis tasks. For example, generating a balanced dataset of stories across the six Pseudo CARP CoOp topic labels (imagery-rich, fighting-heavy, music-themed, family-focused, etc.) by sampling from the corresponding specialized generators, and using this balanced dataset to train a single multi-preference story generator or a preference classifier. The key advantage over manual data collection is cost: generating 10,000 stories across 6 preference categories would cost roughly 60 A100-hours of PPO fine-tuning (6 models × 1 hour per the paper) plus generation time, compared to the cost of commissioning human-written stories with specified preferences. The quality of the generated data would need to be validated against human-written preference-labeled stories, but for tasks where large volumes of preference-diverse story data are useful (training preference-aware retrieval models, building story recommendation systems, studying narrative patterns across preference dimensions), the CARP-based approach offers a scalable alternative to manual annotation.

Character alignment control for educational and therapeutic storytelling. The Alignment CARP CoOp model, which achieves 67.5% aggregate preference identification for character moral alignment (good, neutral, evil) compared to 46% for prompted NeoX-20B and 56.1% for GeDi (Table 2), has direct applicability to domains where controlling the moral valence of generated content matters. Educational storytelling platforms (generating stories that model prosocial behavior for children), therapeutic narrative tools (generating stories where characters navigate moral dilemmas in ways aligned with therapeutic goals), and game narrative systems (generating character dialogue consistent with a specified moral alignment) all require reliable control over the ethical dimensions of generated text. The paper's demonstration that a relatively small model (750M parameters) can achieve substantial alignment control through fine-tuning — and that the control is recognizable to human readers — provides a proof of concept for deployment in resource-constrained settings where running a 20B+ parameter model is infeasible, such as on-device generation for children's reading apps or interactive educational software. The practical caveat is the same as above: the system produces recognizable preference alignment, not necessarily nuanced or contextually appropriate moral content, and deployment in ethically sensitive domains would require additional validation of output quality and safety beyond preference identification rates.

When to Prefer This Method

The paper does not provide a systematic tradeoff analysis against named alternatives with explicit decision criteria, so a formulaic "prefer A when / prefer B when" matrix would be speculative. However, the results do support several conditional recommendations that emerge from the empirical comparisons:

Consider CARP-based RL fine-tuning when: (1) the target preference is subjective, complex, or abstract in ways that make token-level logit manipulation unreliable (as demonstrated by GeDi's poor performance on imagery and fighting, Figure 4); (2) reliability of preference conformance is more important than flexibility across many preferences, since the approach requires per-preference specialized models; (3) a large language model prompted with the preference criteria produces inconsistent or inadequate results, as NeoX-20B does for imagery, fighting, and all alignment categories (Figures 4-5); and (4) you have access to, or can create, a contrastive bi-encoder trained on paired (output, natural language feedback) data in your domain — the approach depends on the existence of a CARP-like model and may not transfer to domains without such a resource.

Prefer prompt-based generation with large LMs when: (1) the range of preferences users need to express is large, diverse, and not known in advance (making per-preference fine-tuning infeasible); (2) the preference can be expressed as a simple, unambiguous instruction that large models attend to reliably (the paper's religion results, where NeoX outperforms CARP CoOp LM, suggest some preferences are better served by prompting); (3) zero-shot preference specification is required and the user cannot provide even the minimal 1,000 examples per class that CoOp training requires; or (4) inference latency and model storage constraints make deploying multiple specialized models impractical relative to a single large model.

The key open question the paper does not resolve is where the crossover point lies: for how many distinct preferences does the total cost of training and serving separate CARP CoOp-tuned models exceed the cost of deploying a single larger prompted model that handles all preferences (albeit unreliably)? Answering this requires the FLOPs-matched cost accounting that the paper does not provide, and it would be a valuable contribution for future work comparing controllable generation paradigms under equalized resource budgets.