Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation

🎯 Pitch

Standard sequential recommenders trained entirely on synthetic, bias-free data outperform those trained on real user logs by 130% on recall@100—revealing that the path to scaling LLMs for recommendation lies not in bigger models but in cleaner data. This principled curriculum finally unlocks predictable power-law scaling, turning recommendation model development from costly guesswork into a forecastable science.

1. Executive Summary

This paper introduces a layered synthetic data framework that transforms noisy, biased user interaction logs into a structured, pedagogical curriculum for continually pre-training LLMs on recommendation tasks, and demonstrates that this high-quality data enables the first scaling laws for LLMs in the recommendation domain. Using the Merrec C2C e-commerce dataset and Qwen3 models from 0.6B to 8B parameters, the framework generates two layers of synthetic data—Layer 1 grounds semantic and collaborative knowledge through item-text alignment and collaborative filtering (CF) association-rule data (e.g., "When a user interacts with item X, there is a 4.9% probability they will also interact with item Y"), while Layer 2 generates unbiased user interaction histories (UIH) via graph-based random walks that eliminate position and popularity biases—and standard sequential models (SASRec, GRU4Rec, NARM, STAMP) trained exclusively on this synthetic data outperform those trained on real data by +130% on recall@100. The authors establish robust power-law scaling of the form ℓ(D) = L∞ + A·D^−α across seven evaluation domains, with UIH data exhibiting the strongest scaling exponents (α ≈ 0.45–0.59) and an asymmetric cross-domain transfer where including CF data alongside UIH reduces asymptotic UIH perplexity by 31% (L∞ = 0.66 vs. 0.95). The FLOPs-matched analysis reveals that recommendation domains are strongly data-dominant rather than model-size-dominant—a reversal of the pattern observed for general text—establishing that predictable scaling for LLM-based recommenders emerges only when training data is explicitly engineered to decouple true user preference signals from system-induced artifacts.

2. Context and Motivation

The Core Problem: No Scaling Laws Exist for LLMs in Recommendation

The fundamental question this paper tackles is both simple and practically urgent: if you want to build an LLM-based recommender system, how do you know how much data and compute to invest? In natural language processing, practitioners have a well-established toolkit for answering this question—the scaling laws pioneered by Kaplan et al. (2020) and refined by Hoffmann et al. (2022) in the Chinchilla paper tell you that if you double your model size, your perplexity will drop by a predictable amount, and what ratio of training tokens to parameters gives you the best return on your compute dollar. These laws aren't just academic curiosities; they are the strategic backbone that enables organizations to forecast costs, justify investments, and allocate resources efficiently before committing millions of dollars to GPU clusters.

In recommendation systems, no such laws exist. This absence is not a minor gap—it means that every team building LLM-powered recommenders operates in the dark, running expensive trial-and-error experiments to determine whether their 3B parameter model is under-trained or their data is insufficient. The paper's opening salvo captures this precisely:

"The development of such systems at scale, however, requires the establishment of predictable scaling laws—indispensable instruments for navigating the substantial investments in data, computation, and engineering that modern LLM development demands. Yet, despite their established significance in NLP, no robust scaling laws have been established for the continual pre-training (CPT) of LLMs within the recommendation domain."

This is not merely an inconvenience. When scaling laws exist, you can extrapolate—you run a small experiment at 100M parameters, fit a power law, and predict what a 7B model will achieve given 1T tokens. Without them, every scale-up is a gamble. For teams considering whether to invest in CPT for recommendation, this uncertainty is paralyzing.

Why Recommendation Scaling Is Harder Than NLP Scaling

The reader might reasonably ask: why can't we just apply NLP scaling laws to recommendation data? The answer lies in the nature of the data itself.

In NLP, training data—web text, books, code—is messy but fundamentally expressive. A sentence on Wikipedia conveys information through the compositional structure of language; the model's job is to learn the patterns of that composition. In recommendation, the raw data is user interaction logs: sequences of clicks, purchases, ratings, and views. These logs are not neutral recordings of user preference; they are confounded records produced by a pre-existing recommendation algorithm operating in a feedback loop. The paper devotes significant attention (Section 3.2) to cataloging exactly how these logs are corrupted:

Position bias: Users click on items at the top of a ranked list far more than items further down, regardless of actual relevance. The paper cites a concrete magnitude: "CTR drops 50%+ from position 1 to 5, even for equally relevant items." A model trained on such data learns that "first position" is a stronger signal of preference than item content—a devastating heuristic for any system that aims to rank items by true user affinity.
Popularity bias: The rich-get-richer dynamic where popular items are recommended more, receive more interactions, and become even more popular. The paper quantifies this: "Top 1% of items receive 80% of interactions; long-tail items never surface." This creates a training distribution where the model sees only a tiny fraction of the catalog repeatedly, starving it of the diversity needed to learn generalizable preference patterns.
Exposure bias: Users can only interact with what they see. If a user loves jazz but the system only recommends pop, the interaction log will contain zero evidence of the jazz preference. The model learns a censored view of reality.
Data incompleteness and sparsity: Users interact across multiple platforms (buying a camera on Amazon, accessories on B&H), but any single platform sees only a fragment of the user's intent. This partial observability means the interaction sequences are fundamentally incomplete stories.
Data noise: Interactions are noisy signals—clicks are accidental, ratings contradict reviews ("5-star rating with review: 'Arrived broken, returning it'"). These contradictions inject conflicting training signals into the model.

The paper's central insight—and the motivation for its entire approach—is that LLMs are exceptionally good at learning precisely the wrong thing from this data. Because LLMs are powerful pattern-matching machines, they don't just fail to filter out bias; they internalize and amplify it. The model's training objective is to minimize prediction error, and the dominant statistical regularities in biased logs are the biases themselves. As the authors put it:

"When an LLM is continually pre-trained on biased user logs, it does not merely learn the bias; its powerful pattern-matching capabilities codify and amplify it. The model internalizes the system's flaws more effectively than simpler, traditional models."

This creates a vicious cycle: the biased LLM produces biased recommendations, which generate more biased interaction logs, which—if used for further training—produce an even more biased model. The paper calls this a "degenerative loop" where "the system's quality degrades over time by continually learning and reinforcing its own errors."

The PLUM Case Study: Empirical Evidence of Scaling Failure

The paper doesn't just theorize about data quality problems—it provides a concrete case study. The PLUM framework (He et al., 2025) represents one of the most prominent recent attempts to adapt LLMs for recommendation through continual pre-training on user activity sequences. The PLUM team built a MoE (Mixture of Experts) architecture at multiple scales, including 900M and 3B parameter variants, and trained them on real user interaction data.

The result was a clear scaling failure: the larger 3B model failed to consistently outperform the smaller 900M model. In classical NLP scaling, doubling or tripling model size yields a reliable perplexity improvement; here, the additional capacity provided no benefit. The paper identifies this as a symptom of what it calls the "sub-scaling" phenomenon:

"This outcome is a clear example of the 'sub-scaling' phenomenon observed in the broader NLP literature. Recent analysis of scaling laws, such as the 'Sub-Optimal Scaling Law' framework proposed in Chen et al. (2025b), explicitly identifies high data density and redundancy as critical factors that cause performance gains to decelerate."

The mechanism is straightforward: when training data lacks diversity and contains heavy redundancy—precisely the characteristics of popularity-biased interaction logs where the same popular items appear in most sequences—the marginal benefit of additional parameters or additional tokens rapidly diminishes. The training distribution is so narrow that a 900M-parameter model already captures nearly all the learnable signal; the extra capacity of a 3B model has nothing useful to learn.

This case study is critical to the paper's motivation because it demonstrates that the problem is not hypothetical. It's not that recommendation teams haven't tried to build LLM recommenders; they have, and they've hit a wall. The PLUM result makes concrete what the data quality analysis predicts: you cannot simply feed raw logs into an LLM and expect scaling laws to emerge.

Prior Approaches and Their Limitations

The paper positions itself against three broad categories of prior work, each of which it argues addresses only part of the problem:

Model-centric approaches (LUM, PLUM): The Large User Model (LUM) paradigm from Yan et al. (2025) introduces a novel training task—"next-condition-item prediction"—designed to better capture contextual user preferences. This is fundamentally an architectural innovation: it changes how the model learns rather than what it learns from. The paper's critique is direct: even the most sophisticated architecture "would struggle to exhibit predictable power-law scaling if trained on the pathologically flawed data we identify as the root inhibitor." Architecture cannot compensate for data that contains no generalizable signal.

Distillation-centric approaches (SUAN): The Stacked Unified Attention Network (Lai et al., 2025) builds a high-capacity teacher model and distills it into a deployable student model. This addresses deployment efficiency but does not address the fundamental question of whether the teacher model itself scales predictably. If the teacher is trained on the same biased data, its performance ceiling is limited regardless of how effectively it's distilled.

System-centric approaches (OneRec series): The OneRec series (Zhou et al., 2025a,b; Liu et al., 2025) studies scaling behavior along model-size and test-time axes using generative recommendation algorithms. While this work contributes to understanding scaling, the paper notes it still operates on data inherited from traditional recommendation pipelines, without systematic attention to data quality as the scaling enabler.

The paper's diagnosis is that all of these prior efforts share a common blind spot: they treat the data as a given and attempt to solve the problem through model innovation. They ask "how can we design a model that is robust to data imperfections?" The paper reframes the question entirely: "how can we design data that is free of imperfections?"

Debiasing approaches: A rich literature exists on debiasing recommender systems through inverse propensity scoring, causal inference, and adversarial training (Section 2.3). The paper acknowledges these methods can mitigate some biases, but argues they are fundamentally palliative. They require "designing more complex model architectures or loss functions that can learn true user preferences despite being trained on confounded data." The implicit critique is that these approaches fight an uphill battle—trying to extract a clean signal from a contaminated source, when one could instead simply purify the source itself.

Connection to Broader Data-Centric Trends in LLM Development

The paper positions its data-centric philosophy within a broader intellectual movement. Recent work in NLP has increasingly recognized that data quality, not just data quantity, governs scaling behavior:

Yang et al. (2024) demonstrated that continued pretraining with next-token prediction fails to teach models certain kinds of knowledge from condensed representations, even with heavy repetition. Instead, generating larger synthetic datasets grounded by source documents enabled successful knowledge acquisition—but only if the synthetic data was sufficiently diverse. Format variation and rewriting proved critical; mere repetition did not work.
Allen-Zhu and Li (2024a) provided theoretical and empirical evidence that data diversity is the critical factor for successful continual pre-training. Their "Physics of Language Models" series established that knowledge capacity and extraction depend fundamentally on how information is presented across training examples.
Muennighoff et al. (2025) studied data-constrained regimes and found that training beyond roughly four repeats of a finite dataset yields diminishing returns, with continued repetition at 20+ repeats actively degrading generalization (Hron et al., 2024).

This body of work provides the theoretical foundation for the paper's thesis: scaling laws only manifest when data has sufficient quality and diversity. The recommendation domain is the perfect stress test for this thesis because its raw data is extraordinarily low-quality by any measure—biased, redundant, and censored. If data quality gates scaling, then the failure to find recommendation scaling laws is exactly what theory predicts.

The paper also connects to the curriculum learning tradition (Soviany et al., 2022; Chen et al., 2025a), which argues that presenting training examples in a structured, progressive order—from simple to complex, from foundational to integrated—improves learning efficiency. The layered synthetic data framework is explicitly designed as a curriculum: Layer 1 teaches the model the basic vocabulary of the domain (what items are, how they relate), while Layer 2 teaches more complex, sequential patterns built on that foundation.

How This Paper Positions Itself

The paper's core move is to shift from a model-centric to a data-centric framing. Rather than asking "how can we build an LLM that scales on recommendation data?", it asks "what would recommendation data need to look like for an LLM to scale on it?" This is a fundamentally different orientation.

Two design principles flow from this shift:

Pedagogy over mimicry: Traditional synthetic data generation for recommendation (using GANs or VAEs) aims to replicate the statistical properties of the source data. The paper explicitly rejects this goal: "The objective is not mimicry but pedagogy." Its synthetic data is designed to teach the model principles of recommendation—semantic item relationships, collaborative patterns, unbiased sequential behavior—rather than reproduce the surface statistics of raw logs.
Purification over debiasing: Instead of building models robust to bias, the paper engineers data free from bias. The two-layer framework is fundamentally a filtering mechanism: Layer 1 extracts the core statistical signal from noisy logs (averaging over many sessions to wash out individual biases), and Layer 2 generates new data from only that purified signal (using random walks that have no concept of position or ranking).

The paper's ambition is not merely to demonstrate that clean data improves performance—that would be unsurprising. The ambition is to demonstrate that clean data enables scaling laws where none existed before, transforming LLM-based recommendation from an unpredictable art to a predictable science. This is what makes the contribution foundational: it doesn't just improve a system; it establishes the conditions under which systematic, quantitative progress becomes possible.

What's At Stake

The practical stakes of this work are substantial. Recommendation systems power a significant fraction of the modern internet economy—e-commerce, content streaming, social media, advertising. If LLMs can be made to scale predictably in this domain, it opens the door to:

Better long-tail recommendations: If popularity bias is eliminated from training, models can surface niche content that current systems systematically suppress.
Cold-start handling: If semantic item understanding is properly grounded (through Layer 1's item-text alignment), new items with no interaction history can be recommended based on their content descriptions alone.
Cross-domain transfer: If the layered curriculum teaches generalizable principles of user preference rather than domain-specific quirks, models might transfer across recommendation domains (e.g., from e-commerce to content streaming) more effectively.
Resource forecasting: Teams can estimate, before training begins, how much compute and data they need to hit a target performance level—enabling the same kind of strategic planning that NLP teams have enjoyed for years.

The paper's bet is that all of these benefits hinge on solving the data quality problem first. Without clean, structured, pedagogical training data, scaling LLMs for recommendation will remain a game of expensive guesswork.

3. Technical Approach

3.1 Reader Orientation

The paper builds a data generation pipeline, not a new recommendation model. The system takes raw, biased user interaction logs as input and produces a multi-layer synthetic training curriculum as output—item-text pairs, collaborative-filtering association statements, and unbiased user interaction histories—that is then fed into a standard continual pretraining pipeline for Qwen3 LLMs. The core problem it solves is that raw recommendation data contains systemic biases (position, popularity, exposure) that prevent LLMs from exhibiting predictable power-law scaling; the solution is a two-layer data purification process that explicitly decouples true user preference signals from system-induced artifacts before the LLM ever sees them.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major stages:

Semantic Tokenization — converts item text descriptions into discrete token sequences (<RECTOKEN> blocks) using a pre-trained Sparse Autoencoder (SAE), establishing a shared vocabulary for all downstream data generation.
Layer 1 Data Generation — produces two foundational data types from raw interaction logs: (a) Item-Text Alignment Data, which pairs each item's token sequence with its textual description to ground semantics; (b) Collaborative Filtering (CF) Data, which mines item-to-item association rules from user histories and formats them as natural-language statements with explicit confidence and lift metrics.
Layer 2 Data Generation — constructs an item-to-item graph using the CF association strengths as edge weights, then runs 2nd-order biased random walks (Node2Vec algorithm) to generate Synthetic User Interaction Histories (UIH) that are realistic in their item co-occurrence patterns but free from position and popularity biases.
Data Mixture Assembly — combines the synthetic recommendation data (Item-Text, CF, UIH) with general-domain text data (SmolLM-Corpus) at a fixed 50%/50% ratio to prevent catastrophic forgetting, with internal recommendation-domain ratios set at approximately 9% Item-Text, 30% CF, 11% UIH.
Continual Pre-Training — feeds the assembled data mixture into Qwen3 models (0.6B, 1.7B, 4B, 8B parameters) using standard autoregressive language modeling with a global batch size of 512 sequences, context windows of 512 tokens, and a peak learning rate of 1e-4 with linear warmup (100 steps) and cosine decay, training all models for 163B total tokens.

Information flows linearly: raw logs → semantic tokenization → Layer 1 (Item-Text + CF) → Layer 2 (UIH via graph random walks) → mixture assembly (with general-domain data) → CPT across model scales → perplexity evaluation on held-out test sets across seven domains.

3.3 Roadmap for the Deep Dive

First, the semantic tokenization process (Section 4.1, Appendix B), because every subsequent data type uses the same token vocabulary to represent items—understanding the SAE-based tokenization is prerequisite to understanding what any training example looks like.
Second, Layer 1's two sub-components (Sections 4.2.1 and 4.2.2): item-text alignment and CF data, since these establish the foundation that Layer 2 builds on, and the CF data specifically defines the edge weights used in the graph random walks.
Third, Layer 2's synthetic UIH generation (Section 4.3.1), including the graph construction, the Node2Vec random walk algorithm with its p and q parameters, and the hyperparameter sweep that selected the final configuration—this is the most technically novel component.
Fourth, the data mixture design and CPT configuration (Sections 6.1, 7.1, 7.2), including the rationale for the 50/50 general-to-recommendation split, the internal recommendation ratios, the choice of 163B training tokens, and the evaluation methodology across seven held-out domains.
Fifth, the scaling law fitting methodology (Section 6.2), including the per-model fits of the form ℓ(D) = L∞ + A·D^−α plus the joint fits ℓ = E + A·N^−α + B·D^−β, the FLOPs accounting, and the compute-optimal allocation condition αA·N^−α = βB·D^−β.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data engineering and empirical analysis paper whose core idea is that high-quality, pedagogically-structured synthetic data enables predictable power-law scaling for LLMs in recommendation, whereas raw user interaction logs—contaminated by position bias, popularity bias, exposure bias, and noise—do not. The technical contribution is the layered data generation framework, the validation of its utility through sequential model baselines, and the first empirical demonstration of recommendation-domain scaling laws across model sizes and data modalities.

Semantic Tokenization: Converting Items into a Shared Vocabulary

Before any synthetic data can be generated, every item in the catalog must be converted into a discrete token sequence that the LLM can process. The paper uses a pre-trained Sparse Autoencoder (SAE) to perform this conversion, selected through an ablation against an in-domain Residual-Quantized VAE (RQ-VAE) and RQ-kmeans approach.

Candidate approaches compared. The paper evaluated three strategies for semantic tokenization (Appendix B):

Approach 1: Pre-trained SAE. This method uses a large-scale SAE originally trained on an internal multi-modal dataset comprising video, photo, and text data. The procedure involves passing an item's textual description through a pre-trained embedding model, then into the SAE. The top-k activated features (concepts) from the SAE are selected as the discrete tokens for that item. The strength of this approach is that it leverages a "foundation" encoder exposed to massive, diverse multi-modal data, potentially capturing rich semantic concepts. The critical risk is semantic mismatch: SAE features learned in a video-centric domain may extract concepts that are mathematically proximal but semantically incongruous for e-commerce product descriptions.
Approach 2: In-Domain RQ-VAE. This method trains a hierarchical Residual-Quantized Variational Autoencoder from scratch specifically on item embeddings from the target dataset. The procedure uses Qwen2.5-7B to generate text embeddings for all items, then trains an RQ-VAE to quantize these embeddings into a sequence of discrete codes using a residual codebook structure. The strength is perfect semantic alignment with the target domain, since the vocabulary is learned directly from item descriptions. The critical risk is data leakage: if the tokenizer is trained on the full dataset including test-set items, test-set information leaks into the training process, invalidating downstream scaling law experiments.
Approach 3: In-Domain RQ-kmeans. Similar to RQ-VAE but using k-means clustering for codebook learning rather than variational inference, described as "more stable and scalable." The paper reports that with 6 layers of codebook and 256 codes per layer, RQ-kmeans achieved a collision rate of 5.33%, while RQ-VAE was stuck at 50.54%—a dramatic difference in codebook utilization.

Selection and justification. The paper selected Approach 1 (Pre-trained SAE) as the primary tokenization method after an ablation comparing SAE against RQ-kmeans (6 layers, 256 codes per layer) on 4B models. Figure 12 in Appendix B shows the comparison across all domains:

"This figure shows SAE consistently outperformed RQ-Kmeans across all domains and steps."

The SAE-based tokenization yielded lower perplexity on General Domain (L∞ = 0.82 vs. 0.83), Item-Text (0.86 vs. 1.25), and UIH Full (0.66 vs. 0.76) at 4B scale. The paper's decision to use a pre-trained SAE despite the semantic mismatch risk suggests that the multi-modal pre-training provided representation quality benefits that outweighed domain-mismatch concerns, at least for the Merrec e-commerce dataset.

Token format. Regardless of the tokenization method, items are represented as blocks delimited by special tokens <RECTOKEN> and </RECTOKEN>, with individual semantic tokens drawn from an added vocabulary (e.g., REC6594, REC5411, etc.). Each item is represented by a sequence of 8 such tokens (visible in all examples throughout Section 4 and the case studies in Section 8). This fixed-length representation means items occupy a consistent number of positions in the LLM's context window, simplifying sequence construction.

RQ configuration sweep. For completeness, the paper reports a hyperparameter sweep for the RQ-kmeans approach in Table 10 (Appendix B), varying the number of layers and codes per layer with a fixed beam size of 5 and progressive search with uniform sampling. The sweep reveals the collision rate as a function of the total codebook capacity:

3 layers × 256 codes = 16M total codes: 82.7% collision rate (clearly insufficient)
3 layers × 512 codes = 134M total codes: 37.4% collision (still high)
5 layers × 256 codes = 1.1T total codes: 9.7% collision (entering acceptable range)
6 layers × 256 codes = 281T total codes: 3.5% collision (low redundancy)
8 layers × 128 codes = 72,057T total codes: 1.2% collision (near-unique)

This sweep illustrates the exponential capacity growth of hierarchical codebooks: adding layers is far more impactful than increasing codes per layer, because each additional layer multiplies the total vocabulary size by the number of codes in that layer.

Layer 1, Component 1: Item-Text Alignment Data

Purpose. The Item-Text alignment data teaches the LLM to associate abstract semantic token sequences with their real-world meaning conveyed through natural language descriptions. This is critical because collaborative filtering traditionally operates on item IDs as opaque symbols—the model learns that "item A co-occurs with item B" without understanding why. By grounding item representations in language, the model can reason about content-based relationships, enabling generalization to new items (cold-start handling) and long-tail items with sparse interaction histories.

Generation methodology. The procedure creates direct pairing between an item's tokenized representation and its descriptive text, formatted as natural language sequences. A training example (from Section 4.2.1) looks like:

"This item <RECTOKEN> REC6594 REC5411 REC1547 REC941 REC7587 REC7639 REC3383 REC6576 </RECTOKEN> is described as [redacted] lotus yoga seed bead bracelet, by Handmade in Women › Jewelry › Necklaces."

The textual descriptions are sourced from product catalogs, content creator metadata, or synthesized by other LLMs to ensure richness and consistency. The <RECTOKEN> and </RECTOKEN> delimiters are added special tokens that mark the boundaries of the item representation, allowing the model to learn exactly which token span corresponds to the item identifier.

Design rationale. This data type serves as the semantic foundation of the entire curriculum. Without it, the LLM would see semantic tokens only in the context of sequences (Layer 2) and CF rules (Layer 1, Component 2), where the tokens' meaning could only be inferred implicitly from co-occurrence patterns. The explicit alignment provides a direct supervised signal: "these 8 tokens mean this specific product." This is analogous to teaching a student vocabulary before asking them to read paragraphs—the Layer 1 Item-Text data builds the basic lexicon.

The paper allocated approximately 2.0B tokens to this data type in the main experiments (Table 2), representing roughly 9% of the total recommendation data and approximately 6.2 repeats over the dataset at the end of training (since 163B total tokens × 9% recommendation-domain allocation / 2.0B unique tokens ≈ 7.3 total exposures, but with the 50% general-domain split, the effective repeats on Item-Text specifically is 163B × 0.09 / 2.0B ≈ 7.3, which is then adjusted for the actual token consumption—the paper reports 6.2 repeats in Table 2).

Layer 1, Component 2: Collaborative Filtering (CF) Data

Purpose. The CF data explicitly teaches the model the statistical logic of collaborative filtering by translating item-to-item association patterns into linguistic statements. Rather than requiring the model to implicitly infer these patterns from noisy interaction sequences (as it would with raw data), the CF data makes the relationships explicit, providing a clear supervised signal. This is the paper's most direct mechanism for purifying the collaborative signal: by aggregating over many user sessions into association rules, the CF extraction process averages out individual-level biases and noise, preserving only the core statistical regularities.

Generation methodology. The process begins by mining association rules from the raw user interaction histories and existing item-to-item (I2I) data. Each rule captures a directional relationship: "given that a user interacted with item A, what is the probability they also interact with item B?" These rules are then formatted as templated natural language statements. A representative example from Section 4.2.2:

"When a user interacts with item <RECTOKEN> REC3078 REC3311 REC7479 REC2862 REC5552 REC4015 REC6914 REC4637 </RECTOKEN>, there is a 4.9% probability they will also interact with item <RECTOKEN> REC3078 REC3311 REC7479 REC4015 REC5211 REC2862 REC1723 REC6914 </RECTOKEN> (confidence: 0.049, lift: 652.45)"

Three metrics are reported alongside each association:

Probability (4.9%): the empirical fraction of users who interacted with the source item and also interacted with the target item, i.e., the support of the co-occurrence divided by the support of the source item.
Confidence (0.049): equivalent to the probability, expressed as a decimal rather than a percentage.
Lift (652.45): the ratio of the observed co-occurrence probability to what would be expected if the two items were independent. A lift of 652.45 means the two items co-occur 652 times more often than chance—an extraordinarily strong association.

What the CF data captures that raw sequences obscure. Consider two items that genuinely appeal to the same user segment but appear at very different positions in real recommendation slates (e.g., item A is presented at position 1, item B at position 50). In raw interaction logs, item A will accumulate many more interactions due to position bias alone, and the sequential pattern "A then B" will be rare because most users never scroll to position 50. The CF mining process, however, looks at any user who interacted with both items across their entire history (potentially across multiple sessions), not just sequential proximity. If users who like A also tend to like B—regardless of when or in what order they encountered them—the association rule will have high confidence and lift. The CF data thus captures a purer signal of item relatedness than sequential data alone.

Why this is "Layer 1" rather than the whole curriculum. The CF data teaches item relationships but does not teach sequential behavior. A model trained only on CF data would know that items A and B are related but would not learn the temporal patterns of how users typically move through a catalog (e.g., that users tend to browse similar items in bursts, or that certain item categories typically precede others in a purchase journey). That sequential knowledge is the domain of Layer 2.

The paper allocated approximately 7.6B tokens to CF data (Table 2), representing roughly 30% of the recommendation data—the largest single component. At 163B total training tokens, the CF data was repeated approximately 7.3 times (163B × 0.30 / 7.6B ≈ 6.4, adjusted to the paper's reported 7.3 repeats in Table 2).

Layer 2: Synthetic User Interaction Histories (UIH)

Purpose. The Synthetic UIH data is the crown jewel of the framework. Its purpose is to generate complete, realistic user interaction sequences that are free from the contaminating artifacts present in raw logs—specifically position bias and popularity bias. Because the sequences are generated algorithmically from an item-relationship graph rather than recorded from real user sessions, they are not tied to any individual's real behavior, providing inherent privacy benefits. The Layer 2 data directly addresses what the paper identifies as the two most damaging data pathologies: missing positives (items the user would have liked but never saw) and position-correlated interaction patterns (items that appear at the top of the screen get clicked regardless of relevance).

Graph construction: from CF data to a navigable item space. The generation process begins by constructing a graph where:

Nodes are individual items, each represented by their 8-token semantic ID.
Edges between items are weighted by the strength of the collaborative filtering relationships mined in Layer 1. An edge exists between item A and item B if the CF association rule mining found a meaningful co-occurrence pattern between them, and the edge weight reflects the confidence or lift of that association.
Walk parameters control how the graph is traversed to generate sequences (described below).

This graph construction is the purification mechanism that separates Layer 2 from raw logs. The graph encodes only the pairwise item relationship strengths—which are themselves averages over many user sessions and therefore less susceptible to individual-level biases—and discards all information about position, presentation order, temporal dynamics, and user-specific context. The random walk generation process then uses this purified graph as its sole basis for producing sequences.

Random walk algorithm: 2nd-order biased walks (Node2Vec). To generate a synthetic user interaction sequence, the system performs a random walk on the item graph. However, a simple 1st-order random walk (where the next step depends only on the current node) would produce sequences that lack the structural properties of real user behavior—users tend to explore similar items in bursts, occasionally jumping to a new category, and rarely oscillate between two items repeatedly. The paper uses the Node2Vec algorithm (Grover and Leskovec, 2016), which employs a 2nd-order biased random walk: the transition probability to a next node depends on both the current node $v$ and the previous node $t$ .

The unnormalized transition probability $\pi_{v,x}$ from current node $v$ to a candidate next node $x$ is given by:

$\pi_{v,x} = \alpha_{p,q}(t, x) \cdot w_{v,x}$

where $w_{v,x}$ is the static edge weight between $v$ and $x$ (derived from the CF association strength), and $\alpha_{p,q}(t, x)$ is the Node2Vec search bias that depends on the relationship between the previous node $t$ , the current node $v$ , and the candidate next node $x$ .

The search bias $\alpha_{p,q}(t, x)$ is controlled by two parameters:

Return parameter $p$ : Controls the likelihood of immediately revisiting the previous node $t$ . When $p > \max(q, 1)$ , the walk is less likely to return to $t$ ; when $p < \min(q, 1)$ , it is more likely. This parameter discourages 2-hop oscillations (A → B → A), which are unnatural in real browsing behavior.
In-out parameter $q$ : Controls whether the walk explores outward (BFS-like behavior) or stays local (DFS-like behavior). When $q > 1$ , the walk prefers nodes close to $t$ (approximately BFS, exploring the neighborhood of the starting point). When $q < 1$ , the walk prefers nodes farther from $t$ (approximately DFS, moving away from the starting point into new territory).

What it computes: At each step of the walk, given the current node $v$ and previous node $t$ , the algorithm considers all neighbors $x$ of $v$ , computes $\pi_{v,x}$ for each, normalizes to obtain a probability distribution, and samples the next node from that distribution. The walk continues until a termination condition is met (controlled by $\alpha_{stop}$ , the probability of stopping at any step), producing a sequence of item nodes.

Why this form: The 2nd-order dependency is critical for generating realistic sequences. A 1st-order walk would produce sequences where item transitions are independent of history, resulting in random-looking trajectories that fail to capture the coherence of real user sessions (where the next item is influenced by what the user has been looking at recently). The $p$ parameter directly eliminates the most unnatural artifact of 1st-order walks: the tendency to bounce back and forth between two highly-connected items. The $q$ parameter controls the exploration-exploitation tradeoff at the sequence level: low $q$ produces meandering walks that explore diverse parts of the catalog (analogous to a discovery-oriented browsing session), while high $q$ produces focused walks that stay within a tight neighborhood (analogous to a goal-directed search session).

Hyperparameter sweep for walk generation. The paper conducted a comprehensive sweep over 14 distinct configurations to select the optimal walk parameters (Appendix A, Table 9). The configurations varied:

$\alpha_{stop}$ (Alpha): The per-step termination probability. Values tested: 0.10, 0.15, 0.20. Higher values produce shorter sequences on average.
path_conf_threshold (Thresh): A minimum cumulative path confidence threshold used to filter out low-quality sequences. Values tested: None (no filtering), 1e-09, 2e-09, 3e-09.
Walk strategy: A 1st-order baseline (DeepWalk-style) and 2nd-order BFS-like strategy with $p = 0.5$ and $q = 2.0$ . The choice of $p = 0.5$ (below 1) means returning to the previous node is discouraged; $q = 2.0$ (above 1) means the walk prefers BFS-like local exploration.

The evaluation metrics for each configuration included:

Items: The number of unique items appearing in the generated sequences (higher is better—indicates catalog coverage).
Gini: The Gini coefficient of item frequency distribution (lower is better—indicates less popularity concentration).
Length: The average sequence length in items.
Tokens: Related to the total token count of generated sequences.
Token Gini: The Gini coefficient of token frequency (lower is better—less concentration on popular tokens).
Geom Lift: A geometric measure of item association strength preserved in the synthetic sequences.

The final selected configuration is highlighted in gray in Table 9: 1st_order_a015_thresh1e-09, which uses:

A 1st-order walk (rather than 2nd-order, interestingly—the ablation found it performed best on the evaluation metrics)
$\alpha_{stop} = 0.15$ (average sequence length of 8.0 items)
path_conf_threshold = 1e-09 (minimal filtering)

This configuration achieved an item count of 180K unique items, a Gini coefficient of 0.635 (lower than the 0.732 of the high-alpha 2nd-order configurations), and a Geom Lift of 385.54. The selection of a 1st-order walk is notable—it suggests that for this particular graph and evaluation criteria, the additional complexity of 2nd-order biasing did not improve the quality of generated sequences as measured by the chosen metrics, possibly because the CF edge weights already encode sufficient structural information about item relationships.

What a generated sequence looks like. Section 4.3.1 provides a concrete example:

"A user interacted with the following sequence of items: <RECTOKEN> REC870 REC5932 REC6271 REC1852 REC1624 REC409 REC6034 REC3608 </RECTOKEN>, <RECTOKEN> REC870 REC5524 REC180 REC4637 REC5552 REC6862 REC6033 REC4948 </RECTOKEN>, ... [8 items total]"

Each item in the sequence is represented by its full 8-token semantic ID block. The sequence contains 8 items (consistent with the average length of 8.0 from the selected configuration), and the model sees this as a single prompt during training: "A user interacted with the following sequence of items: [item1], [item2], ..." This format directly mirrors how recommendation models would need to condition on user history at inference time.

Privacy properties. Because the sequences are generated from an aggregated graph model of item relationships rather than from individual user histories, the synthetic UIH data is decoupled from any real user's activity. No synthetic sequence corresponds to any real user's session; each is a novel traversal of the item graph that respects the learned item-item affinities but has no one-to-one mapping to real behavior. This provides an inherent privacy benefit: if the training data were compromised, an adversary could not reconstruct any individual's browsing history because the training sequences never existed in reality.

Scale. The paper allocated approximately 2.8B tokens to UIH data in the main experiments (Table 2), representing roughly 11% of the total recommendation data. With 163B total training tokens, the UIH data was repeated approximately 6.1 times (Table 2 reports 6.1 repeats explicitly). The apparently small token count relative to CF data (2.8B vs. 7.6B) reflects the fact that each UIH sequence is compact (8 items × 8 tokens per item = 64 semantic tokens, plus natural language framing), whereas each CF rule statement includes extended explanatory text with multiple numerical metrics.

Data Mixture Design and Catastrophic Forgetting Prevention

The 50/50 general-to-recommendation split. The paper mixes general-domain text data with recommendation-specific synthetic data at a fixed ratio of 50% general domain, 50% recommendation domain. This choice is motivated by the well-documented phenomenon of catastrophic forgetting in continual pre-training: if a pre-trained model is fine-tuned exclusively on domain-specific data, its performance on general language understanding tasks degrades as the model's parameters are pulled toward the new distribution, overwriting previously learned capabilities (Ibrahim et al., 2024). The 50/50 split ensures the model continues to see a substantial volume of general-domain text, maintaining its base language capabilities while also acquiring recommendation-specific knowledge.

The general-domain data is sourced from three subsets of the SmolLM-Corpus (Ben Allal et al., 2024):

cosmopedia-v2: A diverse corpus of synthetic and web text covering a broad range of topics.
fineweb-edu-dedup: High-quality educational web text with deduplication applied.
python-edu: Python programming educational content.

These three subsets provide 128B total tokens (Table 2), and since the general domain constitutes 50% of the 163B training budget, each general-domain token is seen less than once on average (the paper reports "< 1" repeats in Table 2)—the general data is not exhausted during training.

Internal recommendation data ratios. Within the 50% recommendation-domain allocation, the three data types are mixed at the following ratios (Table 2):

Item-Text: 9% of recommendation data (2.0B unique tokens)
CF: 30% of recommendation data (7.6B unique tokens)
UIH: 11% of recommendation data (2.8B unique tokens)

These ratios are not arbitrary—they reflect the paper's finding that CF data provides the strongest foundational signal (the 30% allocation is the largest), while UIH provides the most efficient per-token learning but is given a smaller allocation to control overfitting (as explored in the ablation studies of Section 7.2). The Item-Text allocation is relatively small (9%) because the paper found that item semantics saturate quickly in the per-model scaling analysis (α ≈ 0.13–0.21 in Table 5), meaning additional Item-Text tokens provide diminishing returns.

Ablation mixture configurations. Section 7.1 studies four alternative mixture configurations (Table 6):

Item-text + CF + UIH (full mixture): 9%/30%/11% split—the main configuration.
CF + UIH (no Item-Text): 0%/37%/13%, where the Item-Text allocation is redistributed to CF and UIH proportionally. This tests whether Item-Text data is necessary or whether the model can learn item semantics implicitly from CF and UIH data.
Item-text + UIH (no CF): 22.5%/0%/27.5%. This tests whether the CF explicit association rules are necessary or whether UIH sequences alone encode sufficient collaborative signal.
UIH only: 0%/0%/50%. This isolates the contribution of UIH data alone, with no CF or Item-Text support.

All configurations maintain the 50% general-domain split, so the recommendation-domain percentages sum to 50%.

The 163B total token budget. All models are trained for 163B tokens total, which the paper notes is "over 20× (chinchilla scaling multiplier Hoffmann et al. (2022)) for 8B models." This means the 8B model sees approximately 20 tokens per parameter (163B / 8B = 20.4), which is roughly the Chinchilla-optimal ratio for pre-training from scratch. For the smaller models, this represents substantial over-training: the 0.6B model sees 163B / 0.6B ≈ 272 tokens per parameter, far beyond the Chinchilla-optimal ratio. This design choice is deliberate—it enables the paper to study how recommendation-domain scaling behaves across different positions on the under-trained to over-trained spectrum within a single experiment.

Continual Pre-Training Configuration

Base model. All experiments use Qwen3 models (Yang et al., 2025a) at four scales: 0.6B, 1.7B, 4B, and 8B parameters. The paper describes Qwen3 as "one of the best open-weight LLM models available now." These are decoder-only transformer models pre-trained on a large corpus of general-domain text; the CPT experiments adapt them to the recommendation domain while preserving their general language capabilities.

Hardware and training setup. Training is performed on B200 clusters (NVIDIA Blackwell architecture GPUs). The specific number of GPUs and training duration are not reported, but the total compute can be estimated from the standard transformer FLOPs approximation $C = 6ND$ , which yields approximately $6 \times 8\text{B} \times 163\text{B} = 7,824$ exaFLOPs for the 8B model—a substantial but not extreme training run by modern standards.

Hyperparameters. The paper reports the following CPT hyperparameters (Section 6.1):

Global batch size: 512 sequences. This is the total number of sequences processed per optimizer step across all GPUs.
Context window: 512 tokens. Each training sequence is truncated or padded to exactly 512 tokens. This is relatively short by modern LLM standards (where 4K–32K is common) but is appropriate for the recommendation data, where individual sequences (item descriptions, CF rules, UIH sequences) are typically under 200 tokens.
Learning rate: Peak at 1e-4, with a linear warmup over the first 100 steps followed by a cosine decay schedule over the remaining training. The 1e-4 peak rate is relatively standard for continual pre-training of models in this size range.
Optimizer: Not explicitly specified, but the use of cosine decay implies AdamW (the default for nearly all modern LLM training).
Total training tokens: 163B for all model sizes.

Training stability. Figure 6(a) in the paper shows the training loss curves for all four model scales in the main experiment (the one corresponding to Figure 2). All models show "smoothly and monotonically decreasing training loss," indicating stable training without divergence or loss spikes. The larger models achieve lower final training loss (as expected), with the ordering 0.6B > 1.7B > 4B > 8B in final training perplexity.

Evaluation Methodology and Held-Out Test Sets

Perplexity as the primary metric. The paper uses perplexity on held-out test sets as the primary evaluation metric throughout the scaling law experiments. Perplexity is the exponentiated average negative log-likelihood per token: a perplexity of $k$ means the model is as uncertain about each token as if it were choosing uniformly among $k$ equally likely options. Lower perplexity is better, with a theoretical minimum of 1.0 (perfect prediction).

Perplexity is chosen over downstream recommendation metrics (like Recall@K or NDCG) for the scaling law analysis because it is a continuous, well-behaved metric that directly measures the model's language modeling quality on the target domain. The downstream ranking experiments in Section 5.2 (using SASRec, GRU4Rec, etc.) serve to validate that the synthetic data captures useful recommendation signals, but the scaling laws themselves are established using perplexity to enable clean mathematical fitting.

Seven evaluation domains. The paper constructs held-out test sets across seven distinct data modalities (Table 3), each designed to probe a different aspect of the model's learned knowledge:

General Domain: 200K randomly sampled entries (102.4M tokens) from the general-domain data. This measures retention of general language capabilities after CPT.
Item-Text: Item-text pairs for items not present in the training set. 200K entries (102.4M tokens). This measures generalization of the item-text alignment skill to unseen items—if the model has learned the mapping function rather than memorizing specific item-text pairs, it should perform well on new items.
CF Both Seen: Collaborative filtering edges between two items where both items appeared in the training data, but the specific edge (association) was held out. 200K entries (102.4M tokens). This measures whether the model has learned the underlying CF function (inferring that if items A and B are similar in content, they should have high co-occurrence) rather than memorizing specific association rules.
CF One Unseen: CF edges between two items where one item appeared in training and the other did not. 200K entries (102.4M tokens). This is a harder generalization test: can the model predict associations involving items it has never seen in the CF context?
CF Both Unseen: CF edges where neither item nor the edge appeared in training. Approximately 75K entries (38M tokens). This is the hardest CF generalization test, requiring the model to infer item relationships purely from content understanding.
UIH OOD (Out-of-Distribution): UIH sequences sampled only from edges in the CF test set (not used during training). Approximately 48.3K entries (24.7M tokens). This measures whether the model has learned generalizable sequential patterns or has overfit to the specific graph walks in the training data.
UIH Full Graph: UIH sequences sampled from both the CF training edges and CF test edges. Approximately 51.7K entries (26.5M tokens). This is an easier, in-distribution UIH test that includes sequences similar to those seen during training.

Cross-validation and data splitting. The paper's evaluation framework is notable for what it does NOT do: it does not use a standard random train/test split on the raw interaction logs. Instead, it carefully constructs test sets that probe generalization along specific axes. For the CF test sets, the split is at the edge level rather than the item level: edges are held out, but both items in a "Both Seen" edge may appear in training through other edges. This is a more realistic evaluation because in production, the model would need to recommend items that have been seen before but in new combinations, or items that are new to the catalog.

The UIH OOD test set is particularly important for assessing overfitting: because it is generated from CF edges that were explicitly held out, any degradation in UIH OOD perplexity relative to UIH Full Graph perplexity indicates that the model is memorizing specific graph walks rather than learning the underlying sequential behavior generator. The paper exploits this distinction extensively in Section 7.2 when analyzing the effects of data repetition.

Scaling Law Fitting Methodology

The paper fits two types of scaling laws, which operate at different levels of analysis and reveal complementary insights.

Per-model fits: $\ell(D) = L_\infty + A \cdot D^{-\alpha}$ . For each model size and each evaluation domain, the paper fits a three-parameter power-law decay:

$\ell(D) = L_\infty + A \cdot D^{-\alpha}$

where $D$ is the number of training tokens seen (in billions), $\ell(D)$ is the perplexity at that point, and the three fitted parameters are:

$L_\infty$ (asymptotic loss): the irreducible perplexity floor—the theoretical minimum perplexity achievable with infinite data. This captures the intrinsic entropy of the domain: a domain where user behavior is highly predictable will have a low $L_\infty$ ; a domain with high inherent randomness will have a high $L_\infty$ .
$A$ (scale factor): controls the initial gap between the starting perplexity and $L_\infty$ . A larger $A$ means the model starts further from the asymptotic floor and has more room to improve.
$\alpha$ (scaling exponent): the rate at which perplexity decreases per unit of additional training data. A higher $\alpha$ means faster learning—perplexity drops more steeply as more tokens are processed. $\alpha$ is the key metric for comparing data efficiency across domains.

What it computes: Given a set of (training_tokens, perplexity) pairs collected during training, the fitting procedure finds the values of $L_\infty$ , $A$ , and $\alpha$ that minimize the mean squared error between the predicted $\ell(D)$ and the observed perplexity. The resulting curve can be extrapolated to predict what perplexity the model would achieve if trained on more (or fewer) tokens.

Why this form: The power-law form is the standard parameterization in scaling law literature (Kaplan et al., 2020; Hoffmann et al., 2022) because neural network training loss empirically follows power-law decay in the data size. It captures the phenomenon of diminishing returns: each additional doubling of data reduces loss by a smaller absolute amount than the previous doubling. The $L_\infty$ term captures the fact that loss cannot go to zero—there is always some irreducible uncertainty in predicting the next token. Without $L_\infty$ , the power law would predict loss → 0 as $D \to \infty$ , which is physically impossible.

Per-model fit results (Table 5). The fitted parameters reveal a clear hierarchy of data efficiency across modalities:

General Domain: $\alpha \approx 0.020–0.027$ across model sizes. Near-saturation—the pre-trained Qwen3 checkpoint already encodes extensive general-domain knowledge, so additional general-domain tokens provide minimal improvement. $L_\infty$ decreases with model size (0.96 at 0.6B, 0.79 at 8B), consistent with the intuition that larger models capture rarer linguistic patterns.
Item-Text: $\alpha \approx 0.126–0.214$ . Moderate scaling—the model continues to improve its item-text alignment with more data, but the rate is modest. $L_\infty$ ranges from 0.86 at 4B to 0.93 at 8B, with no clear monotonic trend, suggesting that item-text alignment does not strongly benefit from increased model capacity beyond 4B parameters.
CF (three subsets): $\alpha \approx 0.282–0.358$ . Strong scaling—the model benefits substantially from additional CF tokens. $L_\infty$ is remarkably stable across model sizes (0.33–0.36), indicating that even 0.6B parameters are sufficient to represent the underlying CF patterns at convergence. This low $L_\infty$ (0.33–0.36) reflects the low-rank structure of user-item co-occurrence matrices—collaborative filtering is an inherently compressible problem.
UIH (Full and OOD): $\alpha \approx 0.453–0.689$ . The strongest scaling of any domain—the model continues to improve rapidly with additional UIH tokens. $L_\infty$ is stable across model sizes (0.64–0.70), again suggesting that model capacity is not the bottleneck. The UIH OOD exponents are consistently higher than UIH Full exponents (e.g., 0.689 vs. 0.588 at 8B), which is counterintuitive: the harder test set shows faster improvement with data. This likely reflects the OOD test set having a higher initial perplexity (more room to improve) and the model genuinely learning transferable sequential patterns rather than memorizing specific walks.

A notable pattern in Table 5: $\alpha$ increases with model size for CF and UIH, but not for General or Item-Text. At 8B vs. 0.6B, UIH Full $\alpha$ changes from 0.453 to 0.588 and UIH OOD from 0.555 to 0.689. This suggests a model-data interaction where larger models can extract more information per token from complex sequential data, but this advantage does not extend to simpler data types (General text is already well-learned; Item-Text relationships are relatively shallow).

Joint fits: $\ell = E + A \cdot N^{-\alpha} + B \cdot D^{-\beta}$ . The joint scaling law simultaneously models the effects of model size and data size:

$\ell = E + A \cdot N^{-\alpha} + B \cdot D^{-\beta}$

where $N$ is the number of model parameters, $D$ is the number of training tokens, and the fitted parameters are:

$E$ : irreducible loss (domain entropy floor). This is the same concept as $L_\infty$ in the per-model fits but fit jointly across all model sizes.
$A, \alpha$ : model-size scaling parameters. $\alpha$ controls how much loss decreases as model size increases; $A$ scales the contribution of the model-size term to total loss.
$B, \beta$ : data scaling parameters. $\beta$ controls how much loss decreases as training data increases; $B$ scales the contribution of the data term.

What it computes: This is a six-parameter fit ( $E, A, \alpha, B, \beta$ ) to the full set of (model_size, training_tokens, perplexity) observations across all model scales. The additive structure assumes that model-size effects and data-size effects are independent and sum to produce the total loss—an assumption that the paper's own per-model results partially challenge (since $\alpha_{\text{data}}$ varies with model size for CF and UIH), but which provides a useful first-order decomposition.

Why this form: The additive decomposition separates the contributions of model scaling and data scaling, enabling the paper to answer questions like: "For recommendation data, should I invest my compute budget in a larger model or more training tokens?" If $A \cdot N^{-\alpha}$ dominates the total loss, then increasing model size provides more benefit; if $B \cdot D^{-\beta}$ dominates, then increasing data provides more benefit. The standard Chinchilla analysis (Hoffmann et al., 2022) uses this same decomposition to derive the optimal ratio of tokens to parameters.

Joint fit results (Equations 2–5). The fitted joint scaling laws are:

$\ell_{\text{general}} = 0.79 + 16500 N^{-0.511} + 3.85 D^{-0.048}$

$\ell_{\text{item-text}} = 0.473 + 5720 N^{-0.511} + 4.04 D^{-0.070}$

$\ell_{\text{cf}} = 0.193 + 21 N^{-0.277} + 6.79 D^{-0.148}$

$\ell_{\text{UIH}} = 0.514 + 1.89 N^{-0.138} + 63.9 D^{-0.272}$

The most striking pattern is the inverse relationship between $\alpha$ (model-size exponent) and $\beta$ (data exponent) across domains:

General and Item-Text: High $\alpha$ (0.511 for both), low $\beta$ (0.048 and 0.070). These domains are model-size dominant—performance improves primarily by scaling up model capacity. The model-size scaling coefficient $A$ is enormous (16,500 for General, 5,720 for Item-Text), meaning that doubling model size provides a large absolute loss reduction.
UIH: Low $\alpha$ (0.138), high $\beta$ (0.272). This domain is data dominant—performance improves primarily by increasing data coverage. The data scaling coefficient $B$ is large (63.9), meaning that doubling training tokens provides a substantial loss reduction.
CF: Intermediate on both axes ( $\alpha = 0.277$ , $\beta = 0.148$ ). A balanced regime where both model size and data matter, but the smaller $A$ coefficient (21) means that model-size scaling contributes relatively little absolute loss reduction compared to the data term.

The irreducible losses $E$ reveal a domain hierarchy of predictability: CF is most predictable (0.193), followed by Item-Text (0.473), UIH (0.514), and General (0.79). This ordering makes intuitive sense: CF patterns are constrained by the low-rank structure of user-item matrices (highly predictable), while natural language has far more possible valid continuations at each token position (highly unpredictable).

Compute-Optimal Allocation Analysis

FLOPs accounting. The paper uses the standard transformer FLOPs approximation:

$C = 6ND$

where $C$ is the total training compute in FLOPs, $N$ is the number of model parameters, and $D$ is the number of training tokens. The factor of 6 comes from the forward pass (approximately $2ND$ FLOPs per token) plus the backward pass (approximately $4ND$ FLOPs per token) for standard transformer training.

The optimality condition. Given a fixed compute budget $C = 6ND$ , minimizing the joint loss $\ell = E + A N^{-\alpha} + B D^{-\beta}$ subject to this constraint yields the condition that the marginal loss reduction per FLOP must be equal for model size and data:

$\alpha A \cdot N^{-\alpha} = \beta B \cdot D^{-\beta}$

In words: at the compute-optimal point, the benefit you get from spending one additional FLOP on increasing model size equals the benefit you get from spending one additional FLOP on increasing training data. If the left-hand side is larger, you should allocate more compute to model size; if the right-hand side is larger, you should allocate more compute to data.

What it computes: For a given domain, plugging in the fitted $\alpha$ , $A$ , $\beta$ , and $B$ values tells you the optimal ratio of $N$ to $D$ at any compute budget. Equivalently, the quantities $\alpha A$ and $\beta B$ (shown in Table 4) tell you the relative importance of model scaling vs. data scaling: a larger $\alpha A$ means model size matters more; a larger $\beta B$ means data matters more.

Why this form: This derivation is standard in the scaling laws literature (it's how Hoffmann et al. derived the Chinchilla optimal 20 tokens/parameter ratio). The key insight it operationalizes is that model size and data size are substitutable but not equivalent—increasing one while holding the other constant yields diminishing returns, and the optimal strategy balances both.

Results (Table 4). The computed values reveal sharply different optimal strategies:

Domain	$\alpha A$	$\beta B$	Implication
General	8,432	0.185	Overwhelmingly model-size dominant
Item-Text	2,923	0.283	Strongly model-size dominant
CF	5.82	1.005	Moderately model-size dominant
UIH	0.261	17.38	Strongly data dominant

For the General domain, $\alpha A$ is over 45,000× larger than $\beta B$ , meaning that at the current operating point, compute is vastly better spent on increasing model size than on increasing training data. This is expected for pre-trained models undergoing CPT on similar-distribution data—the general-domain knowledge is already well-covered, so more data provides minimal benefit.

For UIH, the situation reverses: $\beta B$ is roughly 67× larger than $\alpha A$ , meaning that compute should be overwhelmingly allocated to data collection. The paper quantifies this concretely:

"Halving the data-dependent loss term for General requires $2^{1/0.048} \approx 1.8 \times 10^6$ times more data, whereas for UIH it requires only $2^{1/0.272} \approx 13$ times more data."

In plain English: to cut the data-contributed portion of the loss in half for the General domain, you would need nearly 2 million times more data—practically impossible. For UIH, you only need 13 times more data—very achievable. This is a stark operational difference: if you're building an LLM-based recommender and want to improve UIH performance, collecting more (synthetic) data is a highly efficient strategy; if you want to improve general-domain performance, you need a bigger model, not more tokens.

Compute-optimal scaling curves (Figure 8). The paper visualizes the compute-optimal frontier by plotting perplexity against compute (EFLOPS) for model sizes from 0.6B to 8B. For General Domain, the Pareto frontier is clear: the 8B model achieves the lowest perplexity at the highest compute, and no smaller model can match it regardless of how much it's overtrained. For recommendation domains, however, the picture is different:

Collaborative Filtering saturates at 4B parameters, with the 4B and 8B models achieving nearly identical perplexity (~1.47 PPL). The compute-optimal strategy would be to train a 4B model rather than an 8B model, since the extra parameters provide no benefit.
UIH OOD shows inverse scaling: the heavily over-trained 0.6B model (~2.01 PPL) actually outperforms the near-Chinchilla-optimal 8B model (~2.10 PPL). The paper attributes this to larger models memorizing repeated behavioral sequences, leading to poorer out-of-distribution generalization, while the extreme over-training of the 0.6B model (454× tokens/parameter) acts as implicit regularization against memorization.

Data Mixture Ratio Experiments (Section 7.2)

Purpose and setup. Section 7.2 investigates how the ratio of UIH data in the training mixture affects scaling behavior and overfitting. The experiments use a reduced UIH dataset of 42M tokens (sampled from the full 2.7B-token UIH dataset), then vary its mixture ratio from 0.5% to 15% of the total training data (Table 7). As the UIH ratio increases while the total UIH tokens remain fixed, the number of repeats over the reduced UIH data increases proportionally:

At 0.5% ratio: 0.82B UIH tokens consumed, 4.1 repeats
At 1% ratio: 1.63B UIH tokens consumed, 8.2 repeats
At 2% ratio: 3.26B UIH tokens consumed, 16.3 repeats
At 5% ratio: 8.15B UIH tokens consumed, 40.8 repeats
At 15% ratio: 24.45B UIH tokens consumed, 122.3 repeats

The other recommendation-domain allocations are adjusted proportionally to maintain a 50% total recommendation ratio.

Key finding: overfitting threshold at ~16 repeats. Figure 5 and Table 7 reveal a clear pattern: UIH perplexity on held-out evaluation sets starts to increase (worsen) once the reduced UIH data has been repeated approximately 16 times, regardless of the absolute mixture ratio or total training tokens:

At 2% ratio: overfitting emerges at ~80B total training tokens (16.3 repeats at that point).
At 5% ratio: overfitting emerges at ~80B total training tokens (40.8 repeats at end of training, but 16 repeats is crossed earlier).
At 15% ratio: overfitting emerges at ~20B total training tokens (far earlier in training, at approximately 16 repeats as well).

The training loss (Figure 6b) continues to decrease monotonically even as evaluation perplexity increases—a classic signature of overfitting: the model is memorizing the training data rather than learning generalizable patterns.

Why ~16 repeats is the threshold. The paper contextualizes this finding against prior work: Yang et al. (2024) found that CPT with 4 repeats could enable knowledge learning given well-constructed synthetic data, while Muennighoff et al. (2025) found that training beyond 4 repeats gives diminished returns and 20+ repeats can decrease generalization (Hron et al., 2024). The paper's 16-repeat threshold is notably higher than the 4-repeat guideline from prior work, suggesting that the synthetic UIH data is of sufficiently high quality and diversity to support more repetition before overfitting sets in. The eventual degradation at 16+ repeats is consistent with Allen-Zhu and Li (2024b) and Yang et al. (2024), who found that "performance degrades sharply when repetitions exceed approximately 100× the original dataset size."

Model scale independence of overfitting threshold. Figure 7 extends the mixture ratio analysis across model scales (0.6B to 32B) at two fixed UIH ratios (2% and 15%). The key finding is that when the perplexity starts to increase depends on the UIH mixture ratio and is independent of model scale:

At 15% ratio: all model sizes begin overfitting at ~20B total training tokens (≈16 repeats).
At 2% ratio: all model sizes begin overfitting at ~160B total training tokens (≈16 repeats).

However, the paper notes that the overfitting effect "becomes more significant as the model gets larger"—larger models, with greater memorization capacity, suffer more severe generalization degradation when the data is over-repeated.

Asymmetric Cross-Domain Transfer (Ablation Study, Section 7.1)

Setup. The ablation study in Section 7.1 trains models with four different data mixtures (Item-Text + CF + UIH, CF + UIH, Item-Text + UIH, UIH only) and evaluates perplexity on all evaluation domains. The critical question is: does training on one data type improve performance on the evaluation set of a different data type?

Key finding: CF → UIH transfer is strong; UIH → CF transfer is negligible. Figure 4 plots UIH evaluation perplexity against the number of UIH training tokens for the four mixture configurations. The results show:

CF + UIH achieves the lowest UIH perplexity (L∞ = 0.66 for UIH Full; L∞ = 0.69 for UIH OOD), outperforming even the UIH-only baseline (L∞ = 0.95 and 0.75 respectively). This means that including CF data alongside UIH training tokens improves the model's ability to model user interaction sequences—a cross-domain transfer effect.
UIH-only training provides zero benefit on CF tasks (α ≈ 0 for CF evaluation when trained on UIH only; the L∞ is stuck at 1.30 regardless of training tokens). This means that learning to predict sequential user behavior does not implicitly teach the model the pairwise item association patterns that CF data explicitly encodes.

The paper quantifies the CF → UIH benefit as a 31% reduction in asymptotic UIH perplexity (L∞ from 0.95 to 0.66 for UIH Full). The interpretation is that CF data provides "complementary signals that benefit sequential user behavior modeling"—knowing which items are related (from CF) helps the model understand why certain item transitions occur in sequences, even when those transitions aren't explicitly present in the UIH training data.

Why the asymmetry matters. This finding validates the paper's layered curriculum design. The two layers are not redundant—they teach different things, and learning Layer 1 (CF) facilitates learning Layer 2 (UIH), but not vice versa. The paper's interpretation is that:

"Collaborative filtering signals, which capture user-item affinity patterns, provide complementary information that benefits sequential user behavior modeling. In contrast, the reverse transfer does not hold: training on UIH data alone yields virtually no improvement on CF tasks."

This suggests a pedagogical ordering: the model benefits from learning the "grammar" of item relationships (CF) before learning the "paragraphs" of user sequences (UIH). The reverse order (learning sequences first, then trying to extract pairwise relationships) is substantially harder, likely because the pairwise signal is diluted by the sequential context.

Domain-specific data remains essential. Despite the positive CF → UIH transfer, the ablation confirms that domain-specific training data remains critical:

Models trained without CF data plateau at CF loss of L∞ = 1.30, compared to L∞ = 0.35 with CF data—a 3.7× higher irreducible loss.
Models trained without Item-Text data show severe degradation on Item-Text evaluation (L∞ ≈ 3.0 vs. L∞ ≈ 1.2 with Item-Text data). The Item-Text evaluation L∞ of ~3.0 with no Item-Text training is essentially random-guessing-level perplexity, indicating that semantic grounding cannot be acquired implicitly from CF or UIH data.

Trade-off: Item-Text data slightly degrades UIH performance. Including Item-Text data in the mixture (the full Item+CF+UIH configuration) yields L∞ = 0.76 for UIH Full, slightly worse than the CF+UIH configuration (L∞ = 0.66). This suggests a modest negative transfer: the 9% of training tokens spent on Item-Text data slightly interferes with the model's ability to learn sequential behavior patterns, possibly because the item-text alignment task competes for model capacity or because the simpler Item-Text patterns create a local minimum that is not optimal for the more complex UIH task. The paper notes this as a "modest trade-off" and suggests that "practitioners should consider their downstream priorities when designing data mixtures."

Summary of Key Design Choices and Their Justifications

Pre-trained SAE over in-domain RQ-VAE/RQ-kmeans for semantic tokenization: The SAE provides superior perplexity across all evaluation domains (Figure 12), despite the theoretical risk of semantic mismatch from its video-centric pre-training. The practical performance advantage outweighed the domain-alignment concern.
50/50 general-to-recommendation split: Catastrophic forgetting prevention is essential for CPT; the equal split ensures the model retains its base language capabilities while acquiring recommendation knowledge. The paper does not experiment with alternative ratios (e.g., 70/30, 30/70), leaving the sensitivity of results to this choice as an open question.
2nd-order biased random walks for UIH generation: The Node2Vec algorithm's ability to control exploration-exploitation tradeoffs via $p$ and $q$ parameters enables the generation of sequences with realistic structural properties (bursty exploration, avoidance of immediate returns). Despite this, the hyperparameter sweep selected a 1st-order configuration (1st_order_a015_thresh1e-09), suggesting that for the Merrec graph, the edge weights alone encode sufficient structure.
CF data as the largest recommendation component (30%): The CF data provides the foundational relational knowledge that enables efficient learning in Layer 2. The ablation study confirms that CF data improves UIH modeling (cross-domain transfer), justifying its prominent allocation.
163B total training tokens across all model sizes: This fixed budget places models at different positions on the Chinchilla spectrum (from heavily over-trained at 0.6B to near-optimal at 8B), enabling analysis of how recommendation-domain scaling interacts with the training-to-model-size ratio.
Perplexity as the scaling law metric rather than downstream ranking metrics: Perplexity provides a continuous, well-behaved objective for fitting power laws and directly measures the model's learning of the training distribution. The TSTR experiments (Section 5.2) separately validate that lower perplexity on recommendation data translates to better downstream ranking performance.
Seven evaluation domains probing different generalization axes: The multi-faceted evaluation (in-distribution vs. OOD, seen vs. unseen items and edges) enables the paper to distinguish genuine learning from memorization—critical for understanding when and why overfitting occurs in the data mixture experiments.

4. Key Insights and Innovations

Innovation 1: Data Quality as the Gating Factor for Recommendation Scaling Laws—A Reframing of the Problem from Architecture to Data Engineering

The paper's most fundamental conceptual move is not a new model, loss function, or training algorithm. It is a diagnostic reframing of why LLMs have failed to scale in recommendation. Prior work—LUM (Yan et al., 2025), SUAN (Lai et al., 2025), the OneRec series (Zhou et al., 2025a,b), and PLUM (He et al., 2025)—all shared an implicit assumption: scaling failures were architectural or algorithmic problems to be solved through better model design. LUM proposed a new training objective; SUAN pursued knowledge distillation; OneRec explored generative architectures. Each asked, in effect, "what model architecture would scale better?"

This paper identifies the shared blind spot: all of these efforts fed their models fundamentally contaminated data. The PLUM result—a 3B MoE model failing to outperform a 900M counterpart—is not presented as a deficiency of the PLUM architecture but as a smoking gun for the data quality hypothesis. If model capacity were the bottleneck, the larger model should have improved. That it didn't implies the training data contained insufficient learnable signal for the extra parameters to exploit.

The reframing is best captured by the paper's distinction between mitigation and purification. The existing debiasing literature (inverse propensity scoring, causal inference, adversarial training—cited in Section 2.3) attempts to make models robust to data flaws. This paper argues such approaches fight an uphill battle: they try to extract a clean signal from a contaminated source. The alternative—engineer the data to be clean at the source—is more direct and, the paper argues, is the prerequisite for scaling laws to emerge. This is not a small tweak; it shifts the engineering focus from the model training pipeline to the data generation pipeline.

Why this matters beyond the paper's own results: if the diagnosis is correct, then the entire research program of building LLM recommenders on raw interaction logs is inherently bounded. No amount of architectural sophistication can compensate for training data where the dominant statistical regularities are the biases themselves (position, popularity, exposure). The paper is arguing, in effect, that the field has been optimizing the wrong thing—and that this explains why scaling laws, which are ubiquitous in NLP, have been conspicuously absent in recommendation. This is a fundamental reframing rather than an incremental improvement: it changes what "the problem" is understood to be.

The evidence anchoring this claim is multi-layered. The PLUM case study (Section 3.3) provides the negative example—scaling failure under the old paradigm. The TSTR experiments (Section 5.2, Figure 1) provide the positive example—standard sequential models trained on synthetic data outperform those trained on real data by large margins (SASRec +130% recall@100). And the scaling law results themselves (Section 6.2, Figures 2–4) are the existence proof: with clean data, power-law scaling emerges where none existed before. The chain of evidence is carefully constructed: failure → diagnosis → cure → validation.

A subtle but important aspect of this innovation: the paper doesn't claim synthetic data is "better" in some abstract sense. It claims synthetic data is better specifically at enabling scaling laws because it removes the data pathologies that cause scaling curves to flatten prematurely. This is a more precise and falsifiable claim than "clean data is good."

Innovation 2: The Concept of a Pedagogical Data Curriculum for Recommendation—Teaching Principles Rather Than Reproducing Surface Statistics

Traditional synthetic data generation for recommendation—using GANs (Bharadhwaj et al., 2018), VAEs (Adouani and Dagdia, 2025), or statistical methods (Qi et al., 2020)—aims to mimic the source distribution. The generator is trained to produce samples indistinguishable from real data; success is measured by statistical fidelity. The paper explicitly rejects this objective, and the rejection is the intellectual core of the framework:

"The objective is not mimicry but pedagogy."

This is a conceptual departure, not just a methodological one. What does "pedagogy" mean in the context of training data? It means the data is designed to teach the model a structured understanding of the domain—principles, relationships, and patterns—rather than to replicate surface statistics. A mimicry approach would try to reproduce the exact distribution of item frequencies in the real logs (including the popularity bias). A pedagogical approach explicitly breaks that distribution: the synthetic UIH data, generated from random walks on a CF graph, has no position bias because the random walk has no concept of position.

The layered structure of the curriculum operationalizes this philosophy. Layer 1 teaches "vocabulary and grammar"—what items are (Item-Text alignment) and how they relate to each other (CF association rules). Layer 2 teaches "paragraphs"—how users move through sequences of items (UIH). This is explicitly compared to curriculum learning (Soviany et al., 2022; Chen et al., 2025a), where training examples are presented in order from simple to complex. But the paper's curriculum is not just an ordering of existing data; it is a constructed progression where each layer builds on the abstractions established by the previous one.

The pedagogical framing has a specific implication that distinguishes this work from prior synthetic data efforts: the synthetic data is not evaluated primarily by its fidelity to real data (though Section 5.1 performs a statistical fidelity check). It is evaluated by its utility as a teaching instrument—measured through TSTR experiments (Section 5.2) and, most critically, through whether it enables scaling laws. A synthetic dataset that perfectly matched real-data statistics (including biases) would fail the utility test because it would reproduce the scaling pathologies. The paper's synthetic data succeeds precisely because it departs from real-data statistics in specific, principled ways.

This innovation is fundamental rather than incremental: it changes the objective function for synthetic data generation in recommendation. The goal shifts from "generate data that looks like the real data" to "generate data that teaches the model what it needs to know." This is a framework-level contribution that other researchers can adopt and extend independently of the specific data generation methods used in this paper.

The evidence for the pedagogical value comes from several sources. The TSTR experiments (Figure 1) show that models trained on pedagogical synthetic data generalize better to real test data than models trained on real training data—despite the distribution mismatch. The ablation study (Section 7.1, Figures 3–4) shows that each curriculum layer provides non-redundant learning signals: removing Layer 1 CF data degrades UIH learning, and removing Item-Text data causes catastrophic semantic forgetting (L∞ ≈ 3.0 on Item-Text evaluation). The asymmetric cross-domain transfer (CF improves UIH, but UIH doesn't improve CF) validates the pedagogical ordering—the model benefits from learning foundational item relationships before tackling sequential behavior.

Innovation 3: The Asymmetric Cross-Domain Transfer Finding—CF Data as a Complementary Signal That Enables More Efficient UIH Learning

The discovery that including CF data alongside UIH data reduces asymptotic UIH perplexity by 31% (L∞ from 0.95 to 0.66) while UIH data provides zero benefit for CF tasks (α ≈ 0) is the paper's most scientifically interesting result, and it carries implications beyond the recommendation domain.

This asymmetry was not hypothesized a priori—the paper's original motivation for the two-layer curriculum was to provide complementary knowledge, but the direction and magnitude of the transfer were empirical discoveries. The finding reveals something fundamental about the relationship between global pairwise preference patterns (CF) and local sequential behavior patterns (UIH): knowing which items tend to be co-preferred by users (CF) helps a model understand why certain items appear in sequences together, but observing many sequences (UIH) does not automatically teach the model which pairs are strongly associated. The global structure provides a scaffold for interpreting local transitions, but not vice versa.

This has a clear pedagogical interpretation: learning the "grammar" of item relationships (CF) facilitates learning the "sentences" of user behavior (UIH), but exposure to many sentences doesn't automatically teach the grammar rules—or at least not efficiently. This asymmetry may explain why models trained purely on raw interaction logs (which are essentially UIH data with no explicit CF signal) exhibit sub-scaling: they are attempting to learn both the pairwise structure and the sequential structure simultaneously from data where the pairwise signal is diluted and contaminated by position and exposure biases.

The finding also suggests a design principle for multi-modal or multi-task training in any domain: when one task captures global, aggregated structure (like CF rules averaged over millions of users) and another captures local, sequential structure (like individual user sessions), presenting the global structure first may create representations that accelerate learning on the local task. This is a testable hypothesis that extends beyond recommendation to any domain with hierarchical data structure—for instance, learning document-level topic models before sentence-level language modeling, or learning graph-level community structure before random-walk-based node embeddings.

This innovation is incremental in scope but fundamental in implication: it's a single empirical finding, but it validates the entire layered curriculum design and provides a principle that other researchers can apply. The evidence is clean and unambiguous (Figure 4, the CF+UIH curve consistently below UIH-only across all training token counts), and the quantification (31% reduction in L∞) gives a concrete magnitude to the benefit.

Innovation 4: The First Empirical Characterization of Overfitting Dynamics for Synthetic Recommendation Data—The ~16-Repeat Threshold

The paper's data mixture experiments (Section 7.2) produce a finding with direct practical implications: for the synthetic UIH data, evaluation perplexity starts to degrade after approximately 16 repeats over the dataset, and this threshold is independent of model scale but the degradation is more severe for larger models.

This finding is not merely an observation; it is a quantitative operational boundary that tells practitioners exactly how much they can reuse a fixed synthetic dataset before returns turn negative. Prior work (Yang et al., 2024; Muennighoff et al., 2025; Allen-Zhu and Li, 2024b) had established that data repetition beyond 4–20× leads to diminishing or negative returns in NLP contexts, but those results were for different data types (knowledge-intensive text, general web corpora). The paper extends this line of inquiry to the recommendation domain and produces a concrete threshold (16 repeats) that is notably higher than the 4-repeat guideline from prior CPT work, suggesting that well-constructed synthetic recommendation data may be more robust to repetition than general text—possibly because the structured, template-like format of CF rules and UIH sequences provides natural variation that staves off memorization.

The model-scale independence of the threshold (Figure 7) is particularly interesting: larger models start overfitting at the same number of repeats as smaller models, but their performance degrades more sharply once overfitting begins. This has a mechanistic interpretation: larger models have greater capacity to memorize specific training sequences, but the onset of memorization (as opposed to generalization) is governed by the number of exposures to each example, not by the model's capacity per se. Once memorization begins, however, the larger model's superior capacity means it memorizes more completely, producing worse generalization.

This finding is incremental—it's a refinement and domain-extension of known repetition-degradation phenomena—but it is practically significant because it gives concrete guidance for resource allocation: if you have a limited UIH dataset, you can safely repeat it up to ~16 times at current mixture ratios, but beyond that you need to either generate more unique synthetic data (by varying walk parameters or expanding the item graph) or reduce the UIH mixture ratio to slow the repetition rate. Without this threshold, practitioners would be operating blind, unsure whether their models are improving or overfitting.

The evidence is contained in Figure 5 and Table 7: UIH Full perplexity at 15% mixture ratio begins increasing at ~20B total training tokens (≈16 repeats), while at 2% ratio it begins increasing at ~160B tokens (also ≈16 repeats). The training loss (Figure 6b) decreases monotonically in both cases, confirming the classic overfitting signature. Figure 7 extends this across model scales, showing the degradation is universal but severity increases with model size.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the Merrec dataset (Li et al., 2025), a publicly available C2C (consumer-to-consumer) e-commerce dataset hosted on HuggingFace containing 1.2 billion user-item interactions from 65.7 million items and 2.6 million users. The dataset provides raw user interaction logs, item metadata (titles, descriptions, categories), and user profiles, making it suitable for both the synthetic data generation pipeline (which mines CF associations from interaction histories) and the downstream ranking evaluation (which tests on held-out real user sequences). For the TSTR experiments in Section 5.2, the test set is drawn from real user interactions filtered to include only items present in the vocabulary of both synthetic and real training sets to ensure fair comparison.

Base model(s). The continual pre-training experiments use Qwen3 models (Yang et al., 2025a) at four parameter scales: 0.6B, 1.7B, 4B, and 8B. The paper describes Qwen3 as "one of the best open-weight LLM models available now." These are decoder-only transformer models pre-trained on a large general-domain corpus. The choice of Qwen3 is pragmatic rather than principled—the paper does not claim any special suitability for recommendation, but rather uses it as a representative strong open-weight model. The range of scales (0.6B to 8B, a 13.3× span) enables fitting of joint scaling laws over model size. For the data mixture experiments (Section 7.2), an additional 14B and 32B scale are tested to examine model-scale effects on overfitting thresholds.

Metrics. The primary metric for scaling law experiments is perplexity on held-out test sets, computed as the exponentiated average negative log-likelihood per token across all tokens in the evaluation corpus. Perplexity is chosen because it is a continuous, well-behaved metric that directly measures the model's language modeling quality and enables clean power-law fitting. For the downstream ranking validation (Section 5.2), the metrics are Recall@K at K = 10, 100, and 1000, measuring the fraction of test sequences where the ground-truth next item appears in the model's top-K predicted items. The answer grading for ranking evaluation uses standard top-K retrieval from the model's predicted item distribution. For semantic ID evaluation (Appendix B), collision rate is reported, measuring the percentage of items that share identical semantic token sequences.

Baselines. The TSTR experiments (Section 5.2, Figure 1) compare four standard sequential recommendation models trained on synthetic data versus trained on real data:

GRU4Rec (Hidasi et al., 2016): Recurrent neural network for session-based recommendation using GRU units.
NARM (Li et al., 2017): Neural Attentive Recommendation Machine with attention over session items.
STAMP (Gao et al., 2025): Short-Term Attention/Memory Priority model capturing both long-term and short-term user interests.
SASRec (Kang and McAuley, 2018): Self-Attentive Sequential Recommendation using transformer self-attention over user history.

Each model is trained in two settings: TSTR (Train on Synthetic, Test on Real) and TRTR (Train on Real, Test on Real). All models are standard architectures from the cited prior work with no modifications for the synthetic data setting. The scaling law experiments (Sections 6–7) have no explicit baselines beyond the internal comparisons across data mixtures and model scales—the central claim is that scaling laws emerge with synthetic data, so the implicit baseline is the absence of scaling laws with raw data (as demonstrated by PLUM's failure in Section 3.3).

Generation budget / compute accounting. For the CPT experiments, compute is measured in total training tokens (fixed at 163B for all model sizes in the main experiments) and in EFLOPS for the compute-optimal analysis (Figure 8). FLOPs are estimated using the standard transformer approximation C = 6ND, where N is parameter count and D is training tokens. The total compute for the 8B model is approximately 6 × 8B × 163B = 7,824 EFLOPS. For the synthetic data generation itself, compute costs are not reported—the paper focuses on the training compute economics rather than the data generation cost, an important omission discussed in the Critical Assessment. For the TSTR experiments, training budgets are not specified beyond the standard training procedures of each sequential model architecture.

Cross-validation / statistical protocol. The paper does not employ k-fold cross-validation for the scaling law experiments, as the models are trained once at each scale and evaluated on fixed held-out test sets. The held-out test sets (Table 3) are constructed by partitioning the data at the edge level for CF (edges are held out while items may appear in both train and test through other edges) and at the walk level for UIH (OOD walks are generated from held-out CF edges). The paper does not report confidence intervals, standard errors, or statistical significance tests for the scaling law fits—the fitted parameters (L∞, α, A, etc.) are reported as point estimates without uncertainty quantification. For the TSTR experiments (Figure 1), the paper reports Recall@K values across four models and multiple K thresholds, but does not report error bars or variance across random seeds.

Main Quantitative Results

Downstream Ranking: Synthetic Data Outperforms Real Data (Section 5.2)

The paper's foundational empirical claim—that the synthetic data captures more generalizable user preference patterns than raw interaction logs—is tested through TSTR vs. TRTR experiments. The headline result appears in Figure 1:

All four sequential models trained exclusively on synthetic data (TSTR) outperform their counterparts trained on real data (TRTR) across all Recall@K thresholds.

The paper reports specific magnitudes: SASRec achieves approximately +130% improvement on Recall@100 when trained on synthetic data versus real data. While exact numbers for each model and K-value are visible only in the figure (the paper does not enumerate them in the text), the qualitative pattern is unambiguous across all four models (GRU4Rec, NARM, STAMP, SASRec) and all three K-values (10, 100, 1000): the blue (TSTR) curves consistently lie above the red (TRTR) curves.

The paper attributes this superiority to the synthetic data's purification of the collaborative filtering signal:

"By learning from the purified collaborative filtering signal rather than the noisy specifics of raw interaction logs, the models trained on synthetic data acquire more generalizable co-occurrence patterns."

This is a striking result because it inverts the typical expectation: synthetic data usually underperforms real data on downstream tasks due to distribution mismatch. Here, the distribution mismatch is a feature rather than a bug—the synthetic data deliberately deviates from the real-data distribution to eliminate biases, and the resulting model generalizes better to real test interactions.

The filtering protocol is important for interpreting these results: both the synthetic and real training sets are constrained to the same item vocabulary, meaning the comparison isolates the quality of the training signal rather than the breadth of item coverage. The synthetic data's advantage comes not from covering more items but from presenting cleaner co-occurrence patterns for the items it does cover.

Scaling Laws: Power-Law Behavior Across All Seven Evaluation Domains (Section 6.2, Table 5, Figure 2)

The central empirical contribution is the demonstration of robust power-law scaling across seven evaluation domains. The per-model fits (Table 5) reveal a clear hierarchy of scaling efficiency:

User Interaction History exhibits the strongest scaling across all model sizes, with α ranging from 0.453 (0.6B) to 0.588 (8B) for UIH Full and from 0.555 to 0.689 for UIH OOD. The UIH OOD exponents are consistently higher than UIH Full exponents (e.g., 0.689 vs. 0.588 at 8B), which the paper attributes to the OOD evaluation having higher initial perplexity and therefore more room for improvement as the model learns generalizable sequential patterns.

Collaborative Filtering shows the second-strongest scaling, with α ≈ 0.282–0.358 across model sizes and CF subsets (Both Seen, One Unseen, Both Unseen). The three CF subsets exhibit nearly identical scaling exponents within each model size, indicating that the model's ability to generalize across different levels of item novelty (both seen, one unseen, both unseen) follows the same learning trajectory.

Item-Text alignment shows moderate scaling (α ≈ 0.126–0.214), and General Domain shows near-saturation (α ≈ 0.020–0.027). The General Domain result is expected: the pre-trained Qwen3 checkpoint already encodes extensive general-domain knowledge, so additional general-domain tokens at the same distribution provide minimal additional benefit.

Two model-size-dependent patterns emerge from Table 5:

α increases with model size for CF and UIH, but not for General or Item-Text. For UIH Full, α increases from 0.453 (0.6B) to 0.588 (8B)—a 30% relative increase—while for Item-Text, α fluctuates without a clear trend (0.169 → 0.141 → 0.126 → 0.214). This suggests that larger models can extract more information per token from complex sequential and relational data, but this capacity advantage does not extend to simpler data types where the patterns are already well-learned at small scales.
L∞ is remarkably stable across model sizes for CF and UIH, but decreases with model size for General and Item-Text. CF L∞ ranges from 0.33–0.36 across all model sizes, and UIH L∞ from 0.64–0.70 (Table 5). Meanwhile, General L∞ drops from 0.96 (0.6B) to 0.79 (8B), and Item-Text L∞ from 1.07 to 0.86–0.93. The stability of L∞ for recommendation domains means that even 0.6B parameters are sufficient to represent the underlying patterns at convergence—the bottleneck is data coverage, not model expressiveness.

Joint Scaling Laws: Recommendation Domains Are Data-Dominant (Section 6.2.2)

The joint fits (Equations 2–5) decompose scaling into model-size and data-size contributions, revealing a sharp inverse relationship between model-size scaling (α) and data scaling (β) across domains:

General Domain: ℓ = 0.79 + 16500 N^−0.511 + 3.85 D^−0.048. High model-size exponent (0.511), very low data exponent (0.048). This domain is overwhelmingly model-size dominant: performance improves primarily by scaling up model capacity.
Item-Text: ℓ = 0.473 + 5720 N^−0.511 + 4.04 D^−0.070. Similar pattern with high model-size exponent (0.511) and low data exponent (0.070).
Collaborative Filtering: ℓ = 0.193 + 21 N^−0.277 + 6.79 D^−0.148. Intermediate on both axes, with a moderate model-size contribution (A = 21, relatively small) and moderate data contribution.
User Interaction History: ℓ = 0.514 + 1.89 N^−0.138 + 63.9 D^−0.272. Low model-size exponent (0.138), high data exponent (0.272). This domain is strongly data-dominant.

The practical implication is quantified through the compute-optimal allocation analysis (Table 4):

Domain	αA	βB	Implication
General	8,432	0.185	Model-size dominant (45,500× ratio)
Item-Text	2,923	0.283	Model-size dominant (10,300× ratio)
CF	5.82	1.005	Moderately model-size dominant (5.8× ratio)
UIH	0.261	17.38	Data dominant (0.015× ratio—i.e., 67× in favor of data)

The paper provides a concrete operationalization: "Halving the data-dependent loss term for General requires 2^(1/0.048) ≈ 1.8 × 10^6 times more data, whereas for UIH it requires only 2^(1/0.272) ≈ 13 times more data."

The irreducible losses (E) reveal the intrinsic predictability of each domain: CF (0.193) is most predictable, consistent with the low-rank structure of user-item matrices; Item-Text (0.473) is moderately predictable; UIH (0.514) reflects moderate stochasticity in user behavior; and General (0.79) has the highest entropy, as expected for natural language.

Compute-Optimal Scaling Curves (Section 7.3, Figure 8)

Figure 8 visualizes perplexity against compute (EFLOPS) for models from 0.6B to 8B across evaluation domains. The key findings:

General Domain follows classic Chinchilla scaling. A clear Pareto frontier emerges: the 8B model achieves the lowest perplexity at highest compute, and no smaller model can match it regardless of over-training. The 0.6B model, despite seeing 454 tokens/parameter, cannot approach the 8B model's perplexity—model capacity is the binding constraint.

Collaborative Filtering saturates at 4B parameters. The 4B and 8B models achieve nearly identical CF perplexity (~1.47 PPL), indicating that CF patterns are fully learnable by a 4B-parameter model given sufficient data repetition. Additional parameters beyond 4B provide no benefit on CF tasks.

UIH OOD exhibits inverse scaling. The most striking finding in Figure 8: the heavily over-trained 0.6B model (~2.01 PPL) outperforms the near-Chinchilla-optimal 8B model (~2.10 PPL) on out-of-distribution UIH evaluation. The paper attributes this to an interaction between data repetition and model capacity:

"Larger models more readily memorize repeated behavioral sequences, leading to poorer OOD generalization, while the extreme over-training regime of smaller models appears to act as implicit regularization against such memorization."

This is a cautionary result: for recommendation domains where unique training data is limited (necessitating repetition), larger models may actually underperform smaller models on generalization tasks due to memorization of training sequences.

Ablation Study: Layered Data Contributions (Section 7.1, Figures 3–4)

The ablation study removes individual data layers to quantify their contributions. The key numerical results:

CF → UIH cross-domain transfer. Including CF data alongside UIH reduces asymptotic UIH Full perplexity from 0.95 (UIH-only) to 0.66 (CF+UIH), a 31% reduction (Figure 4). The CF+UIH configuration achieves the lowest UIH perplexity of all mixtures, even outperforming the full Item-Text + CF + UIH mixture (L∞ = 0.76), suggesting that Item-Text data slightly interferes with UIH learning.

UIH → CF transfer is negligible. Training on UIH data alone produces α ≈ 0 on CF evaluation sets, with L∞ stuck at 1.30 regardless of training tokens (Figure 3, top row). This means sequential behavior data does not implicitly teach pairwise item association patterns.

Domain-specific data is essential. Models trained without CF data plateau at CF loss of L∞ = 1.30 versus L∞ = 0.35 with CF data (3.7× higher). Models without Item-Text data show severe degradation on Item-Text evaluation (L∞ ≈ 3.0 vs. L∞ ≈ 1.2 with Item-Text), which is essentially random-guessing-level perplexity.

Item-Text data introduces a modest trade-off. The full mixture (Item+CF+UIH) yields L∞ = 0.76 for UIH Full, compared to 0.66 for CF+UIH—a degradation of approximately 15% in asymptotic perplexity. The paper suggests practitioners should weigh this trade-off against the benefits of preserving item-text alignment capability.

Data Mixture Ratio Experiments (Section 7.2, Table 7, Figures 5–7)

Using a reduced UIH dataset of 42M tokens with varying mixture ratios:

Overfitting threshold at ~16 repeats. UIH evaluation perplexity begins to increase (worsen) once the reduced UIH data has been repeated approximately 16 times, regardless of the absolute mixture ratio:

At 2% mixture ratio: overfitting emerges at ~160B total training tokens (~16.3 repeats).
At 5% ratio: overfitting emerges at ~80B total training tokens (~16 repeats crossed earlier).
At 15% ratio: overfitting emerges at ~20B total training tokens (~16 repeats).

The training loss (Figure 6b) decreases monotonically in all cases, confirming the classic overfitting signature.

Model-scale independence with severity amplification (Figure 7). The overfitting threshold is independent of model scale—all models from 0.6B to 32B begin degrading at approximately 16 repeats—but larger models suffer more severe degradation once overfitting begins. At 15% ratio, the 32B model's UIH OOD perplexity rises more sharply than the 0.6B model's, consistent with greater memorization capacity in larger models.

Scaling exponents under different mixture ratios (Table 7). For the 4B model, the UIH Full scaling exponent α increases from 0.423 (0.5% ratio) to 0.754 (15% ratio) as the mixture ratio increases. However, this higher α comes at the cost of higher L∞ (0.75 → 0.90), indicating that higher mixture ratios accelerate early learning but lead to worse asymptotic performance due to overfitting.

Tokenization Method Ablation (Appendix B, Figure 12)

The comparison between SAE-based and RQ-kmeans-based tokenization on 4B models shows SAE consistently outperforming:

General Domain: SAE L∞ = 0.82 vs. RQ L∞ = 0.83 (negligible difference)
Item-Text: SAE L∞ = 0.86 vs. RQ L∞ = 1.25 (substantial advantage for SAE—45% higher asymptotic loss for RQ)
CF Both Seen: SAE L∞ = 0.35 vs. RQ L∞ = 0.35 (identical)
UIH Full: SAE L∞ = 0.66 vs. RQ L∞ = 0.76 (15% advantage for SAE)
UIH OOD: SAE L∞ = 0.69 vs. RQ L∞ = 0.80 (16% advantage for SAE)

The SAE advantage is concentrated in text-heavy domains (Item-Text) and sequential domains (UIH), while purely collaborative domains (CF) are insensitive to tokenization method. This pattern is consistent with the SAE's pre-training on multi-modal data providing richer semantic representations that benefit tasks requiring content understanding.

Ablation Studies and Robustness Checks

Synthetic UIH generation hyperparameters: Appendix A (Table 9) sweeps 14 configurations of the Node2Vec random walk, varying α_stop (0.10–0.20), path confidence threshold (None to 3e-09), and walk strategy (1st-order vs. 2nd-order BFS with p = 0.5, q = 2.0). The selected configuration (1st_order_a015_thresh1e-09) achieves 180K unique items, a Gini coefficient of 0.635, average sequence length of 8.0 items, and Geom Lift of 385.54. The selection of a 1st-order walk over 2nd-order alternatives suggests that for this graph, the CF edge weights already encode sufficient relational structure without needing the additional biasing from Node2Vec's return and in-out parameters. The sweep reveals that α_stop = 0.15 with minimal path confidence filtering (1e-09) provides the best balance of catalog coverage (Gini) and sequence coherence (Geom Lift).

CF evaluation across seen/unseen items: Table 5 consistently shows nearly identical scaling exponents for CF Both Seen, CF One Unseen, and CF Both Unseen within each model size (e.g., at 8B: 0.358, 0.342, 0.358). This indicates that the model's CF learning generalizes robustly to items not seen in the CF training data—the learned association function transfers to new items, rather than being memorized for specific item pairs. The L∞ values are also nearly identical (0.35–0.36), confirming that the asymptotic predictability of CF patterns is independent of whether the items were in the training set.

UIH OOD vs. UIH Full evaluation gap: Across all model sizes in Table 5, UIH OOD L∞ is consistently higher than UIH Full L∞ (e.g., at 8B: 0.70 vs. 0.66), while UIH OOD α is consistently higher (0.689 vs. 0.588). The higher L∞ for OOD is expected (harder task), but the higher α is notable: the model improves faster on OOD data than on in-distribution data as training progresses. This suggests that early in training, the model memorizes specific training walks (benefiting UIH Full), but later in training, it learns generalizable sequential patterns that transfer to OOD walks. This is consistent with the overfitting analysis in Section 7.2, where the gap between UIH Full and UIH OOD performance widens as memorization sets in.

General-domain forgetting prevention: The fact that General Domain α ≈ 0.02–0.03 across all model sizes (Table 5) indicates near-saturation—the model is not improving its general language modeling with additional general-domain tokens—but also indicates no degradation. If catastrophic forgetting were occurring, General Domain perplexity would increase with training. The flat scaling curves for General Domain (Figure 2) therefore serve as a robustness check that the 50/50 general-to-recommendation mixture successfully prevents catastrophic forgetting.

RQ codebook capacity sweep (Table 10): The collision rate drops from 82.7% (3 layers × 256 codes, 16M total) to 1.2% (8 layers × 128 codes, 72,057T total), demonstrating the exponential capacity growth of hierarchical codebooks. The 6-layer, 256-code configuration (281T total codes, 3.5% collision) is the paper's recommended RQ-kmeans setting, though SAE was ultimately selected. This sweep is a robustness check showing that the hierarchical quantization approach can, in principle, achieve low collision rates, but the SAE's pre-training provides superior semantic representations.

Training stability: Figure 6(a) shows monotonically decreasing training loss for all four model scales in the main experiment, with no evidence of loss spikes, divergence, or training instability. The smooth curves suggest that the 50/50 data mixture and chosen hyperparameters (peak LR 1e-4, cosine decay) provide stable training across a 13.3× range of model sizes.

Critical Assessment

Does the synthetic data genuinely enable scaling laws where raw data does not?

The paper's central claim is that the layered synthetic data framework enables predictable power-law scaling for LLMs in recommendation, and that this scaling was previously unattainable with raw user interaction logs. The evidence for the positive half of this claim—that scaling laws emerge with synthetic data—is strong. Figures 2–4 and Tables 5 show consistent power-law behavior across seven evaluation domains and four model scales, with fitted curves that closely track empirical data points across three orders of magnitude in training tokens (from ~1B to ~163B). The scaling exponents are stable, the fits are visually tight, and the hierarchy of exponents across domains is internally consistent and interpretable.

However, the evidence for the negative half of the claim—that scaling laws do not emerge with raw data—is indirect. The paper cites the PLUM case study (Section 3.3) as evidence of scaling failure, but PLUM used a different model architecture (MoE), a different base model, a different dataset, and different training procedures. The paper does not run a controlled experiment where the same Qwen3 models are CPT'd on raw Merrec interaction logs to demonstrate the absence of scaling laws under identical conditions. The PLUM result is suggestive but not dispositive—it is possible that raw Merrec data, formatted differently, would produce scaling laws with Qwen3. The paper's core thesis is that data quality is the gating factor; a direct head-to-head comparison (Qwen3 on synthetic vs. Qwen3 on raw, same compute budget, same evaluation) would have been the strongest possible evidence. Its absence is the single largest evidential gap in the paper.

Does the TSTR result demonstrate generalizable preference learning, or does it reflect a specific advantage of the filtered item vocabulary?

The TSTR experiments (Figure 1) show models trained on synthetic data outperforming models trained on real data on real test sets—a genuinely surprising and strong result. However, the test set is filtered to include only items present in both the synthetic and real training vocabularies. This filtering may systematically advantage the synthetic data: the synthetic training data, being generated from CF association rules that capture global item co-occurrence, may represent the relationships among these shared items more cleanly than the real training data, which also contains noisy interactions with items outside the shared vocabulary. Meanwhile, the real training data's advantage—coverage of long-tail items that the synthetic data may miss—is removed by the filtering. This means the TSTR comparison answers the question "which data teaches better patterns for the items both datasets cover?" but does not answer "which data produces better recommendations overall, including for items the synthetic data doesn't represent well?" The +130% SASRec improvement on Recall@100 is impressive but may be an upper bound on the practical advantage in an unfiltered setting.

Are the scaling laws established with sufficient statistical rigor?

The scaling law fits (Table 5, Equations 2–5) are reported as point estimates without confidence intervals, standard errors, or goodness-of-fit statistics. The fitted L∞ values for CF are 0.33–0.36 with differences of 0.01 between model sizes—are these differences statistically distinguishable from noise, or are they effectively identical? Similarly, the joint fit parameters (A = 16,500 for General, A = 1.89 for UIH) differ by four orders of magnitude, but the paper provides no uncertainty quantification that would allow a reader to assess whether these differences are robust to data subsampling or fitting methodology. The absence of error bars is particularly consequential for the compute-optimal allocation analysis (Table 4), where the αA and βB ratios inform resource allocation decisions—if the αA for CF is 5.82 with a wide confidence interval, the recommendation to allocate compute toward model size (rather than data) for CF might be unreliable.

The paper's evaluation sets are also relatively small in certain categories. The CF Both Unseen test set contains only ~75K entries (38M tokens) and the UIH OOD test set contains only ~48.3K entries (24.7M tokens)—these are the smallest evaluation sets and the ones most critical for assessing generalization. Perplexity estimates on 24.7M tokens have non-trivial sampling variance; the paper does not report whether apparent differences in L∞ of 0.01 or 0.02 are within that sampling variance.

The overfitting threshold of ~16 repeats is well-documented, but is it generalizable?

The data mixture experiments (Section 7.2) provide clean evidence for an overfitting threshold at approximately 16 repeats of the reduced UIH dataset, with model-scale independence of the threshold and severity amplification for larger models. This is one of the paper's most actionable findings. However, the threshold is established with a reduced UIH dataset of 42M tokens—less than 2% of the full 2.8B-token UIH dataset. It is possible that the full dataset, with its greater diversity, would support a different (likely higher) repetition threshold. The paper's main experiments use the full UIH dataset at approximately 6.1 repeats (Table 2), which is well below the 16-repeat threshold, so the threshold is never directly validated on the full dataset scale. Practitioners with larger or more diverse synthetic datasets may find the threshold differs.

Additionally, the threshold is established for one specific data format (semantic-token UIH sequences) and one model family (Qwen3). Whether the same threshold holds for different item representations (e.g., natural-language item descriptions instead of semantic tokens) or different base model families (e.g., Llama, Mistral) is unknown.

The inverse scaling finding for UIH OOD is striking but underexplored.

Figure 8's demonstration that the 0.6B model outperforms the 8B model on UIH OOD perplexity is the paper's most counterintuitive result and has significant practical implications: it suggests that for recommendation domains with limited unique training data (requiring substantial repetition), smaller models may generalize better than larger ones. However, the paper provides only a brief mechanistic hypothesis (larger models memorize) and does not explore this phenomenon in depth. Several natural follow-up experiments are absent:

Would intermediate model sizes (e.g., 2B, 3B) show a monotonic relationship between size and OOD performance, or is there an optimal size?
Does the inverse scaling persist if the models are early-stopped before overfitting begins, or is it purely a consequence of training to convergence?
Is the inverse scaling specific to UIH (sequential data) or does it also appear for CF or Item-Text domains?
Would techniques like dropout increase during CPT or data augmentation mitigate the larger models' memorization tendency?

Without these ablations, the inverse scaling result is intriguing but not fully characterized, limiting its practical applicability.

The paper's claim about "eliminating bias" is stronger than what is demonstrated.

The paper describes its synthetic data as "free from bias" (Section 4.3.1) and as having "no intrinsic concept of position or presentation order." This is true of the generation process—the random walk does not have a ranked list—but the generated sequences are not necessarily bias-free in their statistical properties. The CF graph used to generate walks is itself derived from raw logs that contain exposure bias: items that users never saw cannot appear in the CF association rules, so the graph inherits whatever gaps exist in the raw co-occurrence data. A genuinely bias-free dataset would need to represent user preferences for all item pairs, including those never co-exposed in the raw logs. The synthetic data may reduce position and popularity bias, but it does not eliminate exposure bias—it can only represent associations that were observable in the raw logs. The paper acknowledges the layered purification mechanism (Section 4.3.1) but does not empirically quantify residual bias in the synthetic data beyond the statistical fidelity analysis (Section 5.1), which primarily checks distributional similarity rather than bias-specific metrics.

The FLOPs-matched analysis compares against a weak baseline.

The compute-optimal analysis (Figure 8, Table 4) guides resource allocation between model size and data within the paper's own experimental framework, but the loss values are not compared against any external baseline (e.g., a traditional matrix factorization model, a non-LLM sequential recommender trained on raw data). The paper demonstrates that given a fixed LLM architecture and synthetic data, certain allocation strategies are more efficient—but it does not demonstrate that this efficiency makes LLM-based recommenders competitive with or superior to traditional approaches at any given compute budget. The TSTR experiments address this partially for sequential models (GRU4Rec, SASRec, etc. trained on synthetic data outperform those trained on real data), but no TSTR experiment compares an LLM-based recommender against a traditional recommender trained on the same synthetic data. The question "should I use an LLM for recommendation at all?" is not addressed.

The semantic tokenization ablation is under-reported.

The paper selects SAE tokenization over RQ-kmeans based on Figure 12, but the comparison is reported only for the 4B model scale and only for the final fitted L∞ values. Several important details are missing: Were both tokenization methods trained with the same hyperparameters? Did the RQ-kmeans configuration (6 layers, 256 codes per layer) use the same item embedding model (Qwen2.5-7B) as the RQ-VAE approach described in Appendix B? The paper reports that RQ-kmeans achieved 5.33% collision while RQ-VAE was stuck at 50.54%, but does not explain why RQ-VAE failed or whether hyperparameter tuning could have closed the gap. Figure 12 shows SAE substantially outperforming RQ-kmeans on Item-Text (L∞ = 0.86 vs. 1.25) and UIH (0.66 vs. 0.76), but these differences may partially reflect the SAE's access to pre-training on a large multi-modal corpus rather than inherent superiority of the approach—a fair comparison would match the tokenizer pre-training data scale.

Missing baselines and experiments.

Several experiments would have substantially strengthened the paper:

Controlled scaling comparison with raw data. CPT Qwen3 on raw Merrec logs (formatted as natural-language sequences, as in PLUM) with the same compute budget and evaluation domains as the synthetic data experiments, to directly demonstrate that synthetic data enables scaling laws where raw data fails.
Downstream ranking evaluation of the CPT'd LLMs. The paper validates synthetic data quality using traditional sequential models (SASRec, GRU4Rec, etc.) but never evaluates the CPT'd Qwen3 models on a downstream ranking task. The scaling laws are established purely in terms of perplexity; whether lower UIH perplexity translates to better item recommendations from the LLM itself is assumed but not tested.
Sensitivity to the 50/50 general-to-recommendation split. The paper fixes this ratio without justification beyond citing catastrophic forgetting. Varying this ratio (e.g., 70/30, 30/70) would reveal whether the recommendation-domain scaling exponents depend on the fraction of general-domain data in the mixture.
Experiments at larger model scales for the joint scaling law extrapolation. The joint fits in Equations 2–5 are based on four model sizes (0.6B–8B). Extrapolating to, say, a 70B model requires trusting that the power-law form holds far beyond the fitted range. Training a 14B or 32B model (which the paper does for the overfitting experiments in Figure 7 but not for the main scaling law fits) would have provided a validation point for the extrapolation.
Statistical confidence intervals for all fitted parameters. The paper's scaling law claims rest on the fitted parameters in Table 5 and Equations 2–5. Without uncertainty estimates, readers cannot assess whether the reported hierarchy of α values (UIH > CF > Item-Text > General) is statistically robust or could be an artifact of fitting noise.

Conditional scope of the claims.

The paper's claims about scaling laws are empirically supported, but their scope is narrow: they hold for (1) Qwen3 models, (2) the Merrec C2C e-commerce dataset, (3) semantic-token-based item representations, (4) the specific data mixture ratios reported in Table 2, and (5) perplexity as the evaluation metric. The paper does not claim universality—it explicitly positions the work as a first demonstration—but readers should understand that the scaling exponents, overfitting thresholds, and cross-domain transfer patterns may differ for other model families, other recommendation domains (e.g., content streaming, news, social media), other item representation methods (e.g., natural-language descriptions without semantic tokens), or other evaluation metrics (e.g., downstream ranking accuracy).

The FLOPs-matched finding that recommendation domains are data-dominant (Table 4) is particularly sensitive to the joint scaling law fit. The βB/αA ratio of ~67 for UIH implies that compute should be overwhelmingly allocated to data collection, but this conclusion rests on the joint additive model ℓ = E + A·N^−α + B·D^−β. The paper's own per-model fits (Table 5) show that α_data varies with model size for CF and UIH—a model-data interaction that the additive joint formulation cannot capture. If the true relationship includes an interaction term (e.g., larger models benefit more from data), the compute-optimal allocation might be less extremely data-dominant for larger model scales than the additive model predicts.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Exorbitant and Unaccounted For

The assumption or constraint. The entire compute-optimal framework depends on knowing the pass@1-based difficulty bin of each prompt before allocating test-time compute. The paper's method for estimating difficulty involves sampling 2048 complete solutions from the base model per question—either checking correctness against ground truth (oracle) or averaging PRM final-answer scores (predicted). Section 3.2 explicitly acknowledges this cost:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. Generating 2048 samples per question consumes more compute than the largest test-time budgets studied (256–512 generations). The reported 4× efficiency gains over best-of-N are computed after difficulty is known—the difficulty estimation cost is amortized to zero in the paper's accounting. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter, potentially erasing or even reversing the claimed efficiency advantage. A practitioner deploying this method would discover that the "4× cheaper" strategy is actually more expensive than the baseline once difficulty estimation is included—unless they can estimate difficulty far more cheaply than 2048 samples.

What evidence exists in the paper. This limitation is purely an accounting gap; the paper provides no experiment measuring total end-to-end cost including difficulty estimation. The head-to-head curves in Figures 4 and 8 compare compute-optimal scaling against baselines at equal strategy-execution budgets, not equal total budgets. The paper does not report how much the 2048-sample difficulty estimation costs relative to the test-time budgets being studied.

Mitigation status. The paper flags this as a key avenue for future work (Section 8): "pretraining or finetuning models to directly predict difficulty of a question." No such model is developed or evaluated. An adaptive approach—where the first few generations serve simultaneously as difficulty probes and solution attempts—is mentioned in passing but not explored. The mitigation is entirely speculative at this stage.

Hard Problems Remain Completely Unsolved—The Method Cannot Create Capability, Only Amplify It

The assumption or constraint. Test-time compute can improve the probability of finding a correct solution among the model's outputs, but only if the base model produces correct solutions at some non-trivial rate. On problems where the base model's pass@1 is near zero, no amount of search or revision can help—there are no correct solutions in the proposal distribution to find or refine. The paper's data implicitly defines this boundary: difficulty bin 5, the hardest quintile, where pass@1 is essentially zero.

The consequence. For the hardest problems, all methods—search, revisions, compute-optimal combinations—produce accuracy indistinguishable from zero regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods at all budgets. In Figure 7 (right), bin 5 shows 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is flat near 0–5%, and the 14× larger pretrained model also provides minimal improvement. This means the approach offers no path forward for genuinely novel or out-of-distribution reasoning problems. For such problems, test-time compute is simply not the right tool—the model fundamentally lacks the capability, and that capability must be acquired through pretraining. The paper's FLOPs-matched analysis (Section 7) quantifies this: on hard problems at high inference-to-pretraining ratios, test-time compute with the smaller model shows a −52.9% relative disadvantage compared to the 14× larger pretrained model for PRM search (Figure 1, bottom-right bar chart).

What evidence exists in the paper. Bin 5 results consistently show near-zero accuracy across all figures (Figures 3 right, 7 right, 9). The paper is transparent about this boundary, stating explicitly in Section 7 that "test-time compute amplifies existing capability but does not create it." The FLOPs-matched results quantify the failure.

Mitigation status. The paper does not attempt to mitigate this limitation; it is treated as a fundamental boundary condition. The implication is that deployment should route the hardest problems to larger models or human review rather than spending test-time compute on the smaller model—which the paper's difficulty estimator could, in principle, enable. But this routing strategy is not implemented or evaluated.

Verifier Over-Optimization Limits All Search-Based Methods, and the Paper Mitigates Rather Than Solves It

The assumption or constraint. All search methods against the PRM (beam search, lookahead search) are vulnerable to verifier over-optimization: search finds solutions that score highly under the PRM but are actually incorrect. This is the central bottleneck preventing unbounded improvements from additional compute. The compute-optimal policy works around this by routing easy problems (where over-optimization is most severe) away from aggressive search—but does not solve the underlying verifier robustness problem.

The consequence. On medium-difficulty problems where beam search is deployed—the very conditions where the compute-optimal policy says search should be used—over-optimization still limits the scaling ceiling. The beam search curves in Figure 3 (right) for bins 3–4 flatten and sometimes decline well before the budget is exhausted. Lookahead search, the most powerful optimizer, paradoxically performs worst overall (Figure 3, left) because its deeper optimization amplifies verifier errors. The paper's qualitative examples (Appendix M, Figures 29, etc.) show search producing degenerate outputs: repetitive low-information steps at the end of solutions, overly short 1–2 step solutions. These are all symptoms of a verifier whose scores become unreliable under aggressive optimization pressure. This means that improving verifier robustness is the key bottleneck for further scaling test-time compute, but the paper's methods—compute-optimal strategy selection, difficulty-based routing—only sidestep the bottleneck rather than widening it. If verifier reliability could be improved (e.g., through adversarial training, ensemble methods, or better PRM training data), the scaling ceiling would rise for all methods, and the compute-optimal policy might change fundamentally.

What evidence exists in the paper. Figure 3 (right) shows beam search accuracy for bin 1 (easiest) decreasing from ~78% to ~77% as budget goes from 4 to 256 generations—clear over-optimization. Lookahead search underperforms universally in Figure 3 (left). The PRM aggregation analysis (Appendix E, Figure 13) and PRM vs. ORM comparison (Appendix F, Figure 14) quantify verifier quality but do not explore how to improve it beyond the baseline Monte Carlo rollout training.

Mitigation status. The compute-optimal policy partially compensates by using weaker optimization (best-of-N) where the verifier is unreliable (easy problems) and stronger optimization (beam search) only where the verifier signal has more room to guide (medium problems). But it is a routing strategy, not a verifier improvement. The paper does not explore verifier robustness techniques—adversarial training, ensemble verification, KL-constrained search—that could reduce over-optimization. Section 8 flags "developing more robust verifiers" as future work but provides no initial experiments.

Single Benchmark, Single Model Family—Generalization to Other Domains and Architectures Is Unknown

The assumption or constraint. All experiments use the MATH benchmark (500 test questions from high-school competition math) with PaLM 2-S* as the base model, and the 14× larger comparison model is from the same family. The paper states the model is "representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified.

The consequence. Multiple aspects of the findings could be specific to the MATH-PaLM 2-S* combination and not generalize:

The difficulty-dependent optimal strategies (beam search hurts easy problems, revisions help easy problems) depend on the base model's output distribution and error patterns. A model with different calibration—systematically underconfident or overconfident—would exhibit different over-optimization behavior against the same PRM. A model with different in-context learning capabilities would learn differently from revision training trajectories.
The FLOPs-matched comparison between test-time compute and pretraining used a parameter-only-scaled larger model (LLaMA-style, not Chinchilla-optimal), following the same model family. The relative advantage of test-time compute might shrink or reverse against a model from a different architecture family or a compute-optimally trained larger model (scaling both parameters and data).
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic multi-step reasoning. It is unclear whether the patterns—PRM over-optimization on easy problems, the optimal sequential-to-parallel ratio shifting by difficulty, the near-zero benefit on the hardest problems—generalize to code generation (where unit tests provide cleaner verifier signals), logical reasoning (different error patterns), scientific question-answering (requires factual knowledge more than reasoning), or open-ended generation tasks where correctness is ambiguous.
The PRM training procedure (Monte Carlo rollout supervision) depends on having a clean correctness signal for rollouts—available for MATH (exact answer matching) but not for many important real-world tasks. Extending the method to domains without such signals would require a fundamentally different verifier training approach.

What evidence exists in the paper. The paper explicitly acknowledges the single-benchmark limitation (Section 8) but provides no cross-domain or cross-model experiments. The PRM800k finding (Section 5.1)—that a PRM trained on GPT-4 outputs does not transfer to PaLM 2—provides indirect evidence that results may be model-specific, since PRM quality depends on the base model's output distribution. But this is a negative transfer result for one specific cross-model PRM, not a systematic investigation of cross-model or cross-domain generalization.

Mitigation status. The paper does not attempt to demonstrate generalization. It treats MATH-PaLM 2-S* as a valid testbed for establishing the principles of compute-optimal test-time scaling, with the expectation that the framework (difficulty estimation, strategy selection, PRM training via Monte Carlo rollouts, revision training with edit-distance pairing) would transfer to other settings even if the specific numerical results do not. This is a plausible but untested assumption.

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate That Requires Imperfect Workarounds

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct target. At inference time, the model may produce a correct answer in an early revision step, then encounter that correct answer in its own context in a subsequent step—a situation it was never trained to handle. The paper reports that approximately 38% of correct answers get converted back to incorrect ones through this phenomenon (Section 6.1).

The consequence. The revision chain is fundamentally unstable: correctness is not monotonic. Without mitigation, taking the final answer in a revision chain would underperform earlier answers—the model actively degrades its own good outputs. The paper's mitigation—using majority voting or verifier-based selection across the entire chain to pick the best answer—is a post-hoc patch that adds selection overhead and does not prevent the model from wasting compute generating incorrect revisions from correct answers. In a long revision chain, the model may oscillate between correct and incorrect answers multiple times, consuming a substantial fraction of the test-time budget on unproductive revisions.

This instability also creates a tension with sequential revision strategies, which the compute-optimal policy favors for easy problems (Figure 7). On easy problems where the model's initial answers are often correct, a purely sequential revision strategy risks converting many correct initial answers to incorrect ones through repeated revision. The paper's hierarchical selection (within-chain first, then across-chain) partially mitigates this, but the 38% reversion rate means that even with mitigation, a non-trivial fraction of the revision budget is wasted.

What evidence exists in the paper. The 38% figure is reported in Section 6.1 without a detailed breakdown by difficulty bin or revision step—it is unclear whether the reversion rate is higher on easy problems (where correct answers are more common in context) or on hard problems (where the model is more uncertain). The paper does not report an ablation comparing chain-level accuracy with and without the mitigation strategy, so the net effect of the 38% reversion rate on final accuracy is not directly quantified. Figure 6 (left) shows that per-step pass@1 increases gradually across the chain (from ~18.2% at step 1 to ~24–25% by steps 15–20), suggesting the mitigation is partially effective—otherwise per-step accuracy would degrade—but the net cost of reversions is never isolated.

Mitigation status. The paper's mitigation—selection across the entire revision chain—is presented as a workaround, not a solution. It acknowledges the root cause (training data includes only incorrect→correct trajectories) but does not explore alternatives: training the model on "no change needed" trajectories where the current answer is already correct, using the PRM to detect when a revision step degrades the answer and abort, or implementing explicit confidence thresholds below which revision is skipped. Section 8 does not flag correct-to-incorrect reversion as a specific avenue for future work, suggesting the authors view the current mitigation as adequate, even though it leaves a 38% failure mode in the core proposal modification mechanism.

The $14\times$ Larger Pretraining Baseline Is Substantially Weaker Than the Claim Implies

The assumption or constraint. The FLOPs-matched comparison in Section 7 uses a pretraining baseline where model parameters are scaled 14× while training data is held fixed (LLaMA-style scaling), rather than scaling both parameters and data equally (Chinchilla-optimal scaling). Additionally, the 14× larger model uses only greedy decoding—no test-time compute augmentation of any kind. The paper acknowledges this departure:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. The pretraining baseline is weaker than it should be on two fronts. First, a Chinchilla-optimal model trained with 14× more total FLOPs, scaling both parameters and data, would likely outperform a parameter-only-scaled model, making the pretraining baseline stronger and potentially shrinking or reversing the reported advantages of test-time compute (+27.8% on easy questions at R << 1 for revisions, per Figures 1 and 9). Second, the larger model is denied any test-time compute budget—even a modest best-of-8 majority vote, which costs 8× the inference FLOPs of greedy decoding but would be a negligible fraction of the total pretraining FLOPs, could significantly improve the 14× model's performance. The paper's FLOPs-matched comparison thus compares a test-time-augmented smaller model against the weakest reasonable version of a larger model, not against the larger model augmented with its own proportional test-time budget.

This matters because the headline claim—"test-time compute with a smaller model can outperform a 14× larger model"—sounds more dramatic than what is actually demonstrated. A fairer FLOPs-matched comparison would give both models a comparable test-time budget, or at minimum give the larger model a small budget to correct obvious errors. The current setup is analogous to comparing a student who gets extra time on an exam against a student who gets no extra time—the comparison confounds the effect of model size with the effect of test-time compute.

What evidence exists in the paper. The FLOPs-matched results appear in Figures 1 (bar charts), 9 (line plots), and the associated text in Section 7. The paper's internal comparisons (e.g., Figure 4: compute-optimal vs. best-of-N) do hold test-time budgets constant and are valid—the issue is specifically with the pretraining-vs-inference comparison. The paper does not provide an ablation where the 14× larger model is given even a modest test-time budget (e.g., best-of-4, best-of-8) to establish how much of the reported advantage is due to test-time compute per se versus the asymmetry in allowed compute.

Mitigation status. The paper explicitly acknowledges the parameter-only scaling choice and defers the compute-optimal pretraining comparison to future work (Section 7). The greedy-decoding-only baseline is not explicitly defended or acknowledged as a limitation—it is simply the baseline the paper chose. A fairer comparison would require additional experiments (giving the larger model proportional test-time compute) or a clearer framing of the claim's scope: the paper demonstrates that test-time compute can substitute for pretraining compute when the pretrained model is not compute-optimally trained and receives no test-time augmentation, which is a narrower and less general claim than what the abstract and Section 1 suggest.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes the research program for LLM-based recommenders by shifting the bottleneck diagnosis from architecture to data. Prior work—LUM (Yan et al., 2025), SUAN (Lai et al., 2025), PLUM (He et al., 2025), and the OneRec series (Zhou et al., 2025a,b)—all operated under the implicit assumption that scaling failures could be addressed through better model design: novel training objectives, knowledge distillation, or generative architectures. The PLUM result, where a 3B MoE model failed to outperform its 900M counterpart, was interpreted by the field as an architectural scaling challenge. This paper argues it was a data quality problem all along.

The magnitude of this reframing is substantial but not a full paradigm shift—the paper does not propose new training algorithms, loss functions, or model architectures. Instead, it makes a diagnostic intervention: it identifies the root cause of a persistent failure mode (absence of scaling laws) and provides a constructive fix (layered synthetic data curriculum). The shift is from "how do we build models robust to flawed data?" to "how do we build data free of flaws?" This is the "data-centric AI" philosophy (Zha et al., 2023) applied to a domain where its necessity had been theorized but never empirically demonstrated with scaling-law-level evidence.

The paper reconciles a tension that has been building in the recommendation-LLM literature. On one side, proponents of LLM-based recommenders pointed to the rich semantic understanding and sequence modeling capabilities of LLMs as inherently superior to traditional embedding-table architectures. On the other side, practitioners found that scaling these models was unpredictable and expensive—the PLUM result being the most prominent public example. This paper provides a unified explanation: both sides were partially right. LLMs can be superior recommenders, but only when trained on data that teaches recommendation principles rather than reproducing system biases. The failure to find scaling laws was not evidence against LLM-based recommendation; it was evidence that raw interaction logs are pathologically poor training data for language models. This resolution converts a confusing set of contradictory findings into a coherent picture with a clear prescription.

Several research directions become more attractive in light of this work:

Data engineering for recommendation CPT is elevated from a preprocessing afterthought to the central research challenge. The paper demonstrates that investing in data quality produces predictable returns (the scaling laws), while investing in model architecture alone on raw data does not (the PLUM case study). Teams working on LLM recommenders should now expect to allocate a substantial fraction of their engineering effort to synthetic data generation and curriculum design.
Cross-domain recommendation transfer becomes more plausible. If the layered curriculum teaches generalizable principles of user preference rather than domain-specific quirks, a model trained on the synthetic curriculum for one e-commerce platform might transfer to another with minimal additional CPT. The paper's finding that CF data teaches transferable item-relationship functions (CF Both Seen, One Unseen, and Both Unseen have nearly identical scaling exponents in Table 5) suggests that the learned representations are not memorized item-specific patterns but generalizable relational knowledge.
Scaling law extrapolation for resource planning becomes available to recommendation teams for the first time. Just as NLP teams use Chinchilla-style analyses to forecast how much compute they need for a target perplexity, recommendation teams can now use the per-model fits in Table 5 to estimate how many synthetic tokens they need to generate for a target model size. The concrete finding that UIH data requires only ~13× more data to halve the data-dependent loss (versus ~1.8×10^6 for General domain, per Table 4) provides an immediately actionable guideline: for recommendation, collect more data rather than scaling model size.

Conversely, some research directions become less attractive:

Architecture-only approaches to recommendation scaling (new attention mechanisms, novel training objectives, MoE routing strategies) that do not address data quality are implicitly critiqued. The paper's argument implies that such efforts will hit the same sub-scaling wall that PLUM encountered, regardless of architectural sophistication, because the training data lacks sufficient learnable signal.
Debiasing-as-postprocessing approaches (inverse propensity scoring, adversarial debiasing of already-trained models) are reframed as palliative rather than curative. If data can be generated bias-free at the source, the complex machinery of debiasing during or after training becomes unnecessary—a simpler model on cleaner data outperforms a complex model on dirty data, as demonstrated by the TSTR experiments where standard sequential models (SASRec, GRU4Rec) trained on synthetic data achieved +130% Recall@100 over the same models trained on real data (Figure 1).

Follow-Up Research This Work Enables

Direct scaling comparison: synthetic vs. raw data under controlled conditions. The paper's central thesis—that synthetic data enables scaling laws where raw data does not—rests on the PLUM case study as evidence of raw-data scaling failure, but PLUM used a different model architecture, dataset, and training procedure. A definitive experiment would CPT the same Qwen3 models (0.6B–8B) on raw Merrec interaction logs formatted as natural-language sequences, using the identical compute budget (163B tokens), the identical 50/50 general-to-recommendation split, and the identical perplexity evaluation across the seven held-out domains. If raw data produces flat or erratic scaling curves (α ≈ 0 for recommendation domains) while synthetic data produces the clean power laws in Figure 2, the paper's core claim would be unambiguously validated. If raw data also produces scaling laws (even with lower exponents), the paper's contribution would shift from "enabling scaling laws" to "improving scaling efficiency"—a more modest but still valuable finding. This experiment is straightforward to implement and would be the strongest possible evidence for or against the paper's central diagnostic claim.

Downstream ranking evaluation of CPT'd LLMs with controlled comparisons. The paper validates synthetic data quality using traditional sequential models (SASRec, GRU4Rec, etc. in the TSTR experiments) but never evaluates the CPT'd Qwen3 LLMs themselves on a downstream item ranking task. A critical follow-up would measure Recall@K and NDCG@K for the 0.6B–8B CPT'd models on the same filtered Merrec test set used in Figure 1, comparing: (a) CPT'd LLM on synthetic data vs. (b) CPT'd LLM on raw data (from the experiment above) vs. (c) traditional sequential models on synthetic data vs. (d) traditional sequential models on real data. This would answer the question the paper currently leaves open: does lower perplexity on recommendation domains (the scaling law metric) translate to better item recommendations from the LLM itself? It would also reveal whether there is a cross-model transfer gap—do traditional sequential models trained on synthetic data actually outperform CPT'd LLMs on the same synthetic data for the ranking task, suggesting that the LLM's general language capabilities create interference rather than synergy for recommendation?

Verifier-guided search over LLM-generated item sequences. The paper's case studies (Section 8, Figures 9–11) demonstrate that CPT'd models can generate coherent item recommendations given user interaction histories, and beam search produces diverse, relevant responses (Figure 10). This opens a direct parallel to the test-time compute scaling literature: can a process reward model (PRM) trained to score partial item sequences improve recommendation quality through search? A concrete experiment would train a PRM on synthetic UIH data (scoring each item in a sequence for whether it leads to a high-quality final recommendation), then compare greedy decoding vs. best-of-N vs. beam search against the PRM vs. weighted majority voting on a held-out ranking task. The paper's finding that UIH data exhibits strong scaling (α ≈ 0.45–0.59) but eventually overfits at ~16 repeats suggests that test-time search might provide an alternative path to improving recommendations without generating more training data—analogous to how the MATH scaling paper found that test-time compute could substitute for pretraining compute on easy-to-medium problems.

Scaling law validation at larger model sizes and on different model families. The joint scaling laws in Equations 2–5 are fitted on four model sizes (0.6B–8B) from a single model family (Qwen3). Extrapolating to 70B or larger models requires trusting the power-law form far beyond the fitted range. A direct validation would CPT a Qwen3-14B or Qwen3-32B model on the same data mixture and check whether the predicted perplexity from Equation 5 (UIH: ℓ = 0.514 + 1.89 N^−0.138 + 63.9 D^−0.272) matches the observed perplexity. The paper's overfitting experiments in Figure 7 already trained 14B and 32B models, but only for the reduced-UIH mixture ratio experiments, not for the main scaling law configuration. Extending the main experiment to 14B and 32B would directly test the joint fit's extrapolation quality. Additionally, replicating the experiment with Llama-3 or Mistral as the base model would test whether the hierarchy of scaling exponents (UIH > CF > Item-Text > General) and the data-dominant allocation recommendation (Table 4) are Qwen3-specific or generalize across model families. Given that different model families have different pre-training data mixtures and tokenizers, the absolute perplexity values would differ, but the relative ordering of α values across domains should be stable if the paper's thesis about data quality gating scaling is correct.

Cross-domain transfer of the synthetic data framework. The paper's layered curriculum is developed and tested exclusively on the Merrec C2C e-commerce dataset. A strong test of the framework's generality would apply it to a structurally different recommendation domain—content streaming (e.g., MovieLens or a proprietary video platform dataset), news recommendation (e.g., MIND), or music recommendation—where item semantics (movie plots, article text, song lyrics) and user behavior patterns (binge-watching vs. background listening vs. goal-directed reading) differ substantially from e-commerce. The key question is whether the two-layer structure (Item-Text alignment + CF data as Layer 1, synthetic UIH via graph random walks as Layer 2) is a universal pedagogical curriculum or is specific to the purchase-oriented, category-structured nature of e-commerce. A negative result—where the layered curriculum fails to produce scaling laws in a different domain—would help delineate the boundary conditions of the framework. A positive result—where the same data generation pipeline, with domain-appropriate item descriptions and interaction logs, produces similar scaling exponent hierarchies—would substantially strengthen the paper's claim of a foundational methodology.

Dynamic difficulty-adaptive data mixture during CPT. The paper's data mixture experiments (Section 7.2) treat the UIH ratio as fixed throughout training, but Figure 5 shows that overfitting emerges at different points depending on the ratio: at 15% UIH, overfitting begins at ~20B tokens; at 2%, it begins at ~160B tokens. This suggests a curriculum over time: start with a high UIH ratio to rapidly learn sequential patterns (α = 0.754 at 15% ratio per Table 7), then reduce the UIH ratio once the overfitting threshold approaches to avoid memorization, reallocating the budget to CF or Item-Text data that show more stable long-term scaling. A concrete experiment would use a cosine schedule on the UIH mixture ratio, starting at 15% and decaying to 2% over the 163B-token training run, and compare the final perplexity and scaling exponent against the fixed-ratio baselines in Table 7. If the scheduled approach achieves both the high early α of the 15% condition and the low L∞ of the 2% condition, it would demonstrate that the overfitting threshold can be circumvented through dynamic allocation rather than simply staying below it—a practically significant finding for teams with limited synthetic data budgets.

Practical Applications and Downstream Use Cases

Resource forecasting for LLM-based recommender development. The most immediate practical application of this work is enabling teams to estimate, before training begins, how much synthetic data and compute they need to reach a target recommendation quality. A team planning to CPT a 4B-parameter model on e-commerce data can use the fitted scaling law for UIH (Table 5: for 4B, ℓ(D) = 0.66 + A·D^−0.564) to estimate that achieving, say, a UIH Full perplexity of 0.70 would require a specific number of training tokens, which in turn tells them how many unique synthetic UIH sequences to generate (accounting for the ~16-repeat overfitting threshold from Section 7.2). The compute-optimal allocation analysis (Table 4) provides complementary guidance: since UIH is strongly data-dominant (βB = 17.38 vs. αA = 0.261), the team should invest heavily in synthetic data generation rather than scaling to 8B or larger models, at least until the data scaling exponent begins to diminish. This replaces the current trial-and-error approach—"train a 3B model, see if it outperforms the 900M model, adjust"—with a quantitative planning process analogous to what NLP teams have used since Chinchilla.

Cold-start item recommendation via semantic grounding. The paper's Layer 1 Item-Text alignment data explicitly teaches the model to map semantic token sequences to natural-language item descriptions. The ablation study (Figure 3) shows that models without Item-Text data achieve L∞ ≈ 3.0 on Item-Text evaluation—essentially random-guessing perplexity—while models with Item-Text data achieve L∞ ≈ 1.2, confirming that this mapping is learned and not discoverable from CF or UIH data alone. For a production recommender, this means new items added to the catalog (cold-start items with zero interaction history) can be represented by their semantic tokens, and the model will understand their content properties through the learned Item-Text mapping. The model can then use its CF knowledge (Layer 1, Component 2) and sequential behavior knowledge (Layer 2) to recommend the new item based on its semantic similarity to existing items and its fit within typical user sequences—capabilities that traditional collaborative filtering models trained only on interaction IDs cannot provide. The practical benefit is reducing the "cold-start penalty" where new items receive poor recommendations until they accumulate sufficient interaction history, which is a major pain point in fast-moving e-commerce catalogs.

Long-tail item surfacing through debiased training data. The synthetic UIH data is generated from random walks on a CF graph where edge weights reflect co-occurrence strength but the walk process has no concept of item popularity—every neighbor of the current node is considered proportional to its edge weight, not its global interaction count. The paper's hyperparameter sweep (Appendix A, Table 9) explicitly optimized for low Gini coefficient (item frequency concentration) and high unique item count, selecting a configuration that achieves Gini = 0.635 and 180K unique items. In contrast, the raw Merrec logs exhibit the extreme popularity concentration typical of e-commerce: "Top 1% of items receive 80% of interactions" (per Table 1). A model trained on the synthetic data therefore sees a much flatter item distribution during training, learning to make recommendations that reflect item relevance rather than item popularity. For a production recommender, this directly addresses the "rich-get-richer" feedback loop where popular items dominate recommendations, receive more interactions, and become even more popular. The practical outcome is that long-tail and niche items—which are often higher-margin for sellers and more satisfying for users with specific tastes—receive a fairer share of recommendation slots, improving both catalog diversity and user experience for non-mainstream preferences.

Privacy-preserving recommendation model training. The synthetic UIH generation process explicitly decouples training sequences from individual user histories: "the generated user interaction histories are derived from aggregated item-to-item graphs rather than individual user traces" (Section 10, Impact Statement). This means the training data contains no real user sessions—each synthetic sequence is a novel walk on the item graph that respects learned item-item affinities but corresponds to no actual person's behavior. For organizations subject to data privacy regulations (GDPR, CCPA) or handling sensitive recommendation domains (health products, financial services), this provides a path to training recommendation models without exposing individual user data to the training process. The practical workflow would be: (1) mine CF associations from raw logs in a privacy-preserving manner (with differential privacy guarantees if needed), (2) construct the item graph from these aggregated statistics, (3) generate synthetic UIH data from the graph, (4) train the model exclusively on synthetic data, and (5) deploy the model without any real user data ever having been in the training set. The paper's TSTR results (Figure 1) demonstrate that this approach actually improves recommendation quality rather than degrading it—the privacy benefit does not come at a performance cost, at least for the Merrec dataset and the evaluated sequential models.