Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments

ArXiv: 2602.23234

🎯 Pitch

A fine-tuned 3B-parameter model crushes a 30B pretrained one at generating textual relevance labels, achieving an F1 of 0.800 versus 0.382. When millions of these synthetic labels are fed into the App Store’s production ranker, they spark a simultaneous improvement in both behavioral and textual relevance offline, translating to a +0.24% conversion lift online—with the biggest gains concentrated on rare, long‑tail queries where click data is unreliable. This shows that targeted data generation can shift a Pareto frontier in a live, planet‑scale system.

---

1. Executive Summary

This paper deploys and validates an LLM-as-a-Judge paradigm at industrial scale for app-store search ranking, systematically comparing pretrained and fine-tuned in-house language models to generate pointwise textual relevance labels that augment a multi-objective production ranker. The core finding is that a specialized fine-tuned 3B-parameter model significantly outperforms a pretrained 30B-parameter model (F1 of 0.800 vs. 0.382) as a relevance label generator, functioning as a "force multiplier" for scarce human judgments. Augmenting the production ranker's training data with millions of these LLM-generated labels yields a Pareto frontier shift — simultaneously improving offline NDCG for both textual and behavioral relevance — and translates to a statistically significant +0.24% increase in conversion rate in a worldwide A/B test. The online gains concentrate disproportionately on tail queries, establishing that LLM-generated relevance labels provide the strongest signal precisely where behavioral relevance data is most sparse or absent.

2. Context and Motivation

The Core Problem: Behavioral Relevance Abundance vs. Textual Relevance Scarcity

This paper addresses a fundamental asymmetry in how large-scale commercial search ranking systems are trained: behavioral relevance signals are plentiful, but textual relevance signals are scarce. To understand why this asymmetry matters, we need to understand what each signal captures and why both are essential.

Behavioral relevance refers to the implicit feedback users provide through their interactions — clicks on results, app downloads, dwell time, and so on. When millions of users search for "photo editor" and consistently tap on a particular app, that app is behaviorally relevant to the query. These signals are abundant because they are generated passively by every user session in the App Store; the paper describes them as coming from "aggregated App Store search logs" covering a large time window (Section 3.2, 4.1). Behavioral relevance is powerful because it captures revealed preference — what users actually do, not what they say they want.

Textual relevance, by contrast, captures the semantic fit between a query and a result — whether the app is genuinely about what the user searched for. This is assessed by trained human judges who examine the query, review the app's metadata (title, description, keywords, category), and assign a graded relevance label according to a defined rubric. The paper's prompt template (Figure 1) shows five relevance levels (label_1 through label_5), indicating that human judges provide granular, pointwise assessments of how well an app matches a query's intent.

The problem is one of scale: behavioral relevance labels can be collected continuously from every user interaction, but textual relevance labels require human judges to individually examine and rate query-app pairs. The paper is explicit about the consequences of this imbalance: "the textual relevance objective [is] under-powered in multi-objective training" (Section 1). In other words, when training a ranker that optimizes for both objectives simultaneously, the behavioral signal — by virtue of its sheer volume — can overwhelm the textual signal, causing the ranker to prioritize click-bait or popularity-driven results over semantically appropriate ones.

This is not merely an academic concern. In a digital marketplace like the App Store, where millions of apps compete for visibility and users issue billions of queries, a ranker that cannot adequately distinguish between "this app is genuinely relevant to the query" and "this app gets clicked because it has a compelling icon and is listed first" will produce systematically degraded discovery experiences. Users searching for a niche productivity tool may be shown a popular game instead simply because the game has more behavioral data. The textual relevance signal exists precisely to prevent this — but it can only do so if it is present at sufficient scale.

Why This Problem Is Important: The Long Tail and Pareto Efficiency

The importance of this problem crystallizes when we examine where behavioral labels fail. The paper's Figure 2 provides the critical insight: conversion rate improvements from LLM-generated textual labels are concentrated in the tail queries — low-frequency searches that individually appear rarely but collectively represent a substantial fraction of all queries. For head queries like "Instagram" or "YouTube," behavioral data is overwhelming and reliable; the ranker knows exactly what users want because millions of them have demonstrated it. But for tail queries like "vector illustration app for scientific figures" or "ADHD time management with pomodoro," behavioral data is sparse, noisy, or entirely absent. A user searching for such an app may see results dominated by unrelated popular apps simply because the ranker has no signal to distinguish relevance.

This connects to a broader phenomenon known in information retrieval: the long-tail problem, where the cumulative mass of rare queries exceeds that of frequent ones, yet statistical ranking models (which depend on repeated observations) systematically degrade on these rare cases. Textual relevance labels solve this problem because they do not depend on user traffic — a single human judgment can provide a reliable relevance signal for a tail query-app pair that may only appear in a handful of searches per year. The paper's contribution is showing that LLMs can scale this single-judgment approach to millions of query-app pairs, effectively providing robust textual relevance signals across the entire frequency spectrum, with the greatest marginal value accruing precisely where behavioral signals are weakest.

The problem is also important from a multi-objective optimization perspective. The paper's production ranker uses scalarization — a technique that combines two objectives (behavioral and textual relevance) by mixing their training data at a tunable ratio (Section 3.2). When one objective's signal is dramatically underrepresented in the training data, the Pareto frontier — the set of solutions where improving one objective requires sacrificing the other — is inherently constrained. You cannot explore tradeoffs between behavioral and textual relevance if the textual relevance signal is too sparse to meaningfully influence the model's parameters. By generating millions of textual labels, the paper effectively expands the optimization landscape, enabling the ranker to discover parameter configurations that were previously unreachable. The offline results in Table 2 confirm this: the LLM-augmented model achieves strictly higher NDCG on both behavioral and textual relevance — a "Pareto improvement" that is theoretically significant because it means the production model was not on a true Pareto frontier at all; it was operating below the frontier due to insufficient textual supervision.

Prior Approaches and Their Limitations

The paper situates itself against several strands of prior work, each of which addresses parts of the problem but falls short of a complete solution.

LLMs as real-time rerankers. A substantial body of work has explored using LLMs as pointwise, pairwise, or listwise rerankers that process search results at inference time [1, 4, 14, 15, 18–20, 25, 32, 33]. In this paradigm, each user query is sent to an LLM along with candidate results, and the LLM scores, compares, or reorders them on the fly. While this approach can improve relevance, it introduces latency constraints — the LLM inference must complete within the tight time budget of a user-facing search request (typically tens to hundreds of milliseconds). This forces tradeoffs between model quality (larger models are slower) and response time, and it multiplies inference costs by the volume of user traffic, which in a global App Store is enormous.

The paper explicitly differentiates itself from this line of work: "our work differs from these approaches as they mainly focus on deploying LLMs as real-time rerankers; instead, we use an LLM as an offline label generator to create large-scale textual relevance for a production ranker" (Section 2, paragraph 1 of "LLMs as Judges"). The key insight is that the LLM's judgment is needed once per query-app pair (offline), not once per user request (online). This decouples the computational cost of LLM inference from the volume of user traffic, making it feasible to use sophisticated models (like the fine-tuned 3B-parameter model) without latency concerns.

LLM-as-a-Judge for evaluation. Another strand of work has demonstrated that LLMs can substitute for human evaluators in assessing output quality, relevance, or correctness [2, 5, 7, 8, 10, 13, 22, 30]. These approaches typically use LLMs to score or rank system outputs for benchmarking purposes — for example, G-Eval [13] uses GPT-4 to evaluate NLG quality with chain-of-thought prompting, and ARES [22] automates evaluation of retrieval-augmented generation systems. While this line of work establishes that LLMs can produce judgments aligned with human preferences, it stops at evaluation — the judgments are used to measure system performance, not to improve it.

The paper positions itself as extending this paradigm from evaluation to training data generation: "Our paper distinguishes itself by using a fine-tuned, in-domain LLM not just for evaluation, but as a force multiplier to generate millions of textual relevance labels used directly as training data for our production ranker" (Section 2, "LLMs as Judges"). This is a conceptual leap: rather than using LLM judgments to answer "how good is our system?", the paper uses them to answer "how can we make our system better?"

Knowledge distillation approaches. RRADistill [6] represents an alternative strategy for addressing label scarcity: distilling the semantic understanding of large LLMs into smaller, specialized ranking models through architectural modifications. The core idea is to train a student model to mimic the relevance assessments of a teacher LLM, but with lower inference cost. While effective, this approach requires modifying the ranker's internal architecture to accommodate the distillation objective. The paper's approach is architecturally agnostic: "our work utilizes pointwise labels to train the production ranker without modifying its internal architecture" (Section 2, "LLMs as Judges"). The LLM-generated labels are simply additional rows in the training data, compatible with any ranking architecture that accepts pointwise supervision. This is a practical advantage in industrial settings where the ranker architecture may be a complex, heavily optimized component that cannot be easily altered.

Multi-objective LTR with content signals. The paper by Liu et al. [12] is the closest prior work, as it also combines behavioral and LLM-generated content signals for product search ranking. However, their approach differs in a critical detail: they "apply a sigmoid transformation to the LLM-generated label" before integrating it into the training objective. The paper's approach is simpler and more direct: "we utilize a data-mixing approach within our multi-objective framework" (Section 2, "Multi-Objective Learning to Rank"). Rather than transforming the labels through a nonlinear function, the LLM-generated labels are treated as equivalent to human-generated labels and mixed into the training data at a tunable ratio. This design choice reflects a philosophical difference: the paper trusts its fine-tuned LLM judgments to be sufficiently aligned with human judgments that no calibration transformation is needed.

Preference alignment for ranking. A growing body of work explores how to align LLM preferences with ranking-specific objectives, using techniques like permutative preference alignment [29] or prompt engineering for ranking goals [21]. These methods aim to improve the LLM's own ability to act as a ranker. The paper draws inspiration from alignment techniques in prompt and model design but explicitly states the goal is "fundamentally different. We do not seek to deploy the LLM as a ranker, but to obtain relevance labels that align with the existing human-rated rubric and can be used for training the ranker" (Section 2, "Preference Alignment and Prompting for Ranking Objectives"). This clarification is important because it positions the LLM as a data generation tool rather than a ranking system, sidestepping the challenges of real-time LLM inference, calibration, and deployment.

Where Prior Approaches Collectively Fall Short

Synthesizing across these strands, the paper identifies a gap that no prior work fully addresses: how to use LLM-generated textual relevance labels at massive scale to directly improve a production multi-objective ranker without architectural changes or latency penalties. Each prior approach solves part of the problem but leaves a critical piece missing:

• Real-time LLM rerankers introduce latency and cost that make them impractical for high-volume production search.
• LLM-as-a-Judge for evaluation provides labels but doesn't demonstrate their effectiveness as training data for a learned ranker.
• Distillation approaches like RRADistill improve efficiency but require architectural modification and still depend on a teacher LLM at training time.
• Multi-objective LTR with content signals (Liu et al.) demonstrates the concept but uses label transformations that may discard information and hasn't been validated at the scale of a global App Store.

The paper's contribution is therefore not any single technique in isolation, but rather the end-to-end validation of a pipeline: fine-tune a domain-specific LLM → generate millions of labels → mix them into multi-objective ranker training → shift the Pareto frontier → validate with A/B testing → demonstrate tail-query gains. This pipeline leverages the strengths of each prior approach (LLM judging, multi-objective optimization, offline label generation) while avoiding their weaknesses (latency, architectural change, scale limitations).

How the Paper Positions Itself: The "Force Multiplier" Metaphor

The paper's central metaphor — "force multiplier" — is revealing. A force multiplier in military strategy is a capability that amplifies the effectiveness of existing forces without requiring proportional increases in personnel or equipment. Applied to this work: human judges are the "existing forces" — expensive, scarce, but producing high-quality, rubric-aligned labels. The fine-tuned LLM is the force multiplier — it takes the limited set of human judgments as training data, learns to approximate their decision-making, and then generates orders of magnitude more labels at near-zero marginal cost.

This metaphor captures why the paper's approach is fundamentally different from replacing human judges with LLMs. The human judgments remain essential as training and calibration data for the LLM. The LLM does not replace the judge; it extends their reach. The paper's experimental design reinforces this: the LLM is fine-tuned on a training set of human judgments, evaluated against a held-out validation set of human judgments, and the labels it generates are treated as equivalent to human labels in the ranker training pipeline. The human judge's rubric, standards, and domain expertise are preserved and scaled, not discarded.

The paper also positions itself within the broader trajectory of industrial search systems. The introduction establishes that "high-quality textual relevance labels are scarce and expensive to produce, creating a scalability bottleneck" (Section 1). This bottleneck is not unique to the App Store — it affects any large-scale search or recommendation system where semantic relevance matters. By providing a "practical blueprint for other large-scale search systems to overcome relevance-label scarcity" (Section 6), the paper aims to establish a generalizable methodology rather than a product-specific optimization. The blueprint is: (1) fine-tune a domain-specific LLM on existing human judgments, (2) use it to generate labels at scale, (3) augment multi-objective ranker training with these labels, (4) validate with offline metrics and online A/B testing, (5) expect the largest gains on tail queries.

Crucially, the paper does not claim to have invented the LLM-as-a-Judge concept or the multi-objective LTR framework. Its contribution is the industrial-scale validation — proving that these ideas, when combined and executed at the scale of a global App Store with a fine-tuned in-house model, produce measurable improvements in both offline and online metrics. The significance lies in closing the gap between academic demonstrations (small-scale, controlled datasets) and production reality (millions of queries, real-time constraints, multiple objectives, rigorous A/B testing).

3. Technical Approach

3.1 Reader Orientation

This paper builds an offline label generation pipeline that uses a fine-tuned large language model to produce millions of pointwise textual relevance labels for query-app pairs, which are then injected as additional training data into an existing multi-objective production ranker for the App Store. The system solves the problem of textual relevance label scarcity — where human judges can only annotate a tiny fraction of query-app pairs — by training a specialized 3B-parameter model on historical human judgments and then using that model as a "force multiplier" to generate high-quality labels at massive scale, decoupling the cost of textual relevance assessment from the limited throughput of human annotators.

3.2 Big-Picture Architecture (Diagram in Words)

The system is a two-stage offline pipeline followed by an online deployment. The major components are:

1. Human Judgment Dataset — a small, high-quality set of query-app pairs labeled by trained human judges with graded relevance scores (label_1 through label_5). This serves as both fine-tuning data for the LLM and evaluation ground truth.
2. LLM Judge — a language model (pretrained 3B, pretrained 30B, or fine-tuned 3B) that takes a query, app metadata, and optional few-shot examples as input and outputs a predicted relevance label. This is the force multiplier that scales human judgment.
3. Aggregated Search Logs — a large-scale dataset of millions of query-app pairs from historical App Store session logs, containing no textual relevance labels. These are the candidates for which the LLM generates new labels.
4. Behavioral Relevance Labels — abundant implicit feedback (clicks, downloads) derived from the same search logs, used as the primary training signal for the production ranker.
5. Training Data Mixer — a configurable mechanism that combines behavioral labels, existing human textual labels, and LLM-generated textual labels into a unified training dataset with a tunable mixing ratio.
6. Multi-Objective Ranker — the production ranking model, trained via scalarization on the mixed dataset to jointly optimize for both behavioral and textual relevance.
7. A/B Test Infrastructure — the online deployment framework that serves the LLM-augmented ranker to a fraction of worldwide traffic and measures conversion rate.

Information flow: Human judgments → fine-tune LLM Judge → LLM Judge scores millions of query-app pairs from search logs → LLM-generated labels join behavioral labels and human labels in the Training Data Mixer → mixed data trains the Multi-Objective Ranker → ranker is deployed in A/B test → conversion rate measured, with gains concentrated on tail queries.

3.3 Roadmap for the Deep Dive

• First, the LLM relevance label generation pipeline — how prompts are constructed, what configurations are tested, how fine-tuning works, and why the fine-tuned 3B model outperforms the pretrained 30B model. This is the core technical contribution and must be understood before the labels can be used.
• Second, the multi-objective ranker training framework — how scalarization works as an MOO technique, how the data mixing ratio serves as a tunable hyperparameter, and why "strict separation" of label sources during gradient computation is crucial. This provides the mechanism by which LLM-generated labels influence the ranker.
• Third, the evaluation methodology for the LLM labels — how precision, recall, and F1 are computed against held-out human judgments, and why these metrics are the right way to validate label quality before integrating them into the ranker.
• Fourth, the offline ranker evaluation protocol — how NDCG@k is computed separately for textual and behavioral relevance using distinct validation sets, and why this separation demonstrates a true Pareto improvement rather than a tradeoff.
• Fifth, the A/B test design and tail-query analysis — how conversion rate is defined, how query frequency buckets are constructed, and what the concentration of gains in tail buckets tells us about the mechanism of improvement.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an industrial system paper whose core idea is that a fine-tuned domain-specific LLM can serve as a scalable offline annotator, generating textual relevance labels that, when mixed into multi-objective ranker training, shift the behavioral-textual Pareto frontier outward and produce measurable online conversion gains, especially for tail queries where behavioral signals are sparse.

---

LLM Relevance Label Generation Pipeline

The label generation pipeline transforms a large pool of unlabeled query-app pairs into a labeled dataset suitable for ranker training. The input is historical search logs containing $n \times m$ candidate query-app pairs (where $n$ is the number of queries and $m$ is the number of apps considered per query). The output is a pointwise textual relevance label for each pair, assigned by an LLM configured to mimic the decision-making of human judges.

Prompt Construction. The LLM is prompted using a structured template (Figure 1 in the paper) that provides:

• A role specification: "Imagine you are an App Store evaluator."
• The task description: given a user search term and an app with its metadata, choose one of five relevance labels.
• The relevance rubric: descriptions of what each label level means (label_1 through label_5, with label_1 presumably being "excellent" or "perfect" relevance and label_5 being "poor" or "irrelevant" — the exact rubric text is referenced in the prompt template but not reproduced in full in the paper).
• A strict output constraint: "Your response should only be the label and nothing else." This constraint is critical because it prevents the LLM from generating explanatory text, hedging, or alternative formatting that would require post-processing to extract the label.
• Few-shot examples (when using few-shot prompting): a set of previously rated query-app pairs from human judges, presented with their associated metadata and assigned labels. The paper states that "few-shot prompts with textual labels performed best" (Section 4.2.2), meaning the label values are presented as text strings (e.g., "Excellent" rather than "5").
• The target query-app pair: the specific query and app metadata for which the model must generate a label.

The app metadata available to the LLM matches what human judges see: three metadata fields (labeled app_metadata_1, app_metadata_2, app_metadata_3 in Figure 1). The paper does not disclose the exact nature of these fields (they could include app title, description, keywords, category, developer name, or other signals), but the key design choice is that the LLM receives exactly the same information that human judges use. This ensures that any differences in labeling behavior reflect the model's judgment, not asymmetric information access.

Model Configurations Tested. The paper evaluates three LLM configurations (Section 4.2.1):

1. Pretrained 3B ("PT-3B"): An in-house 3-billion parameter language model used in its pretrained state without any fine-tuning on the human judgment dataset. This serves as a baseline for what a moderately-sized model can achieve with only its pretraining knowledge of language and relevance.

2. Pretrained 30B ("PT-30B"): An in-house 30-billion parameter model, also in its pretrained state. This is ten times larger than the 3B model and serves to test whether raw model scale can compensate for lack of domain-specific fine-tuning. The computational cost of inference with this model is substantially higher (roughly 10× more FLOPs per forward pass, assuming similar architecture families).

3. Fine-tuned 3B ("FT-3B"): The 3B model after being fine-tuned on the training set of human judgments. This model receives the same prompts as the pretrained models but has been explicitly trained to predict the labels that human judges assigned, effectively learning the specific relevance rubric and judgment patterns of the App Store's human evaluation team.

Fine-Tuning Procedure. The paper fine-tunes the 3B model on the training split of the human judgment dataset (Section 4.1, 4.2.1). While specific hyperparameters are not disclosed in the main text (the ACM format limits this to a 5-page paper), the fine-tuning objective is implicit: the model is trained to map (query, app_metadata_1, app_metadata_2, app_metadata_3) → human-assigned label, using a standard sequence-to-sequence or classification objective depending on whether the labels are treated as generated text or predicted classes.

The training data construction follows the same prompt format that will be used at inference time (Figure 1), meaning the model sees exactly the prompt structure — role description, rubric, examples, and target — during fine-tuning. This is a deliberate design choice: the model learns not just "what label to assign" but "what label to assign given this specific prompt format." Training on the exact prompt template ensures that the model's behavior at inference time is in-distribution with respect to its fine-tuning data, reducing the risk of format-induced errors.

Why Fine-Tuning Works: The Domain Gap. The paper does not provide an extensive theoretical analysis, but the logic of why fine-tuning is necessary is clear from the experimental design. A pretrained model, regardless of its scale, possesses general knowledge about relevance — it understands that "photo editor" should match apps about photo editing. However, it lacks knowledge of the App Store's specific relevance rubric: what distinguishes a label_2 from a label_3 in this particular annotation scheme, what edge cases human judges consider, what metadata fields are most diagnostic, and how to calibrate its judgments to match the distribution of labels that human judges assign.

The fine-tuning process bridges this domain gap. By training on examples of (query, app, human_label), the model learns:

• The label distribution of the human judges (how often each label appears).
• The decision boundaries between adjacent relevance levels.
• The metadata interpretation patterns that human judges use.
• The edge cases where surface-level semantic similarity does not imply true relevance (e.g., an app named "Photo Vault" might semantically match "photo" but is actually a privacy app, not a photo editor).

The pretrained 30B model, despite having an order of magnitude more parameters, cannot acquire this domain knowledge from its pretraining corpus because that corpus does not contain the App Store's specific relevance rubric or human judgment patterns.

Label Generation at Scale. Once the optimal configuration (FT-3B with few-shot textual prompts) is identified through offline evaluation, it is used to perform inference on the large-scale aggregated query-app log data (Section 4.1). This dataset contains "millions of query-app pairs" — the exact size is not specified, but the qualifier "millions" indicates at least $10^6$ and likely $10^7$ or more pairs.

The generation is entirely offline, meaning:

• There is no latency constraint from user-facing requests.
• The full computational cost is a one-time batch inference job, amortized across all future uses of the generated labels.
• The labels can be validated against held-out human judgments before being integrated into ranker training.

This offline design is a critical architectural decision that differentiates the paper's approach from real-time LLM reranking. The paper explicitly states: "By performing this generation offline, we create a force multiplier for human annotation without the latency constraints of real-time LLM reranking" (Section 3.1). The computational cost of inference — even with a 3B-parameter model — would be prohibitive if incurred per user query, but as a batch job it is manageable.

The generated labels are pointwise: each label applies to a single query-app pair independently. This contrasts with pairwise approaches (which would rank pairs of apps against each other for the same query) and listwise approaches (which would assign relevance scores across a full list of results for a query). The paper chooses pointwise labels because they are:

• Simpler to generate: each inference call produces exactly one label, with no need to maintain list context.
• Compatible with any ranker architecture: pointwise labels can be used to train pointwise, pairwise, or listwise rankers, whereas pairwise labels constrain the training objective.
• Directly analogous to human judgments: human judges also produce pointwise labels, so the LLM's output format matches the existing data format exactly.

Force Multiplier Quantification. The paper's central claim is that the FT-3B model is a "force multiplier" — but what is the multiplication factor? The paper does not specify the exact number of human judgments in the training set, but the implication is clear: a team of human judges produces a limited set of labels (likely thousands to tens of thousands, based on typical industrial annotation budgets), and the FT-3B model scales this to millions. If, for example, 10,000 human judgments enable generation of 10,000,000 LLM labels, the force multiplication factor is $1000\times$. The key insight is that the marginal cost of one additional LLM-generated label is essentially zero (just the inference cost of a single forward pass), whereas the marginal cost of one additional human-generated label is high (recruiting, training, quality control, and paying a human judge for their time).

---

Multi-Objective Ranker Training via Scalarization

The production ranker is trained within a multi-objective optimization (MOO) framework that must simultaneously optimize for two distinct relevance signals: behavioral relevance (what users click and download) and textual relevance (how well results match the query semantically). The paper's approach to combining these objectives is scalarization — converting the multi-objective problem into a single-objective problem by constructing a weighted training dataset.

The Training Data Structure. The training data is constructed from two distinct label sources (Section 3.2):

1. Behavioral relevance labels: Derived from aggregated App Store search logs. These are implicit signals — a query-app pair receives a behavioral relevance score based on aggregated user clicks and downloads. The exact score derivation is not disclosed, but typical approaches include click-through rate (CTR), download rate, or engagement-weighted combinations thereof. Critically, these labels are abundant because they are generated passively by every user session.

2. Textual relevance labels: Explicit relevance judgments from two sub-sources:

   • Human judges: A limited set of manually labeled query-app pairs, serving as the gold standard.
   • LLM-generated labels: The millions of labels produced by the FT-3B model, designed to mimic human judgments.

The crucial structural property of this training data is that the same query-app pair can appear multiple times — once with a behavioral relevance label and once with a textual relevance label. The paper states: "Crucially, the same query-app pair, represented by an identical feature vector, can appear in the training data multiple times — once with a behavioral relevance label and once with a textual relevance label" (Section 3.2). This means that if a particular (query, app) pair has both a click history and a textual relevance judgment, the ranker sees both labels for the same input features, and the training objective receives gradients from both objectives.

Scalarization via Data Mixing. The paper uses a specific form of scalarization: rather than explicitly combining loss functions with a weighted sum, it mixes the training data at a tunable ratio and trains on the combined dataset. This is described as:

"We employ a common MOO technique known as scalarization, where we combine the two objectives into a single objective function by creating a weighted mix of the training data" (Section 3.2).

Operationalizing this: if the desired mix is $70-30$ (70% behavioral, 30% textual), then during each training epoch, 70% of the examples in each batch are drawn from the behavioral label source and 30% from the textual label source. The model sees both types of labels, computes a single loss function (presumably a pointwise or pairwise ranking loss), and backpropagates gradients that reflect the mixed distribution.

This is mathematically equivalent to minimizing:

$L_{\text{total}} = \alpha \cdot L_{\text{behavioral}} + (1 - \alpha) \cdot L_{\text{textual}}$

where $\alpha$ is the mixing weight (e.g., 0.7 for a 70-30 mix), $L_{\text{behavioral}}$ is the loss computed on behavioral examples, and $L_{\text{textual}}$ is the loss computed on textual examples. However, rather than explicitly weighting the losses, the paper achieves the same effect by controlling the proportion of each label type in the training data. This approach has the practical advantage of not requiring modifications to the training objective — the ranker's loss function remains unchanged, and only the data composition changes.

Strict Separation of Label Sources. The paper emphasizes that "strict separation of label sources during comparison is crucial, as it ensures the gradients for each objective are computed independently" (Section 3.2). This means that when the model encounters a behavioral example, it computes the loss using only the behavioral label, and the resulting gradient reflects only the behavioral objective. Similarly, textual examples produce gradients reflecting only the textual objective. The model never sees a "combined label" that merges behavioral and textual relevance into a single value.

Why is this separation crucial? In multi-objective optimization, conflating the objectives would make it impossible to control their relative influence. If behavioral and textual signals were merged into a single composite label per example, the training process would optimize for the average of the two, potentially settling at a point on the Pareto frontier that reflects the composite label construction rather than a deliberate tradeoff between the two objectives. By keeping the labels separate, the data mixing ratio $\alpha$ serves as an interpretable, tunable hyperparameter that directly controls the ranker's position on the behavioral-textual Pareto frontier.

The Data Mixing Ratio as a Hyperparameter. The paper states: "By sampling or limiting rows from each data source, we can construct datasets with any desired mix (e.g., 90-10, 70-30, 50-50), enabling us to systematically train different models that correspond to different points on the Pareto frontier" (Section 3.2). This is a powerful capability: by training multiple models with different mixing ratios, the team can trace out the Pareto frontier and select the model that best balances the two objectives for the production deployment.

The paper reports (Section 4.3.2) that augmenting with LLM-generated labels shifts the Pareto frontier outward — meaning that for any given level of behavioral relevance, the LLM-augmented model achieves higher textual relevance than the production model, and vice versa. Further experimentation revealed that "by varying the proportion of LLM-generated labels, we could then move along this new, superior frontier, rather than shifting the frontier further outward" (Section 4.3.2). This suggests that once the frontier is shifted outward by adding LLM labels, the mixing ratio can be tuned to select the desired operating point on the new frontier.

What the Ranker Architecture Looks Like (Inferred). The paper does not disclose the ranker's internal architecture, but from the described training setup, we can infer several properties:

• The ranker takes a query-app feature vector as input. This vector includes app metadata features (the same fields available to the LLM and human judges), query features (likely embeddings or n-gram representations), and possibly cross-features (interaction terms between query and app features).
• The ranker is trained with pointwise supervision — each training example is a single query-app pair with a label, and the model predicts a relevance score for that pair.
• The loss function is likely a pointwise regression or classification loss (e.g., mean squared error for continuous behavioral scores, cross-entropy for discrete textual labels), or a pairwise loss that compares the predicted scores of two apps for the same query.
• The multi-objective aspect is handled entirely through data mixing, not through architectural modifications or specialized loss functions.

This architectural agnosticism is a deliberate design choice highlighted in the related work discussion: "our work utilizes pointwise labels to train the production ranker without modifying its internal architecture" (Section 2). The LLM-generated labels are simply additional rows in the training table, compatible with any ranking architecture that accepts pointwise supervision.

---

Offline Evaluation of LLM Label Quality

Before the LLM-generated labels can be trusted for ranker training, their quality must be validated against held-out human judgments. The paper conducts this evaluation using standard classification metrics (Section 4.2.1).

Evaluation Setup. The human judgment dataset is split into training and validation sets (Section 4.1). The training split is used to fine-tune the FT-3B model and to construct the few-shot prompts for all configurations. The validation split is held out and used only for evaluation. Each model configuration (PT-3B, PT-30B, FT-3B) generates labels for the validation set queries, and these predicted labels are compared against the ground-truth human labels.

Metrics. The paper reports precision, recall, and F1 scores (Table 1). While the paper does not specify how these metrics are computed for a multi-class classification problem with five ordered labels, the standard approach is:

For a given relevance level (e.g., label_1 = "perfect match"):

• Precision measures: of all query-app pairs that the model assigned label_1, what fraction did human judges also assign label_1? This penalizes false positives (the model being too generous with high-relevance labels).
• Recall measures: of all query-app pairs that human judges assigned label_1, what fraction did the model also assign label_1? This penalizes false negatives (the model being too conservative with high-relevance labels).
• F1 is the harmonic mean of precision and recall: $F1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}}$.

The reported F1 scores (Table 1) are likely macro-averaged across the five relevance levels (equal weight to each label class) or micro-averaged (weighted by class frequency). The paper does not specify which, but macro-averaging is more conservative for imbalanced datasets because it does not let common classes dominate the metric.

Results Interpretation. The results (Table 1) are striking:

Several observations:

1. Scale alone is insufficient. The pretrained 30B model achieves an F1 of 0.382 — better than the pretrained 3B (0.287) but still far below human-level agreement (which would be the inter-annotator agreement among human judges, not reported but typically 0.7-0.9 for well-defined rubrics). A 10× increase in parameters yields only a 33% improvement in F1 (0.287 → 0.382), suggesting diminishing returns to pretraining scale for this domain-specific task.

2. Fine-tuning is transformative. The FT-3B model achieves an F1 of 0.800 — more than double the pretrained 3B (0.287 → 0.800) and more than double the pretrained 30B (0.382 → 0.800). This is a 179% improvement over the same-sized pretrained model and a 109% improvement over the 10× larger pretrained model. The gap between FT-3B and PT-30B (0.800 vs. 0.382) represents the value of domain-specific fine-tuning over raw scale.

3. Precision and recall are balanced. The FT-3B model achieves nearly identical precision (0.802) and recall (0.798), indicating no systematic bias toward over-generous or over-conservative labeling. The pretrained models both show a slight recall advantage over precision (0.309 vs. 0.300 for PT-3B; 0.402 vs. 0.424 for PT-30B with the larger model showing slightly higher precision), suggesting they may be slightly conservative in assigning high-relevance labels — possibly because they lack confidence in the domain-specific rubric.

Why These Metrics Matter for the Pipeline. The label quality evaluation serves as a gate: if the LLM-generated labels are too noisy or poorly aligned with human judgments, injecting them into ranker training would add noise rather than signal, potentially degrading rather than improving the ranker. The FT-3B's F1 of 0.800 indicates strong but imperfect alignment with human judges — the model agrees with humans on 80% of labeling decisions (by F1, which balances precision and recall). The remaining 20% disagreement could come from:

• Ambiguous cases where even human judges would disagree (inter-annotator disagreement).
• Systematic biases in the LLM's judgment that differ from human biases.
• Cases where the LLM lacks contextual knowledge that human judges possess.

The paper implicitly argues that this level of agreement is sufficient for training data augmentation: the LLM-generated labels are "good enough" to strengthen the textual relevance signal, and the residual errors are outweighed by the benefit of dramatically increased label coverage, especially for tail queries where the alternative is no label at all.

Prompt Engineering Details (Partially Disclosed). The paper notes that "few-shot prompts with textual labels performed best" (Section 4.2.2) but does not provide full ablation results for different prompt configurations. The configurations explored include:

• Zero-shot vs. few-shot: Whether to include example query-app pairs with their human-assigned labels in the prompt. Few-shot outperforms zero-shot, likely because the examples serve as implicit calibration — they show the model what the label distribution looks like and how the rubric is applied in practice.
• Textual vs. numeric labels: Whether the labels are represented as text strings (e.g., "Excellent," "Good," "Fair") or numeric values (e.g., "5," "4," "3"). Textual labels perform better, which is consistent with the LLM's pretraining — it has been trained on natural language text, not on numeric rating scales, so textual representations are more in-distribution and allow the model to leverage its understanding of the semantic content of label descriptors.

The paper acknowledges the omission: "For brevity, we omit detailed results on prompt design" (Section 4.2.2). This is a limitation for replicability, as the exact prompt format (wording of the rubric, selection of few-shot examples, formatting of metadata) can significantly affect LLM performance.

---

Offline Ranker Evaluation Protocol

Once the LLM-generated labels are produced and validated, the next stage is evaluating their impact on the ranker itself. The paper conducts a controlled offline experiment comparing two ranker variants (Section 4.3.1):

• prod: The production model, trained on behavioral labels from aggregated search logs plus the limited set of human textual labels.
• llm-augmented: The same architecture and training procedure, but with the training data further augmented by millions of LLM-generated textual labels.

Evaluation Metric: NDCG@k. The paper uses Normalized Discounted Cumulative Gain at rank $k$ (NDCG@k) as the offline metric (Section 4.3.1, citing Järvelin and Kekäläinen, 2002). NDCG@k measures ranking quality by comparing the model's top-$k$ results against an ideal ranking where all results are perfectly ordered by relevance. It is defined as:

$NDCG@k = \frac{DCG@k}{IDCG@k}$

where $DCG@k$ (Discounted Cumulative Gain) is:

$DCG@k = \sum_{i=1}^{k} \frac{\text{rel}_i}{\log_2(i + 1)}$

and $IDCG@k$ is the ideal DCG — the DCG of the ground-truth ranking where items are sorted by their true relevance scores in descending order.

Here, $i$ is the rank position (from 1 to $k$), $\text{rel}_i$ is the relevance score of the item at position $i$, and the denominator $\log_2(i + 1)$ is the discount factor that penalizes relevant items appearing at lower ranks (the discount grows logarithmically, so the drop from rank 1 to rank 2 is more severe than from rank 10 to rank 11).

What NDCG@k computes operationally: For a given query, the ranker produces an ordered list of apps. Each app at position $i$ has a true relevance score $\text{rel}_i$ (from human judges for textual NDCG, from aggregated behavioral data for behavioral NDCG). The DCG sums these relevance scores, weighted by a logarithmic position penalty. This sum is divided by the DCG of the perfect ranking (where all apps are in descending order of true relevance). The result is a value between 0 and 1, where 1 indicates perfect ranking and lower values indicate that relevant items are buried at lower positions.

Why NDCG@k for this evaluation: The choice of NDCG@k is appropriate for a search ranking system because:

• It accounts for position bias (users are more likely to interact with top-ranked results).
• The logarithmic discount matches empirical observation that user attention decays roughly logarithmically with rank.
• Normalization by IDCG makes scores comparable across queries with different numbers of relevant items.
• Reporting at multiple $k$ values (@1, @3, @7 in Table 2) shows how the improvement propagates through the ranking — from the very top result (what the user sees first) to deeper in the list.

Separate Relevance Scores for Each Objective. A critical design choice in the evaluation is that "the relevance score for each item used in the NDCG calculation is defined differently for each objective" (Section 4.3.1):

• Textual NDCG: The relevance score $\text{rel}_i$ is the human judge's label for that query-app pair. This means textual NDCG measures how well the ranker surfaces apps that human judges (not the LLM) consider textually relevant.
• Behavioral NDCG: The relevance score $\text{rel}_i$ is "a score derived from aggregated user clicks and downloads." This means behavioral NDCG measures how well the ranker surfaces apps that users actually click and download.

The evaluation data is constructed to be independent of the fine-tuning data: "the judgments used for computing the NDCG@k for the textual relevance are not included in the ones we used for finetuning and validation of the LLM relevance labels, as they are two distinct sets" (Section 4.3.1). This is essential to avoid circular evaluation — if the same human judgments were used to fine-tune the LLM and to evaluate the ranker trained on LLM-generated labels, the evaluation would be contaminated. By using separate human judgment sets for LLM training/validation and for ranker evaluation, the paper ensures that the ranker evaluation measures genuine generalization to unseen human judgments.

Interpreting the Pareto Improvement (Table 2). The results show that the LLM-augmented model achieves strictly higher NDCG on both objectives at all rank cutoffs:

The improvements are small in absolute terms (e.g., +0.001 to +0.007 across the various metrics) but consistent across all six numbers. This is precisely what a Pareto improvement looks like: no metric degrades and at least one improves. In multi-objective optimization, this is the gold standard — it means the production model was not on the true Pareto frontier, and the additional textual labels revealed a strictly better region of parameter space.

Why the Behavioral NDCG Also Improves. A natural question arises: why does adding textual relevance labels improve behavioral NDCG? The paper offers an explanation in its interpretation of the results (Section 4.3.2): "the increase of the training data with LLM-generated labels not only increases the semantic relevance of the results to the query, but also increases the likelihood of users downloading or clicking the results."

The mechanism is likely indirect but important: by teaching the ranker to better distinguish textually relevant apps from textually irrelevant ones, the augmented training data helps the model learn feature representations that are also useful for behavioral prediction. For example, if the ranker learns that "vector illustration app for scientific figures" should return apps whose metadata contains terms like "vector," "illustration," and "scientific" rather than apps that happen to have high click-through rates on unrelated queries, this improved semantic understanding may also help it identify apps that users are more likely to download because they are genuinely relevant, not just because they are popular. In other words, better textual relevance acts as a form of regularization against pure popularity bias in the behavioral objective.

The Data Mixing Ratio's Effect on the Frontier. The paper reports that "by varying the proportion of LLM-generated labels, we could then move along this new, superior frontier, rather than shifting the frontier further outward" (Section 4.3.2). This means that once the LLM labels are added, the mixing ratio $\alpha$ controls where on the new frontier the model operates:

• A higher proportion of LLM-generated labels (lower $\alpha$, more weight on textual objective) pushes the model toward higher textual NDCG at the expense of behavioral NDCG.
• A lower proportion (higher $\alpha$, more weight on behavioral objective) pushes toward higher behavioral NDCG.
• The fact that both can be improved simultaneously (the frontier shifts outward) means the LLM labels are not merely trading one objective for the other — they are genuinely expanding the model's capabilities.

However, the paper states that varying the proportion moves the model along the new frontier rather than further outward. This implies that the specific LLM-augmented model reported in Table 2 already operates at a frontier point, and that adding even more LLM labels does not continue to shift the frontier — it just changes the operating point. The frontier shift is achieved by adding LLM labels at all; the quantity of labels beyond some threshold determines the position on the frontier.

---

A/B Test Design and Tail-Query Analysis

The offline gains are validated through a large-scale online A/B test that measures the real-world impact on user behavior. The paper's A/B test design provides the critical link between offline metric improvements and business outcomes.

A/B Test Setup. The test ran on worldwide traffic, comparing the production model (prod) against the LLM-augmented model (llm-augmented) (Section 5). Key details:

• Traffic split: Not explicitly stated, but a typical large-scale A/B test randomizes users (or sessions) into treatment and control groups, with the treatment group served by the LLM-augmented model and the control group served by the production model.
• Duration and scale: Worldwide traffic over a sufficiently long period to achieve statistical significance on the primary metric. The paper reports results as "statistically significant" (Table 3), which implies the sample size was large enough to detect the observed effect.
• Primary metric: Conversion rate, defined as "the proportion of search sessions with at least one app download" (Section 5). This is a session-level binary metric: each search session either results in at least one download (conversion = 1) or does not (conversion = 0). The reported +0.24% improvement is the relative increase in this proportion.

Why Conversion Rate as the Primary Metric? Conversion rate is the standard North Star metric for app store search because it directly measures the system's core purpose: helping users find and download apps they want. It is:

• User-centric: It reflects actual user behavior, not intermediate proxies like click-through rate.
• Revenue-aligned: Downloads drive the app ecosystem, making conversion rate directly connected to business outcomes.
• Sensitive to relevance: If search results are more relevant, users are more likely to find and download apps, increasing conversion rate.
• Robust to position bias: Unlike click-through rate, which can be inflated by clickbait, downloads require genuine user intent, so a conversion rate improvement is less susceptible to gaming.

Definition and Significance of +0.24%. The paper reports a +0.24% increase in conversion rate (Table 3). The authors explicitly contextualize this: "While this number may appear small, it is considered a significant improvement for a mature industrial ranker" (Section 5). In a mature system like the App Store, where the ranker has been heavily optimized over years of iteration, most obvious improvements have already been captured. A +0.24% lift represents a genuine advance that required a novel approach (LLM-generated labels at scale) rather than incremental tuning of existing features.

To appreciate the magnitude: if the baseline conversion rate is, say, 20% of search sessions resulting in a download, then +0.24% means approximately 0.048 percentage points absolute improvement (20% → 20.048%). This might seem trivially small, but at the scale of the App Store (billions of search sessions per year), it translates to millions of additional downloads annually — a substantial business impact.

Storefront Coverage. The gain "was observed in 89% of storefronts" (Section 5). The App Store operates across many countries and languages (storefronts), and the LLM-augmented model improved conversion rate in the vast majority of them. This suggests the approach generalizes across languages and markets, which is expected because the LLM generates textual relevance labels using the same app metadata and query text available in each storefront. The 11% of storefronts that did not see improvement may represent markets where:

• The baseline behavioral signal is exceptionally strong, leaving little room for textual relevance improvement.
• The LLM's language capabilities are weaker for those specific languages.
• Statistical noise (some storefronts may have smaller traffic, making it harder to detect significant differences).

Tail-Query Analysis (Figure 2). The most insightful result in the paper is the analysis of conversion rate improvements across query frequency buckets. The paper divides queries into frequency buckets based on how often they appear in search logs:

• Low-numbered buckets correspond to tail queries — low-frequency, niche searches that individually appear rarely.
• High-numbered buckets correspond to head queries — high-frequency, popular searches that appear millions of times.

The x-axis of Figure 2 is labeled "Log Query frequency," indicating the buckets are logarithmically spaced, which is standard practice for query frequency analysis because query frequencies follow a heavy-tailed (typically Zipfian) distribution.

The y-axis shows "Conversion rate absolute difference" between the LLM-augmented and production models. A positive value means the LLM-augmented model has a higher conversion rate in that bucket.

The key finding is that "the most substantial improvements occur in the tail" (Section 5, Figure 2). The curve shows a clear downward trend: the leftmost (low-frequency, tail) buckets show the largest positive differences, and as frequency increases (moving right), the difference shrinks. The highest-frequency buckets show near-zero or slightly negative differences.

Why Gains Concentrate in the Tail. The paper's explanation is precise and well-supported by the system design: "tail queries, by definition, lack sufficient user traffic to generate reliable behavioral relevance signals. Our llm-augmented model excels here because the newly added textual relevance labels provide a robust and accurate signal where the behavioral signal is sparse or absent, effectively closing the relevance gap" (Section 5).

Let's unpack this mechanism:

• Head queries (e.g., "Instagram," "YouTube," "TikTok"): Millions of users search for these terms and consistently click/download the same apps. The behavioral signal is overwhelming — the ranker knows with high confidence which apps users want because the data is abundant and consistent. Adding textual relevance labels for these queries is redundant; the behavioral signal already captures the correct ranking, so the LLM labels provide little additional information.

• Tail queries (e.g., "vector illustration app for scientific figures," "ADHD time management with pomodoro," "app to track houseplant watering schedule"): These queries appear rarely (perhaps dozens or hundreds of times per year across the entire App Store). The behavioral signal for these queries is sparse (few observations), noisy (high variance due to small sample size), or entirely absent (zero observations for new or very rare queries). Without behavioral data, the ranker must rely on textual relevance — but without sufficient human-labeled examples, the textual relevance signal is also weak. The ranker may default to showing popular apps or apps with high behavioral relevance on other queries, which are likely poor matches for the specific tail query.

The LLM-generated labels solve this problem by providing textual relevance signals for millions of query-app pairs, including tail queries, before those queries ever appear in user sessions. Because the LLM can generate a label for any query-app pair (given the query text and app metadata), it does not need to wait for user traffic to accumulate. A new tail query can have textual relevance labels from day one, enabling the ranker to surface relevant apps immediately rather than waiting months for behavioral data to accumulate.

The "Relevance Gap" and Why It Matters. The "relevance gap" the paper refers to is the difference between the ranker's performance on head queries (where behavioral data is abundant) and tail queries (where it is not). Without textual relevance labels, this gap is inherent: you cannot have good behavioral relevance on queries with no behavioral data. The LLM-generated labels bridge this gap by providing a non-behavioral relevance signal that works uniformly across the frequency spectrum. The result is that the ranker's performance on tail queries rises to approach its performance on head queries — not by degrading head query performance, but by elevating tail query performance.

This finding has a profound implication: the marginal value of an additional relevance label is inversely proportional to the amount of existing behavioral data for that query. A textual relevance label for "Instagram" is nearly worthless because the behavioral signal is already perfect; a textual relevance label for a rare, specific query is extremely valuable because it may be the only relevance signal available. The LLM-as-a-Judge approach is therefore efficiently targeted: it generates labels across all queries, but the benefit of those labels is concentrated exactly where it is most needed.

Statistical Significance and Practical Significance. The paper reports that the +0.24% conversion rate increase is "statistically significant" (Table 3). With worldwide traffic, even a small effect size can be statistically significant because the enormous sample size provides high statistical power. The more important question is practical significance: does +0.24% matter? The authors argue yes, both because of the absolute volume it represents at App Store scale and because "it is considered a significant improvement for a mature industrial ranker" where large gains are rare. The tail-query analysis strengthens this argument: the +0.24% is an average across all queries, masking much larger improvements in the tail. If tail queries represent 20-30% of total search volume (typical for heavy-tailed query distributions), a +2% improvement in tail query conversion rate combined with 0% change in head queries could produce an overall +0.24% average. The tail-specific gains are therefore the mechanism driving the overall improvement.

---

Summary of Key Design Choices and Their Justifications

• Offline label generation over real-time LLM reranking: avoids latency constraints, amortizes LLM inference cost across all uses of the labels, and decouples label quality from serving speed.
• Fine-tuning over pretrained scaling: the domain gap between general language understanding and the App Store's specific relevance rubric is too large for pretraining scale alone to bridge (FT-3B F1 of 0.800 vs. PT-30B F1 of 0.382, despite the 30B model having 10× more parameters).
• Pointwise labels over pairwise/listwise: compatible with any ranker architecture, simpler to generate (one inference per query-app pair, no need to maintain list context), and directly analogous to existing human judgment format.
• Data mixing scalarization over explicit loss weighting: achieves the same multi-objective optimization effect without modifying the ranker's loss function, making it architecturally agnostic and easy to tune by adjusting data proportions.
• Strict separation of label sources during training: ensures independent gradient computation for each objective, making the mixing ratio an interpretable, tunable hyperparameter that controls position on the Pareto frontier.
• Independent evaluation sets for LLM validation and ranker evaluation: prevents circular evaluation where the same human judgments used to validate the LLM would also measure ranker performance, ensuring that ranker improvements represent genuine generalization.
• Conversion rate as the primary online metric: directly measures the system's core purpose (helping users download apps), is robust to position bias, and reflects actual user behavior rather than intermediate proxies.
• Query frequency bucket analysis for understanding mechanism: decomposes the overall conversion rate lift to reveal that gains concentrate in the tail, confirming the hypothesis that LLM-generated labels matter most where behavioral signals are weakest.
• Few-shot prompting with textual labels: examples calibrate the LLM to the human judge's label distribution and rubric application; textual labels are more in-distribution for the LLM's pretraining than numeric labels.

4. Key Insights and Innovations

Innovation 1: Reframing LLM-as-a-Judge from Evaluation Tool to Training Data Force Multiplier

The dominant paradigm in prior work on LLM-based relevance assessment has been evaluation-oriented: use LLMs to score or rank system outputs for benchmarking purposes. G-Eval [13] uses GPT-4 to evaluate NLG quality, ARES [22] automates RAG system evaluation, and numerous other works [2, 5, 7, 8, 10, 30] treat LLM judgments as a substitute for human evaluation metrics. The unstated assumption across this literature is that LLM judgments answer the question "how good is our system?"

This paper makes a fundamental conceptual pivot: LLM judgments should answer "how can we make our system better?" — and the mechanism is training data augmentation, not evaluation. The distinction matters because it completely changes the economics. Evaluation labels are consumed once (to produce a metric number) and discarded. Training labels are invested — they become part of the model's parameterization and generate returns indefinitely through improved ranking quality. This is what the paper means by "force multiplier": a single human judgment, when used to fine-tune an LLM that then generates millions of training labels, produces a compounding return that a pure-evaluation use case would never achieve.

This reframing is not merely a different application of the same technique. It imposes different requirements on label quality, different tolerance for error, and different scaling dynamics. For evaluation, a noisy label directly corrupts the metric; systematic biases in the LLM's judgment can lead to misleading conclusions about system performance. For training data augmentation, label noise is absorbed into the overall training signal and can be partially mitigated by the presence of other objectives (in this case, behavioral labels). The paper's results implicitly validate this framing: an F1 of 0.800 against human judgments (Table 1) indicates 20% disagreement, yet the augmented ranker improves on both textual and behavioral NDCG (Table 2). If these labels were used for evaluation, 20% noise would be unacceptable; as training data, it is sufficient to shift the Pareto frontier.

This conceptual move connects to a broader trend in machine learning — using learned models as data generators (e.g., data augmentation in computer vision, synthetic data in NLP) — but applies it to a domain (search relevance) and a mechanism (multi-objective LTR via scalarization) where the approach had not been validated at industrial scale. The paper's contribution is demonstrating that the LLM-as-a-Judge paradigm is viable not just for benchmarking but for production model improvement, and that the path from LLM judgment to business metric impact goes through training data volume rather than evaluation fidelity.

Evidence anchor: Table 2 (Pareto improvement in both behavioral and textual NDCG) combined with Table 3 (+0.24% conversion rate) closes the loop from label quality → offline metric → online business outcome.

---

Innovation 2: Demonstrating That Domain-Specific Fine-Tuning Dominates Pretraining Scale for Judgment Alignment

The paper's side-by-side comparison of PT-3B, PT-30B, and FT-3B (Table 1) produces a result that is individually striking and collectively challenges a common assumption about LLM capabilities. The raw numbers — FT-3B F1 of 0.800 versus PT-30B F1 of 0.382 — represent a 109% improvement from a model one-tenth the size, achieved solely through fine-tuning on in-domain human judgments.

This is not merely "fine-tuning helps," which would be trivial. The finding is more specific and counterintuitive: pretraining scale does not substitute for domain alignment when the task requires internalizing a specific, institution-specific evaluation rubric. The pretrained 30B model, despite having an order of magnitude more parameters and presumably stronger general reasoning capabilities, cannot infer the App Store's relevance taxonomy, label boundaries, or edge-case handling patterns from its pretraining corpus. These are not general semantic knowledge; they are organizational conventions — the specific way a particular team of human judges has been trained to apply a five-point relevance scale to query-app pairs with specific metadata fields.

This finding challenges the implicit assumption behind much LLM-as-a-Judge work that larger pretrained models will naturally produce better judgments by virtue of their superior "understanding." The paper shows that for judgment tasks where the evaluation criteria are institution-specific rather than universal, fine-tuning on even a modest number of in-domain examples is more effective than scaling model size by an order of magnitude. This has significant industrial implications: it suggests organizations with proprietary relevance rubrics should invest in fine-tuning smaller, cheaper models rather than paying the inference cost of large pretrained models, and that the human judgments they already possess are more valuable as fine-tuning data than as evaluation benchmarks.

The paper does not develop this into a full theory of when pretraining scale versus domain fine-tuning dominates, but the empirical result is strong enough to serve as a diagnostic reference point for future work. It also implicitly explains a tension in the LLM-as-a-Judge literature: some papers report strong alignment with human judgments (typically when the LLM is evaluated on tasks close to its pretraining distribution), while others report poor alignment (when the task requires domain-specific conventions). The missing variable may be the degree to which the judgment rubric is inferable from general language understanding versus requiring institution-specific calibration.

Evidence anchor: Table 1, specifically the F1 column (PT-3B: 0.287, PT-30B: 0.382, FT-3B: 0.800). The gap between PT-30B and FT-3B is the core evidence.

---

Innovation 3: The "Data Mixing as Pareto Frontier Control" Insight for Multi-Objective LTR

Multi-objective learning to rank is an established area [9, 16, 17, 24, 26, 27], and scalarization — combining objectives into a single weighted loss — is the most common technique. The standard formulation is explicit loss weighting:

$L_{\text{total}} = \alpha L_{\text{behavioral}} + (1 - \alpha) L_{\text{textual}}$

where α is tuned as a hyperparameter. This requires modifying the training objective, which may be architecturally invasive in a production system with a complex, heavily optimized loss function.

The paper's innovation is not scalarization itself but a specific implementation insight: the same effect can be achieved purely through data composition, without touching the loss function. By mixing behavioral examples and textual examples in the training data at a tunable ratio, and maintaining "strict separation" so that each example contributes gradients to exactly one objective, the data mixing ratio becomes a direct proxy for the loss weight. The ranker's loss function remains unchanged; only the data sampling distribution varies.

This seems like a minor implementation detail, but in industrial contexts it is a significant architectural simplification. Production rankers often have loss functions that incorporate multiple terms (regularization, auxiliary tasks, calibration penalties) that have been carefully tuned and validated. Modifying the loss function to accommodate a new objective weight requires re-validation of the entire training pipeline. Changing the data composition requires no such re-validation — the same training script runs on a differently sampled dataset.

The paper also demonstrates a nuanced dynamic that is not obvious from the scalarization literature: adding LLM-generated labels shifts the entire Pareto frontier outward, while varying the mixing ratio moves along the frontier. The paper reports (Section 4.3.2) that "by varying the proportion of LLM-generated labels, we could then move along this new, superior frontier, rather than shifting the frontier further outward." This implies that the frontier shift and the position-on-frontier are controlled by different variables — the presence of LLM labels shifts the frontier, while the proportion of LLM labels selects the operating point. This is a diagnostic finding rather than a theoretical contribution: it tells practitioners that adding LLM labels is not merely a way to upweight the textual objective (which would be a tradeoff along the existing frontier) but a way to achieve strictly better performance on both objectives simultaneously.

This framing provides a principled answer to the question "how many LLM labels should we generate?" — enough to shift the frontier outward. Beyond that threshold, additional labels change the operating point but not the frontier itself, and the operating point should be chosen based on business priorities rather than label volume.

Evidence anchor: Table 2 (simultaneous improvement on all six NDCG numbers) demonstrates the Pareto frontier shift. The paper's description of varying proportions to move "along this new, superior frontier" (Section 4.3.2) supports the distinction between frontier shift and operating point selection.

---

Innovation 4: The Tail-Query Diagnosis — Marginal Value of Labels Is Inversely Proportional to Behavioral Data Abundance

Prior work on search relevance has long recognized that tail queries are challenging due to data sparsity, and that textual relevance signals can help [6, 12]. However, the typical framing is qualitative: "textual relevance helps on tail queries because behavioral data is sparse." The paper's contribution is a quantitative diagnosis that reveals the shape of this relationship: the conversion rate improvement from LLM-generated labels is a decreasing function of query frequency, with the largest gains concentrated at the lowest frequencies (Figure 2).

This finding is more than confirmation of the expected pattern. It has a specific economic implication: the return on investment for generating an LLM label depends on the query's frequency. Labels for head queries (where behavioral data is already perfect) have near-zero marginal value; labels for tail queries (where behavioral data is absent) have high marginal value. This suggests that the label generation budget — which LLM inferences to run on which query-app pairs — should not be allocated uniformly but should be concentrated on tail queries.

The paper does not implement this optimization (it generates labels for millions of query-app pairs broadly), but the Figure 2 result provides the diagnostic foundation for it. A future system could prioritize LLM label generation for queries in low-frequency buckets, achieving similar ranker improvements with fewer total LLM inferences. This is an instance of an important principle: when labels serve as training data rather than evaluation, their value depends on information overlap with existing signals. A label that provides no new information beyond what the behavioral signal already captures is worthless as training data; a label that provides information entirely orthogonal to behavioral data is maximally valuable. Tail queries instantiate the latter case.

The paper does not formalize this as an information-theoretic principle, but the empirical pattern in Figure 2 points toward it. The practical guidance is clear: if you are going to invest in LLM-generated relevance labels, focus on the queries where you lack behavioral signal. The paper's worldwide A/B test, with gains observed in 89% of storefronts and concentrated in tail buckets, validates that this is not just a theoretical insight but a deployed reality.

Evidence anchor: Figure 2 (conversion rate difference vs. log query frequency, showing monotonic decrease from tail to head buckets) and the paper's explicit interpretation (Section 5): "the llm-augmented model excels here because the newly added textual relevance labels provide a robust and accurate signal where the behavioral signal is sparse or absent."

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation uses two primary datasets (Section 4.1): (1) a large-scale dataset of millions of query-app pairs from historical App Store session logs aggregated over a large time window, which provides behavioral relevance labels and serves as the pool for LLM label generation, and (2) a much smaller dataset of historical textual relevance judgments from human judges, which is split into training and validation sets for fine-tuning and evaluating the LLM judge, and further split for ranker evaluation to ensure independence (the ranker's textual NDCG validation set is distinct from the LLM fine-tuning and validation sets).

• Base model(s). Three in-house language model configurations are compared (Section 4.2.1): (1) Pretrained 3B (PT-3B), a 3-billion parameter pretrained model; (2) Pretrained 30B (PT-30B), a 30-billion parameter pretrained model with 10× the parameters; and (3) Finetuned 3B (FT-3B), the 3B model fine-tuned on the training set of human judgments. The production ranker's architecture is not disclosed, but it operates within a multi-objective learning-to-rank framework trained on behavioral and textual relevance labels.

• Metrics. For LLM label quality evaluation, the paper reports precision, recall, and F1 scores against held-out human judgments (Section 4.2.1, Table 1). For offline ranker evaluation, Normalized Discounted Cumulative Gain at rank k (NDCG@k) is computed separately for textual and behavioral relevance (Section 4.3.1, Table 2), where textual relevance scores are derived from human judge labels and behavioral relevance scores are derived from aggregated user clicks and downloads. For online A/B testing, the primary metric is conversion rate, defined as the proportion of search sessions with at least one app download (Section 5, Table 3), with analysis broken down by query frequency buckets (Figure 2).

• Baselines. For LLM label quality: PT-3B serves as the same-size pretrained baseline, PT-30B serves as the scale baseline (10× larger pretrained). For ranker evaluation: the production model (prod), trained on behavioral labels from aggregated search logs plus the limited set of human textual labels, serves as the primary baseline against the LLM-augmented model (llm-augmented). No explicit comparison is made against prior published methods (e.g., Liu et al. [12] or RRADistill [6]), as the goal is industrial validation against the current production system rather than academic benchmarking.

• Generation budget / compute accounting. The LLM label generation is performed as an offline batch inference job with no latency constraints (Section 3.1). The paper does not report the exact inference cost (FLOPs, GPU-hours, or tokens generated) for producing the millions of labels. The computational tradeoff is framed qualitatively: fine-tuning a smaller model (3B) is more cost-effective than running inference with a larger pretrained model (30B), offering "a clear path to production with lower computational and operational costs" (Section 4.2.2). The ranker training cost is not reported, and no direct comparison is made between the cost of generating LLM labels versus acquiring additional human labels.

• Cross-validation / statistical protocol. For LLM label quality evaluation, a simple train-validation split of the human judgment dataset is used (Section 4.1). The paper explicitly notes that the validation set for LLM evaluation is "a held-out validation set of judgments from human judges" and that the "judgments used for computing the NDCG@k for the textual relevance are not included in the ones we used for finetuning and validation of the LLM relevance labels, as they are two distinct sets" (Section 4.3.1). For the A/B test, statistical significance is reported for the +0.24% conversion rate improvement (Table 3), but the specific statistical test, p-value threshold, confidence intervals, and power analysis are not disclosed. The test ran on worldwide traffic with results observed in 89% of storefronts, implying enormous sample sizes that would make even small effects statistically detectable.

Main Quantitative Results

LLM Label Quality Comparison

The headline result (Table 1) is that the fine-tuned 3B model (FT-3B) achieves an F1 of 0.800 against held-out human judgments, dramatically outperforming both the pretrained 3B (F1 = 0.287) and the pretrained 30B (F1 = 0.382).

Side-by-side comparison:

The FT-3B achieves a 109% improvement over PT-30B on F1 (0.800 vs. 0.382) despite having one-tenth the parameters. The precision and recall for FT-3B are nearly identical (0.802 and 0.798), indicating no systematic bias toward over-generous or over-conservative labeling relative to human judges. In contrast, the pretrained models show lower precision than recall (PT-3B: 0.300 vs. 0.309; PT-30B: 0.424 vs. 0.402), with PT-30B showing slightly higher precision, suggesting different calibration patterns in pretrained models when applied to this domain-specific rubric.

The paper notes that few-shot prompting with textual labels performed best among the prompt configurations tested, but "for brevity, we omit detailed results on prompt design" (Section 4.2.2). This means we cannot assess how much of the performance gain comes from fine-tuning versus prompt engineering, nor how sensitive the results are to specific prompt choices (number of few-shot examples, rubric wording, metadata formatting).

Offline Ranker Performance: Pareto Frontier Shift

The headline result (Table 2) is that augmenting the production ranker's training data with millions of LLM-generated labels yields strictly higher NDCG on both textual and behavioral relevance at all measured rank cutoffs, representing a Pareto improvement where no metric degrades.

Side-by-side comparison (Table 2):

Several observations:

• The absolute improvements are small but consistent across all six numbers. No metric degrades, which is the defining characteristic of a Pareto improvement.
• Behavioral NDCG improvements are larger in absolute terms than textual NDCG improvements (+0.004–0.006 vs. +0.001–0.002). This is notable because the augmented labels are explicitly textual — the fact that behavioral relevance improves more than textual relevance suggests that better textual understanding helps the ranker learn feature representations that generalize to behavioral prediction, rather than simply upweighting the textual objective at the expense of behavioral performance.
• The improvements persist across all rank cutoffs (@1, @3, @7), indicating that the benefit is not limited to the very top result but propagates through the ranking.
• The paper states (Section 4.3.2) that "by varying the proportion of LLM-generated labels, we could then move along this new, superior frontier, rather than shifting the frontier further outward." Specific numbers for different mixing ratios are not reported, so we cannot assess how sensitive the Pareto frontier position is to the exact proportion of LLM-labeled data.

Online A/B Test: Conversion Rate and Tail Queries

The headline result (Table 3) is a statistically significant +0.24% increase in conversion rate for the LLM-augmented model compared to the production model in a worldwide A/B test.

The paper contextualizes this number: "While this number may appear small, it is considered a significant improvement for a mature industrial ranker" (Section 5). The gain was observed in 89% of storefronts, indicating broad generalization across languages and markets. No absolute conversion rates are reported, and no confidence intervals are provided.

The query frequency analysis (Figure 2) reveals that the conversion rate improvement is not uniform across query popularity:

• The x-axis shows log query frequency, with lower-numbered buckets representing tail (low-frequency) queries and higher-numbered buckets representing head (high-frequency) queries.
• The y-axis shows the absolute conversion rate difference between the LLM-augmented and production models.
• The curve shows a clear downward trend: tail-query buckets show positive differences (the LLM-augmented model has higher conversion rates), while head-query buckets show near-zero or slightly negative differences.
• The paper states that "the most substantial improvements occur in the tail" (Section 5), where behavioral relevance signals are sparse or absent and the LLM-generated textual labels provide the only reliable relevance signal.

The exact number of frequency buckets, the absolute conversion rates per bucket, and the magnitude of the tail-specific improvement are not reported quantitatively in the paper text or Figure 2 (the figure shows a relative curve without labeled y-axis values). This limits the ability to assess effect sizes separately for head and tail queries.

Ablation Studies and Robustness Checks

Model scale vs. fine-tuning: The comparison of PT-3B, PT-30B, and FT-3B (Table 1) serves as an implicit ablation showing that fine-tuning on in-domain human judgments (FT-3B F1 = 0.800) dominates model scale (PT-30B F1 = 0.382). However, the natural complement — fine-tuning the 30B model — is explicitly acknowledged as not performed: "While fine-tuning the 30B model remains a compelling direction for future work, our findings confirmed that the finetuned 3B model provided a sufficiently strong and cost-effective solution" (Section 4.2.2). This leaves open the question of whether FT-30B would further improve label quality beyond FT-3B, or whether the 3B model already captures all available information in the human judgment training set.

Prompt configuration (zero-shot vs. few-shot, textual vs. numeric labels): The paper states that "few-shot prompts with textual labels performed best" (Section 4.2.2) but provides no ablation table, no quantitative comparison of different prompt configurations, and no details on the number of few-shot examples used or how they were selected. This is a significant omission for replicability. The performance difference between zero-shot and few-shot could indicate whether the LLM struggles with the rubric itself or with calibration to the label distribution; the difference between textual and numeric labels could indicate whether the LLM leverages its pretraining knowledge of semantic label descriptors. Without these numbers, practitioners cannot determine whether the prompt engineering or the fine-tuning is the dominant factor in FT-3B's performance.

Data mixing ratio variation: The paper reports (Section 4.3.2) that varying the proportion of LLM-generated labels allows moving "along this new, superior frontier" but does not provide a sweep of mixing ratios with corresponding NDCG numbers. This is a missed opportunity to characterize the Pareto frontier quantitatively — to show, for example, that a 70-30 mix achieves one NDCG tradeoff while a 50-50 mix achieves another. Without these numbers, readers cannot determine how sensitive the ranker is to this hyperparameter or what mixing ratio was used for the reported llm-augmented model in Table 2 and the A/B test.

Storefront-level consistency: The A/B test gain was observed in 89% of storefronts (Section 5). The paper does not analyze the 11% of storefronts that did not see improvement — whether these represent specific languages, markets with particular query distributions, or simply statistical noise due to lower traffic volumes. This analysis could reveal boundary conditions on the approach's effectiveness (e.g., whether it fails for languages where the LLM's fine-tuning data was sparse).

Label quality threshold for ranker improvement: The paper does not ablate the relationship between LLM label quality (as measured by F1 against human judgments) and downstream ranker improvement. It is possible that a model with F1 of 0.600 would achieve similar ranker gains, or that an F1 of 0.800 is necessary. Without this ablation, the paper cannot claim that the specific FT-3B quality level is required — only that it is sufficient.

Negative result (implicit): The PT-30B model, despite having 10× more parameters than FT-3B, underperforms dramatically on label quality (F1 0.382 vs. 0.800). This is effectively a negative result for the hypothesis that larger pretrained models can serve as effective relevance judges without domain-specific fine-tuning. The paper does not frame it as such explicitly, but it is one of the strongest findings: scaling pretraining compute does not substitute for alignment with an institution-specific rubric.

Critical Assessment

Claim from the executive summary: "A specialized fine-tuned 3B-parameter model significantly outperforms a pretrained 30B-parameter model (F1 of 0.800 vs. 0.382)"

This claim is directly and strongly supported by Table 1. The experimental design cleanly compares the three configurations on the same held-out validation set. However, the experiment demonstrates a narrower point than the claim might suggest: it shows that this specific 3B model fine-tuned on this specific human judgment dataset outperforms this specific pretrained 30B model. The 30B model is not fine-tuned — and the paper acknowledges this gap by calling FT-30B "a compelling direction for future work." The claim therefore establishes that fine-tuning matters more than scale for this task, but does not establish the ceiling on what fine-tuning a larger model could achieve. If the 30B model were also fine-tuned and underperformed FT-3B, the claim would be stronger; as presented, the comparison is fine-tuning vs. no fine-tuning, not small vs. large.

Additionally, the paper does not report the inter-annotator agreement among human judges (e.g., Cohen's kappa or Fleiss' kappa). An F1 of 0.800 may be approaching or even exceeding human-level agreement, in which case further model improvements (through FT-30B or otherwise) would be impossible. Alternatively, if human agreement is 0.95, there is substantial room for improvement that a larger fine-tuned model might capture. Without this reference point, the F1 of 0.800 is difficult to interpret in absolute terms.

Claim from the executive summary: "Augmenting the production ranker's training data with millions of these LLM-generated labels yields a Pareto frontier shift — simultaneously improving offline NDCG for both textual and behavioral relevance"

This claim is supported by Table 2, which shows strictly non-negative improvements on all six NDCG numbers. The Pareto improvement interpretation is valid: no metric degrades, and at least one improves. However, several qualifications are necessary:

• The absolute improvements are very small (+0.001 to +0.006 across NDCG numbers that range from 0.403 to 0.867). The paper does not report whether these improvements are statistically significant in the offline setting (no confidence intervals, no bootstrap estimates, no cross-validation variance across folds). At this magnitude, the improvement could be within the noise floor of NDCG estimation, especially if the validation set is small.
• The paper does not report the size of the validation set used for ranker evaluation. If it is small (e.g., a few thousand queries), the observed improvements may not generalize. The independence of the ranker evaluation set from the LLM fine-tuning set is correctly established, but the absolute size of the evaluation set is unknown.
• The mechanism of behavioral NDCG improvement is not experimentally validated. The paper's interpretation — that better textual understanding helps behavioral prediction — is plausible but untested. An alternative explanation is that the LLM-augmented training data changes the optimization dynamics (e.g., providing more training examples overall, or changing the effective learning rate by altering batch composition) in ways unrelated to textual relevance. An ablation where the same number of random labels (or behavioral-only labels) is added would distinguish these explanations, but is not performed.
• The claim of a "Pareto frontier shift" would be stronger if the paper showed multiple points on both the original and new frontiers (e.g., by sweeping the data mixing ratio for both the prod and llm-augmented models and plotting textual vs. behavioral NDCG). The current evidence shows one point on each frontier, which is consistent with a Pareto improvement but does not characterize the shape or extent of the shift.

Claim from the executive summary: "Translates to a statistically significant +0.24% increase in conversion rate in a worldwide A/B test"

This claim is supported by Table 3 and the statement of statistical significance. The worldwide deployment and observation in 89% of storefronts add credibility. However, important details are missing:

• The specific statistical test, p-value threshold, and confidence interval are not reported. Without these, readers cannot assess whether the +0.24% is precisely estimated or has a wide confidence interval (e.g., +0.24% ± 0.20%).
• The baseline conversion rate is not reported. A +0.24% relative improvement means very different things if the baseline is 5% (absolute +0.012 percentage points) versus 50% (absolute +0.12 percentage points).
• No guardrail metrics are reported. A legitimate concern with any ranking change is whether improvements in conversion rate come at the expense of other objectives (e.g., user satisfaction, diversity of results, developer fairness, latency). The paper does not address whether any such metrics degraded.
• The A/B test compares exactly one model (llm-augmented with a specific mixing ratio) against the production baseline. There is no online sweep of the data mixing ratio to validate that the offline Pareto improvement translates to online gains across the frontier. The reported +0.24% is therefore a single point estimate that may not generalize to different mixing ratios.

Claim from the executive summary: "The online gains concentrate disproportionately on tail queries"

This claim is supported by Figure 2, which shows higher conversion rate differences in lower-frequency buckets. The qualitative pattern is clear, but the quantitative support is limited:

• The y-axis values in Figure 2 are not numerically labeled, preventing readers from assessing the magnitude of the tail-specific improvements.
• The paper does not report the number of frequency buckets, the query volume per bucket, or the absolute conversion rates per bucket for each model. Without these, the claim "concentrate disproportionately" is a visual observation rather than a quantified finding.
• The paper does not report whether the tail-query improvements are individually statistically significant. It is possible that the overall +0.24% is significant but the per-bucket differences are not, especially if tail queries have small sample sizes (which they do, by definition).

What experiments would have strengthened the paper:

1. FT-30B evaluation: Fine-tuning the 30B model on the same human judgment dataset would reveal whether label quality saturates at FT-3B's F1 of 0.800 or whether larger fine-tuned models can extract more signal from the training data. This is the most obvious missing experiment, acknowledged by the authors themselves.

2. Mixing ratio sweep with full Pareto frontier visualization: Reporting textual and behavioral NDCG for both prod and llm-augmented models at multiple mixing ratios (e.g., 90-10, 70-30, 50-50, 30-70, 10-90) would transform the single-point Pareto improvement claim into a frontier-level characterization, showing how far outward the frontier shifts and whether the shift is uniform across the frontier or concentrated in certain regions.

3. Human inter-annotator agreement baseline: Reporting the agreement rate among human judges on the same task would contextualize the LLM's F1 of 0.800. If human agreement is, say, 0.85, then the LLM is approaching human-level performance. If it is 0.95, substantial room for improvement remains.

4. Label quality vs. ranker performance ablation: Training the ranker with labels from PT-3B (F1 = 0.287), PT-30B (F1 = 0.382), and FT-3B (F1 = 0.800) separately would reveal the relationship between label quality and downstream ranker improvement. Does the ranker need F1 > 0.8, or would F1 > 0.4 suffice? This has direct practical implications for whether organizations need to invest in fine-tuning or can use off-the-shelf pretrained models.

5. Label volume ablation: The paper generates "millions" of labels but does not ablate the number of labels added. Would 100K labels achieve similar gains? Would 10M labels shift the frontier further? Without this, the "force multiplier" claim lacks a quantified multiplication factor.

6. Real-time LLM reranker baseline: Comparing the offline label generation approach against a real-time LLM reranker (as in [1, 4, 14]) would strengthen the paper's architectural argument that offline generation is preferable. The comparison would need to account for both ranking quality and computational cost/latency.

7. Generalization to other ranker architectures: The paper claims architectural agnosticism, but all experiments use the same (undisclosed) production ranker. Testing with a different ranking architecture (e.g., a gradient-boosted tree ranker vs. a neural ranker) would validate that the approach generalizes beyond the specific production system.

Genuine weaknesses in the experimental design:

• Opaque baselines and undisclosed details: The production ranker architecture, the behavioral label derivation method, the exact human judgment rubric, the number of human judgments, the ranker validation set size, and the specific prompt engineering choices are all undisclosed. This is understandable for an industrial paper (proprietary systems), but it fundamentally limits the scientific reproducibility and the ability to assess whether the findings are specific to the App Store's particular setup or generalizable.

• Small absolute NDCG improvements without variance estimates: The offline improvements in Table 2 are +0.001 to +0.006 on metrics measured to three decimal places. Without standard errors, confidence intervals, or cross-validation variance, readers cannot distinguish a genuine improvement from sampling noise. This is a critical omission given the claim of a "Pareto frontier shift."

• Single-point A/B test without sensitivity analysis: The online validation tests exactly one model configuration. There is no evidence that the chosen mixing ratio is optimal, that the result is robust to different traffic splits or time periods, or that alternative configurations (e.g., using PT-30B labels instead of FT-3B labels) would produce different outcomes.

• No cost analysis: The paper's central value proposition is industrial scalability — the LLM as a "force multiplier" that is cheaper than human annotation. Yet no cost comparison is provided: what did the millions of LLM labels cost to generate (in GPU-hours, dollars, or engineer-time) versus what the equivalent number of human labels would have cost? Without this, the economic argument is asserted rather than demonstrated.

6. Limitations and Trade-offs

1. Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Gains

The assumption or constraint. The paper frames the fine-tuned 3B LLM as a "force multiplier" that transforms a small set of human judgments into millions of labels at near-zero marginal cost. However, the cost of producing the FT-3B model — specifically, the human judgments required for fine-tuning — is treated as a sunk cost that already exists in the production pipeline. The paper states in Section 4.2.1 that the FT-3B is "fine-tuned on the training set of human judgments dataset described in Section 4.1," and this dataset is "a much smaller dataset of historical textual relevance judgments on query-app pairs from our human judges" (Section 4.1). The paper never reports the size of this dataset, the cost of acquiring it, or how many human judgments are needed to achieve the reported F1 of 0.800.

The consequence. A practitioner considering this approach faces a chicken-and-egg problem: the FT-3B model requires a human judgment dataset for fine-tuning, but the whole motivation for the approach is that human judgments are scarce and expensive. If an organization does not already possess a sufficiently large and high-quality human judgment dataset, the upfront cost of creating one may exceed the cost of simply purchasing more human labels directly. The paper provides no guidance on the minimum number of human judgments needed, how label quality (F1) scales with fine-tuning dataset size, or whether the approach is viable for teams with, say, only 100 labeled examples versus 10,000.

What evidence exists in the paper. None. The paper does not ablate the size of the fine-tuning dataset against FT-3B's label quality (F1) or downstream ranker performance. The only reported numbers are the F1 scores in Table 1 for the three model configurations, with no sweep of fine-tuning data volume. The paper also does not compare the cost of the human judgments used for fine-tuning against the cost of the LLM-generated labels, making the "force multiplier" framing qualitative rather than quantitative — we know the LLM multiplies labels, but not at what cost ratio.

Mitigation status. Not addressed. The paper treats the existing human judgment dataset as a given and does not discuss the bootstrapping problem. The authors do not suggest whether transfer learning from other relevance judgment datasets (e.g., public IR benchmarks) could reduce the need for in-domain human labels, nor do they explore whether the pretrained 30B model could serve as a teacher for the 3B model via distillation, bypassing the need for human fine-tuning data entirely.

---

2. No Evidence That the Method Works Beyond the App Store Domain or Model Family

The assumption or constraint. All experiments are conducted on a single domain (App Store search relevance), using a single model family (an undisclosed in-house LLM with 3B and 30B variants), and a single production ranker architecture (also undisclosed). The paper claims in Section 6 that it provides "a practical blueprint for other large-scale search systems to overcome relevance-label scarcity," but the blueprint is validated on exactly one system. The paper does not test on public benchmarks (e.g., MS MARCO, TREC), on different search domains (e.g., web search, product search, enterprise document retrieval), or with different LLM families (e.g., LLaMA, Mistral, Gemma).

The consequence. Three distinct generalization failures are possible, and the paper provides evidence against none of them:

• Domain generalization: The App Store relevance rubric — built around app metadata (title, description, keywords) and a five-point relevance scale — may not transfer to domains where relevance is defined differently. In web search, relevance depends on page content, authority, and freshness; in product search, it depends on product attributes, price, and availability. A "blueprint" that works only for app-store-style metadata-and-query matching is not a general blueprint.

• Model family generalization: The in-house model's pretraining data, architecture, and fine-tuning behavior may differ from publicly available LLMs. The finding that FT-3B >> PT-30B (Table 1) could be specific to this model family's pretraining distribution — a different family might have stronger zero-shot relevance assessment capabilities (narrowing the gap) or weaker fine-tuning responsiveness (widening it).

• Ranker architecture generalization: The paper emphasizes that the pointwise label approach "utilizes pointwise labels to train the production ranker without modifying its internal architecture" (Section 2, "LLMs as Judges"). This architectural agnosticism is asserted, not tested. A gradient-boosted tree ranker, a pairwise neural ranker, or a two-tower embedding model might respond differently to augmented textual labels than the paper's (undisclosed) production ranker.

What evidence exists in the paper. The A/B test result that gains were "observed in 89% of storefronts" (Section 5) provides weak evidence of cross-language and cross-market generalization within the App Store domain, but it does not address cross-domain or cross-model-family generalization. The storefront coverage is a within-domain robustness check, not a cross-domain validation.

Mitigation status. Not addressed. The paper does not position this as a limitation, does not call for replication on other domains or with other model families, and does not qualify its "blueprint" claim with the caveat that all evidence comes from a single system.

---

3. The Absolute Offline NDCG Improvements Are Very Small and Not Accompanied by Variance Estimates

The constraint and what is reported. Table 2 reports NDCG improvements between the production and LLM-augmented models ranging from +0.001 to +0.006 across the six metric-model combinations. These are reported to three decimal places without standard errors, confidence intervals, bootstrap estimates, or any other measure of statistical uncertainty. The size of the validation set used for these NDCG calculations is not disclosed.

The consequence. At this magnitude, two concerns arise that the paper cannot address without variance estimates:

• Statistical noise vs. genuine improvement: NDCG is an average over queries, and its variance depends on the number of queries in the validation set and the variance of per-query NDCG. If the validation set contains, say, 5,000 queries, an improvement of +0.001 in NDCG@1 (from 0.867 to 0.868) could easily fall within a 95% confidence interval of the difference, meaning the result is not distinguishable from zero. The paper's claim of a "Pareto improvement" (Section 4.3.2) requires that no metric degrades — but if the confidence intervals include zero or negative values for some metrics, the Pareto improvement interpretation is not statistically supported.

• Practical significance of tiny NDCG deltas: Even if the improvements are statistically significant (which is possible with a very large validation set), NDCG differences of 0.001–0.006 may not correspond to user-perceptible ranking changes. The paper's online A/B test provides the translation (+0.24% conversion rate), but the offline-to-online correlation cannot be assessed because only one offline model configuration was tested online. If a future model variant showed an NDCG improvement of +0.0005, would it predict a conversion rate lift? We cannot know from these results.

What evidence exists in the paper. Table 2 reports the point estimates. No statistical analysis is provided. The paper does not report the validation set size, the standard deviation of per-query NDCG, or any test for whether the differences are statistically significant. The online A/B test is reported as "statistically significant" (Table 3) without specifics, but this applies to the conversion rate metric, not to the offline NDCG numbers. A reader cannot determine whether the offline improvements would replicate on a different validation split.

Mitigation status. Not addressed. The paper treats the consistent direction of improvement across all six numbers as sufficient evidence of a real effect, but without variance estimates, consistency could arise from correlated measurement error rather than genuine model improvement. The authors do not flag this as a limitation or suggest that future work should include uncertainty quantification.

---

4. The FT-3B Model Is Not Compared Against a Fine-Tuned 30B Variant, Leaving the Quality Ceiling Unknown

The assumption. The paper establishes that FT-3B (F1 = 0.800) outperforms PT-30B (F1 = 0.382) and concludes that "fine-tuning a smaller, more efficient model proved more effective than using its much larger, pretrained counterpart, offering a clear path to production with lower computational and operational costs" (Section 4.2.2). This conclusion conflates two separate variables: model scale (3B vs. 30B) and training regimen (fine-tuned vs. pretrained-only). The comparison demonstrates that fine-tuning matters more than scale for this task, but it does not demonstrate that a smaller fine-tuned model is inherently preferable to a larger fine-tuned model.

The paper acknowledges this: "While fine-tuning the 30B model remains a compelling direction for future work, our findings confirmed that the finetuned 3B model provided a sufficiently strong and cost-effective solution" (Section 4.2.2).

The consequence. The practical decision facing a team deploying this approach is not "FT-3B vs. PT-30B" (the paper's comparison) but "FT-3B vs. FT-30B vs. FT-7B vs. some other configuration." Without an FT-30B result, we cannot answer:

• Does label quality saturate at F1 = 0.800? If human inter-annotator agreement is, say, 0.85, then FT-3B may already be near the performance ceiling, and further scale would yield negligible gains. But if human agreement is 0.95, FT-30B might achieve F1 of 0.90+, which could translate to substantially larger ranker improvements.
• Does better label quality produce measurably better ranker performance? The paper does not ablate label quality against downstream NDCG or conversion rate. If a model with F1 = 0.600 produces similar ranker gains to one with F1 = 0.800, then the FT-3B's quality advantage is overkill. If ranker performance is highly sensitive to label quality in the 0.7–0.9 range, then FT-30B might be worth the additional inference cost.
• What is the cost-quality Pareto frontier for fine-tuned model scale? FT-3B is cheaper per inference than FT-30B would be, but the paper is generating labels offline in batch — if FT-30B labels cost 10× more to generate but improve the ranker's conversion rate by +0.30% instead of +0.24%, the additional cost may be justified by the incremental revenue. Without the FT-30B result, this tradeoff cannot be evaluated.

What evidence exists in the paper. None beyond the FT-3B vs. PT-30B comparison in Table 1. The paper does not fine-tune any model other than the 3B, does not report the number of human judgments used for fine-tuning or the compute budget for fine-tuning, and does not analyze whether FT-3B is approaching ceiling performance or still improving with more fine-tuning data.

Mitigation status. The authors acknowledge the missing experiment as "a compelling direction for future work" (Section 4.2.2) and argue that FT-3B is "sufficiently strong and cost-effective." This is a fair engineering argument — the system works well enough to ship — but it leaves the scientific question of how label quality scales with fine-tuned model capacity unanswered.

---

5. No Analysis of Negative Side Effects, Guardrail Metrics, or Regressions

The constraint. The paper reports a single online metric: conversion rate (+0.24%, Section 5, Table 3). No other online metrics are reported — no engagement metrics beyond downloads (e.g., time spent in app, user retention, repeat search behavior), no diversity metrics (e.g., concentration of downloads among top apps, exposure for new or small developers), no fairness metrics (e.g., differential impact across user demographics or app categories), and no latency or infrastructure cost metrics.

The consequence. Improving conversion rate is the stated goal of the system, but search ranking involves inherent tradeoffs. The LLM-augmented model strengthens the textual relevance objective, which teaches the ranker to favor apps that are semantically similar to the query. This could have several unintended consequences that the paper does not investigate:

• Popularity bias reduction or amplification: Textual relevance should reduce reliance on behavioral popularity signals, which could increase exposure for niche, high-quality apps that lack download volume. This would be a positive side effect. Alternatively, if the LLM's relevance judgments inherit popularity biases from its training data, it could reinforce rather than counteract popularity concentration. The paper provides no evidence either way.
• Developer ecosystem effects: If the LLM-augmented model systematically promotes apps with well-written metadata over apps with sparse or poorly written metadata (because the LLM relies on metadata to assess relevance), this could disadvantage smaller developers who lack resources for App Store Optimization (ASO). This would be a fairness concern that the paper does not address.
• Query category heterogeneity: The tail-query analysis (Figure 2) shows that gains concentrate in low-frequency queries, but this aggregates across all query types. It is possible that certain categories of tail queries (e.g., navigational queries where users search for a specific app name) are unaffected or negatively impacted, while others (e.g., exploratory queries where users describe a need) benefit substantially. Without category-level analysis, practitioners cannot anticipate which query types will improve.
• Latency and infrastructure cost: The paper states that label generation is offline and does not affect serving latency (Section 3.1). However, the ranker trained on augmented data may have different computational characteristics at serving time (e.g., if the additional training data changes the model's feature utilization patterns). No serving cost comparison is provided.

What evidence exists in the paper. The +0.24% conversion rate lift (Table 3) and the tail-query frequency analysis (Figure 2). No guardrail metrics are reported. The paper does not discuss whether any metrics were monitored and found to be neutral, or whether the experimental design only tracked conversion rate. The observation that gains were "observed in 89% of storefronts" (Section 5) implies that 11% of storefronts did not see improvement, but the paper does not investigate whether these storefronts experienced regressions on other metrics or were simply flat.

Mitigation status. Not addressed. The paper's scope is explicitly focused on demonstrating that LLM-generated labels can improve conversion rate, and it achieves this goal. However, in a production search system, a ranking change that improves conversion rate but degrades developer fairness, user satisfaction diversity, or infrastructure efficiency would not be deployable. The absence of guardrail analysis limits the practical actionability of the results for teams that must consider these tradeoffs.

---

6. Prompt Engineering Details and Sensitivity Are Undisclosed, Making Replication Dependent on Unspecified Design Choices

The assumption. The paper reports that "few-shot prompts with textual labels performed best" (Section 4.2.2) and that the FT-3B model achieved an F1 of 0.800 using this configuration. However, the paper does not report:

• The number of few-shot examples used in the prompt (Figure 1 shows placeholders like "Example 1" and "Example 2," but the actual count is unspecified).
• How few-shot examples were selected (randomly from the training set? stratified by relevance label? chosen to cover diverse query types?).
• The exact wording of the relevance rubric descriptions ("Description of relevance levels is as follows: ..." in Figure 1 is a placeholder).
• The exact app metadata fields provided ("app_metadata_1, app_metadata_2, app_metadata_3" in Figure 1 are placeholders — are these title, description, keywords, category, rating, something else?).
• Ablation results comparing different prompt configurations quantitatively.

The paper acknowledges this omission: "For brevity, we omit detailed results on prompt design" (Section 4.2.2).

The consequence. Prompt engineering is known to significantly affect LLM performance on judgment tasks, sometimes rivaling the effect of model scale or fine-tuning. Without the specific prompt template used, a practitioner attempting to replicate the approach cannot determine:

• Whether FT-3B's performance is robust to prompt variation: If F1 drops from 0.800 to 0.600 when the rubric wording changes slightly, the system is brittle and requires careful prompt maintenance. If F1 is stable across reasonable prompt variations, the approach is more robust.
• Whether the "best" prompt configuration is cherry-picked: Without an ablation table showing performance across multiple configurations (zero-shot, few-shot with 1/2/5/10 examples, textual vs. numeric labels), the reader cannot assess whether the reported result reflects a genuine configuration advantage or post-hoc selection of the best-performing variant from many attempts.
• How to adapt the prompt to a new domain: The paper's "blueprint" claim (Section 6) implies that other teams can follow the same methodology. But without the specific prompt template — the role specification, rubric phrasing, and metadata formatting — a team in a different domain (e.g., web search, product search) would need to rediscover effective prompt design through trial and error, which is expensive and may not converge to the same quality level.

This limitation is particularly significant given the paper's industrial setting: proprietary data and models are expected to be undisclosed, but prompt templates and ablation methodology are typically shared even in industry papers because they are essential for scientific replication and do not reveal business-sensitive information.

What evidence exists in the paper. The qualitative statement that "few-shot prompts with textual labels performed best" (Section 4.2.2) with no supporting data. Table 1 reports the aggregate model comparison but does not disaggregate by prompt configuration.

Mitigation status. Not addressed. The paper offers no justification for omitting the prompt details beyond "brevity" and does not suggest that prompt templates will be released or that a companion technical report will provide the missing ablations. For a paper that positions itself as a "practical blueprint" (Section 6), the omission of the specific prompt design — the most transferable component of the methodology — is a significant barrier to adoption.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new ranking algorithm, a novel loss function, or a fundamentally different model architecture. Its contribution is subtler but, for the field of industrial search and recommendation, arguably more consequential: it provides the first end-to-end validation at global scale that LLM-generated judgments can serve as training data for a production multi-objective ranker, not merely as evaluation benchmarks or real-time rerankers. The shift is from "LLMs can evaluate relevance" (a claim well-established by prior work [2, 5, 7, 8, 10, 13, 22, 30]) to "LLMs can improve relevance when their judgments are invested as training data rather than consumed as metrics."

This is a conceptual reframing with practical consequences. In the evaluation paradigm, an LLM judgment answers a backward-looking question: "how well did our system perform on this query?" In the training paradigm, the same judgment answers a forward-looking question: "how can we make our system perform better on this query in the future?" The difference is not merely semantic — it changes the economics, the tolerance for label noise, and the scaling dynamics. An evaluation label is consumed once and discarded; a training label is embedded in model parameters and generates returns indefinitely through improved rankings. The paper's core metaphor — "force multiplier" — captures exactly this compounding dynamic: a single human judgment, by enabling the fine-tuning of an LLM that then generates millions of training labels, produces a return that vastly exceeds its acquisition cost.

This work also resolves a latent tension in the LLM-as-a-Judge literature that the paper itself does not explicitly call out but that becomes visible when its results are juxtaposed with prior findings. Multiple papers have reported strong alignment between LLM and human relevance judgments [2, 5, 10, 13], while others have found the opposite [7, 30] — LLMs disagree with humans in systematic ways that make them unreliable judges. The paper's Table 1 provides a diagnostic that reconciles these findings: pretrained models (even large ones) show only moderate alignment with domain-specific human judgments (PT-30B F1 = 0.382), while fine-tuned models achieve strong alignment (FT-3B F1 = 0.800). The missing variable in the prior literature is whether the LLM's evaluation rubric is inferable from general pretraining knowledge or requires institution-specific calibration. When the rubric is generic (e.g., "is this summary factually consistent with the source?"), pretrained models may perform well because the judgment criteria are represented in their training corpus. When the rubric is institution-specific (e.g., the App Store's particular five-point relevance taxonomy with its specific metadata interpretation conventions and edge-case handling), pretrained models cannot infer the criteria from general language understanding — they need fine-tuning on in-domain examples. This single diagnostic explains much of the variance in prior LLM-as-a-Judge results and should shift the field's methodology: future work should not ask "are LLMs good judges?" in general, but rather "for this specific judgment rubric, how much of the evaluation criteria is inferable from pretraining versus requiring domain-specific calibration?" The paper's experimental design — comparing PT-3B, PT-30B, and FT-3B on the same task — provides a template for answering this question.

The paper also changes the cost calculus for industrial search teams. Prior to this work, a team facing textual relevance label scarcity had three options: (1) hire more human judges (expensive, scales linearly), (2) deploy an LLM as a real-time reranker (introduces latency and per-query inference cost), or (3) accept that the textual relevance objective would remain underpowered. This paper establishes a fourth option: use existing human judgments to fine-tune a small LLM, generate labels offline at massive scale, and augment the existing multi-objective ranker training pipeline without architectural changes or latency penalties. The paper's demonstration that a 3B fine-tuned model produces higher-quality labels than a 10× larger pretrained model (F1 0.800 vs. 0.382, Table 1) makes this option economically viable — the inference cost of a 3B model in an offline batch setting is modest compared to the cost of human annotation or real-time 30B inference. This is not a paradigm shift in the Kuhnian sense; the underlying techniques (fine-tuning, scalarization, multi-objective LTR) are all established. But it is a validated industrial methodology that closes the gap between academic demonstrations and production reality, and its validation at the scale of the global App Store (worldwide A/B test, +0.24% conversion rate, 89% storefront coverage) sets a new bar that future work in this area will need to meet.

Finally, the paper's tail-query diagnosis (Figure 2) introduces a new efficiency principle for training data generation: the marginal value of a relevance label is inversely proportional to the amount of existing behavioral data for that query. Labels for head queries — where behavioral data is already overwhelming — are nearly worthless as training data; labels for tail queries — where behavioral data is sparse or absent — are extremely valuable. This principle has immediate implications for resource allocation: teams generating LLM labels should prioritize queries in low-frequency buckets, achieving larger ranker improvements per label generated. The paper does not implement this prioritization (it generates labels broadly across millions of query-app pairs), but Figure 2 provides the empirical foundation for it. This principle may also generalize beyond search ranking to any domain where synthetic training data supplements real user feedback — the synthetic data's value is highest where the real feedback is weakest.

One research direction that becomes less attractive after this paper is real-time LLM reranking for high-volume production search. The paper's approach achieves measurable conversion rate improvements without adding latency to user-facing requests, without multiplying inference costs by query volume, and without requiring the ranker architecture to accommodate an LLM in the serving path. This does not mean real-time LLM reranking is always inferior — for low-volume, high-stakes search tasks (e.g., legal document retrieval, medical literature search), the latency and cost of real-time LLM inference may be acceptable. But for high-volume consumer search (app stores, e-commerce, web search), the paper provides strong evidence that offline label generation is the more practical path to leveraging LLM capabilities, and real-time reranking faces an increasingly steep burden of proof to justify its additional operational complexity.

Follow-Up Research This Work Enables

Quantifying the fine-tuning data efficiency curve for LLM-as-a-Judge label quality. The paper establishes that fine-tuning the 3B model on human judgments produces an F1 of 0.800 (Table 1), but it provides no information about how F1 scales with the number of fine-tuning examples. This is the single most important missing piece for practitioners considering adoption: if an organization has 100 human-labeled examples, 1,000, or 10,000, what label quality (F1) can they expect? A strong follow-up study would fine-tune the FT-3B model (or a publicly available comparable model, e.g., LLaMA-3B) on systematically varied subsets of a human judgment dataset — 50, 100, 200, 500, 1,000, 2,000, 5,000, and 10,000 examples — and measure F1 on a fixed held-out set at each training size, producing a scaling curve. This curve would reveal whether the approach is viable for teams with small annotation budgets (does F1 plateau at 500 examples, or does it continue improving to 10,000?) and would provide the data needed for a cost-benefit analysis: given a budget of X dollars for human annotation, is it more effective to fine-tune on those X dollars' worth of labels and generate millions more, or to simply use the X dollars' worth of labels directly in ranker training? The paper's current results show what is achievable with an existing human judgment dataset of unknown size; the scaling curve would show what is achievable at any annotation budget.

Fine-tuning the 30B model and measuring the label quality ceiling. The paper explicitly acknowledges this gap: "While fine-tuning the 30B model remains a compelling direction for future work" (Section 4.2.2). The experiment is straightforward: fine-tune the 30B model on the same human judgment training set using the same procedure as FT-3B, evaluate F1 on the same held-out validation set, and compare against both FT-3B (F1 = 0.800) and human inter-annotator agreement (which the paper should also report). Three outcomes are possible, each with different implications: (1) FT-30B F1 is significantly higher than FT-3B (e.g., 0.90+), which would justify the additional inference cost for label generation in quality-sensitive applications; (2) FT-30B F1 is approximately equal to FT-3B, suggesting that the 3B model already captures all signal in the fine-tuning data and that additional capacity provides no benefit — this would be a strong practical finding that the cheaper model is sufficient; (3) FT-30B F1 is approximately equal to human inter-annotator agreement, establishing a ceiling on achievable label quality and showing that FT-3B is already near that ceiling. Without this experiment, the paper's claim that FT-3B is "sufficiently strong" is an assertion, not a demonstrated optimum.

Label quality vs. downstream ranker performance ablation. The paper shows that FT-3B labels (F1 = 0.800) improve the ranker (Table 2, +0.24% conversion rate), but it does not establish the relationship between label quality and ranker improvement. A critical follow-up would train three separate versions of the production ranker, each augmented with labels from a different LLM configuration: PT-3B labels (F1 = 0.287), PT-30B labels (F1 = 0.382), and FT-3B labels (F1 = 0.800). For each, measure offline textual and behavioral NDCG (as in Table 2) and, ideally, run an A/B test to measure conversion rate impact. This experiment would answer the question that every practitioner needs to answer: how good do the LLM labels need to be for the ranker to benefit? If PT-30B labels (F1 = 0.382) produce similar ranker improvements to FT-3B labels (F1 = 0.800), then the expensive fine-tuning step is unnecessary — any moderately capable pretrained model suffices. If ranker improvement is highly sensitive to label quality and only FT-3B-level quality produces measurable gains, then fine-tuning is essential and the paper's emphasis on it is justified. This experiment would also provide an empirical curve relating label F1 to NDCG improvement, which would be the key deliverable for teams deciding how much to invest in label quality.

LLM-generated labels for pairwise or listwise ranker training. The paper generates pointwise labels because they are simple and architecturally agnostic. But the prompt template in Figure 1 can be extended: instead of asking the LLM to assign a label to a single query-app pair, present it with a query and two apps (pairwise) or a query and a list of apps (listwise) and ask it to rank them. This would produce training data for pairwise or listwise ranking losses, which are known to be more effective than pointwise losses for many ranking tasks. A follow-up study would compare three rankers trained with: (1) FT-3B pointwise labels (the current approach), (2) FT-3B pairwise preferences (for each query, sample pairs of apps and ask the LLM which is more relevant), and (3) FT-3B listwise rankings (for each query, present the full list of candidate apps and ask the LLM to order them). The comparison would measure offline NDCG and conversion rate for each approach, revealing whether the additional structure in pairwise/listwise labels translates to better ranker performance. The paper's acknowledgment in Section 6 — "we plan to experiment with ... pairwise and listwise configurations" — confirms this is a natural extension.

Cross-domain replication on publicly available benchmarks. The paper's results are entirely on App Store search with proprietary data, models, and ranker architecture. To validate the "practical blueprint" claim (Section 6), the methodology needs replication on a publicly available dataset with open models. A strong replication study would: (1) take a publicly available search relevance benchmark with human judgments (e.g., MS MARCO passage ranking, TREC Deep Learning track, or Amazon ESCI for product search); (2) fine-tune an open LLM (e.g., LLaMA-3B, Mistral-7B) on a subset of the human judgments; (3) generate millions of labels for unlabeled query-document pairs; (4) train a standard neural ranker (e.g., a cross-encoder or BERT-based ranker) on a mix of behavioral labels (e.g., clicks from the dataset's logs) and LLM-generated textual labels; (5) compare against the same ranker trained without LLM labels, measuring NDCG on the benchmark's test set. This replication would test whether the paper's findings — FT > PT for label quality, Pareto improvement from augmented training, concentration of gains in tail queries — generalize beyond the App Store domain and beyond proprietary models. A negative result (e.g., LLM labels not improving a BERT-based ranker on MS MARCO) would not invalidate the paper's App Store findings but would reveal domain-specific prerequisites that the current "blueprint" language obscures. Either outcome is scientifically valuable.

Difficulty-aware label generation budget allocation. The paper's Figure 2 demonstrates that LLM label value is concentrated in tail queries, but the label generation process treats all query-app pairs uniformly. A follow-up study would implement a query-frequency-aware label generation policy: rank all candidate queries by their frequency in search logs, allocate the LLM inference budget disproportionately to low-frequency queries (e.g., 80% of the budget to queries in the bottom 20% of the frequency distribution), and compare the resulting ranker against one trained on uniformly generated labels with the same total label count. If the paper's tail-query diagnosis is correct, the frequency-aware allocation should achieve equivalent or better ranker performance with fewer total labels, or strictly better performance at the same label count. This is a direct test of the principle that training data value is highest where existing signals are weakest, and a positive result would provide a concrete efficiency gain for practitioners. This experiment also connects to the compute-optimal inference literature: just as the optimal test-time strategy depends on prompt difficulty, the optimal label generation strategy may depend on query frequency, and a unified framework could allocate a fixed LLM inference budget across queries to maximize downstream ranker improvement.

Practical Applications and Downstream Use Cases

Any large-scale search or recommendation system with a "cold start" relevance problem. The paper's tail-query finding (Figure 2) generalizes beyond App Store search to any system where user behavior signals are abundant for popular items but absent for new, niche, or rare items. An e-commerce search engine, for example, has millions of behavioral signals for "iPhone case" but virtually none for "left-handed ergonomic potato peeler." Using the paper's methodology, the e-commerce team would fine-tune an LLM on their existing human relevance judgments (which they likely already collect for evaluation), use it to generate textual relevance labels for millions of product-query pairs, and augment their production ranker training data. The expected benefit, based on the paper's results, is improved conversion rate on tail queries — exactly where the revenue opportunity from capturing niche purchase intent is highest, and where behavioral-only rankers systematically fail. The methodology requires no changes to the serving infrastructure, no additional latency, and no architectural modifications to the ranker. The upfront investment is the fine-tuning of a domain-specific LLM and a one-time batch inference job; the ongoing cost is zero beyond periodic re-generation as the product catalog and query distribution evolve.

Multi-objective ranker training in domains where one objective is label-starved. The paper's scalarization-via-data-mixing approach (Section 3.2) is immediately applicable to any multi-objective ranking system with asymmetric label availability. For instance, a video recommendation system might have abundant engagement labels (watch time, likes, shares) but scarce labels for content quality, age-appropriateness, or factual accuracy. By fine-tuning an LLM on the limited available quality/appropriateness/accuracy judgments and generating millions of additional labels, the system can strengthen the underpowered objectives without requiring changes to the recommendation model architecture or serving pipeline. The paper's Pareto improvement finding (Table 2) is particularly relevant here: strengthening the label-starved objective may also improve the label-rich objective, because better representations of content quality help the model disentangle genuine user preference from superficial engagement signals. The paper's finding that behavioral NDCG improved more than textual NDCG (+0.004–0.006 vs. +0.001–0.002 in absolute terms, Table 2) is an existence proof of this synergy.

Automated evaluation pipeline modernization for mature search systems. Many mature search systems maintain a human evaluation pipeline that produces textual relevance judgments for offline metric computation but does not feed those judgments back into model training (because the volume is too low to move the training distribution). The paper provides a blueprint for converting this evaluation-only judgment pipeline into a training-plus-evaluation pipeline. The existing human judgments serve double duty: they continue to serve as the ground truth for NDCG computation (evaluation), but they also serve as fine-tuning data for an LLM that then generates training labels at scale (training). This conversion requires no new data collection — it uses judgments the organization is already paying for — and the LLM generation cost is a one-time expense. The paper's offline (Table 2) and online (Table 3) results provide evidence that this conversion from evaluation-only to training-plus-evaluation yields measurable ranking quality improvements, making the business case for the one-time engineering investment straightforward to justify.

When to Prefer This Method

The paper does not explicitly position its approach against named alternatives with a clear decision rule (e.g., "use offline LLM labels instead of real-time LLM reranking when X, Y, Z"). The related work section (Section 2) describes alternatives — real-time LLM reranking, distillation approaches like RRADistill, preference alignment for ranking — but the paper's evaluation compares only against its own production baseline, not against these alternatives implemented in the same system. The paper's implicit argument is that offline label generation is preferable to real-time reranking when latency constraints and per-query inference cost make serving-time LLM calls impractical, and that it is preferable to distillation approaches when architectural changes to the ranker are undesirable. But these are qualitative tradeoffs asserted in the paper's positioning, not quantitative comparisons supported by experimental evidence. The paper does not run an A/B test comparing the LLM-augmented ranker against a real-time LLM reranker or an RRADistill-style distilled model, so any "prefer A when B" decision rule would be extrapolation beyond the paper's experimental scope. Given this, a formulaic tradeoff matrix would be inappropriate boilerplate. The paper's contribution is demonstrating that its specific approach works at industrial scale, not establishing that it dominates all alternatives under measurable conditions.