Fine-tuning Small Language Models as Efficient Enterprise Search Relevance Labelers

ArXiv: 2601.03211

🎯 Pitch

A fine-tuned 3.8B-parameter small language model can match GPT-4o in labeling enterprise search relevance—delivering a 17× throughput gain and 19× cost reduction—by distilling skills from a synthetic data pipeline that teaches it to navigate the ambiguous, persona-specific queries that break web-trained models.

---

1. Executive Summary

This paper introduces a synthetic data generation pipeline that leverages an LLM—specifically GPT-4o—to synthesize realistic enterprise queries from seed documents, applies BM25 to retrieve challenging negatives, and uses a teacher LLM to assign graded relevance labels, producing query–document–label triplets that are distilled into a fine-tuned small language model (SLM) for efficient relevance labeling. Evaluated on a proprietary benchmark of 923 enterprise query–document pairs annotated by trained human labelers, the fine-tuned Phi-3.5 Mini Instruct achieves agreement with human judgments on par with or better than the teacher LLM, with a 17× throughput increase and 19× cost reduction—reaching an SLM–Human NDCG of 0.953 and pairwise accuracy of 63.81 compared to GPT-4o's 0.944 NDCG and 62.58 accuracy—establishing that compact, well-trained SLMs can match frontier LLM labeling quality while enabling production-scale offline ranking evaluation, with the primary training signal derived from the synthetic enterprise query pipeline and multi-task tuning providing only auxiliary generalization benefits.

2. Context and Motivation

The Core Problem: Enterprise Search Relevance Labeling Is Stuck Between Two Unsustainable Extremes

The fundamental challenge this paper tackles is practical and acute: how do you produce high-quality relevance labels for enterprise search at scale when neither human annotation nor large language model (LLM) labeling is economically or operationally viable?

To understand why this matters, we need to appreciate what enterprise search actually looks like compared to the more familiar domain of web search. The paper draws a sharp contrast in Section 1 and Table 1. In web search, a query like "Juno release date" has a clear, shared, common-knowledge referent—the 2007 film—and relevance can be assessed by comparing document content against that publicly understood intent. But in enterprise search, the same query could mean "find files containing the release date of a project code-named Juno" or "find emails from a colleague named Juno." The difference, as the paper explains, is that enterprise queries are "often ambiguous, persona-specific, and heavily context-dependent, requiring systems to interpret user intent while accounting for content, metadata, and both personal and organizational context." This means that the mapping from query text to relevant documents isn't purely semantic—it depends on knowledge of internal entities (people, projects, code names, file structures), organizational relationships, and metadata patterns like author names, folder hierarchies, and document types.

This distinction has concrete consequences for labeling. A web search relevance labeler can often rely on general world knowledge and textual similarity. An enterprise labeler must understand that "iris" might refer to a person or project rather than a flower, that "Lisa budget report" is a pattern combining an author name with a document type, and that two documents with near-identical content but different metadata (different folder, different author) may have very different relevance to a given query. Table 1 captures this precisely: both enterprise and web queries can be semantic, but enterprise queries "further require enterprise-specific knowledge (e.g., 'iris' = colleague/project)" rather than relying on "global/common knowledge (e.g., 'iris' = part of the eye)."

This makes the labeling problem both harder and more consequential, because enterprise search systems power the information retrieval that knowledge workers depend on daily—finding the right document, email, or chat message—and poor relevance directly translates to lost productivity.

Why Existing Solutions Fail

The paper identifies three approaches to relevance labeling and explains why each is inadequate for enterprise search at scale:

Manual human annotation is infeasible at scale. While human labelers produce the highest-quality judgments (the paper uses human-annotated data as its gold standard for evaluation), the practical barriers in enterprise settings are severe. The paper states in Section 3: "Manual annotation is often infeasible in real-world enterprise search due to strict privacy policies and the substantial effort required to go through large-scale datasets." Enterprise documents contain sensitive, confidential information—emails, financial reports, project plans—that cannot be shared with external annotators. And even for internal annotators, the volume of query–document pairs in a production search system (where new content arrives continuously and ranking models need regular evaluation and retraining) makes exhaustive human labeling cost-prohibitive. This isn't just a budget problem; it's a fundamental privacy barrier that rules out the traditional IR approach of crowdsourcing or hiring dedicated annotation teams.

LLM-based labeling is high-quality but computationally expensive and slow. The paper acknowledges that "LLM-based judgments have become a standard approach for enterprise relevance labeling" (Section 1), citing works like Thomas et al. (2024) which showed LLMs can accurately predict searcher preferences. But this approach has "significant drawbacks: LLMs are computationally expensive and have limited throughput, making them inefficient for large-scale labeling tasks" (Section 1). The throughput numbers are revealing: while the paper's fine-tuned SLM achieves 873 requests per minute on a single A100 GPU (extrapolating to nearly 7,000 RPM on an 8-GPU cluster), typical LLM labelers are "generally limited to the hundreds of RPM range." And the cost comparison is stark: GPT-4o costs $2.50 per 1M input tokens and$10.00 per 1M output tokens, while the fine-tuned Phi-3.5 Mini costs $0.13 and$0.52 respectively—roughly 19× cheaper on both input and output tokens (Section 4, Cost Analysis). When you need to label tens or hundreds of thousands of query–document pairs for training data or offline evaluation, these differences compound dramatically.

Public benchmark datasets don't capture enterprise characteristics. The IR community has developed excellent resources for web search—the paper cites MS MARCO with over 400,000 passages and TREC-CAsT with 4,000 conversational search examples—but as the authors note in Section 3, "there is no publicly available dataset for enterprise search that captures its unique characteristics." These public datasets are designed for semantic web search, where queries express common-knowledge intent and relevance depends primarily on textual content. They lack the keyword-metadata hybrid patterns, the entity ambiguity (is "Juno" a project or person?), and the domain-specific context that characterize enterprise queries. As the ablation in Table 4, row 7 demonstrates starkly: fine-tuning on public data alone (INTERS, TREC-CAsT, MS MARCO) without the synthetic enterprise query dataset yields almost no improvement over the vanilla SLM (NDCG moves from 0.815 to only 0.826, accuracy from 42.16 to 42.88). The authors note this confirms that "existing open-source datasets are primarily designed for web search query understanding and thus do not solely capture the characteristics of enterprise search relevance labeling."

The Additional Challenge: Query Logs Are Scarce, Biased, or Unavailable

Beyond the labeling problem, there's a deeper data scarcity issue. In web search, query logs provide a rich source of training signal—real user queries with implicit feedback from clicks. But in enterprise settings, as the paper explains in Section 3, there are multiple compounding problems:

• Query logs may not exist. "While some enterprise systems may collect query logs, this is not always the case."
• When they do exist, they contain sensitive information. Logs "contain sensitive, confidential user information which makes it untenable to collect human relevance judgements."
• Even if usable, click logs are biased. "Interaction data such as click logs are inherently biased, as they primarily reflect user engagement rather than true relevance," with the well-known phenomenon that "users often click on higher-ranked results regardless of their quality" (citing Craswell et al., 2008; Vardasbi et al., 2020).

This triple bind—no labels, no usable query logs, and domain-specific requirements that public datasets don't meet—means that conventional approaches to training or evaluating search relevance models are largely unavailable in the enterprise context. The paper's synthetic data generation pipeline is designed specifically to circumvent all three constraints simultaneously.

How Prior Work Falls Short on This Specific Problem

The paper positions itself against several lines of prior work, acknowledging their contributions while identifying specific gaps:

General LLM relevance labeling research (MacAvaney and Soldaini, 2023; Abbasiantaeb et al., 2024; Thomas et al., 2024; Farzi and Dietz, 2025): These works established that LLMs can produce useful relevance judgments, but they focus on open-domain web search settings. The paper does not argue that these methods are wrong—rather, they don't address the enterprise-specific challenges of metadata-dependent relevance, entity ambiguity, and privacy constraints.

SLM distillation for ranking and retrieval (Choi et al., 2024; Fitte-Rey et al., 2025; Samarinas and Zamani, 2025; Weller et al., 2025): These works demonstrated that small models can be distilled from large ones for re-ranking and relevance tasks. RRADistill (Choi et al., 2024) distilled LLMs into SLMs for long-tail query re-ranking. Rank1 (Weller et al., 2025) used reasoning-based LLMs and MS MARCO traces to train smaller, explainable re-rankers. The augmented relevance datasets work (Fitte-Rey et al., 2025) showed that fine-tuned small LLMs can improve relevance dataset quality. But as the paper notes in Section 2, "the exploration of SLMs for relevance labeling has been largely limited to open-domain, semantically driven retrieval tasks." None of these works tackled the hybrid keyword-semantic nature of enterprise queries or the challenge of generating training data without query logs.

Synthetic query generation for IR (Bonifacio et al., 2022; Dai et al., 2023; Jeronymo et al., 2023; Chandradevan et al., 2024): Methods like InPars, Promptagator, and DUQGen showed that LLM-generated synthetic queries could train effective retrievers, often outperforming BM25. But these approaches target document retrieval, not relevance labeling. They generate queries to train retrievers to find documents, not to train models to assess graded relevance on a 0–4 scale. Moreover, they operate in web domains where semantic similarity is the primary signal. The enterprise setting requires modeling metadata-pattern matching, entity resolution, and keyword-based relevance alongside semantic understanding.

SLM query generation for enterprise (the paper's own first approach, Section 3.1): Interestingly, the paper itself provides evidence for a failed approach that illuminates why the problem is hard. When the authors fine-tuned Phi-3.5 Mini as a query generator—conditioning it on document metadata and target relevance levels—the model struggled fundamentally. It achieved only 60.8% binary relevance accuracy, with a strong bias toward generating positive-sounding queries even when asked to produce negatives. The paper diagnoses two causes: (1) Positivity bias in pretraining, where instruction-tuned SLMs are optimized to be helpful and cooperative, and when asked to "generate a query for a document that should not be retrieved," they "may still attempt to generate something that sounds relevant or plausible"; and (2) limited model capacity, where the SLM "may struggle to internalize nuanced instructions like generating 'key-word-based queries that appear plausible but are not relevant,' especially in enterprise contexts where relevance boundaries can be subtle." This negative result is informative: it shows that directly fine-tuning an SLM for query generation is insufficient, motivating the more sophisticated pipeline approach that ultimately succeeds.

How the Paper Positions Its Contribution

The paper does not claim to invent LLM-based labeling, SLM distillation, or synthetic query generation. Rather, it fills a specific, previously unaddressed gap: applying these techniques to enterprise search relevance labeling under realistic constraints, and demonstrating that the combination works in practice.

The positioning is practical and systems-oriented. The contribution is a pipeline—a sequence of components that together solve a deployment problem. The paper explicitly frames this around throughput and cost (Section 4): "Our fine-tuned SLM achieved 873.33 RPM on a single A100 GPU... This throughput is an order of magnitude faster than typical LLM labelers." And the 19× cost reduction "combined with strong alignment to human judgments, makes trained SLM an attractive option for scalable enterprise relevance labeling."

The paper also makes a specific methodological choice that distinguishes it from prior work on synthetic query generation: rather than generating queries from scratch, it anchors query generation in document metadata and query pattern templates. As described in Section 3.2, the pipeline samples from a template table of common query patterns (e.g., <author name><file name>, <folder name><document type><keyword>) with probability proportional to observed frequency, then fills those templates with actual document metadata. This ensures the synthetic queries respect the structural patterns of real enterprise queries, not just their semantic surface forms. The subsequent BM25 negative mining and LLM labeling steps then provide the graded relevance signal that teaches the SLM to distinguish truly relevant documents from topically adjacent ones—a distinction that pure semantic approaches miss.

Finally, the paper's contribution is validated by a specific failure mode: the initial SLM query generator approach (Section 3.1) failed precisely because it could not generate diverse negative queries and keyword-based enterprise patterns. The successful approach (Section 3.2) succeeded because it offloaded query generation to a more capable LLM (GPT-4o) and used BM25 to handle negative mining, reserving the SLM's role for the labeling task where its compact size provides the throughput advantage. This division of labor—LLM for data generation, SLM for deployment—is the core architectural insight, and the paper's ablation studies (Table 4) systematically validate that each component (synthetic enterprise queries, query refinement, multi-task tuning, data size) contributes meaningfully to the final result.

3. Technical Approach

3.1 Reader Orientation

The system being built is a fine-tuned small language model (SLM) that can judge the relevance of any enterprise document to any enterprise query on a 0–4 scale, matching the quality of a frontier LLM (GPT-4o) while running 17× faster and costing 19× less. The problem it solves is that high-quality relevance labels are essential for evaluating and training enterprise search systems, but existing solutions are either too expensive (LLMs), too slow (LLMs), or impossible due to privacy constraints (human annotators) — and the "shape" of the solution is a synthetic data pipeline that generates realistic training data without requiring real user query logs, then distills that data into a compact, deployable model.

3.2 Big-Picture Architecture (Diagram in Words)

The system has two distinct phases: a data generation phase (offline, expensive, run once or periodically) and a deployment phase (online, cheap, run continuously). The data generation phase is the core contribution; the deployment phase is standard supervised fine-tuning.

Phase 1 — Synthetic Training Data Generation:

1. Seed Document Collection: Start with 1,500 proprietary enterprise documents (with metadata like author, title, file type, folder path, and content) that have been cleared for "eyes-on" review — meaning they can be used for data generation under privacy constraints.

2. Query Pattern Template Table: Maintain a ranked list of common enterprise query patterns (e.g., <author name><file name>, <folder name><document type><keyword>) with their observed frequencies in real search traffic. These patterns encode the structural templates that users follow when forming queries — combining metadata fields and keywords in specific orders.

3. Synthetic Query Generator (GPT-4o): For each seed document, sample a query pattern from the template table (weighted by frequency), fill it with the document's actual metadata and GPT-4o-extracted keywords, then ask GPT-4o to generate three distinct enterprise queries for that document-pattern pair. Apply a second GPT-4o refinement step to diversify phrasing and ensure each query is unique.

4. BM25 Negative Document Miner: For each synthetic query, run BM25 retrieval against the full document corpus to find the top-4 highest-scoring documents (excluding the source document). These are plausible retrieval candidates that span a range of relevance levels — some highly relevant, some partially related, some irrelevant — creating a natural mixture of hard negatives, weak positives, and near-miss cases.

5. LLM Labeler (GPT-4o): For each synthetic query–document pair (the source document plus the 4 BM25-retrieved candidates), prompt GPT-4o to assign a graded relevance label on a 0–4 scale. Apply quality control: re-label any pair where formatting is invalid, and filter out synthetic queries where the source document itself receives a low relevance score after re-labeling.

6. Output: A set of roughly 14,000–24,000 query–document–label triplets that form the core enterprise-specific training dataset.

Phase 2 — SLM Fine-Tuning:

7. Multi-Task Pre-Training: Fine-tune Phi-3.5 Mini Instruct on the INTERS dataset (20 query/document understanding tasks) to improve general robustness.

8. Instruction Fine-Tuning: Further fine-tune the multi-task-tuned model on a mixture of: (a) the synthetic enterprise triplets from Phase 1, (b) TREC-CAsT human-labeled web search pairs, and (c) MS MARCO passages with GPT-4o-generated synthetic queries.

Phase 3 — Deployment:

9. Fine-Tuned SLM Labeler: The resulting model takes a query–document pair as input and outputs a relevance score from 0–4 at 873 requests per minute on a single A100 GPU.

3.3 Roadmap for the Deep Dive

• First, the formal problem definition (Section 2.1 of the paper) — what exactly "relevance labeling" means mathematically, including the input/output specification and the 0–4 ordinal scale. This grounds everything that follows.

• Second, the synthetic query generation pipeline — how GPT-4o is prompted to generate realistic enterprise queries from seed documents, including the query pattern template mechanism, the keyword extraction step, and the query refinement stage. This is the most novel engineering contribution.

• Third, the BM25-based negative document mining — how hard negatives are retrieved, why $k = 4$ is chosen, and how this step creates a natural relevance spectrum without requiring explicit negative labels.

• Fourth, the LLM labeling step — how GPT-4o is prompted to assign 0–4 relevance scores, the quality control mechanisms, and the post-labeling filtering that ensures training data consistency.

• Fifth, the SLM fine-tuning procedure — the two-stage training (multi-task pre-training followed by instruction tuning), the dataset mixture, the specific hyperparameters, and the design choice to exclude explanation generation from the SLM's output format.

• Sixth, the failed approach — the SLM-as-query-generator experiment (Section 3.1), why it failed, and what that failure teaches us about the division of labor between LLMs and SLMs in this pipeline.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a practical systems paper whose core idea is that high-quality synthetic training data for enterprise relevance labeling can be generated using only seed documents and a teacher LLM, without requiring real user query logs, and that an SLM distilled on this data matches the teacher's labeling quality at a fraction of the cost.

---

Problem Definition: What the System Must Compute

Before diving into the pipeline, the paper formally defines the relevance labeling task (Section 2.1). This formalization is important because it specifies exactly what the fine-tuned SLM must learn to compute, and it establishes the mathematical structure that the training data must support.

Given a user query $q$, the objective is to determine the relevance of a set of documents $D = \{d_1, d_2, ..., d_n\}$ to the query. We define a relevance function $r(q, d_i)$ that maps each query–document pair to a relevance score:

$r(q, d_i) \in \mathbb{R}$

where $q \in \mathcal{Q}$ is a query from the space of all user enterprise queries, $d_i \in \mathcal{D}$ is a document from the space of enterprise documents, and $\mathbb{R}$ is the space of relevance scores.

What it computes: for any given pair of an enterprise query and an enterprise document, this function assigns a real-valued score representing how relevant that document is to that query. In this paper, the output range is constrained to the discrete ordinal scale $\{0, 1, 2, 3, 4\}$, where 0 means "bad quality, should never be shown" and 4 means "ideal quality, should be the ideal result." The function $r(q, d_i)$ is designed to approximate human judgments and align with the quality of state-of-the-art LLMs such as GPT-4o.

Why this form: the 0–4 graded relevance scale is standard in information retrieval evaluation (it's the scale used in TREC and MS MARCO), which means that models trained on this scale produce labels directly compatible with established ranking metrics like NDCG. A binary relevant/irrelevant scale would be simpler but would lose the fine-grained distinction that enterprise search requires — for example, distinguishing between a document that exactly matches the query (score 4), one that partially addresses the question (score 2), and one that is only topically adjacent (score 1). The graded scale enables evaluation metrics (like NDCG) to reward ranking models that place the 4-document above the 2-document above the 1-document, rather than treating all non-zero-relevance documents identically.

The paper's goal, as stated in Section 2.1, is "to develop a practical and scalable approach for enterprise search relevance labeling, enabling rapid and cost-effective evaluation of various IR models while ensuring high-quality and consistent supervision in real-world information filtering systems." In operational terms: the fine-tuned SLM serves as a drop-in replacement for either GPT-4o or human annotators when you need to label query–document pairs for offline evaluation of ranking models.

---

Synthetic Query Generation: How GPT-4o Creates Realistic Enterprise Queries

This is the most novel engineering component of the paper, and it solves a specific constraint: we cannot use real user queries (they may not exist, they may be sensitive, or they may be biased), but we need training queries that faithfully capture the structural patterns of enterprise search. The solution is to generate queries from documents rather than the other way around, using GPT-4o guided by query pattern templates.

Step 1: The Query Pattern Template Table

The paper assumes access to a template table that encodes how real users form queries in enterprise search. These templates are constructed by "entity resolution of query text, ensuring that each segment corresponded to an entity in the file metadata" (Section 3.2). Examples of patterns include:

• <author name><file name> → "Lisa budget report"
• <folder name><document type><keyword> → "Projects folder spreadsheet Q4"
• <keyword><file type> → "deployment plan pdf"

The table is ranked by frequency observed in actual search traffic, but the paper explicitly notes that even without real query logs, "synthetic template tables can be generated by systematically enumerating combinations of document metadata fields and keywords, which we found to produce realistic and diverse query structures." This is an important practical note: the method does not depend on having access to real user query data, only on knowing what metadata fields exist in the document store.

During generation, templates are sampled with probability proportional to their frequency (weighted sampling with replacement), so more common patterns appear more often in the training data — preserving the distributional properties that make the synthetic queries realistic.

Step 2: Keyword Extraction and Template Filling

For each seed document, a template is sampled. The document has associated metadata (author, title, file type, parent folder, filename) and text content. GPT-4o is prompted with the structure shown in Box 3.1 to perform two tasks sequentially:

First, keyword extraction: the model extracts up to 6 relevant single-word keywords from the document content, excluding stop words. If the content is empty, it returns an empty list. These keywords capture the topical essence of the document — the terms that a user searching for this document might use.

Second, query generation: the model uses the template's metadata string in exact order, replacing any content placeholders with the extracted keywords, and generates three distinct queries without reordering the metadata parts. The prompt includes few-shot examples (the specific wording is not disclosed due to confidentiality, but the structure is outlined in Box 3.1).

Why this approach over alternatives: the key design choice is that the template constrains the structural form of the query while GPT-4o provides the creative variation. If GPT-4o were asked to generate queries from scratch given only the document, it would likely produce semantically rich but structurally uniform queries (e.g., natural language questions like "What is the budget for Q4?"). These would miss the keyword-metadata hybrid patterns that characterize real enterprise search — queries like "Lisa budget 2023 Q4 xlsx" or "Projects finance report". The template enforces structural diversity; GPT-4o ensures naturalness and content-grounding.

Step 3: Query Refinement for Diversity

The paper observed a specific failure mode: "for more complex patterns, those with more components or rare metadata combinations, we observed that the initial queries sometimes lacked variation. For example, if the sampled query template was long, the LLM tended to repeatedly select the same portion of each component, even when explicitly instructed to vary them."

To fix this, a second GPT-4o call performs query revision (Box 3.2). The revision prompt instructs the model to: (1) validate that queries follow the metadata order and structure, (2) ensure no redundancy and that queries stay short and natural, and (3) rewrite queries so that "each metadata part is phrased differently across the three queries." Figure 2 illustrates the effect: the initial generation produces three queries that reuse the same metadata segments (limited diversity), while the refinement step yields three queries with varied phrasing of each metadata component.

Effectiveness evidence: The ablation in Table 4, row 6 vs. row 2, quantifies the impact. When using 14K raw synthetic queries without refinement (row 6), the fine-tuned SLM achieves NDCG of 0.943 and pairwise accuracy of 60.97. With refinement (row 2), these improve to 0.953 and 63.81. The accuracy gain of nearly 3 percentage points is substantial given the already-strong baseline, underscoring that "improving data quality has a greater impact than simply adding more data, once the dataset surpasses a reasonably large threshold."

---

BM25 Negative Document Mining: Creating a Realistic Retrieval Distribution

Once we have a synthetic query $q$ generated from a seed document $d_{\text{seed}}$, the next challenge is to construct a set of candidate documents for labeling that covers the full relevance spectrum. The paper uses BM25, the classic lexical retrieval algorithm, to solve this.

The retrieval procedure:

For each synthetic query $q$, BM25 is run against the full document corpus. The top-$k$ highest-scoring documents are retrieved, with $k = 4$ as the default. The seed document $d_{\text{seed}}$ (from which $q$ was generated) is excluded from retrieval — it already has a known relationship to the query, and including it would create a trivially-positive example that doesn't teach the model to discriminate.

Why BM25 rather than a dense retriever or random sampling:

The paper makes a deliberate choice that is worth understanding. BM25 is a lexical retriever — it scores documents based on term frequency, inverse document frequency, and document length normalization, without any semantic understanding. This means it retrieves documents that share vocabulary with the query, regardless of whether they address the same underlying information need.

This property is exactly what makes BM25 useful for negative mining. In enterprise search, as the paper explains, many queries combine keywords with metadata patterns. A query like "Lisa budget report" will retrieve documents containing "Lisa," "budget," and "report" in various combinations — some by Lisa Morrison about budgets (highly relevant), some by Lisa Chen about a different topic (partially relevant), and some about budgets authored by someone else (weakly relevant or irrelevant). This creates a natural relevance spectrum without requiring any prior labeling.

The paper notes that "BM25 tends to return documents with varying degrees of lexical and semantic overlap with the query. Some retrieved documents may be highly relevant (e.g., discussing the same entity or topic), while others are only partially related or largely irrelevant." These retrieved documents "naturally form a mixture of hard negatives, weak positives, and near-miss cases, reflecting the ambiguity commonly observed in real-world enterprise search."

Why $k = 4$:

The choice of $k = 4$ is empirical and motivated by label distribution balance. The paper states that "this empirically yields a uniform distribution of relevance labels across levels 0–4 after LLM-based annotation. This balance is desirable for training a robust relevance labeler, as it prevents over-representation of trivially irrelevant examples while encouraging the model to learn subtle distinctions between closely related documents."

If $k$ were set too high, the later BM25 results would be mostly irrelevant (trivial negatives that don't teach the model to make fine distinctions). If $k$ were set too low, the training data would consist mostly of highly relevant documents, under-representing the negative and partially-relevant examples needed to learn what not to retrieve.

The paper also notes that $k$ is adjustable: "if the LLM-based annotation results in an over-representation of higher relevance scores, we can increase $k$ to retrieve additional candidate documents and annotate only the newly introduced query–document pairs, while retaining previously labeled pairs, thereby incrementally rebalancing the dataset without reprocessing the entire corpus." This incremental rebalancing strategy is important for practical deployment, since re-running the full pipeline would be expensive.

The outcome of this step: for each synthetic query, we now have one known-positive pair $(q, d_{\text{seed}})$ (the query was generated from this document) and four unknown-relevance pairs $(q, d_{\text{BM25},1}), ..., (q, d_{\text{BM25},4})$. These five pairs per query proceed to LLM labeling, where GPT-4o determines the actual relevance of each pair — including potentially assigning low scores to the seed document if the generated query was poorly formed.

---

LLM Labeling: How GPT-4o Assigns Graded Relevance Scores

The third major component is the labeling step, where GPT-4o acts as a teacher to produce the relevance labels that the SLM will learn from. The prompt template is outlined in Box 3.3.

The labeling prompt structure:

GPT-4o is given the role of "an enterprise search quality rater evaluating file/message relevance." For each query–document pair, it receives:

• The query text
• The document's metadata (author, title, file type, folder, etc.)
• Content highlights from the document (likely key passages or summaries)

It must output a single integer score from 0 to 4, defined on a graded scale:

• 4: ideal quality, should be the ideal result
• 3: excellent quality, highly relevant
• 2: good quality, relevant
• 1: fair quality, partially relevant
• 0: bad quality, should never be shown

(The paper provides only the endpoints of this scale explicitly; the intermediate descriptions are indicated by ellipsis in Box 3.3, suggesting there are fuller definitions in the actual prompt that aren't disclosed due to confidentiality.)

Quality control mechanisms:

The paper implements two quality assurance steps after initial labeling:

1. Format validation: Out of roughly 24,000 samples, only 15 cases contained invalid outputs (e.g., missing scores, malformed responses). These were automatically re-labeled. This step is simple but critical for training data pipeline reliability — a single malformed training example can cause issues during SLM fine-tuning.

2. Semantic consistency filtering: For each synthetically generated positive query from Step 1, if GPT-4o assigned a low score (0 or 1) to the $(q, d_{\text{seed}})$ pair — meaning the query generated from a document was judged irrelevant to that same document — the pair was re-labeled using the same prompt. If the low score persisted after re-labeling, the pair was retained as a valid negative case (the generated query was genuinely poor). If the score changed, the query was discarded.

This filtering step is subtle and important. It serves two purposes: (a) it removes genuinely bad synthetic queries from the training data (queries that don't match their source document), preventing the SLM from learning that "irrelevant" is the correct label for a document-query pair where the query was literally derived from that document; (b) it identifies accidental negatives — cases where GPT-4o made an initial labeling error that was corrected on re-labeling. By using GPT-4o both as generator and labeler with this cross-check, the pipeline achieves a form of self-consistency validation.

The paper notes that "this filtering process prevents the model from learning contradictory signals and enforces a consistent training distribution." Without it, the SLM would encounter examples where $(q_{\text{generated from doc A}}, doc A) \rightarrow \text{score 0}$, which contradicts the fundamental premise that generated queries are relevant to their source documents.

Why GPT-4o for labeling rather than a cheaper model:

The paper never directly compares labeling quality across different LLMs, but the choice of GPT-4o as the teacher is motivated by the need for the highest-quality labels possible. Since the entire pipeline rests on the assumption that the teacher's judgments are a valid proxy for human judgments, using the strongest available LLM minimizes the risk that poor teacher labels propagate errors into the SLM. The cost of running GPT-4o on 14,000–24,000 query–document pairs (roughly 70,000–120,000 individual labeling decisions, since each query produces ~5 document pairs) is a one-time expense amortized over all subsequent SLM usage.

The output of this step: a dataset of $(q, d_i, \text{score}_i)$ triplets where $\text{score}_i \in \{0, 1, 2, 3, 4\}$ represents GPT-4o's best assessment of relevance. This dataset plus the public datasets (INTERS, TREC-CAsT, MS MARCO) forms the complete training corpus for SLM fine-tuning.

---

The Failed Approach: SLM-as-Query-Generator (Section 3.1)

Before arriving at the successful pipeline, the paper explored a more direct approach: fine-tune the SLM itself to generate queries given a document and a target relevance score, then use those generated queries to train the SLM labeler. This approach failed, and understanding why it failed is instructive for two reasons: it validates the complexity of the enterprise query generation task, and it explains the specific division of labor in the successful pipeline (LLMs generate, SLMs label).

The experimental setup:

The authors fine-tuned Phi-3.5 Mini Instruct on synthetic query generation data produced by GPT-4o. GPT-4o was prompted to generate both positive and negative enterprise search queries using a carefully designed prompt. The SLM was then trained to generate queries conditioned on document metadata and a target relevance level — essentially learning to answer the question "given this document and the instruction to produce a relevant query, what query would someone type?" and similarly for irrelevant queries.

The failure and its diagnosis:

When evaluated against GPT-4o gold-standard judgments (using GPT-4o as a judge to classify whether each SLM-generated query was truly relevant or irrelevant to its source document), the SLM achieved only 60.8% binary relevance accuracy. Even worse, the failure mode was asymmetric: the model showed a "strong bias toward generating positive or plausible-sounding queries, even when the prompt explicitly instructed the model to produce irrelevant (negative) ones."

The paper provides a concrete example (with fabricated metadata to avoid confidentiality issues):

• Document: "A document about how to add a page in Word"
• Metadata: Filename "AddPage.docx", Author "Lisa Morrison", Title "AddPage", File type "docx", Parent folder "Word Tutorial"
• Keywords: page, Word, add

When instructed to generate negative queries (queries for which this document should NOT be retrieved), the SLM produced: "Lisa tutorial Docs," "Add Page," and "Lisa Loop page." All three are actually relevant to the document — "Lisa" matches the author, "tutorial" relates to the Word Tutorial folder, "Docs" is a variant of "docx," and "page" and "add" match the content.

Diagnosed causes:

The paper attributes this failure to two factors:

1. Positivity bias in pretraining: "Instruction-tuned SLMs like Phi-3.5 Mini are optimized to be helpful and cooperative. When asked to generate a query 'for a document that should not be retrieved,' the model may still attempt to generate something that sounds relevant or plausible, rather than something truly irrelevant even after fine-tuning." This is a specific instance of a broader phenomenon in language model behavior — models trained to be helpful and follow instructions have difficulty executing tasks that require them to be intentionally unhelpful or produce deliberately bad outputs. Generating a genuinely irrelevant query requires the model to understand what makes a query relevant, and then deliberately avoid those properties — a kind of adversarial reasoning that SLMs with limited capacity struggle with.

2. Limited model capacity: "As a small model, Phi-3.5 Mini may struggle to internalize nuanced instructions like generating 'key-word-based queries that appear plausible but are not relevant,' especially in enterprise contexts where relevance boundaries can be subtle." The enterprise domain makes this particularly challenging because relevance depends on fine-grained metadata matching. A query might share keywords with a document but be irrelevant because it refers to a different project, person, or time period — distinctions that require world knowledge about the enterprise's internal entities that an SLM fine-tuned on a small number of examples may not acquire.

Why this failed approach is reported:

Negative results are often buried, but the paper includes this one because it illuminates the central design principle of the successful pipeline: LLMs handle data generation (where broad knowledge, strong instruction following, and the ability to understand subtle intent are critical), while SLMs handle the labeling task at deployment (where speed, cost, and consistency matter most). The SLM's failure at query generation is not a failure of the overall approach — it's evidence that the task decomposition is correct. The SLM is asked to perform the simpler, better-defined task of scoring relevance given an existing query–document pair, not the harder, more open-ended task of generating realistic queries from scratch.

This finding also connects to the broader literature on synthetic data generation: smaller models can effectively fine-tune on LLM-generated data, but they may not be able to replace the LLM in the generation role itself. The generation step requires the kinds of capabilities (broad knowledge, nuanced instruction following, adversarial reasoning) that scale more steeply with model size than the classification/judgment capabilities needed for labeling.

---

SLM Fine-Tuning: Two-Stage Training with Multi-Task Data

The final component is the fine-tuning procedure that converts Phi-3.5 Mini Instruct into an enterprise relevance labeler. The training follows a two-stage curriculum depicted in Figure 3.

Stage 1: Multi-Task Pre-Training on INTERS

The first stage fine-tunes the base Phi-3.5 Mini Instruct on the INTERS dataset (Zhu et al., 2024). INTERS consists of approximately 250,000 samples across 20 different tasks related to query and document understanding — tasks like query classification, document summarization, query-document relationship prediction, and other IR-relevant skills.

Purpose of Stage 1: The paper describes this as "multi-task tuning to enhance its robustness and overall capabilities." The intuition is that before the model specializes in enterprise relevance labeling, it should develop a broad foundation in understanding queries, documents, and their relationships. This is analogous to the "intermediate task training" paradigm in NLP, where training on related auxiliary tasks improves downstream performance by building generalizable representations.

The INTERS dataset is particularly well-suited because it covers diverse query and document understanding scenarios, exposing the model to many different ways that relevance and relationship signals can appear — not just the enterprise-specific patterns that dominate the later training.

Stage 2: Instruction Fine-Tuning on a Dataset Mixture

After Stage 1, the multi-task-tuned model is further fine-tuned on a mixture of three datasets:

1. Synthetic enterprise query–document–label triplets: The output of the pipeline described above, roughly 14,000–24,000 examples. This is the primary training signal — it teaches the model the specific patterns, metadata dependencies, and relevance criteria of enterprise search.

2. TREC-CAsT human-labeled pairs: Approximately 4,000 examples from the TREC Conversational Assistance Track, consisting of semantic web query–passage pairs with human annotations on a 0–4 scale. These provide high-quality human judgment signal for open-domain semantic relevance, which the paper believes "helps the model develop a stronger understanding of query–document relevance beyond strictly keyword-matching setting."

3. MS MARCO passages with GPT-4o-generated synthetic queries: The MS MARCO dataset provides over 400,000 passages. The paper uses GPT-4o to generate synthetic queries on a 0–4 relevance scale for randomly selected passages, "based on a chain-of-thought idea." This creates an additional source of semantic relevance training data, complementing the keyword-heavy enterprise data and the human-labeled web data.

Why include web search datasets in an enterprise search task:

The paper explicitly addresses this design choice: "Although our primary goal is enterprise query labeling, we deliberately incorporated semantic web query datasets from the public domain. We believe this broader supervision helps the model develop a stronger understanding of query–document relevance beyond strictly keyword-matching setting, thereby improving both generalization and robustness."

This is a form of regularization through data diversity. Without the web search data, the model might overfit to the keyword-metadata patterns in the synthetic enterprise data, performing poorly on queries that require semantic understanding (e.g., natural language questions where the relevant document uses different vocabulary than the query). The web search data teaches the model that relevance can also be purely semantic, while the enterprise data teaches it that relevance often depends on structured metadata and exact lexical matching. The combination produces a model that can handle both types — which is exactly what enterprise search requires, since real users issue both keyword-based queries and natural language questions.

Training hyperparameters (from Section 3.2, Step 4):

The paper specifies the training configuration precisely:

• Epochs: 2
• Maximum sequence length: 4096 tokens
• Effective batch size: 32, achieved by setting per-device batch size to 4 and using 8 gradient accumulation steps
• Training log frequency: every 40 steps
• Evaluation frequency: every 80 steps
• Evaluation dataset: a separate human-labeled test dataset not used for training (no early stopping or feedback loop from the evaluation set)
• Hardware: cluster of eight A100 GPUs
• Training speed: approximately 5.84 samples per second on the GPU cluster

Output format design choice:

A notable detail is that the SLM is prompted using the same template as the LLM labeling prompt (Box 3.3), but "after removing the instruction for the model to generate explanations alongside the predicted relevance score." The authors found through empirical analysis that "including explanations, though useful for interpretability, led to inconsistent training signals and introduced unnecessary verbosity in the trained SLM's output."

This is an important practical finding. Many LLM labeling approaches use chain-of-thought or explanation generation to improve label quality, and it's natural to try to distill this reasoning into the SLM. But the paper's experience suggests that for SLM fine-tuning, forcing the model to generate explanations alongside scores creates two problems: (1) the explanation generation introduces additional variance — two equally valid relevance scores might be accompanied by different explanations, creating conflicting training signals; (2) the verbosity increases sequence length and slows inference, partially undermining the throughput advantage that motivates using an SLM in the first place. By training the SLM to output only the score, the fine-tuning signal is cleaner and the deployed model is faster.

Why the two-stage curriculum:

The paper does not provide an ablation comparing one-stage vs. two-stage training directly, but the structure follows established best practices in instruction tuning: broad multi-task pre-training develops general capabilities, then task-specific fine-tuning specializes the model. The key evidence for the multi-task tuning's contribution comes from Table 4 (row 2 vs. row 4): with the same 14K synthetic examples, adding multi-task tuning improves NDCG from 0.946 to 0.953 and accuracy from 62.41 to 63.81. These gains are consistent but modest (roughly 0.7% relative improvement in NDCG, 2.2% relative improvement in accuracy), confirming the paper's characterization that "multi-task tuning offers an auxiliary benefit by improving robustness via multi-task tuning" rather than being the primary driver of performance.

---

Data Scaling and Quality Trade-offs

The ablation studies in Table 4 provide several insights into how data size and quality affect the final SLM performance. These are not separate pipeline components, but their implications inform the design choices throughout the pipeline.

Scaling synthetic data from 14K to 24K examples yields minimal gains. Comparing row 2 (14K examples with multi-task tuning and refinement) against row 3 (24K examples, same configuration): NDCG is essentially unchanged (0.953 vs. 0.954) and accuracy slightly decreases (63.81 vs. 63.55). Without multi-task tuning (rows 4 vs. 5), the pattern is similar — NDCG actually drops slightly (0.946 to 0.943), while accuracy is nearly flat (62.41 vs. 62.50). This suggests that 14K carefully generated enterprise examples are sufficient for the SLM to learn the enterprise relevance labeling task, and additional synthetic data beyond this point provides negligible benefit. The paper characterizes this as "diminishing returns" and notes that "improving data quality has a greater impact than simply adding more data, once the dataset surpasses a reasonably large threshold."

Query refinement contributes more than data scaling. As discussed in the query generation section, removing refinement (row 6: 14K raw queries without refinement, no multi-task tuning) reduces accuracy from 62.41 (row 4: refined, no multi-task) to 60.97 — a drop of 1.44 percentage points. This is larger than the gain from doubling the dataset size with refinement (row 4 vs. row 5: accuracy changes from 62.41 to 62.50, a gain of only 0.09 points). The paper interprets this as evidence that "for fine-tuning the SLM, improving data quality has a greater impact than simply adding more data."

Synthetic enterprise queries are the primary training signal. Row 7 reveals what happens when the model is fine-tuned with multi-task tuning and all public datasets, but zero synthetic enterprise queries: NDCG reaches only 0.826 (up from 0.815 for the vanilla model), and accuracy reaches only 42.88 (up from 42.15). This is almost no improvement — a gain of roughly 1.3% in NDCG and 1.7% in accuracy. The paper states this "confirms that our synthetic enterprise queries provide the primary training signal, while multi-task tuning offers an auxiliary benefit."

This finding has important implications for anyone attempting to replicate the approach: the public datasets alone (INTERS, TREC-CAsT, MS MARCO) are insufficient for learning enterprise relevance labeling. The synthetic enterprise data generated through the pipeline is essential — without it, the model never learns the keyword-metadata patterns, entity-specific knowledge, or structural query templates that distinguish enterprise search from web search.

---

From Pipeline to Deployment: What the Fine-Tuned SLM Actually Does

At deployment time, the fine-tuned SLM operates as a straightforward scoring function. Given a query $q$ and a document $d_i$ (with its metadata and content), the model outputs a single integer from 0 to 4. There is no chain-of-thought, no explanation, no retrieval step — just a direct mapping from $(q, d_i)$ to $\text{score}$.

The throughput measurement is reported as 873.33 requests per minute (RPM) on a single A100 GPU, which "extrapolates to nearly 7K RPM on a cluster of eight A100s." Each request consists of one query–document pair being scored. For comparison, typical LLM labelers are "generally limited to the hundreds of RPM range" — meaning the SLM processes at least an order of magnitude more query–document pairs per minute than a frontier LLM on comparable hardware.

The cost comparison uses OpenAI's API pricing: the fine-tuned Phi-3.5 Mini costs $0.13 per 1M input tokens and$0.52 per 1M output tokens, compared to GPT-4o's $2.50 per 1M input tokens and$10.00 per 1M output tokens. This represents a roughly 19× reduction on both input and output token costs. For a production pipeline labeling millions of query–document pairs per month, this difference is the difference between feasible and infeasible.

What the SLM does not do: it does not generate queries, retrieve documents, or rank results. Its sole function is to produce a relevance label given a query and a document. This narrow scope is intentional — by restricting the SLM to a well-defined classification task, the fine-tuning signal is clean, the model size requirements are modest, and the deployment footprint is small. The more complex tasks (query generation, retrieval, initial labeling) are handled by the LLM and BM25 components in the offline data generation phase, which runs infrequently and can absorb the higher computational cost.

This division of labor — expensive, powerful models for one-time data generation; cheap, fast models for ongoing deployment — is the central architectural principle of the paper, and it generalizes beyond enterprise search to any domain where the target task is well-defined enough that synthetic data can be generated by a teacher model.

4. Key Insights and Innovations

Innovation 1: The SLM-as-Labeler Framing Inverts the Standard Division of Labor Between LLMs and Small Models

The dominant assumption in the LLM-for-IR literature is that large models handle the "hard parts" (judgment, reasoning, evaluation) while small models, if used at all, handle narrow retrieval or embedding tasks. This paper inverts that framing: the LLM is best deployed as a one-time data factory, while the SLM becomes the production-grade judge. The intellectual move is not that SLMs can be fine-tuned — that is well-established — but rather the specific claim that in enterprise relevance labeling, the SLM can match or exceed the teacher LLM's judgment quality, not merely approximate it at degraded performance.

Prior work on SLM distillation for IR (Fitte-Rey et al., 2025; Choi et al., 2024; Samarinas and Zamani, 2025) has shown that small models can "approach" or "approximate" larger ones, always with an acknowledged quality gap. The implicit bargain is: accept lower accuracy in exchange for speed and cost. This paper's results break that bargain. The fine-tuned Phi-3.5 Mini achieves SLM-Human NDCG of 0.953 vs. GPT-4o's 0.944 and pairwise accuracy of 63.81 vs. GPT-4o's 62.58 (Figure 4), with the statistical non-inferiority analysis (Section 4) rejecting the null hypothesis that the SLM is worse than GPT-4o by more than negligible margins (p = 0.012 for accuracy, p = 0.00098 for NDCG).

Why is this conceptually important beyond the performance number? Because it changes the role of the LLM in the system architecture. If the SLM is strictly worse than its teacher, the LLM remains the gold standard you fall back to when quality matters most, and the SLM is a budget option for when you can tolerate errors. If the SLM is statistically non-inferior, the LLM becomes purely an offline tool — a data generation engine that operates once or periodically — while the SLM becomes the authoritative labeler at deployment. This shifts the LLM from a runtime dependency (every labeling decision requires an expensive API call) to a development-time investment (amortized over all subsequent SLM inferences).

The mechanism enabling this inversion matters: the paper's pipeline (Section 3.2) generates training data that is better calibrated to the enterprise domain than what the LLM produces zero-shot at inference time. The template-guided query generation ensures structural fidelity to real enterprise query patterns. The BM25 negative mining creates a graded relevance spectrum that teaches fine-grained discrimination. The LLM labeling with quality-control filtering removes contradictory signals. The SLM, trained on this curated, domain-aligned data, can outperform the LLM acting as a general-purpose labeler at inference time — not because the SLM is inherently more capable, but because the training data pipeline compensates for the LLM's lack of enterprise-specific calibration at inference time. The paper's qualitative analysis (Section 4) provides a concrete example: "the original SLM and even GPT-4o tend to avoid assigning the lowest relevance score, even for documents that are completely off-topic, leading to overly optimistic relevance judgments." The fine-tuned SLM corrects this calibration error — it "is able to correctly assign a score of 0 when a passage is entirely irrelevant, thereby aligning more closely with human annotations." This suggests the LLM's zero-shot judgments are biased (likely a positivity/helpfulness bias similar to what the paper diagnosed in the SLM query generator), and the fine-tuning pipeline de-biases them.

The significance of this inversion extends beyond enterprise search. It suggests a general principle: when a task requires domain-specific calibration that a general-purpose LLM lacks at inference time, a domain-curated synthetic data pipeline can produce an SLM that outperforms the LLM used as a zero-shot judge. The LLM's role transitions from judge to teacher, and the training data pipeline serves as the calibration mechanism. This principle likely applies to any domain where relevance, quality, or correctness judgments depend on domain-specific norms that a general-purpose LLM does not fully internalize — legal document review, medical coding, financial compliance checking, and so on.

---

Innovation 2: BM25 as a Relevance Spectrum Generator Rather Than a Retriever

The paper repurposes BM25 in a way that is conceptually novel within the relevance labeling literature: not as a retrieval method to find relevant documents, but as a lexical probe that generates a natural, unlabeled relevance spectrum for downstream annotation. This is a fundamentally different use case from BM25's traditional role, and it solves a specific data construction problem that prior synthetic data pipelines did not address.

In standard IR pipelines, BM25 is used to retrieve documents that are likely relevant to a query — it serves as a first-stage ranker whose output is then re-ranked or judged. In synthetic data generation for retrieval (InPars, Promptagator, DUQGen), the focus is on generating positive query–document pairs through LLM-based query generation from documents; negatives are typically sampled randomly from the corpus or taken from the top-ranked results of a separate retriever. These approaches implicitly treat "relevance" as a binary or near-binary property: the generated positive is relevant, and everything else serves as background negatives whose exact degree of irrelevance is not modeled.

The paper's insight is that for graded relevance labeling on a 0–4 scale, you need a training distribution that covers all relevance levels — not just clear positives and clear negatives, but the ambiguous middle where partial relevance, topical adjacency, and near-miss patterns live. Simply pairing each synthetic query with its source document (score 4) and a set of random negatives (score 0) would produce a training set that teaches the model only to distinguish the obvious from the obviously-not, missing the difficult boundary cases where domain expertise matters most.

BM25, because it is purely lexical, naturally produces exactly this spectrum. Given a synthetic query like "Lisa budget report," BM25 retrieves documents that share vocabulary: some by the right author on the right topic (relevant), some by the right author on a different topic (partially relevant due to author match), some on the right topic by a different author (partially relevant due to content match), and some sharing only incidental terms (irrelevant). The paper notes (Section 3.2) that choosing k = 4 "empirically yields a uniform distribution of relevance labels across levels 0–4 after LLM-based annotation." That uniform distribution is not an accident — it means BM25, by design, is retrieving documents that span the full relevance scale without any prior labeling or explicit negative mining strategy.

What is intellectually distinctive here is the reversal of causal direction: in standard IR, you use a retriever to find relevant documents. In this paper, you use BM25's failure modes — its tendency to retrieve lexically similar but semantically irrelevant documents — as a feature that generates the ambiguous, boundary-case training examples needed for graded relevance discrimination. The harder BM25's job (i.e., the more enterprise queries rely on metadata and entity resolution rather than simple keyword matching), the better the training distribution it produces, because the gap between lexical match and true relevance widens, creating more informative training examples.

This is a small but fundamental reframing: BM25 is not being evaluated for retrieval quality; it is being evaluated for how well it spans the relevance distribution. The fact that BM25's top-4 results are not all relevant is not a bug — it is the mechanism that generates the training data's structure. This insight is likely transferable to any domain where relevance is graded rather than binary and where lexical overlap is an imperfect but systematic proxy for relevance.

---

Innovation 3: The SLM Query Generator Failure as a Diagnostic for Task Complexity

The paper's inclusion and analysis of the failed SLM query generator approach (Section 3.1) is itself a contribution — not to method performance, but to our understanding of which sub-tasks in a synthetic data pipeline require LLM-scale capabilities and which can be handled by SLMs. Negative results that illuminate task decomposition boundaries are rare in the literature (most papers report only successes), and this one is unusually informative because the authors diagnose the failure precisely rather than simply moving on.

The SLM query generator achieved only 60.8% binary relevance accuracy, with a systematic positivity bias: it generated relevant-sounding queries even when explicitly instructed to produce negative examples. The paper attributes this to two causes: (a) positivity bias from instruction tuning (the model is optimized to be helpful and cooperative, making it resist generating "bad" outputs), and (b) limited capacity to internalize the nuanced distinction between plausible-sounding and genuinely irrelevant in enterprise contexts where relevance boundaries depend on entity resolution and metadata matching.

Why this failure is conceptually significant: it provides evidence for a capability threshold model of the LLM-SLM division of labor. The query generation task requires the model to perform adversarial reasoning (what would a query look like that seems relevant to this document but actually refers to a different entity/project/person?) and to maintain consistent intent across a complex instruction (generate diverse queries, some relevant and some irrelevant, with specific structural constraints). The SLM, with its limited capacity and positivity-biased pretraining, cannot sustain this adversarial frame — it collapses toward generating plausible, helpful-sounding outputs regardless of the instruction.

The labeling task, by contrast, requires the model to perform a more bounded form of judgment: given a query and a document, assess their relationship. This is classification, not generation; it evaluates rather than constructs; it does not require maintaining an adversarial frame. The SLM can learn this from curated examples even if it cannot generate those examples itself.

This finding matters because it provides a principled basis for deciding when distillation from LLM to SLM is likely to succeed versus when it will hit a capability wall. If the target task requires generating diverse, instruction-dependent outputs with fine-grained control over properties like relevance/irrelevance, an SLM may fundamentally struggle. If the target task involves classifying, scoring, or evaluating within a well-defined input space, SLM distillation is more promising. The paper does not state this as a general principle, but the contrast between the failed approach (Section 3.1) and the successful one (Section 3.2) strongly implies it, and the diagnosis language — "positivity bias in pretraining," "limited model capacity," "struggle to internalize nuanced instructions" — provides a vocabulary for characterizing task complexity that future work can build on.

---

Innovation 4: Quality-Control Filtering as an Implicit Teacher-Student Calibration Mechanism

The paper's post-labeling filtering procedure — re-labeling pairs where the synthetic query's source document receives a low score, and discarding queries that fail re-labeling — is easy to overlook as a mundane data-cleaning step. But at the conceptual level, it implements something more interesting: a self-consistency check that identifies and removes training examples where the teacher LLM's judgments contradict the premise under which the data was generated. This is a calibration mechanism, not just noise reduction.

The premise of the synthetic query generation step is: we generate queries from documents, so the generated query should be relevant to its source document. If GPT-4o, acting as labeler, assigns a low score to the (query, source_document) pair, one of two things has happened: either the query generation step produced a genuinely poor query (the query doesn't match the document it was derived from), or the labeling step made an error (the query is actually relevant but GPT-4o misjudged it). The re-labeling procedure disambiguates: if the low score persists, the query is genuinely bad and is retained as a negative example (teaching the SLM that some generated queries are poor, which is a valid signal); if the score changes, the original labeling was an error and the query is discarded (preventing the SLM from learning that relevant queries are irrelevant).

Why this is conceptually interesting beyond basic data cleaning: it uses the teacher LLM to check its own consistency across two different roles (generator and labeler), surfacing examples where the model's capabilities in one role don't align with its capabilities in the other. This is a lightweight form of cross-modal consistency validation — the model's generative capability (can it produce a query that matches this document?) is cross-checked against its evaluative capability (does it judge this query as matching this document?). When the two disagree, the example is either informative (a genuinely bad query) or unreliable (a labeling error), and the re-labeling step distinguishes the two cases.

The practical impact is documented indirectly: the paper notes that "out of roughly 24,000 samples, only 15 cases contained invalid or incomplete outputs" from formatting issues — but the semantic consistency filtering discards an unspecified number of additional examples (the paper doesn't report the exact count of queries that failed re-labeling). The point is that without this step, the training data would contain contradictory signals: examples where the SLM is taught that a query-document pair is irrelevant even though the query was derived from that document. The SLM, lacking the broader world knowledge to recognize this as a data error, would internalize the contradiction, likely degrading its ability to learn the correct relevance mapping.

This calibration mechanism is specific to synthetic data pipelines where the same model serves as both generator and evaluator, and where the generation premise (queries are relevant to their source documents) provides a ground-truth anchor that can be checked against the evaluator's judgments. It is a small but transferable methodological contribution: any pipeline that generates training examples from a teacher model and then uses the same teacher model to label those examples should include a cross-consistency check that identifies and resolves cases where the teacher contradicts its own generative premise.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation uses a proprietary benchmark of 923 enterprise query–document pairs annotated by trained human labelers (described in Table 2). These pairs span 228 distinct enterprise queries, with each document receiving a graded relevance judgment on a 0–4 scale. The benchmark was curated internally to reflect real enterprise search scenarios across heterogeneous document types (emails, chats, files, knowledge bases), and it is kept entirely separate from the training data — there is no overlap between documents used for synthetic data generation and those in the evaluation set.

• Base model(s). The primary model evaluated is Phi-3.5 Mini Instruct (Abdin et al., 2024), a small language model with a few billion parameters. The paper chose this model because it "represents a strong exemplar of the compact SLM class" and has demonstrated competitive performance in recent retrieval and reasoning studies (Section 2). GPT-4o serves as both the teacher LLM (providing labels during training data generation) and a baseline labeler at evaluation time. The vanilla (non-fine-tuned) Phi-3.5 Mini Instruct is evaluated as an additional baseline to measure the impact of fine-tuning.

• Metrics. The paper employs two complementary evaluation metrics, computed at the query level and then averaged across queries. Normalized Discounted Cumulative Gain (NDCG) measures ranking quality by assigning higher gains to more relevant documents and discounting them logarithmically by their rank position — placing relevant items earlier in the ranked list is rewarded more. The paper uses full NDCG (not truncated at @k), meaning all retrieved documents for each query are included in the computation, providing a comprehensive evaluation of ranking effectiveness. Pairwise Accuracy measures how consistently the model preserves the correct relative ordering of documents compared to ground-truth labels. For each query, all possible document pairs (A, B) are examined, and a pair is counted as correctly aligned if the model's predicted relative relevance (A > B, A < B, or A = B) matches the ground truth. The proportion of correctly aligned pairs is computed per query, yielding a score between 0 and 1, then averaged across queries. Table 3 provides the full computation matrix: a cell value of 1 indicates perfect agreement between model prediction and ground truth for that pairwise comparison; 0 indicates mismatch. Additionally, Requests Per Minute (RPM) is reported as a throughput metric, defined as the number of total labeling requests processed divided by elapsed time in minutes on declared hardware.

• Baselines. The evaluation compares against two baselines directly. GPT-4o (Hurst et al., 2024) serves as the primary teacher model baseline — it is the frontier LLM that provided labels during training data generation, and its zero-shot relevance judgments on the evaluation set represent the state-of-the-art in LLM-based labeling. Vanilla Phi-3.5 Mini Instruct (the non-fine-tuned base model) quantifies the performance achievable without any domain-specific training. The paper does not compare against other SLM labelers, other fine-tuning approaches (e.g., LoRA-based adaptation), or other LLM labeling methods (e.g., few-shot prompted labeling with chain-of-thought), since the central claim is about the pipeline's ability to match its own teacher rather than about general superiority over all possible labeling approaches.

• Generation budget / compute accounting. The paper does not use a "generation budget" in the sense of controlling the number of sampled outputs per query — the labeler produces a single relevance score per query–document pair. Instead, efficiency is measured through throughput (RPM) and cost per token. The fine-tuned SLM's throughput is measured at 873.33 RPM on a single A100 GPU, extrapolating to nearly 7,000 RPM on an 8-GPU cluster. Cost is compared using OpenAI API pricing: for the fine-tuned Phi-3.5 Mini, $0.13 per 1M input tokens and$0.52 per 1M output tokens; for GPT-4o, $2.50 per 1M input tokens and$10.00 per 1M output tokens — a roughly 19× reduction on both input and output.

• Cross-validation / statistical protocol. The evaluation does not use cross-validation for model selection (the training and evaluation datasets are completely separate, and there is no early stopping or feedback loop from the evaluation set during training). For statistical significance testing, the paper conducts a one-sided paired non-inferiority test using the Wilcoxon signed-rank test (Wilcoxon, 1949). The null hypothesis is that the fine-tuned labeler performs worse than GPT-4o by more than a predefined non-inferiority margin Δ, i.e., H₀: E(δ_q) ≤ -Δ where δ_q = labeler_q - GPT-4o_q is the per-query performance difference. The paper adopts conservative non-inferiority margins of Δ = 0.1% for pairwise accuracy and Δ = 0.0001 for NDCG, which represent negligible differences relative to the scale of the respective metrics. The test is conducted across the 228 distinct queries in the evaluation set, and p-values below 0.05 are considered statistically significant evidence that the fine-tuned labeler is non-inferior to GPT-4o.

---

Main Quantitative Results

Overall Fine-Tuning Impact: SLM Matches and Marginally Exceeds GPT-4o

The headline result appears in Figure 4, which compares three labelers — vanilla Phi-3.5 Mini Instruct, fine-tuned Phi-3.5 Mini Instruct, and GPT-4o — against human ground-truth labels on the 923-pair enterprise evaluation benchmark. The fine-tuned SLM achieves an SLM–Human NDCG of 0.953, compared to 0.944 for GPT-4o and 0.815 for the vanilla SLM. On pairwise accuracy, the fine-tuned SLM reaches 63.81, compared to 62.58 for GPT-4o and 42.15 for the vanilla model.

These numbers represent the core empirical claim of the paper. The fine-tuned SLM not only closes the gap with the teacher LLM — it slightly exceeds it on both metrics. The NDCG improvement over GPT-4o is approximately 0.009 (less than 1% relative), while the pairwise accuracy improvement is approximately 1.23 percentage points (roughly 2% relative). These are not large margins, and the paper does not claim the SLM is substantially better than GPT-4o — rather, the statistical non-inferiority analysis demonstrates that the SLM is at least as good as GPT-4o within negligible margins.

The improvement from vanilla to fine-tuned SLM is dramatic: NDCG increases from 0.815 to 0.953 (a 16.9% relative improvement), and pairwise accuracy jumps from 42.15 to 63.81 (a 51.4% relative improvement). This confirms that the synthetic data pipeline provides the training signal needed to transform the SLM from a model that is essentially guessing (42% pairwise accuracy is not far from random on a 5-class ordinal scale where ties exist) to one that reliably aligns with human judgments.

The paper also reports two additional pairwise comparisons in Table 4, though these are discussed more fully in the ablation analysis:

• SLM–LLM NDCG/Accuracy: Comparing the fine-tuned SLM's labels against GPT-4o's labels (rather than human labels) yields NDCG of 0.951 and accuracy of 66.16 (Table 4, row 2). This measures how well the SLM reproduces its teacher's judgments — the high NDCG indicates strong agreement, while the accuracy of 66.16 (higher than SLM–Human accuracy of 63.81) suggests that the SLM and GPT-4o agree with each other more than either agrees with humans, likely because both models share some systematic biases that humans do not.

• LLM–Human NDCG/Accuracy: GPT-4o achieves 0.944 NDCG and 62.58 accuracy against human labels (Table 4, row 1, rightmost columns). This serves as the upper bound that the SLM aims to match or exceed, and it also establishes that even a frontier LLM does not achieve perfect agreement with human relevance judgments in enterprise search — there remains a substantial gap between LLM labels and human labels.

Ablation Studies: Data Size, Quality, and Multi-Task Tuning Contributions

The paper's ablation results are presented in Table 4, which systematically varies three factors: the presence or absence of multi-task tuning, the inclusion or exclusion of the query refinement step, and the size of the synthetic training dataset (0, 14K, or 24K examples). The findings are organized around four key comparisons:

Data scaling from 14K to 24K yields diminishing returns. With multi-task tuning and query refinement enabled (rows 2 vs. 3), expanding the synthetic dataset from 14K to 24K examples changes NDCG from 0.953 to 0.954 and pairwise accuracy from 63.81 to 63.55 — essentially flat, with accuracy actually decreasing slightly. Without multi-task tuning (rows 4 vs. 5), the same expansion changes NDCG from 0.946 to 0.943 (a slight decrease) and accuracy from 62.41 to 62.50 (a negligible gain). The paper interprets this as evidence that "beyond a certain threshold additional synthetic data offers diminishing returns" and that 14K carefully generated examples are sufficient for the SLM to learn the enterprise relevance labeling task.

Multi-task tuning provides a modest but consistent benefit. Comparing the model trained with multi-task tuning against the same model trained without it, holding dataset size and refinement constant (row 2 vs. row 4: 14K with refinement), NDCG improves from 0.946 to 0.953 (a 0.74% relative gain) and accuracy from 62.41 to 63.81 (a 2.2% relative gain). The same pattern holds at 24K (row 3 vs. row 5): NDCG improves from 0.945 to 0.954 and accuracy from 62.70 to 63.55. The gains are small enough that the paper characterizes multi-task tuning as providing "an auxiliary benefit" rather than being essential, but the consistency of the improvement across both dataset sizes suggests the effect is real.

Query refinement has a larger impact than doubling the dataset size. With multi-task tuning disabled, the model trained on 14K refined queries (row 4) achieves NDCG 0.946 and accuracy 62.41. The model trained on 14K raw (unrefined) queries (row 6) achieves NDCG 0.943 and accuracy 60.97 — a drop of 1.44 accuracy points. Compare this to the gain from doubling the dataset from 14K to 24K without multi-task tuning (row 4 vs. row 5): accuracy changes from 62.41 to 62.50, a gain of only 0.09 points. The paper's interpretation is that "improving data quality has a greater impact than simply adding more data, once the dataset surpasses a reasonably large threshold." The refinement step contributes more to model quality than an additional 10,000 training examples do.

Synthetic enterprise queries are the primary training signal; public data alone is insufficient. Row 7 of Table 4 shows the model trained with multi-task tuning and all public datasets (INTERS, TREC-CAsT, MS MARCO with synthetic queries), but with zero synthetic enterprise queries. The results are nearly identical to the vanilla SLM: NDCG improves only from 0.815 to 0.826, and accuracy from 42.15 to 42.88. This is a critical finding: it demonstrates that the public datasets, despite containing relevance-labeled query–document pairs on a 0–4 scale, teach almost nothing about enterprise search relevance labeling. The authors state this "aligns with our expectation: existing open-source datasets are primarily designed for web search query understanding and thus do not solely capture the characteristics of enterprise search relevance labeling." The synthetic enterprise queries generated through the pipeline provide essentially all of the domain-specific signal; multi-task tuning and public data contribute only small auxiliary improvements on top of that signal.

Throughput and Cost Efficiency

The throughput measurement is reported in Section 4, "Throughput Analysis": the fine-tuned SLM achieves 873.33 RPM on a single A100 GPU, which the paper extrapolates to nearly 7,000 RPM on a cluster of eight A100s. For comparison, the paper states that "typical LLM labelers" are "generally limited to the hundreds of RPM range," meaning the SLM provides roughly an order of magnitude higher throughput on equivalent hardware. The paper does not report GPT-4o's exact RPM for direct comparison, but the order-of-magnitude framing is sufficient to establish the practical advantage: labeling 100,000 query–document pairs would take the SLM roughly 1.9 hours on a single A100 versus potentially a day or more using an LLM.

The cost analysis (Section 4, "Cost Analysis") uses OpenAI API pricing as of the paper's writing: the fine-tuned Phi-3.5 Mini costs $0.13 per 1M input tokens and$0.52 per 1M output tokens, compared to GPT-4o's $2.50 per 1M input tokens and$10.00 per 1M output tokens. This represents a ~19× reduction on both input and output tokens. The paper notes that this cost efficiency, combined with the strong alignment to human judgments, "makes trained SLM an attractive option for scalable enterprise relevance labeling." For a production pipeline processing millions of query–document pairs per month, a 19× cost difference is the difference between a labeling budget measured in hundreds of dollars versus thousands of dollars — or, at larger scales, between feasible and infeasible.

Importantly, the paper does not account for the one-time cost of generating the synthetic training data (GPT-4o inference for query generation, refinement, and labeling across 14K–24K examples). This is a reasonable omission for a deployment-focused evaluation (the training data generation cost is amortized over all subsequent SLM inferences), but it means the 19× cost reduction applies only at inference time. A full total-cost-of-ownership analysis would need to factor in how frequently the training data pipeline must be re-run (e.g., when new document types or query patterns emerge).

Statistical Non-Inferiority to GPT-4o

The statistical significance analysis (Section 4, "Statistical Significance Analysis") tests whether the fine-tuned SLM is non-inferior to GPT-4o within small, pre-specified margins. Using the 228 distinct queries in the evaluation set, the per-query performance difference δ_q = labeler_q - GPT-4o_q is computed for both pairwise accuracy and NDCG.

For pairwise accuracy, with a non-inferiority margin of Δ = 0.1%, the one-sided Wilcoxon signed-rank test yields a p-value of 0.012. For NDCG, with a margin of Δ = 0.0001, the p-value is 0.00098. Both are below the 0.05 significance threshold, meaning the null hypothesis (that the SLM is worse than GPT-4o by more than the specified margin) is rejected for both metrics. The paper interprets this as providing "statistical evidence, from both pairwise accuracy and NDCG, that our fine-tuned model achieves performance comparable to the strong GPT-4o labeler."

The choice of non-inferiority margins deserves scrutiny. For pairwise accuracy, Δ = 0.1% means the test checks whether the SLM's accuracy is within 0.1 percentage points of GPT-4o's — an extremely tight margin. For NDCG, Δ = 0.0001 means the test checks whether the SLM's NDCG is within 0.0001 NDCG units of GPT-4o's — also an extremely tight margin. The fact that the SLM passes non-inferiority tests with such small margins is a strong result: it means the SLM is not just "close enough for practical purposes" but genuinely indistinguishable from GPT-4o in terms of alignment with human judgments, given the statistical power available from 228 queries.

However, it is worth noting that non-inferiority does not imply superiority. The SLM's point estimates (0.953 NDCG, 63.81 accuracy) are slightly higher than GPT-4o's (0.944, 62.58), but the statistical test is designed to show that the SLM is not meaningfully worse, not that it is meaningfully better. A superiority test would be needed to claim the SLM outperforms GPT-4o, and the small margins (0.009 NDCG, 1.23 accuracy points) are unlikely to reach significance given the sample size. The paper appropriately frames the result as "non-inferior" and "comparable" rather than superior.

---

Ablation Studies and Robustness Checks

Multi-task tuning (Table 4, rows 2/4 and 3/5): Adding multi-task pre-training on INTERS before instruction fine-tuning improves NDCG by 0.007–0.009 and pairwise accuracy by 0.85–1.40 percentage points, depending on dataset size. The effect is consistent but modest — the paper characterizes it as an "auxiliary benefit" that "improves robustness" rather than a primary performance driver. This is an expected finding: multi-task tuning provides general query-document understanding capabilities that complement the enterprise-specific signal, but the enterprise signal is what teaches the model the domain-specific relevance criteria.

Query refinement (Table 4, row 4 vs. row 6): Removing the GPT-4o-based query refinement step (which diversifies phrasing and ensures each generated query is structurally varied) reduces pairwise accuracy from 62.41 to 60.97, a drop of 1.44 percentage points. This is larger than the gain from adding multi-task tuning (which added 1.40 points in the equivalent comparison) and substantially larger than the effect of doubling the dataset size (0.09 point accuracy gain from 14K to 24K without multi-task tuning). The paper concludes that "improving data quality has a greater impact than simply adding more data," a finding that has practical implications for practitioners who might be tempted to scale data volume rather than invest in data quality.

Training dataset size (Table 4, rows 2/3 and 4/5): Expanding the synthetic dataset from 14K to 24K examples produces negligible or slightly negative changes across all configurations. With multi-task tuning and refinement, NDCG changes from 0.953 to 0.954 and accuracy from 63.81 to 63.55. Without multi-task tuning, NDCG changes from 0.946 to 0.945 and accuracy from 62.41 to 62.50. The paper identifies 14K examples as a "reasonably large threshold" beyond which additional data provides diminishing returns. This is a practically useful finding — it suggests that the pipeline does not need to be run at massive scale to achieve strong results, reducing the upfront LLM inference cost for data generation.

Public data only, no synthetic enterprise queries (Table 4, row 7): This is the most informative ablation. When the model is trained with multi-task tuning plus all public datasets (INTERS, TREC-CAsT, MS MARCO) but zero synthetic enterprise queries, the results are nearly indistinguishable from the vanilla SLM: NDCG is 0.826 (up from 0.815) and accuracy is 42.88 (up from 42.15). The gain is approximately 1.3% relative in NDCG and 1.7% relative in accuracy — essentially no learning. This confirms that the public datasets, despite containing relevance-labeled query–document pairs on the same 0–4 scale, do not transfer to the enterprise domain. The enterprise-specific synthetic queries generated through the pipeline are the essential training signal; everything else provides only marginal auxiliary benefits.

SLM–LLM agreement (Table 4, middle column group): Across all configurations, the SLM–LLM NDCG (SLM labels compared to GPT-4o labels) is consistently high — ranging from 0.948 to 0.951 for the trained models — and the SLM–LLM pairwise accuracy ranges from 66.05 to 66.83. These numbers are notably higher than the SLM–Human accuracy (60.97–63.81), indicating that the trained SLM agrees with its teacher more than either agrees with humans. This is the expected signature of distillation: the student model learns to reproduce the teacher's judgments, including the teacher's systematic biases that differ from human judgments. The gap between SLM–LLM accuracy (~66%) and SLM–Human accuracy (~63%) quantifies the degree to which GPT-4o's judgments deviate from human annotations — about 3 percentage points of the disagreement is attributable to teacher bias rather than student error.

Qualitative analysis of calibration improvement (Section 4, "Qualitative Analysis"): The paper reports that the vanilla SLM and GPT-4o both exhibit a reluctance to assign the lowest relevance score (0), even for documents that are completely off-topic. The fine-tuned SLM, by contrast, "demonstrates greater calibration and confidence: it is able to correctly assign a score of 0 when a passage is entirely irrelevant, thereby aligning more closely with human annotations." This is not quantified with a distributional analysis (the paper does not report the frequency of score-0 predictions for each model), but it provides a qualitative explanation for why the fine-tuned SLM can outperform GPT-4o despite being distilled from GPT-4o's labels: the training pipeline (particularly the BM25 negative mining and the quality-control filtering) produces a training distribution with a balanced representation of all relevance levels, which corrects the teacher's tendency toward positivity bias. The LLM-as-teacher provides the labels, but the data curation pipeline provides the calibration.

The failed SLM query generator (Section 3.1): Although not an ablation of the final system, the negative result from the SLM query generator experiment provides an important robustness insight. The fine-tuned SLM query generator achieved only 60.8% binary relevance accuracy, with a systematic bias toward generating positive-sounding queries. This failure establishes that the pipeline design choice to use GPT-4o for query generation (rather than an SLM) is not arbitrary — the SLM lacks the capacity to perform adversarial query generation (generating diverse queries across the full relevance spectrum, including genuinely irrelevant ones that still appear plausible). This justifies the division of labor in the final pipeline: LLMs for generation, SLMs for scoring.

---

Critical Assessment

Does the paper demonstrate that the fine-tuned SLM achieves agreement with human judgments "on par with or better than" GPT-4o?

The quantitative evidence supports this claim with important qualifications. The point estimates (NDCG 0.953 vs. 0.944, accuracy 63.81 vs. 62.58) favor the SLM marginally, and the non-inferiority test with tight margins (Δ = 0.1% for accuracy, Δ = 0.0001 for NDCG) rejects the hypothesis that the SLM is meaningfully worse than GPT-4o (p = 0.012 and p = 0.00098). This is strong evidence of non-inferiority. However, the margins are extremely small — a 0.009 NDCG difference and a 1.23 percentage point accuracy difference — and the paper does not conduct a superiority test. The appropriate interpretation is that the SLM is statistically indistinguishable from GPT-4o on these metrics, not that it is demonstrably better. The "on par with" portion of the claim is well-supported; the "or better than" portion is a point-estimate observation that lacks statistical significance testing.

A more subtle issue: the evaluation is conducted on 923 query–document pairs across 228 queries, all drawn from the same enterprise domain for which the synthetic training data was generated. This tests the SLM's ability to label queries and documents that are in-distribution with respect to the training data generation process. It does not test whether the SLM generalizes to new query patterns, new document types, or new enterprise domains that were not represented in the 1,500 seed documents. The claim should be understood as "on par with GPT-4o on enterprise queries and documents drawn from the same distribution as the training seed documents," which is the practically relevant case for a deployed system but is narrower than an unqualified "on par with GPT-4o for enterprise relevance labeling."

Does the paper demonstrate a 17× throughput increase and 19× cost reduction?

The throughput measurement of 873.33 RPM on a single A100 is clearly reported, and the paper frames the comparison as "an order of magnitude faster than typical LLM labelers, which are generally limited to the hundreds of RPM range." However, the paper does not report GPT-4o's actual RPM under the same evaluation conditions. The 17× figure appears in the abstract and executive summary but is not derived from a side-by-side measurement — it appears to be based on the extrapolated 7,000 RPM on 8 A100s compared against an unstated LLM baseline RPM. This makes the 17× claim somewhat imprecise. The order-of-magnitude advantage is well-established by the SLM throughput number alone, but the specific multiplier should be treated as approximate.

The 19× cost reduction is more concretely grounded, using publicly available API pricing (GPT-4o at $2.50/$10.00 per 1M input/output tokens vs. the fine-tuned Phi-3.5 Mini at $0.13/$0.52). However, this pricing comparison is between a hosted API service (GPT-4o) and a self-hosted model (the fine-tuned SLM running on A100s). The GPT-4o API price includes OpenAI's infrastructure, profit margin, and operational costs, while the SLM cost appears to be based on the fine-tuning API pricing for Phi-3.5 Mini, which is also a hosted service. If the SLM is deployed on owned hardware, the marginal inference cost would be even lower (effectively electricity and hardware depreciation), making the 19× figure conservative. If the SLM is accessed through the same fine-tuning API, the pricing comparison is fair but depends on the specific pricing tiers available at the time. The cost advantage is real and substantial, but the exact multiplier depends on deployment architecture.

Does the paper demonstrate that the synthetic data pipeline is the essential driver of performance?

Yes, and this is the strongest evidential claim in the paper. Table 4, row 7 shows unequivocally that without the synthetic enterprise queries, the model achieves essentially the same performance as the vanilla SLM (NDCG 0.826 vs. 0.815, accuracy 42.88 vs. 42.15). The public datasets (INTERS, TREC-CAsT, MS MARCO) provide negligible benefit alone. This is a clean ablation that isolates the contribution of the pipeline's output and demonstrates that it is not merely helpful but necessary for learning enterprise relevance labeling.

Is the evaluation missing important comparisons or ablations?

Several gaps are notable:

No comparison against fine-tuning GPT-4o itself. If the goal is to demonstrate that an SLM can match GPT-4o's labeling quality, a natural question is whether fine-tuning GPT-4o on the same synthetic data would produce an even stronger labeler, against which the SLM would then be compared. The paper implicitly assumes that fine-tuning GPT-4o is impractical (due to cost, API limitations, or deployment constraints), but this is not stated explicitly. If fine-tuned GPT-4o substantially outperforms both zero-shot GPT-4o and the fine-tuned SLM, then the SLM's non-inferiority to zero-shot GPT-4o is less compelling — it tells us the SLM matches a baseline that could itself be improved.

No comparison against other fine-tuning methods. The paper uses full fine-tuning (all parameters updated) of Phi-3.5 Mini. Parameter-efficient methods like LoRA (Hu et al., 2022) or QLoRA could potentially achieve similar performance with lower training cost and smaller storage footprint. The paper does not compare against these alternatives, leaving open the question of whether the 2-epoch full fine-tuning on 8 A100s is necessary or whether lighter-weight adaptation would suffice.

No analysis of performance by query type. The paper distinguishes between keyword-based queries (metadata-driven patterns) and semantic queries (natural language questions) in its motivation (Section 1, Table 1), but the evaluation results are not broken out by query type. It would be informative to know whether the fine-tuned SLM's advantage over GPT-4o is concentrated in keyword-based queries (where the template-guided training data provides domain-specific patterns that GPT-4o lacks) and whether it lags on purely semantic queries (where GPT-4o's broader training might provide an advantage). The paper includes web search data in training to address semantic queries, but does not evaluate whether this strategy actually closes the semantic gap.

No analysis of performance by relevance level. The qualitative analysis notes that GPT-4o and the vanilla SLM avoid assigning score 0, but there is no quantitative breakdown of per-class accuracy, confusion matrices, or calibration curves. Such an analysis would reveal whether the fine-tuned SLM's improvement is concentrated in better discrimination of the lowest relevance levels (as the qualitative analysis suggests) or is uniform across the scale.

No analysis of the 15 filtered cases or the number of queries discarded by quality control. The paper reports that only 15 out of ~24,000 samples had formatting issues, but does not report how many synthetic queries were discarded by the semantic consistency filter (the step where queries whose source documents received low relevance scores were re-labeled and potentially removed). Knowing the discard rate would help assess the pipeline's efficiency and the teacher LLM's self-consistency.

The evaluation set is small for statistical power on subgroup analyses. With 923 query–document pairs across 228 queries, the average query has approximately 4 associated documents. This limits the granularity of per-query analysis — many queries may have only one or two documents in the evaluation set, making per-query NDCG unreliable. The paper reports that significance testing is conducted at the query level (228 observations), which is appropriate but means that any query-type or difficulty-level breakdown would have very small sample sizes. The paper does not attempt such breakdowns, which is a reasonable choice given the data constraints, but it means that claims about the model's behavior on different query types remain unsupported.

What experiments would have strengthened the paper?

A human–GPT-4o agreement baseline for context. The paper reports GPT-4o–Human NDCG of 0.944 and accuracy of 62.58, but does not report human–human agreement (inter-annotator agreement among the trained human labelers). If human labelers agree with each other at, say, 0.96 NDCG and 70% pairwise accuracy, then the gap between GPT-4o and humans (0.944, 62.58) represents the ceiling for what an automated labeler can achieve, and the SLM's 0.953 NDCG likely reflects noise rather than genuine superiority. If human agreement is lower, then both GPT-4o and the SLM are approaching or exceeding the practical ceiling of label consistency. This context would substantially improve the interpretation of the headline numbers.

A deployment-scale evaluation. The paper demonstrates strong performance on a fixed 923-pair benchmark, but a key claim is that the SLM enables "production-scale offline ranking evaluation." A real deployment-scale test — labeling, say, 100,000 unseen query–document pairs and comparing system-level ranking metrics (NDCG of the search system evaluated using SLM labels vs. using human labels) against the same metrics computed with GPT-4o labels — would be much stronger evidence of practical utility. The current evaluation shows that the SLM's per-pair labels align with humans, but does not demonstrate that using the SLM as an evaluation tool leads to the same conclusions about ranking system quality as using human labels would.

A test of temporal robustness. Enterprise document collections and query patterns evolve over time (new projects, new people, new document types). A useful property of a relevance labeler is robustness to such distribution shift. The paper does not test whether the SLM trained on synthetic data from seed documents maintains its performance when new documents and query patterns appear that were not represented in the training data. This is a practical concern for any deployed system and would require either periodic re-running of the synthetic data pipeline or an evaluation of the model's ability to generalize to out-of-distribution enterprise content.

An analysis of the BM25 retrieval composition. The paper states that k = 4 "empirically yields a uniform distribution of relevance labels across levels 0–4," but does not show this distribution or report what fraction of the BM25-retrieved documents fall into each relevance level after LLM labeling. Providing this breakdown would strengthen the claim that BM25 naturally produces a useful relevance spectrum, and would help practitioners calibrate k for their own domains.

Where do the claims hold conditionally?

The claims hold under the following conditions, which are mostly met by the paper's experimental design but should be made explicit:

• The enterprise domain is represented in the seed documents. The synthetic data pipeline requires 1,500 curated enterprise documents with rich metadata. If the seed documents do not cover the query patterns, metadata structures, or content types that appear in deployment, the SLM will not learn to handle those cases. The paper uses an "eyes-on" reviewed internal document set; the generalizability to other enterprise environments depends on the representativeness of those 1,500 documents.

• A reasonable query pattern template table can be constructed. The paper uses an internal template table derived from real query traffic analysis. The authors note that "synthetic template tables can be generated by systematically enumerating combinations of document metadata fields and keywords," and state that this produces "realistic and diverse query structures." However, this claim is not empirically validated — the paper does not compare SLM performance when trained with real-traffic-derived templates versus systematically-enumerated templates. If the template table must be derived from real query logs to achieve the reported performance, the approach is less applicable to organizations that lack query log data entirely.

• The teacher LLM (GPT-4o) produces reasonably calibrated relevance labels. The entire pipeline depends on GPT-4o generating useful labels for the synthetic query–document pairs. If GPT-4o's labeling quality degrades for certain query types, document types, or relevance levels, those errors propagate into the SLM's training data. The paper mitigates this with quality-control filtering, but does not independently validate GPT-4o's labeling quality against human judgments for the synthetic training pairs — the only GPT-4o–Human comparison is on the separate evaluation benchmark. The assumption that GPT-4o labels the training data as accurately as it labels the evaluation data is plausible but untested.

• The enterprise search task can be adequately captured by a 0–4 graded relevance scale. The paper adopts this standard IR scale, and the evaluation uses human labels on the same scale. If enterprise relevance is more nuanced (e.g., depending on user role, time sensitivity, or task context in ways that a single ordinal score cannot capture), then even perfect agreement with human labels on the 0–4 scale may not translate to improved search quality for actual users. This is a limitation of the relevance labeling paradigm itself, not specific to this paper, but it bounds the practical significance of the results.

• The 14K training examples represent the diversity of enterprise queries and documents sufficiently. The paper shows that expanding from 14K to 24K examples yields diminishing returns, suggesting that 14K covers the relevant patterns. However, this conclusion is specific to the 1,500 seed documents and their associated query patterns. A larger or more diverse document collection might require proportionally more training data. The paper does not test whether the 14K saturation point generalizes or is specific to the particular seed document set used.

Overall, the experimental analysis is thorough for a systems paper focused on practical deployment. The central claims — that the fine-tuned SLM matches GPT-4o labeling quality, that the synthetic data pipeline is essential, and that the SLM provides substantial throughput and cost advantages — are supported by the evidence presented. The most significant limitations are the single-domain evaluation (one enterprise, one set of seed documents, one model family), the absence of human inter-annotator agreement baselines, and the lack of deployment-scale validation beyond the 923-pair benchmark. These limitations are common in industry systems papers and do not undermine the paper's practical contribution, but they should be considered when assessing the generalizability of the approach to other enterprise search environments.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted For and May Dominate Deployment Economics

The assumption or constraint. The entire compute-optimal framework depends on estimating each prompt's difficulty before allocating the test-time compute budget. The paper's method for doing so—generating 2048 samples per question and averaging either ground-truth pass@1 (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference… our experiments do not account for this cost largely for simplicity"

The consequence. In any realistic deployment, the total cost is difficulty estimation plus strategy execution. The difficulty estimation step (2048 samples per question) actually exceeds the test-time compute budgets studied (256–512 generations) by a factor of 4–8×. This fundamentally undermines the paper's central claim of 4× efficiency gains, since those gains are computed after difficulty is already known, without amortizing the cost of learning it. A practitioner implementing this method would find that the apparent savings evaporate once the difficulty estimation overhead is included—unless difficulty can be predicted far more cheaply. For batch evaluation where many questions are scored, the amortization could work (pay the estimation cost once per question, then use compute-optimal allocation for many samples). But for single-use inference, the overhead makes the approach strictly worse than simply spending the equivalent budget on best-of-N.

What evidence exists. The paper provides no measurement of the total end-to-end cost including difficulty estimation. The 2048-sample procedure is described in Section 3.2, and the generation budgets in Figures 3, 4, 6, 7, and 8 are reported without the difficulty estimation cost factored in. The "4× efficiency gain" claim in the abstract and Section 1 simply compares the compute-optimal curve against the best-of-N curve at equivalent accuracy, ignoring the upfront cost.

Mitigation status. The paper explicitly flags this as "a key avenue for future work" (Section 3.2) and suggests training models to predict difficulty directly from the question text. However, no such model is developed or evaluated. The predicted difficulty bins (Section 3.2) reduce the dependency from ground-truth labels to PRM scores, but still require generating 2048 samples and scoring them—addressing only the oracle vs. deployable gap, not the cost gap. A dynamic or coarse difficulty estimation scheme (e.g., using a small number of initial samples to estimate difficulty before allocating the remaining budget) is suggested but not implemented. Until cheap difficulty estimation is demonstrated, the headline efficiency numbers should be treated as upper bounds on potential gains rather than realized deployment improvements.

Hard Problems Remain Essentially Unsolved—The Method Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The entire framework assumes that the base model can produce correct solutions at some non-trivial rate. All test-time strategies—search, revisions, and their compute-optimal combinations—operate by finding or refining solutions that the model already generates. They amplify existing capability; they do not create it from scratch. The paper explicitly notes this as a boundary condition in Section 7 (the FLOPs-matched comparison) and in the discussion of difficulty bin 5, where "no method makes meaningful progress."

The consequence. On problems where the base model's pass@1 is near zero (difficulty bin 5—the hardest quintile of MATH questions), no amount of test-time compute provides any benefit. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets (4, 16, 64, 256 generations). In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio at 128 generations. The compute-optimal policy cannot assign a strategy that solves these problems because no strategy works. For a practitioner, this means the method has a hard ceiling: if your problem distribution includes a substantial fraction of genuinely hard questions (those outside the base model's capability range), test-time compute scaling will not help—you need a more capable base model, and pretraining remains the only viable path.

What evidence exists. The difficulty-bin analyses in Figures 3 (right) and 7 (right) clearly show the flat, near-zero performance on bin 5 for all methods and all budgets. The FLOPs-matched comparison in Section 7 and Figure 9 provides the strongest evidence: on the hardest questions (bin 5), the 14× larger model substantially outperforms the smaller model with compute-optimal test-time scaling across all values of R = D_inference / D_pretrain. The paper's own takeaway box in Section 7 states this explicitly:

"On the hardest questions, pretraining is almost always more effective. Test-time compute provides minimal gains on problems that are fundamentally outside the base model's capability range."

The quantitative gap is stark: in the FLOPs-matched comparison with PRM search at R ≫ 1, hard questions show a −52.9% relative disadvantage for test-time compute compared to the larger model.

Mitigation status. None. This is not a "limitation" that can be fixed by improving the method—it is a fundamental property of test-time compute scaling. The paper is transparent about this boundary, and the difficulty estimation mechanism (Section 3.2) implicitly serves as a routing mechanism: easy-to-medium questions get test-time compute, hard questions would ideally be routed to a larger model or flagged for human review. But the paper does not implement or evaluate such a routing system. A practitioner deploying this method needs some external mechanism to handle the hard-question regime—either accepting near-zero performance on those questions, or investing in a fallback pipeline (larger model, human review) with the associated cost.

Verifier Over-Optimization Limits Scaling and Is Not Solved by Compute-Optimal Allocation

The assumption or constraint. All search-based methods (beam search, lookahead search, best-of-N weighted) depend on a Process Reward Model (PRM) to score solution steps or complete solutions. The PRM is an imperfect proxy for correctness—trained via Monte Carlo rollouts (Section 5.1, Appendix D) rather than ground-truth correctness—and its scores can be exploited by aggressive search. The PRM's training data distribution (solutions generated by the base model via standard sampling) differs from the distribution of solutions encountered during search (which are selected by the PRM itself), creating a distribution shift that can cause over-optimization.

The consequence. The paper documents concrete failure modes of verifier over-optimization in Section 5.3 and Appendix M:

• Beam search degrades performance on easy problems at high budgets. In Figure 3 (right, bin 1), beam search accuracy decreases from roughly 78% at 4 generations to 77% at 256 generations, while best-of-N weighted increases from 68% to 88%. This is attributed to the PRM being exploited—search finds solutions that score highly under the PRM but are actually incorrect.

• Lookahead search, the most powerful optimizer, paradoxically performs worst overall. Figure 3 (left) shows that lookahead search (both k = 1 and k = 3) generally underperforms simpler methods at equivalent generation budgets because its stronger optimization pressure amplifies verifier errors.

• Qualitative degeneracy. Appendix M (Figures 29 and surrounding discussion) shows search producing degenerate outputs—low-information repetitive steps at the end of solutions, overly short 1–2 step solutions—that score highly under the PRM but are substantively wrong.

The compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N instead of beam search), but it does not solve the underlying verifier quality problem. On medium-difficulty problems (bins 3–4) where beam search is deployed, over-optimization still limits the scaling ceiling: the beam search curves in Figure 3 (right) flatten and sometimes decline well before the budget is exhausted. The compute-optimal approach is fundamentally bounded by verifier quality, and the paper provides no mechanism for improving that quality beyond the initial Monte Carlo rollout training.

What evidence exists. The difficulty-dependent search results in Figure 3 (right) provide the primary evidence: beam search (solid line) falls below best-of-N weighted (dashed line) on easy questions (bins 1–2) at high budgets, while the opposite pattern holds on medium questions (bins 3–4). The lookahead search underperformance is documented in Figure 3 (left) across all budgets. Specific failure modes are shown in Appendix M with qualitative examples. The paper explicitly attributes these findings to "over-optimization of the PRM—search finds solutions that score highly under the PRM but are actually incorrect" (Section 5.3).

Mitigation status. The paper acknowledges the over-optimization problem (Section 5.3, Section 8) but does not attempt to solve it. The compute-optimal policy essentially works around the verifier's limitations—using weaker optimization where the verifier is fragile (easy problems) and stronger optimization only where the verifier still has room to provide guidance (medium problems). The authors suggest future work on "improving verifier robustness" and "constrained search methods that penalize solutions deviating too far from the base model's typical output distribution" (discussed in the broader implications sections), but no such improvements are evaluated. The ReST^EM experiment in Appendix K (Figure 16) reveals an additional complication: attempting to optimize the revision model with RL-style training caused performance to degrade substantially, suggesting that the relationship between training optimization and test-time optimization is complex and fragile. A practitioner relying on this approach should expect that verifier quality—not search algorithm sophistication—is the binding constraint on test-time compute scaling, and that pushing beyond moderate budgets (roughly 64–128 generations) on easy-to-medium problems risks over-optimization.

Single Benchmark, Single Model Family, Moderate-Sized Test Set

The assumption or constraint. All experiments use the MATH benchmark (Hendrycks et al., 2021) with a test set of 500 questions, evaluated exclusively on PaLM 2-S* models. The authors state in Section 4 that they "believe this model is representative of the capabilities of many contemporary LLMs," but this assumption is not tested.

The consequence. Several aspects of the findings could be domain-specific or model-specific in ways that limit generalization:

• MATH consists entirely of competition-level math problems requiring symbolic multi-step reasoning. It is unclear whether the difficulty-dependent patterns—beam search hurting easy problems, revisions helping easy problems more than hard problems—generalize to other reasoning domains (code generation, logical deduction, scientific QA) or, more critically, to tasks requiring factual knowledge or open-ended generation rather than problems with verifiable correct answers.

• PRM quality and over-optimization behavior are model-dependent. PaLM 2-S*'s output distribution—its error patterns, calibration properties, and the types of mistakes it makes—directly shapes the PRM's training data and, consequently, the search behavior. A model with different calibration or different failure modes might exhibit different optimal strategies at equivalent difficulty levels, changing the compute-optimal policy.

• The revision model's behavior depends on in-context learning capabilities. The paper's revision model is fine-tuned on trajectories of incorrect-to-correct answer sequences. The model's ability to benefit from seeing its own previous mistakes in context likely varies across model families and scales. A model with weaker in-context learning might benefit less from sequential revisions, shifting the optimal sequential-to-parallel ratio.

• The test set size limits the reliability of per-bin strategy selection. The 500-question test set, split into five difficulty quintiles (~100 questions each), then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed gains are statistically reliable at this sample size or whether the selected strategies would differ with a larger or different test set.

What evidence exists. The paper provides replication across two independent mechanisms (search and revisions) that show similar difficulty-dependent patterns, which is the strongest internal evidence for generality—the difficulty-dependence is not specific to one method. However, all results are on MATH with PaLM 2-S*. No results on other benchmarks (e.g., GSM8K, MMLU, HumanEval) or other model families (e.g., LLaMA, GPT, Gemini) are reported. The cross-validation protocol (Section 3.2) is sound for the given test set but does not address the underlying question of whether the 500-question sample is representative of MATH's difficulty distribution or whether the selected strategies would generalize to a new sample.

Mitigation status. The paper explicitly acknowledges the single-benchmark, single-model limitation (Section 8) and calls for future work to "replicate these findings on other benchmarks and model families." The authors argue that PaLM 2-S* is "representative," but this is an assertion, not a demonstrated property. A practitioner considering deploying this approach should conduct their own calibration: the difficulty bins, optimal strategies per bin, and over-optimization thresholds are likely to shift for different models and different problem domains. The general principle—difficulty-conditioned allocation improves efficiency—is well-supported, but the specific strategy lookup tables (beam search for medium problems, revisions for easy problems) may not transfer directly.

Revisions and Search Are Studied Independently, Not Combined

The assumption or constraint. The paper studies two complementary mechanisms—PRM-guided search (Section 5) and iterative revisions (Section 6)—but evaluates them separately. Section 8 explicitly acknowledges this:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The two mechanisms have complementary strengths that suggest combination would outperform either alone. Revisions modify the proposal distribution, improving the quality of generated candidates by conditioning on previous (incorrect) attempts. PRM search modifies the selection mechanism, using step-level verifier scores to guide which candidates to explore or select. A natural combined approach—using the revision model as the proposal distribution within beam search, or using the PRM to determine when to continue versus restart a revision chain—could yield gains beyond either method alone. Since the paper never evaluates such combinations, the reported results represent a lower bound on what an integrated system could achieve. A practitioner following this work would face an open question: should I invest in search, revisions, or both, and how should they interact? The paper provides strong evidence that each works individually under specific difficulty conditions, but offers no guidance on integration.

What evidence exists. The paper provides suggestive but indirect evidence for complementarity. The difficulty-bin analyses show that search is most effective on medium problems (Figure 3, right, bins 3–4) while revisions are most effective on easy problems (Figure 7, right, bins 1–2). The FLOPs-matched comparison (Figure 9) shows revisions outperforming PRM search in the R ≪ 1 regime, particularly on easy and medium problems. These patterns imply that a difficulty-adaptive system that chooses between search, revisions, or a combination might outperform either alone. But no direct combination experiment is reported.

Mitigation status. The paper acknowledges this as a future direction in Section 8 but provides no results toward it. The authors note the practical challenge: "PRM tree-search techniques in combination with revisions" would require training a PRM that works well on the revision model's output distribution, and the paper already documents (Appendix J, Figure 15a) that the base-model PRM underperforms when scoring revision-model outputs due to distribution shift. This means combination is not merely a matter of plugging components together—it would require addressing the distribution shift problem in the PRM, possibly by training a revision-specific PRM or by using the revision-specific ORM (already discussed in Section 6.1). A practitioner would need to solve this engineering challenge to realize the potential gains from combination.

Sequential Revision Latency Is Not Accounted For—Throughput and Wall-Clock Time Are Not Distinguished

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a proxy for total FLOPs but ignores latency—the wall-clock time required to produce a final answer. Sequential revisions are inherently serial: each revision depends on the previous one, so a chain of 64 sequential revisions takes 64× longer wall-clock time than 64 parallel independent samples, assuming sufficient hardware to run the parallel samples simultaneously. The paper reports throughput improvements (Section 4), but throughput in a batch-processing setting (where many independent problems can be parallelized) is not the same as latency for a single query.

The consequence. The compute-optimal policy frequently favors sequential-heavy strategies, particularly on easy problems where purely sequential revisions are optimal (Figure 7, right, bins 1–2) and at low-to-moderate budgets where fully sequential settings dominate (Figure 7, left, budgets 8–32). In a latency-sensitive application—interactive tutoring systems, real-time decision support, customer-facing chatbots—a strategy that allocates 128 generations as a single chain of 128 revisions would be completely impractical regardless of its accuracy advantages, because the user would wait 128× longer than for a single generation. The compute-optimal policy, as presented, optimizes for accuracy-per-FLOP but ignores accuracy-per-second, which is the metric that matters for interactive use.

For batch processing (e.g., evaluating thousands of MATH problems offline), latency is less critical, and throughput—the total number of solutions produced per unit time—is the relevant metric. In this setting, sequential strategies are less problematic because multiple independent chains can run in parallel across a GPU cluster. But the paper does not distinguish between these two deployment scenarios when reporting efficiency gains.

What evidence exists. The sequential-to-parallel ratio sweep (Figure 7, left) shows that at high budgets (128–256), a moderate sequential-to-parallel ratio is optimal—for example, at 256 generations, the optimal ratio is around 2^1 to 2^3 (2:1 to 8:1 sequential-to-parallel). This partially mitigates the latency concern because truly massive sequential chains are not selected at high budgets. However, at lower budgets (8–32 generations), fully sequential strategies are optimal, meaning the latency penalty applies most strongly in the regime where the total compute budget is small and users are most likely to be latency-sensitive. The paper does not provide any analysis of wall-clock time, GPU utilization, or latency under different allocation strategies.

Mitigation status. None. The paper does not discuss the latency-throughput distinction, does not report wall-clock time for any experiment, and does not consider latency constraints in the compute-optimal objective (Equation 1 in Section 3.1 optimizes only for accuracy). A practitioner deploying this method must independently decide how to trade off sequential depth against latency requirements, potentially overruling the compute-optimal policy for latency-sensitive applications. The paper provides no framework for making this tradeoff.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around enterprise information retrieval evaluation from a binary choice—expensive but high-quality LLMs versus cheap but unreliable heuristics—to a three-tier architecture where LLMs serve as data factories, SLMs serve as production-grade judges, and the pipeline connecting them becomes the object of engineering optimization. This is not a paradigm shift in the sense of introducing a new model architecture or training objective, but it is a methodologically important reframing of how the field should think about the relationship between model scale and task quality in domain-specific evaluation.

The conceptual move is this: rather than asking "can an SLM match an LLM on task X?" (the standard distillation framing, which typically answers "almost, with a quality gap"), the paper demonstrates that in the specific context of enterprise relevance labeling, the right question is "can a curated synthetic data pipeline produce training data that de-biases the SLM relative to the LLM's zero-shot errors?" The answer, from Figure 4 and the non-inferiority analysis (p = 0.012 for accuracy, p = 0.00098 for NDCG), is yes—and the mechanism is not the SLM's inherent capability but the calibration signal embedded in the training data construction process.

This finding reconciles a latent tension in the LLM-for-IR literature that the paper itself surfaces. On one hand, works like Thomas et al. (2024) and MacAvaney and Soldaini (2023) have shown that LLMs can serve as effective relevance labelers, establishing them as practical alternatives to human annotation in web search settings. On the other hand, works on SLM distillation for ranking (Fitte-Rey et al., 2025; Choi et al., 2024; Samarinas and Zamani, 2025) have consistently shown that distilled models underperform their teachers, particularly on nuanced or domain-specific judgments. The implicit message from prior work was: small models can approximate large ones, but you trade quality for efficiency. This paper breaks that tradeoff—at least for enterprise relevance labeling—by showing that the training data pipeline can correct the teacher's own calibration errors (specifically, GPT-4o's positivity bias and reluctance to assign the lowest relevance scores, documented in the qualitative analysis). The SLM doesn't merely imitate the LLM; it improves on the LLM's alignment with human judgments in a specific domain. The mechanism is the interaction between BM25 negative mining (which generates a balanced relevance spectrum) and quality-control filtering (which removes contradictory teacher labels), neither of which is present when the LLM operates as a zero-shot labeler at inference time.

The practical implication is that LLM-as-a-judge is not the ceiling for automated evaluation. There is headroom above zero-shot LLM judgments that can be captured through domain-specific synthetic data pipelines, even when the downstream labeler has dramatically fewer parameters. This matters because it means that organizations deploying enterprise search systems need not accept either the cost of human annotation, the latency and expense of LLM labeling, or the quality degradation of naive SLM distillation. They can invest in a one-time synthetic data generation pipeline—using their own seed documents, their own query pattern templates, and a teacher LLM—and deploy an SLM that matches or exceeds the teacher's quality at a fraction of the per-query cost. The 19× cost reduction and 17× throughput increase are not merely "nice to have"; they are what makes production-scale offline ranking evaluation economically viable for organizations that label millions of query–document pairs per month.

The paper also redirects research attention from model architecture to data generation methodology for domain-specific evaluation tasks. Prior work on SLM distillation for IR has focused heavily on training efficiency (LoRA, QLoRA, knowledge distillation loss functions) and on scaling laws for retrieval quality. This paper suggests that the more impactful variable—once a reasonable instruction-tuned SLM is available—is the structure and quality of the synthetic training data itself. The ablation in Table 4 provides concrete evidence: query refinement (a data quality intervention) contributes more to accuracy than doubling the dataset size (a data quantity intervention), and multi-task tuning on public IR datasets (a model training intervention) contributes less than either. The conclusion, as the paper states, is that "improving data quality has a greater impact than simply adding more data, once the dataset surpasses a reasonably large threshold." For the enterprise IR community, this shifts the research frontier from "how do we build better small rankers/labelers?" toward "how do we construct training data that captures the right calibration signal?"

Finally, the paper validates the failed SLM query generator experiment (Section 3.1) as a diagnostic tool. The 60.8% binary relevance accuracy and the systematic positivity bias provide evidence for a capability threshold: generating diverse, instruction-dependent queries with adversarial properties (both relevant and genuinely irrelevant) requires capabilities that the tested SLM lacks, even after fine-tuning. This negative result is practically useful because it defines a boundary: in synthetic data pipelines, use LLMs for generation and SLMs for judgment, and do not expect SLMs to perform the generation role effectively for tasks requiring adversarial reasoning or fine-grained intent control. This division of labor is likely to generalize beyond enterprise search to any domain where synthetic training data must span both positive and negative examples with subtle distinctions.

Follow-Up Research This Work Enables

Cheap difficulty estimation for compute-optimal test-time allocation in ranking evaluation. The paper from the reference example (Kang et al.'s work on test-time compute scaling) identifies difficulty estimation cost as a major deployment bottleneck—generating 2048 samples per question to estimate difficulty is more expensive than the test-time compute budget itself. This enterprise labeling paper does not face the same problem, since the SLM scores each query–document pair in a single forward pass without iterative search, but the conceptual parallel is instructive. The BM25 negative mining step in this paper's pipeline serves a function analogous to difficulty estimation: it generates a distribution of query–document pairs that spans the full relevance spectrum, enabling the training data to cover easy, medium, and hard cases without requiring explicit difficulty labels. A natural extension would investigate whether BM25 retrieval scores or the distribution of teacher LLM labels within a BM25-retrieved set can serve as a proxy for query difficulty, enabling dynamic budget allocation at deployment time. Concretely: for a new query, run BM25 to retrieve the top-k candidate documents, then use the SLM labeler (or the teacher LLM's historical labeling behavior on similar BM25 score distributions) to estimate whether this query is "easy" (documents have clear relevance separations) or "hard" (documents cluster at similar relevance levels). On hard queries, route to a larger model or flag for human review. This would close the loop between difficulty estimation and compute-optimal allocation in a setting where the cost of estimation is amortized over the retrieval step that must happen anyway. A strong follow-up would measure: (1) the correlation between BM25 score variance within a retrieved set and the SLM's labeling confidence, (2) the accuracy improvement from adaptive routing compared to uniform SLM labeling, and (3) the end-to-end cost including the routing overhead.

Extension of the BM25-as-spectrum-generator insight to dense retrieval and multi-stage ranking evaluation. The paper's core insight—that BM25's purely lexical matching produces a natural relevance spectrum when applied to hybrid keyword-metadata enterprise queries—is a specific instance of a broader principle: when the retrieval method's failure modes are systematic and predictable, those failure modes can be exploited to generate graded training data. A natural follow-up question is whether the same principle applies when the "retriever" in the negative mining step is a dense embedding model (e.g., a fine-tuned BERT-based retriever or a ColBERT-style late-interaction model). Dense retrievers have different failure modes than BM25—they tend to retrieve semantically related but lexically dissimilar documents, missing exact keyword matches while capturing paraphrases and topic-level associations. Would swapping BM25 for a dense retriever in the pipeline change the distribution of relevance labels produced by the teacher LLM? Would the resulting SLM labeler be better at semantic relevance judgments but worse at keyword-metadata matching? A strong follow-up experiment would run the identical pipeline (same seed documents, same query generation, same teacher LLM) but vary the retriever in the negative mining step—BM25, a dense retriever, a hybrid retriever—and measure: (1) the resulting relevance label distribution (is it still uniform across 0–4?), (2) the SLM's performance broken out by semantic vs. keyword queries, and (3) whether combining BM25-mined and dense-retriever-mined training data produces a labeler that is robust across both query types. This would test whether the "relevance spectrum" property is specific to lexical retrieval or is a general property of any retriever that systematically over- and under-matches certain relevance dimensions.

Cross-enterprise generalization: how many seed documents are needed, and how similar must they be to the target domain? The paper's evaluation is conducted entirely within a single enterprise's document collection—the seed documents used for synthetic data generation and the evaluation benchmark come from the same underlying distribution. The positive result (SLM matches GPT-4o) demonstrates that the pipeline works for in-distribution enterprise search, which is the practical use case for a company deploying its own relevance labeler. But an important open question is how much the pipeline depends on the specific documents, query patterns, and metadata structures of that enterprise. If Company B adopts this approach, starting with their own 1,500 seed documents, will they achieve the same 0.953 NDCG? Or does performance vary systematically with corpus characteristics (size, document type diversity, query pattern complexity, metadata richness)? A strong follow-up would deploy the pipeline across multiple enterprises with differing characteristics—small vs. large document collections, structured (SharePoint-like) vs. unstructured (email-heavy) corpora, single-language vs. multilingual environments—and measure the correlation between corpus properties and final SLM labeling quality. A particularly informative negative result would be: if an enterprise's query patterns are substantially different from the template table's distribution, does performance degrade, and can the template table be adapted using only a small sample of real query logs? This would establish the portability of the approach and the minimum requirements for successful deployment in a new context.

Human inter-annotator agreement as an interpretability ceiling for automated labelers. The paper reports that GPT-4o achieves 0.944 NDCG and 62.58 pairwise accuracy against human labels, and that the fine-tuned SLM achieves 0.953 and 63.81. These numbers are difficult to interpret without knowing the ceiling: what is human–human agreement on the same 923-pair benchmark? If trained human labelers agree with each other at, say, 0.98 NDCG and 75% pairwise accuracy, then both GPT-4o and the SLM have substantial room for improvement, and the SLM's marginal edge over GPT-4o is real but small. If human agreement is, say, 0.95 NDCG and 65% accuracy, then both automated labelers are effectively at the ceiling of achievable performance given the inherent ambiguity of enterprise relevance judgments, and the SLM's "improvement" over GPT-4o is within the noise range of human disagreement. This is not merely a calibration detail—it determines whether further investment in synthetic data quality or teacher LLM improvements can yield meaningful gains, or whether the enterprise relevance labeling problem is fundamentally bounded by inter-annotator variability. A strong follow-up would collect multiple independent human annotations on the same 923-pair benchmark (or a subset of it) and compute standard inter-rater agreement metrics (Cohen's kappa, Krippendorff's alpha, pairwise accuracy between annotators). The resulting ceiling would immediately contextualize all future work on automated enterprise relevance labeling and would indicate whether research effort should shift from improving labeler quality to accepting a natural ceiling and focusing on other dimensions (latency, cost, interpretability).

Temporal drift: what happens when the enterprise document collection and query patterns evolve? Enterprise environments are dynamic—new projects launch, new people join, new document types and communication channels emerge, and the distribution of queries shifts accordingly. The paper's pipeline generates training data from a static set of 1,500 seed documents and a static query pattern template table. If a deployed SLM labeler is used for months after training, will its performance degrade as the document collection and user query patterns drift away from the training distribution? This is a practical reliability question that the paper does not address. A strong follow-up would simulate temporal drift: train the SLM on synthetic data from seed documents collected at time t, then evaluate it on a human-labeled benchmark drawn from documents and queries at time t + Δ, where Δ ranges from weeks to months. Measure the degradation in NDCG and pairwise accuracy as a function of Δ. Additionally, test lightweight retraining strategies: if only a small number of new seed documents (e.g., 100–200) are available at t + Δ, can the SLM be efficiently fine-tuned (without re-running the full GPT-4o pipeline) to recover performance? This would establish the maintenance cost of the pipeline and whether it can be deployed as a "set and forget" system or requires periodic refreshes. A related question: can the teacher LLM detect its own calibration drift by monitoring the distribution of assigned relevance scores over time? If the average relevance score shifts systematically, that might signal domain drift and trigger a retraining cycle.

Adversarial robustness: can the SLM labeler be exploited by ranking models optimized against it? This paper treats the SLM labeler as an evaluation tool—a drop-in replacement for human annotators to score the output of ranking systems. But in the broader IR ecosystem, ranking models are often trained to optimize against automated evaluation metrics. If a ranking model is trained to maximize NDCG as measured by the fine-tuned SLM labeler, will it learn to exploit the SLM's systematic biases (e.g., any residual positivity bias, keyword-matching over-reliance, or metadata-pattern shortcuts) rather than genuinely improving relevance? This is the classic reward hacking problem applied to evaluation labelers rather than reward models. A strong stress-test would: (1) train a ranking model to maximize SLM-labeled NDCG, (2) measure the ranking model's performance against held-out human labels, and (3) compare the human-label NDCG of the SLM-optimized ranker against a ranker optimized directly on human labels (or teacher LLM labels). If the SLM-optimized ranker achieves high SLM-NDCG but significantly lower human-NDCG, the labeler is not safe for use as a training objective—it is only suitable for offline evaluation. If the SLM-optimized ranker matches human-NDCG of a directly human-optimized ranker, the labeler is robust enough to serve as both an evaluation metric and a training signal, dramatically expanding its practical utility. The paper's finding that the SLM agreement with the teacher LLM (SLM–LLM accuracy ~66%) is higher than agreement with humans (SLM–Human accuracy ~63%) suggests that the SLM has learned some of the teacher's idiosyncratic biases that may be exploitable—quantifying that exploitability would directly address the safety of deploying this labeler in a training loop.

Practical Applications and Downstream Use Cases

Production-scale offline evaluation of enterprise ranking models. This is the paper's primary intended application, and the numbers are compelling enough to make it actionable immediately for organizations with enterprise search deployments. The traditional workflow for evaluating a new ranking model in enterprise settings requires either: (1) human annotators to label query–document pairs, which is slow (days to weeks for thousands of pairs), expensive (tens of dollars per query), and often infeasible due to privacy constraints; or (2) GPT-4o API calls, which at $2.50/$10.00 per 1M input/output tokens becomes cost-prohibitive at scale—labeling 100,000 query–document pairs with an average of 500 input tokens and 5 output tokens per pair costs roughly $130 for input and$25 for output, totaling ~$155, versus ~$8 for the fine-tuned SLM. More importantly, the 873 RPM throughput on a single A100 means that 100,000 pairs can be labeled in approximately 1.9 hours, versus potentially a full day or more with an LLM API (which imposes rate limits and has variable latency). This throughput difference fundamentally changes the cadence of experimentation: ranking teams can run nightly evaluations across dozens of model variants, conduct hyperparameter sweeps with hundreds of configurations, and perform A/B tests on live traffic with SLM-based evaluation rather than waiting days for human labels or paying substantial API costs for each evaluation cycle. The practical benefit is a tighter iteration loop for enterprise search quality improvement—the same dynamic that made rapid experimentation possible in web search (where click logs and public benchmarks enabled fast evaluation) becomes available in the privacy-constrained enterprise context.

Training data labeling for learning-to-rank models in enterprise settings. Beyond evaluation, the fine-tuned SLM can serve as a labeling engine for training data generation in learning-to-rank (LTR) pipelines. Modern enterprise search systems often use gradient-boosted trees or neural rankers trained on labeled query–document pairs. Acquiring those labels at scale—for thousands of queries, each with dozens or hundreds of candidate documents—has been the primary bottleneck preventing enterprises from deploying sophisticated LTR systems. The standard approach of using clicks as implicit labels is biased (users click on position, not relevance), and explicit human labeling is too expensive. An SLM labeler that processes 873 pairs per minute at $0.13/$0.52 per 1M tokens makes it feasible to label, say, 10,000 queries × 50 candidate documents = 500,000 query–document pairs in approximately 9.5 hours on a single A100, at a total token cost of roughly $50–100. This is a one-time cost to generate training data for a production ranker that will serve millions of queries. The paper validates this use case indirectly: the SLM–Human NDCG of 0.953 means that rankers trained on SLM labels will optimize for a target that is highly correlated with human preferences. An organization deploying this would: (1) sample representative queries (from logs or synthetic generation), (2) retrieve candidate documents using their production first-stage retriever, (3) run the SLM labeler to assign 0–4 labels, and (4) train an LTR model on the resulting pairs. The entire pipeline operates within the enterprise's infrastructure, addressing privacy constraints.

Cross-lingual and multi-lingual enterprise search evaluation. The paper's pipeline operates on English enterprise documents and queries, but the methodology is language-agnostic—it requires only that the teacher LLM supports the target language and that a query pattern template table can be constructed. Many large enterprises operate across multiple languages, with employees issuing queries in their local language against documents that may be in English, the local language, or mixed. An SLM labeler that can evaluate relevance across languages would enable unified ranking evaluation without requiring separate human annotation pipelines for each language. The practical path: (1) construct language-specific query pattern template tables (or use the same templates translated), (2) use a multilingual teacher LLM (GPT-4o supports ~50 languages) or a language-specific LLM for query generation and labeling, and (3) fine-tune a multilingual SLM (Phi-3.5 Mini supports multiple languages, though the paper only evaluates English). The 19× cost reduction is even more impactful in the multilingual setting, where human annotators for less-common enterprise languages are scarce and expensive, and LLM API costs accumulate per language. A practical deployment would first pilot in English to validate the pipeline, then extend to the top 3–5 languages of the enterprise, using the same seed document collection (which may already contain multilingual content) and measuring SLM–Human agreement in each language against a small human-labeled evaluation set. The paper's finding that 14K synthetic examples suffice for English suggests that a modest per-language labeling effort could produce effective multilingual labelers.

Automated quality assurance for enterprise search result pages. Enterprise search systems often serve structured result pages (document cards, people cards, project summaries) where relevance is only one dimension of quality—others include freshness, authority, diversity, and the absence of stale or duplicate results. The paper's SLM labeler focuses on a single dimension (graded relevance), but the pipeline methodology extends naturally: for each quality dimension, define a labeling prompt, generate synthetic examples (possibly using the same query–document pairs but with different labeling instructions), and fine-tune a separate SLM head or a multi-task SLM that outputs scores for multiple quality dimensions simultaneously. The practical deployment scenario: a search team wants to monitor whether a new ranking model introduces regressions in result diversity or surface quality. They deploy a multi-dimensional SLM labeler that processes the top-10 results for a sample of queries nightly, producing dashboards of per-dimension scores over time. When a dimension degrades, the team can investigate before the regression reaches users. The throughput advantage (873 RPM per A100) means that scoring 10,000 queries × 10 results × 5 quality dimensions = 500,000 individual judgments takes roughly 9.5 hours on a single A100—a nightly batch job that fits comfortably within off-peak compute windows. The paper's demonstration that the SLM can match or exceed GPT-4o quality on the relevance dimension is the proof of concept; extending to other dimensions would require constructing appropriate labeling prompts and evaluation benchmarks for each, but the pipeline infrastructure (seed document collection, template table, teacher LLM labeling, SLM fine-tuning) transfers directly.

When to Prefer This Method

The paper does not explicitly articulate a "prefer our SLM labeler over GPT-4o labeling" decision rule—its positioning is that the SLM matches GPT-4o quality with better throughput and cost, making it strictly preferable for the enterprise relevance labeling task when the deployment constraints align with the pipeline's requirements. However, the practical conditions under which the approach succeeds (or fails) are discernible from the methodology and results:

• Prefer the fine-tuned SLM labeler when: (1) you have access to a curated set of seed enterprise documents with rich metadata (author, title, file type, folder path, content) that are representative of your document collection—the paper uses 1,500 such documents; (2) you can construct or approximate a query pattern template table that captures how users form queries in your enterprise search system—the paper shows this can be done by systematic enumeration of metadata field combinations even without real query logs; (3) your teacher LLM (GPT-4o or equivalent) produces reasonably calibrated relevance labels for your domain—the paper's quality-control filtering can catch some errors but assumes the teacher is fundamentally competent; (4) you need to label query–document pairs at scale (tens of thousands to millions of pairs), where the upfront cost of the synthetic data pipeline (GPT-4o generation and labeling of ~14K examples) is amortized over the SLM's 19× cheaper per-query cost; and (5) your evaluation or training needs are in-distribution—the queries and documents you will label at deployment time are drawn from the same patterns (metadata structures, query templates, entity types) as the seed documents used for training.

• Prefer GPT-4o (or another teacher LLM) zero-shot labeling when: (1) you need to label a small number of query–document pairs (hundreds, not thousands) and the upfront cost of building the synthetic data pipeline exceeds the cost of direct LLM labeling; (2) your enterprise document collection or query patterns are evolving rapidly, and the cost of periodically re-running the synthetic data pipeline to capture new patterns outweighs the per-query savings; (3) you need supplementary signals that the SLM does not provide—for example, natural language explanations for why a document is or is not relevant, which the paper explicitly removed from the SLM's output format because it introduced training instability and verbosity; or (4) you are operating in a domain where query patterns are predominantly semantic and well-represented in public datasets (the paper's row 7 ablation shows that public data alone is insufficient for enterprise, but for web search labeling, public data may suffice, making the synthetic pipeline unnecessary).

• Prefer human labeling when: (1) the labeling task requires subjective or context-dependent judgments that cannot be reduced to a 0–4 scale with written criteria—for example, judgments about whether a document's tone is appropriate for a particular audience, or whether a retrieved email would violate privacy norms if shown; (2) you are establishing the gold-standard evaluation benchmark against which automated labelers (SLM, LLM) will be validated—the paper's own evaluation depends on a 923-pair human-labeled dataset, and any enterprise deploying this approach would similarly need a small human-labeled set for initial validation; or (3) regulatory or policy requirements mandate human review for certain types of content or decisions.

The unstated but implicit tradeoff is between upfront investment in pipeline construction (seed document curation, template table creation, GPT-4o inference for query generation and labeling, SLM fine-tuning on 8 A100s) and per-query inference cost. The paper's 14K–24K training examples required roughly 70,000–120,000 GPT-4o labeling calls (each query produces ~5 document pairs), plus query generation and refinement calls, which at GPT-4o pricing might cost on the order of hundreds of dollars in API fees—a modest upfront cost for an organization that will subsequently label millions of pairs. An organization with sustained, high-volume labeling needs will prefer the SLM; an organization with sporadic, low-volume needs will find the upfront investment harder to justify. The paper provides the numbers to make this calculation but does not formalize the break-even analysis—a practitioner would need to estimate their own labeling volume, seed document availability, and template table coverage to determine the crossover point.