ArXiv: 2510.22101

🎯 Pitch

A 10× throughput boost—achieved by pruning the model 40%, summarizing items with a specialized RL agent, and stripping out autoregressive generation—makes LLM-based ranking viable at millions of requests per second. Naive content compression actually hurts more than it helps, but aligning the summarizer to the ranker preserves quality within 2%.

1. Executive Summary

This paper analyzes techniques for deploying Small Language Models (SLMs) as relevance rankers in a production semantic search system at LinkedIn, using a decoder-only 0.6B model on the job search ranking task. The authors develop three complementary efficiency mechanisms—structured pruning (removing 50% of MLP hidden neurons and up to 12 transformer blocks via OSSCAR), context compression via RL-based summarization (training a 1.7B actor model with GSPO to produce item description summaries aligned to the ranker's output distribution), and serving infrastructure optimization (batch tokenization, decode-loop bypass, amortized prefix KV-cache reuse, and traffic shaping)—that together deliver a 10× throughput improvement in online deployment, increasing GPU capacity from approximately 200 to 2,000 items scored per second per H100 GPU. The combined model-and-context compression reduces the SLM from 600M to 375M parameters and the average prompt length by roughly 4× while losing less than 2% in NDCG@10 quality, establishing that aggressive compression coupled with RL-aligned summarization can make LLM-based ranking economically viable at millions of requests per second—provided the summarization model is explicitly trained to preserve the SLM's relevance signal rather than relying on generic prompt-engineering approaches that the paper shows degrade performance substantially (e.g., 2.5% NDCG loss at only 52% compression with a naive "summarize" prompt).

2. Context and Motivation

The Core Problem: LLM-Quality Ranking at Internet Scale Is Economically Infeasible

The fundamental tension this paper contends with is deceptively simple: LLMs are excellent relevance judges for semantic search, but serving them online at production scale is prohibitively expensive. The authors frame this around LinkedIn's Semantic Job Search system, which processes millions of ranking requests per second across a global user base. Each request requires computing a relevance score between a user's free-text query and potentially hundreds of candidate job postings, each containing verbose natural-language descriptions. Running a decoder-only language model—even a small one at 0.6B parameters—for every (query, job) pair in this pipeline would require a GPU fleet of impractical size and cost.

This is not merely an academic concern. The paper reports concrete production requirements (Section 2): the ranking system must score 3.15 million items per second across multiple product facets. At the unoptimized throughput of roughly 200 items per second per H100 GPU (as implied by the 10× improvement figure in Section 6), this would require approximately 15,750 H100 GPUs running continuously—an economically absurd proposition for a relevance ranking service. The gap between what the model quality demands and what the infrastructure budget permits is the central problem.

The economic dimension is more acute than in many LLM deployment papers because this is a prefill-only, high-volume classification task embedded in a latency-sensitive user-facing pipeline. Unlike chatbot applications where users tolerate some response delay, search ranking directly shapes the user experience: milliseconds of additional latency translate to perceptible slowness in search results loading. Unlike batch inference pipelines where cost-per-query can be amortized over time, this system must handle bursty, real-time traffic patterns (peak hours for job searching). And unlike research benchmarks where accuracy is the sole metric, this system operates under a hard quality bar—the ranking must be accurate enough to improve user engagement and reduce poor matches, or the entire semantic search investment is undermined.

Conflicting Pressures: Quality Demands Complexity, Complexity Demands Resources

The paper's specific use case embodies a broader tension in industrial information retrieval. On the quality side:

  • Semantic understanding genuinely helps. The paper shows that moving from keyword-matching (embedding-based retrieval alone) to an SLM-based cross-encoder reranker delivers substantial gains: +5.54% NDCG@10 and a 19.84% reduction in poor match rate for the uncompressed SLM (Table 9). Users demonstrably benefit from the model's ability to understand intent rather than matching vocabulary.

  • Item descriptions carry critical signal. When the authors remove the item description entirely from the prompt, NDCG@10 drops by 9% (Table 4). These free-text job descriptions—with a median length of 900 tokens and maxima exceeding 2,100 tokens—contain nuanced information about required skills, responsibilities, qualifications, and company culture that structured fields (title, location, company) cannot capture. The quality case for including them is unambiguous.

  • The teacher model is expensive. The ground-truth relevance labels are generated by a 7B LLM that produces graded relevance judgments with rationales (Section 2). This teacher is far too large to serve online directly, creating a distillation problem: the SLM must approximate the teacher's quality while operating under tight compute constraints.

On the efficiency side, these same quality drivers create resource demands that compound multiplicatively:

  • Prompt length dominates compute cost. Because the self-attention mechanism scales quadratically with input length, the 900-token median job descriptions mean that over 94% of the SLM's prompt tokens come from a single field (Section 4.2). Reducing model size helps, but reducing input length helps quadratically more for the prefill latency.

  • Prompt variability creates batching challenges. Job descriptions vary wildly in length—from very short to over 2,100 tokens, with 10% of prompts being truncated at the 2048-token budget (Section 4.2). This length heterogeneity makes efficient batching difficult, as the longest prompt in a batch determines the compute time for the entire batch.

  • High RPS means every optimization matters. At millions of requests per second, even tiny per-request overheads (tokenization, memory copies, GPU kernel launch gaps) aggregate into major system bottlenecks. The paper's finding that tokenization alone can become a dominant cost (Section 5.2) underscores this reality—tasks that are negligible in low-throughput settings become critical path at scale.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior work along multiple dimensions:

Distillation to non-LLM architectures is the dominant workaround, but it sacrifices flexibility. Wang et al. (2024) used an LLM as a relevance judge at Pinterest but distilled it into a smaller feed-forward neural network for online serving, explicitly avoiding the challenge of serving the LLM directly. Dey et al. (2025) similarly distill an LLM relevance judge into a non-LLM architecture. Shang et al. (2025) distill into a BERT-style model. The paper acknowledges these approaches but argues they represent a capitulation to the efficiency problem rather than a solution to it. Distillation to a feed-forward or encoder-only architecture forces the model's input representation to be pre-structured (fixed feature vectors), losing the flexibility to process free-text job descriptions naturally. The authors note that Shang et al. (2025) "dropped certain features such as item description from their online serving stack due to efficiency reasons"—precisely the feature this paper's summarization technique preserves. By deploying a decoder-only model directly, this work retains the architectural capacity to ingest and reason over natural language at inference time, unlocking quality from item descriptions that other systems abandon.

Generic prompt compression fails for heterogeneous item descriptions. The prompt compression literature offers techniques like Gisting (Mu et al., 2023), which compresses prompts into smaller sets of "gist" tokens that can be cached and reused. However, the paper notes a critical limitation: gisting "is usually used for general system prompt / task descriptions, and cannot be generalized to a vast amount of heterogeneous item descriptions" (Section 3.2). A system prompt is fixed across all queries; a job description is unique per job. Nano-Capsulator (Chuang et al., 2024) compresses prompts into natural language while preserving semantics, but the paper argues that "text descriptions can be verbose and contain information that isn't useful for ranking, which cannot be distinguished with semantic loss alone" (Section 3.2). In other words, generic compression optimized for reconstructing the original text may preserve details that are semantically interesting but rank-irrelevant, wasting tokens on content that doesn't improve relevance predictions.

Prompt engineering alone is insufficient. The paper's own experiments (Section 4.2.1, Table 4) demonstrate this concretely. Simply prompting the summarization model to "summarize" the job description achieves 52% compression but at a 2.5% NDCG loss—already beyond the 2% acceptable threshold. A YAML-structured prompt compresses 74% but loses 2.7%. A "key phrases" approach compresses 61% with a 1.5% loss. These results establish that off-the-shelf summarization, however cleverly prompted, does not preserve the specific information the ranker needs. The summarization model has no awareness of the downstream relevance task, so it compresses based on general linguistic importance rather than rank-relevance importance.

Pruning literature focuses on general LLM capabilities, not task-specific fine-tuned models. While model pruning for LLMs is well-studied (Frantar & Alistarh, 2023; Sun et al., 2023; Meng et al., 2024a), prior work largely evaluates pruned models on broad benchmarks (perplexity, downstream task accuracy) rather than on a specific production ranking task where the model has been fine-tuned for a particular input-output mapping. The paper's finding that last transformer layers are significantly less important than early layers for the relevance task (Table 2: removing the last layer causes only a -0.0009 NDCG change vs. -0.3356 for the first layer) is task-specific knowledge that generic pruning studies cannot provide. This layer-sensitivity profile directly informs the structured pruning strategy, and it is only discoverable through task-specific experimentation.

Serving infrastructure for prefill-only workloads is underexplored. Most LLM serving research and engineering (including the SGLang engine the paper builds on; Zheng et al., 2024) is optimized for autoregressive decode-heavy workloads typical of chatbots and text generation. The paper's workload is fundamentally different: it is prefill-only (only the last token's logits are needed for classification, per Equation 2), operates at extremely high RPS (orders of magnitude more than chatbot serving), and features shared prefixes across items for the same query (all (query, job) pairs for a given query share the system prompt and query text). These characteristics create optimization opportunities—amortized prefix KV-cache reuse, decode-loop bypass, batch tokenization—that are not addressed in standard serving configurations. The paper's contribution here is not inventing these techniques de novo but rather identifying which standard assumptions to violate and which domain-specific properties to exploit.

How This Paper Positions Itself

The paper positions itself at the intersection of model compression, context compression, and serving systems engineering, arguing that these are not independent concerns but must be co-optimized for real-world deployment. This is a departure from the typical research paper structure where each technique is studied in isolation and evaluated on a benchmark. Here, the evaluation criterion is whether the combined system meets a throughput target while staying within a quality budget (<2% NDCG loss), and the paper's structure reflects this: each compression technique is developed with awareness of how it will compose with the others.

Several aspects of this positioning are notable:

The quality budget is explicit and fixed. Rather than treating accuracy as a metric to maximize, the paper treats it as a constraint: keep NDCG@10 within 2% of the uncompressed baseline. All efficiency gains are measured relative to this constraint. This reflects industrial reality: a ranking model that is fast but inaccurate is useless, and one that is perfectly accurate but too slow to serve is equally useless. The acceptable loss threshold is determined by product requirements, not by optimization convenience.

Compression is aligned to the downstream task, not to reconstruction. The RL-based summarization approach (Section 3.2) is the clearest expression of this philosophy. The reward signal in Equation (4) has two terms: a length penalty and a KL divergence between the SLM's output distribution on summarized vs. raw context. Crucially, the summarization model is not trained to reproduce the original description; it is trained to produce summaries that cause the SLM to make the same predictions it would have made with the full description. This is a form of task-aware compression where the information bottleneck is shaped by relevance rather than by semantic fidelity. The paper contrasts this with the prompt engineering baselines which, lacking access to the SLM's output, optimize a generic notion of summary quality.

The meta-lesson is that co-design matters. The paper's finding that SFT after pruning using summarized data marginally improves NDCG compared to SFT using full descriptions (Table 6: 0.8788 vs. 0.8786) is a small but telling result. It suggests that the model compression (pruning) and context compression (summarization) pipelines interact: a pruned model trained on compressed inputs learns to use the available capacity more effectively than one trained on full inputs and then fed compressed inputs at inference time. This "co-design" insight—that you should compress the model and the context together rather than sequentially—is a practical takeaway that purely algorithmic pruning or summarization papers would not surface.

The work is explicitly positioned as deployment engineering, not algorithmic novelty. The authors do not claim to invent pruning (they use OSSCAR; Meng et al., 2024b), RL-based summarization (they use verl and GSPO; Sheng et al., 2024; Zheng et al., 2025), or efficient serving (they build on SGLang; Zheng et al., 2024). Their contribution is the integration and adaptation of these techniques to a specific production workload at a scale that stresses every component. This is legitimate but important to recognize: the paper's value is not in new algorithms but in the empirical demonstration that aggressive, co-designed compression can bridge the gap between LLM quality and production feasibility for ranking—a gap that prior work either avoided (via distillation to non-LLMs) or failed to close (via prompt engineering alone). Table 9's online A/B results—showing the compressed SLM actually outperforming the uncompressed version on user engagement metrics—validates that the engineering investment translates to real user value, not just cost savings.

3. Technical Approach

3.1 Reader Orientation

This paper describes a production system that uses a decoder-only Small Language Model (SLM) as a cross-encoder relevance ranker for job search at LinkedIn, scoring how well each candidate job posting matches a user's free-text query. The core problem is that deploying even a modestly-sized SLM (0.6B parameters) to score millions of (query, job) pairs per second is economically infeasible without aggressive optimization; the solution is a three-pronged compression strategy—model pruning to reduce parameters, context summarization to reduce input length, and serving infrastructure optimization to extract maximum throughput from GPU hardware—co-designed so that the combined system stays within a 2% quality budget while improving throughput by 10×.

3.2 Big-Picture Architecture (Diagram in Words)

The system consists of five major components arranged in a serving pipeline:

  1. The Teacher LLM (7B parameters) — a large language model that generates graded relevance labels and rationales for (query, job) pairs offline, defining the quality target the system must approximate. It is never served online; its outputs train everything else.

  2. The SLM Ranker (0.6B → 375M parameters) — a decoder-only transformer fine-tuned to output "yes"/"no" logits for a (query, job) prompt, producing a relevance score via softmax. This is compressed via structured pruning and then further fine-tuned (SFT) to recover quality.

  3. The Item Description Summarizer (1.7B parameters) — a separate language model trained via reinforcement learning to compress verbose job descriptions (median 900 tokens) into short, rank-relevant summaries. It uses the frozen SLM as a reward model: summaries that cause the SLM to make the same predictions it would have made with full descriptions receive higher reward, while length is penalized.

  4. The Summarization RL Training Loop — built on the verl framework with GSPO optimization, this offline pipeline trains the summarizer by generating summaries, scoring them through the SLM, computing a combined reward (KL divergence preservation + length penalty), and updating the actor model. The SLM parameters are frozen throughout.

  5. The Online Serving Stack (SGLang on H100 GPUs) — the deployment infrastructure that receives queries, looks up pre-computed item summaries and features from a distributed cache, constructs prompts, batches them, and runs the pruned SLM in a prefill-only mode to produce relevance scores. This incorporates batch tokenization, decode-loop bypass, amortized prefix KV-cache reuse, score caching, dynamic depth control, and traffic shaping.

Information flows as follows: offline → the teacher labels training data, the SLM is fine-tuned and pruned, the summarizer is RL-trained against the SLM, and summaries are pre-computed for all job postings; online → a user query arrives, candidate jobs are retrieved via embedding-based search, their pre-computed summaries and features are fetched from distributed storage, prompts are constructed and batched through SGLang, the pruned SLM computes yes/no logits, softmax converts them to relevance scores, and items are ranked by score before being passed to the auction layer.

3.3 Roadmap for the Deep Dive

  • First, the SLM ranking mechanism (Equation 1 and 2) — how a decoder-only language model is converted into a relevance classifier using yes/no token logits — since this is the core inference operation everything else must optimize.
  • Second, the SLM training pipeline (distillation, soft labels, KL-divergence loss; Equation 3) — because the quality targets and training objectives determine what the pruning and summarization techniques must preserve.
  • Third, structured model pruning (OSSCAR for MLP neurons, layer removal) — since this reduces the model's computational footprint and is the first compression axis.
  • Fourth, context summarization via reinforcement learning (Equation 4, the RL reward design, the length penalty formulations P1 and P2; Equations 5–6) — because this is the most technically novel compression mechanism and requires understanding the alignment between summarizer and ranker.
  • Fifth, the co-design integration (pruned model SFT on summarized data) — since the paper's key practical insight is that model and context compression interact and must be combined thoughtfully.
  • Sixth, serving infrastructure optimization (batch tokenization, decode-loop bypass, memory synchronization, garbage collection, amortized prefix caching, score caching, dynamic depth, traffic shaping) — because these system-level techniques amplify the gains from model and context compression.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems engineering paper whose core idea is that LLM-quality ranking can be made economically feasible at internet scale by co-designing three compression axes—model size, context length, and serving overhead—around a fixed quality budget, using task-aligned RL to ensure that context compression preserves the specific information the ranker needs rather than generic semantic content.

The SLM Relevance Classification Mechanism

The fundamental inference operation in the system is converting a (query, job) pair into a relevance score using a decoder-only language model. The mechanism described in Section 2 works as follows.

For each query $q$ and job posting $i$, the system constructs a structured prompt:

prompt(q,i)=system prefix,q,metadatai,desci,suffix\text{prompt}(q, i) = \text{system prefix}, q, \text{metadata}_i, \text{desc}_i, \text{suffix}

where $\text{system prefix}$ and $\text{suffix}$ are fixed strings containing chat-template formatting tags and instructions directing the model to judge relevance, $\text{metadata}_i$ encodes structured job attributes (title, company, location, employment type, remote-work eligibility), and $\text{desc}_i$ is the free-text job description—by far the dominant field, comprising over 94% of tokens in the uncompressed prompt (Section 4.2), with a median length of approximately 900 tokens and a maximum exceeding 2,100 tokens.

How the score is computed. The prompt is passed through the decoder-only SLM, producing a hidden state at the final input token position. From this hidden state, two specific logits are extracted: $\text{logit}_{\text{yes}}$ and $\text{logit}_{\text{no}}$, corresponding to the tokens "yes" and "no" in the model's vocabulary. These logits represent the unnormalized model confidence that the job matches the query (yes) versus does not match (no). The conversion to a relevance probability uses a two-class softmax restricted to these two tokens:

(pyes,pno)=Softmax(logityes,logitno)(p_{\text{yes}}, p_{\text{no}}) = \text{Softmax}(\text{logit}_{\text{yes}}, \text{logit}_{\text{no}})

where $\text{Softmax}(x_1, x_2)$ is the standard two-element softmax: $p_k = \frac{e^{x_k}}{e^{x_1} + e^{x_2}}$ for $k \in \{\text{yes}, \text{no}\}$.

What this computes. Given only the logits for the yes/no tokens at the final position, the two-class softmax produces normalized probabilities $p_{\text{yes}}$ and $p_{\text{no}}$ that sum to one. The value $p_{\text{yes}}$ is interpreted as the model's confidence that job $i$ is relevant to query $q$. Items are then ranked in descending order of $p_{\text{yes}}$. The system never generates any text autoregressively—it only needs the logits at the final input position, making this a pure prefill workload with no decode phase.

Why this form. This "verbalizer" approach—mapping a classification task onto the model's existing yes/no token vocabulary rather than adding a classification head—has several practical advantages for this deployment. First, it requires no architectural modification to the base model: the same decoder-only transformer that was pretrained and fine-tuned for text generation can serve as a classifier without additional parameters. Second, it leverages the model's pretrained understanding of affirmative/negative semantics: the yes and no tokens have rich representations learned during pretraining that capture nuanced notions of correctness and agreement. Third, it produces calibrated probabilities via softmax that can be directly used for ranking without additional scaling. The alternative—adding a linear classification head on top of the final hidden state—would require training new parameters from scratch and would discard the semantic knowledge already encoded in the yes/no token embeddings. Prior work on LLM-based relevance ranking (Wang et al., 2024; Zhang et al., 2025) uses this same verbalizer pattern, and the paper adopts it as a known-good approach.

SLM Training Pipeline: Distillation and Supervised Fine-Tuning

The SLM is never trained directly on human-labeled relevance data. Instead, the paper describes a multi-stage distillation pipeline (Section 2) that transfers knowledge from an expensive 7B teacher model to the compact 0.6B student.

Stage 1: Teacher label generation. A 7B LLM is prompted to evaluate (query, job) pairs and produce 5-point graded relevance scores across multiple dimensions (e.g., skill match, seniority match, industry match), along with natural-language rationales explaining its judgments. These dimension-wise grades are then aggregated into a single final label per pair. The teacher-generated labels define the ground truth for all subsequent training; this is a form of weak supervision where the 7B model's judgments serve as a proxy for actual user relevance feedback.

Stage 2: Initial distillation to a smaller reasoning model. The 7B teacher is distilled into a 0.6B model that generates both graded labels and rationales—essentially mimicking the teacher's full output format. This step produces a smaller model that can reason about relevance in natural language, but it is still too expensive for online serving because generating rationales requires autoregressive decoding.

Stage 3: KL-divergence fine-tuning as a binary classifier. The graded labels from the teacher are converted into "soft labels" $(p^*_{\text{yes}}, p^*_{\text{no}})$ that represent the teacher's relevance judgment as a probability distribution over yes/no. For example, a teacher grade of 4 out of 5 might be mapped to $p^*_{\text{yes}} = 0.8, p^*_{\text{no}} = 0.2$, while a grade of 1 out of 5 maps to $p^*_{\text{yes}} = 0.2, p^*_{\text{no}} = 0.8$. The SLM is then fine-tuned with supervised learning to minimize the Kullback-Leibler (KL) divergence between the teacher's soft label distribution and the SLM's predicted distribution:

loss=KL((pyes∗,pno∗)∥(pyes,pno))\text{loss} = \text{KL}((p^*_{\text{yes}}, p^*_{\text{no}}) \parallel (p_{\text{yes}}, p_{\text{no}}))

where $\text{KL}(p \parallel q) = \sum_i p_i \log(p_i / q_i)$. In expanded form for the two-class case:

loss=pyes∗log⁡(pyes∗pyes)+pno∗log⁡(pno∗pno)\text{loss} = p^*_{\text{yes}} \log\left(\frac{p^*_{\text{yes}}}{p_{\text{yes}}}\right) + p^*_{\text{no}} \log\left(\frac{p^*_{\text{no}}}{p_{\text{no}}}\right)

What this computes. The KL divergence measures the information lost when the SLM's predicted distribution $(p_{\text{yes}}, p_{\text{no}})$ is used to approximate the teacher's target distribution $(p^*_{\text{yes}}, p^*_{\text{no}})$. It penalizes the SLM both for placing low probability on tokens the teacher considers likely and for placing high probability on tokens the teacher considers unlikely. The resulting scalar loss is minimized during fine-tuning, driving the SLM's yes/no predictions toward the teacher's graded judgments.

Why this form over cross-entropy with hard labels. Using KL divergence with soft labels rather than standard cross-entropy with binary hard labels has a specific justification for this use case. The teacher produces ordinal grades (1–5), which encode more information than a binary relevant/irrelevant decision. A grade of 4 and a grade of 5 are both "relevant" but with different degrees of confidence. By converting these grades to soft probability targets—e.g., $(0.8, 0.2)$ for a strong match versus $(0.6, 0.4)$ for a borderline match—the KL loss preserves this graded information during distillation. The SLM learns not just the binary classification boundary but the teacher's confidence calibration, which is important for ranking because items with similar binary labels may need to be ordered by relevance strength. Standard cross-entropy with hard 0/1 labels would discard this ordinal information and produce a model that only distinguishes relevant from irrelevant without capturing relevance degree.

Training data and hyperparameters. The model is trained on 200,000 (query, job) examples for up to 5 epochs. The prompt in Equation (1) is used throughout, with the job description $\text{desc}_i$ truncated so that the total prompt length does not exceed 2,048 tokens, the maximum context length supported by the model architecture. During both training and inference, approximately 10% of prompts are affected by this truncation due to extremely verbose job descriptions exceeding 2,100 tokens (Section 4.2).

Evaluation. Quality is measured using NDCG@10 (Normalized Discounted Cumulative Gain at rank position 10) on a holdout set labeled by the teacher. NDCG@10 evaluates ranking quality by comparing the order produced by the SLM's $p_{\text{yes}}$ scores against the ideal ordering from teacher labels, with a logarithmic discount factor that weights top-ranked positions more heavily. This metric directly captures what matters for the user experience: whether the most relevant jobs appear at the top of the search results page.

Structured Model Pruning

The first compression axis reduces the SLM's parameter count through structured pruning, which removes entire model components (neurons, attention heads, transformer layers) rather than individual weights. Structured pruning is chosen over unstructured pruning because the resulting model can run on standard GPU hardware without requiring specialized sparse matrix kernels.

The paper applies two complementary pruning strategies to a 0.6B model that has already been fine-tuned on the relevance task:

MLP hidden neuron pruning via OSSCAR. The OSSCAR algorithm (Meng et al., 2024b) identifies and removes 50% of the hidden neurons in all Feed-Forward MLP layers of the transformer. Each transformer block in a decoder-only architecture contains an MLP sublayer with an intermediate hidden dimension that is typically 4× the model's hidden dimension. Pruning these hidden neurons reduces the intermediate size, directly decreasing the FLOPs and memory required for the MLP computation, which is the dominant cost in the transformer forward pass.

Calibration data. OSSCAR uses a small calibration dataset to determine which neurons can be removed with minimal impact on model output. Following prior work (Behdin et al., 2025), the authors use in-domain data for calibration—approximately 40 million tokens from their training prompts. This is critical: using generic text for calibration would not reveal which neurons are important for the specific relevance classification task. The calibration data is processed through the model, and OSSCAR's combinatorial optimization selects a subset of neurons to retain such that the pruned model's activations remain close to the original model's activations on this data.

Recovery fine-tuning. After pruning, the model undergoes SFT (Supervised Fine-Tuning) on the same relevance training data to recover any accuracy lost during compression. The paper reports (Table 1) that pruning 50% of MLP neurons without SFT causes a -0.0095 NDCG@10 drop, but SFT reduces this to -0.0046—a recovery of more than half the loss. The pruned model has approximately 460 million parameters, a 23% reduction from the original 600M.

Whole transformer block removal. In addition to MLP neuron pruning, the paper systematically evaluates the impact of removing entire transformer blocks (layers) from the model. Each block contains self-attention, MLP, and layer normalization sublayers, so removing a block eliminates all associated computation and parameters. The paper tests removing one layer at a time and evaluates the resulting quality.

The key finding (Table 2) is that layer importance is highly position-dependent:

  • First layer removal: -0.3356 NDCG@10 — catastrophic quality loss. The first layer performs initial token embedding transformations that cannot be compensated for by remaining layers.
  • Second layer removal: -0.0296 — still substantial, much larger than middle-layer removal.
  • Third layer removal: -0.0133, and tenth layer removal: -0.0166 — middle layers show moderate sensitivity, likely because they perform task-specific intermediate computations that are somewhat redundant but not fully so.
  • Last layer removal: -0.0009, one before last: -0.0004, two before last: -0.0001 — the final three layers contribute almost nothing to the relevance task. Their removal causes negligible quality degradation.

Why last layers are dispensable for classification. This finding has a clear mechanistic interpretation. In a decoder-only model trained for next-token prediction, the final layers specialize in projecting hidden representations onto the vocabulary space for token generation. However, this paper's use case only requires logits for two specific tokens (yes/no) at the final position. The penultimate-layer representations already contain sufficient information for this binary classification; the final few layers are performing vocabulary-projection work that is largely redundant when only two output dimensions matter. This is fundamentally different from a text generation workload, where every token in the vocabulary is a potential output and the full projection through all layers is necessary.

Combined pruning pipeline. The paper integrates MLP pruning and layer removal into a sequential pipeline (Table 3):

  1. Start with the 600M fine-tuned SLM.
  2. Apply OSSCAR to prune 50% of MLP hidden neurons.
  3. Perform SFT to recover quality post-MLP-pruning.
  4. Remove the last 8, 10, or 12 transformer blocks.
  5. Perform a second round of SFT to recover quality post-layer-removal.

The results show that removing 50% of MLP neurons plus the last 8 layers (producing a 375M model, a 37.5% reduction) costs only -0.0079 NDCG@10. Removing 12 layers (330M, 45% reduction) costs -0.0080. The degradation is remarkably flat across these configurations, suggesting that the combination of MLP pruning and layer removal targets redundant capacity that the relevance task does not fully utilize.

Design choice: pruning order matters. The paper applies MLP pruning first, then layer removal, not the reverse. This ordering is motivated by the calibration data: OSSCAR's neuron selection depends on the model's activation patterns, which change after layer removal. Pruning neurons first establishes a "compressed activation profile" that the subsequent layer removal can adapt to during SFT. The alternative (remove layers first, then prune neurons) would risk OSSCAR selecting neurons based on activation patterns that will be altered by layer removal, leading to suboptimal neuron retention.

Context Summarization via Reinforcement Learning

This is the most technically involved compression mechanism and the paper's primary methodological contribution. The goal is to train a separate language model (the "actor" or "summarizer") to compress verbose job descriptions into short summaries that preserve the relevance signal—the information the SLM ranker uses to make its yes/no decision—rather than preserving general semantic content.

Why prompt engineering fails. The paper's preliminary experiments (Section 4.2.1, Table 4) establish the need for RL. A naive "summarize" prompt achieves 52% compression but at -2.5% NDCG loss. A YAML-structured extraction prompt compresses 74% but loses -2.7%. A "key phrases" approach loses -1.5% at 61% compression. None achieve substantial compression (e.g., >80%) within the 2% quality budget. The fundamental problem is that off-the-shelf summarization optimizes for general readability and factual completeness, not for preserving the specific features the downstream ranker uses. A summary might faithfully capture the job's responsibilities while omitting or distorting the skill keywords or seniority indicators that drive the relevance prediction.

RL training setup. The training pipeline (Figure 2) involves two frozen models and one trainable model:

  1. Actor model (1.7B parameters, trainable): A separate LLM that takes a full job description as input and generates a summarized version. This is the model being optimized. It is initialized from a pretrained 1.7B open-weights language model.
  2. SLM ranker (0.6B, frozen): The relevance classifier described earlier. It serves as the reward model—its output distribution on summarized versus raw descriptions defines the quality component of the reward.
  3. RL framework (verl + GSPO): The training loop is built on the verl framework (Sheng et al., 2024) and uses Group Sequence Policy Optimization (GSPO; Zheng et al., 2025) as the optimization algorithm.

The training proceeds as follows: For each training example (a (query, job) pair with the full job description $\text{desc}_{\text{raw}}$), the actor generates a summarized description $\text{desc}_{\text{sum}}$. Two prompts are then constructed: one using the raw description and one using the summarized description (all other prompt components—system prefix, query, metadata, suffix—are identical). Both prompts are fed through the frozen SLM, producing two output distributions: $p_{\text{raw}} = (p_{\text{yes}}^{\text{raw}}, p_{\text{no}}^{\text{raw}})$ from the raw-description prompt and $p_{\text{sum}} = (p_{\text{yes}}^{\text{sum}}, p_{\text{no}}^{\text{sum}})$ from the summarized-description prompt. The reward is then computed from these distributions and the summary length, and GSPO updates the actor's parameters to maximize expected reward.

The reward function. The reward signal combines two competing objectives—preserve ranker behavior and reduce length—into a single scalar:

reward=−KL(psum∥praw)−w(lensumlenraw)2\text{reward} = -\text{KL}(p_{\text{sum}} \parallel p_{\text{raw}}) - w \left(\frac{\text{len}_{\text{sum}}}{\text{len}_{\text{raw}}}\right)^2

where:

  • $\text{KL}(p_{\text{sum}} \parallel p_{\text{raw}})$ is the Kullback-Leibler divergence from the raw-context SLM distribution to the summarized-context SLM distribution: $p_{\text{yes}}^{\text{raw}} \log(p_{\text{yes}}^{\text{raw}} / p_{\text{yes}}^{\text{sum}}) + p_{\text{no}}^{\text{raw}} \log(p_{\text{no}}^{\text{raw}} / p_{\text{no}}^{\text{sum}})$
  • $\text{len}_{\text{sum}}$ is the token length of the generated summary
  • $\text{len}_{\text{raw}}$ is the token length of the original job description
  • $w$ is a hyperparameter controlling the trade-off between quality preservation and length reduction

What this computes. The first term, $-\text{KL}(p_{\text{sum}} \parallel p_{\text{raw}})$, measures how much the SLM's relevance prediction changes when using the summary instead of the full description. When the summary perfectly preserves all rank-relevant information, $p_{\text{sum}} \approx p_{\text{raw}}$ and the KL divergence is near zero, yielding a reward near zero for that component. When the summary distorts rank-relevant information, the KL term becomes negative, penalizing the actor. The second term, $-w(\text{len}_{\text{sum}} / \text{len}_{\text{raw}})^2$, penalizes summary length quadratically. A summary that is 50% of the original length incurs a penalty of $-w \cdot 0.25$; one that is 10% of the original length incurs $-w \cdot 0.01$. The total reward is maximized when the summary is simultaneously short (small length ratio) and rank-preserving (small KL divergence).

Why this form over standard summarization losses.

  • KL divergence vs. semantic similarity. A standard approach would be to train the summarizer to minimize some semantic distance between the original and summarized text (e.g., embedding cosine similarity, ROUGE score, or reconstruction loss). The problem, as the paper notes (Section 3.2), is that "text descriptions can be verbose and contain information that isn't useful for ranking, which cannot be distinguished with semantic loss alone." A job description might contain a long paragraph about company culture that is semantically rich but rank-irrelevant for most queries; a semantic loss would penalize the summarizer for dropping this content, forcing it to waste tokens on information that doesn't improve relevance predictions. The KL divergence against the SLM's output naturally focuses the summarizer on content that changes the ranker's decision, which is exactly what matters for the downstream task.

  • Quadratic length penalty vs. linear penalty. The quadratic form $(\text{len}_{\text{sum}} / \text{len}_{\text{raw}})^2$ is chosen over a linear penalty $\text{len}_{\text{sum}} / \text{len}_{\text{raw}}$ because (as the paper explains in Section 4.2.2) "it minimizes the expected overshoot $\mathbb{E}[(r - \tau)_+]$, whereas a linear form primarily reduces the tail probability $\Pr(r > \tau)$." Here $r = \text{len}_{\text{sum}}/\text{len}_{\text{raw}}$ is the compression ratio and $\tau$ is a target threshold. The quadratic penalty makes large violations (summaries much longer than the target) disproportionately costly, driving the expected magnitude of threshold violations down. A linear penalty would only control how often violations occur, not how severe they are. For throughput optimization, the severity matters more: a few very long summaries in a batch can bottleneck the entire batch's latency, so penalizing the magnitude of length excess is the right objective.

  • Asymmetric reward signal. Unlike standard RLHF where the reward model approximates human preferences, here the reward model (the SLM) is the exact downstream consumer of the summaries. There is no gap between what the reward model prefers and what matters for the application—the SLM's output is what matters. This makes the RL training unusually well-aligned with the deployment objective.

Length penalty formulations: P1 and P2. The paper experiments with two specific length penalty functions (Equations 5 and 6 in Section 4.2.2), motivated by different assumptions about the desired compression behavior:

undefined