Decoding ML Decision: An Agentic Reasoning Framework for Large-Scale Ranking System

ArXiv: 2602.18640

🎯 Pitch

Major gains in ranking optimization aren’t stuck on better models, but on a critical translation bottleneck: turning fuzzy product goals into testable hypotheses. GEARS cracks this by letting operators steer systems with high-level ‘vibes’ while an agent autonomously discovers deployable policies—but only after a strict stability gate filters out strategies that overfit short-term noise.

---

1. Executive Summary

This paper introduces GEARS (Generative Engine for Agentic Ranking Systems), a framework that reframes large-scale ranking optimization as an autonomous discovery process within a programmable experimentation environment, evaluated across 20 internal experiments with 100 synthesized policy-selection instructions using Claude Sonnet as the backbone LLM. GEARS contributes two named mechanisms—Specialized Agent Skills (modular, filesystem-based resources that encapsulate expert ranking knowledge into reusable reasoning capabilities, enabling operators to steer systems via high-level intent rather than low-level parameter tuning) and Deterministic Lifecycle Governance (validation hooks that audit feature stability over a 6-month window and filter out policies that overfit transient signals)—alongside a Tolerance-Based Frontier Expansion algorithm that admits near-Pareto candidates into the selection process. The full framework achieves an NDCG@1 of 0.94 and Top-1 Accuracy of 0.86, substantially outperforming the strongest baseline (Code-as-Action at NDCG@1 of 0.77 and Top-1 Accuracy of 0.68), and delivers statistically significant metric lifts across nine production surfaces—establishing that agent-driven ranking optimization can discover deployable, stable policies that generalize beyond short-term experimental windows only when candidates are subjected to deterministic stability governance rather than purely statistical selection.

2. Context and Motivation

The Core Problem: The Engineering Context Constraint Bottleneck

Modern large-scale ranking systems at companies like Meta operate at staggering scale: they orchestrate heterogeneous product surfaces (recommendation feeds, social interfaces, discovery surfaces) serving a billion-scale global user base with multifaceted preferences. Over decades of iterative development, these systems have accumulated highly intricate architectures where numerous optimization layers concurrently target diverse and often conflicting metrics. The paper argues that this complexity has shifted the primary bottleneck to system advancement away from pure signal estimation (i.e., building better predictive models) and toward what it calls the engineering context constraint: the arduous translation of product intuition and domain expertise into auditable, executable hypotheses.

What does "engineering context constraint" mean concretely? Consider a scenario where a product manager wants to improve long-term user engagement on a recommendation surface without degrading short-term session quality. A data scientist might identify 50 statistically promising user cohorts and treatment combinations through uplift modeling. But only a fraction of these are actually deployable. The reasons are manifold and deeply practical:

• Feature instability: A cohort defined by a behavioral feature that shifts substantially month-over-month (e.g., a user's content consumption pattern that changes with seasonal trends) will produce a policy that degrades rapidly after deployment.
• Operational constraints: The ranking infrastructure may only support a certain number of concurrent treatment groups, or may have latency budgets that preclude complex per-cohort logic.
• Non-differentiable business "vibes": Qualitative goals like "the experience should feel natural" or "we don't want to cannibalize our other surface" are difficult to encode as loss functions but critical to deployment decisions.
• Cohort-treatment interactions: A treatment that improves engagement for power users may simultaneously harm the experience for new users, and the trade-off must be explicitly negotiated.

The paper's key observation is that current workflows remain tethered to manual intervention. Domain experts must manually navigate the combinatorial complexity of multi-objective trade-offs, treatment interpretability, and alignment with evolving business criteria. This creates a scalability barrier: the space of possible policies is combinatorially vast (cohorts × treatments × constraints), but human bandwidth for evaluation is fixed and limited. Consequently, high-value policies remain undiscovered not because they don't exist, but because no human has the time to find them.

Why This Problem Matters: The Deployment Gap

The practical significance of this problem is immense, and the paper articulates it through the concept of a deployment gap: the disconnect between statistically promising policies identified offline and policies that are actually deployable in production. This gap has real economic consequences — it represents optimization potential that is left on the table, directly translating to suboptimal user experiences and missed engagement opportunities at billion-user scale.

Consider the workflow that GEARS aims to replace. In a traditional setup, after an A/B experiment concludes, a data scientist might spend days or weeks:

1. Running uplift models to identify responsive cohorts.
2. Manually checking each promising cohort for feature stability (is the segmentation feature deprecated? Does it shift over time?).
3. Cross-referencing against infrastructure constraints (can this cohort be targeted in production?).
4. Iterating with engineers to understand why certain policies that look good offline fail to ship.
5. Writing documentation justifying the chosen policy to stakeholders.

The paper estimates that this process can take multiple weeks per experiment for an expert-driven discovery workflow. At Meta's scale, with hundreds of experiments running concurrently across dozens of surfaces, the cumulative engineering overhead is staggering. The paper's case study in Section 5.1 makes this concrete: deploying a cohort-targeted policy that balances two competing engagement metrics required GEARS to automate what was "previously a multi-week, expert-driven discovery process."

Beyond the immediate efficiency gains, there is a theoretical significance to this problem. The deployment gap exposes a fundamental limitation in how the field conceptualizes ranking optimization. Traditional approaches like uplift modeling and heterogeneous treatment effect (HTE) estimation frame the problem as static model selection: find the policy that maximizes expected reward given historical data. But in real-world systems, the objective function is not just "maximize metric lifts" — it includes implicit constraints about stability, interpretability, and operational feasibility that are difficult to formalize mathematically. The paper's central insight is that bridging this gap requires not better optimization algorithms, but a different paradigm entirely: one that treats the experimentation ecosystem as an interactive environment where agents can reason about deployment feasibility alongside statistical performance.

Prior Approaches and Where They Fall Short

The paper situates GEARS against three broad categories of prior work, each of which addresses a piece of the problem but leaves critical gaps.

Uplift Modeling and HTE Estimation

Traditional uplift modeling approaches — meta-learners (Künzel et al., 2019), tree-based methods (Zhao et al., 2017; Athey and Imbens, 2016), and neural network-based methods (Wei et al., 2024; Louizos et al., 2017) — seek to identify user segments most responsive to specific interventions. These methods have been deployed at Meta for over a decade through systems like Smart Scorer (Peysakhovich and Lada, 2016; Lada et al., 2019).

The paper identifies three critical limitations of these approaches:

1. Static feature dependence: Most existing approaches "rely heavily on offline features and static user profiles, which limits their ability to respond to real-time shifts in user behavior or to dynamically adapt intervention strategies." A model that performs well on historical data may select policies based on feature distributions that have since shifted, producing recommendations that are optimal on paper but brittle in deployment.

2. Blindness to ranking context: The paper explicitly states that methods like Smart Scorer "are only intended to solve algorithmic problems, without incorporating any ranking context." In large-scale ranking systems, any deployable solution must account for business objectives, existing ranking policies, and cannibalization across products and systems. An uplift model that identifies a statistically optimal cohort-treatment pair cannot assess whether deploying that policy would violate a guardrail metric on a related surface or conflict with an existing product strategy.

3. Manual deployment workflows: Even when uplift models identify promising policies, "the design and deployment of these models often require manual workflows that are time-consuming and resource-intensive." The algorithmic output (a set of cohort-treatment pairs with estimated lifts) must still be manually vetted, interpreted, and translated into production configurations — a process that reintroduces the engineering context constraint at the deployment stage.

A crucial nuance the paper highlights: uplift models can identify policies that are statistically optimal but operationally undeployable. For example, a policy might target users based on a transient behavioral feature that shifts 50% over six months (the paper documents exactly this scenario with "Feature 4" in Table 2). A pure statistical approach would eagerly select this policy for its high lift, unaware that the segmentation will degrade before the policy can be shipped.

Adaptive Experimentation (AE)

Adaptive Experimentation methods like Multi-Armed Bandits and Bayesian Optimization take a different approach: rather than learning a static model, they optimize outcomes through sequential decision-making and active exploration, balancing exploration and exploitation to identify optimal configurations dynamically. Meta has invested substantially in this area through platforms like Ax (Olson et al., 2025; Wu et al., 2022; Bakshy et al., 2018).

AE addresses the "static" limitation of uplift modeling — it can adapt to feedback over time. However, the paper identifies a different gap: AE methodologies are parameter-space optimizers, not context-aware reasoning systems. They excel at efficiently searching vast parameter spaces (e.g., tuning ranking model weights, exploring treatment variants), but they do not reason about the broader context that determines whether an optimal configuration is deployable. An AE system might converge on a treatment that maximizes engagement lift, but it cannot assess whether the features used to target that treatment are stable over time, whether the treatment violates infrastructure constraints, or whether the policy aligns with qualitative product intent.

The paper's framing is instructive here: AE solves the exploration efficiency problem, but not the deployment feasibility problem. GEARS builds on top of AE-style candidate generation (through GAS, described in Section 3.1) but adds the reasoning and validation layers that AE methods lack.

LLM-Based Agents and Context Engineering

The paper also positions itself against emerging "LLM-as-Ranker" approaches and general-purpose agent frameworks. Recent work on Tool-Integrated Reasoning (Qian et al., 2025; Mialon et al., 2023; Dong et al., 2025), self-refinement mechanisms (Madaan et al., 2023; Shinn et al., 2023), and memory systems (Packer et al., 2023; Zhong et al., 2023) demonstrates that LLMs can interact with environments, execute tools, and improve their own outputs through iterative feedback.

However, the paper identifies a critical failure mode it terms "context rot": the performance degradation LLMs experience with overly long prompts as they accumulate context over extended reasoning trajectories. In the ranking domain, where decisions require synthesizing statistical evidence, feature metadata, infrastructure constraints, and historical learnings, the context window can easily become bloated with information that is semantically plausible but insufficiently grounded. The paper notes that generic LLM agents "often suffer from hallucination and 'context rot', generating policies that are semantically plausible but operationally brittle."

More fundamentally, prior LLM agent work lacks three capabilities that GEARS specifically targets:

1. Domain-specific skill encapsulation: General-purpose agents rely on prompting or tool descriptions to access domain knowledge. GEARS introduces modular, filesystem-based Specialized Agent Skills that encode structured instructions, operational procedures, and references to internal artifacts (SQL scripts, codebases) — a much richer form of knowledge grounding than simple tool descriptions.

2. Deterministic validation: Existing agent frameworks typically evaluate outputs through LLM-based critics or self-reflection (which can hallucinate) or through generic tool execution (which can succeed technically while producing operationally invalid results). GEARS introduces deterministic hooks that audit feature stability, cohort consistency, and performance persistence against predefined, reproducible criteria — not LLM judgments.

3. Deployment-oriented lifecycle management: Prior work treats agent outputs as terminal. GEARS embeds governance checks throughout the agent's decision process, filtering candidates before they reach the selection stage and providing structured feedback that triggers refinement loops when policies fail validation.

The recent finding by Cemri et al. (2025) — that multi-agent LLM systems often produce outputs that appear locally reasonable yet fail to generalize due to insufficient verification — is cited as direct evidence for why governance is necessary. This paper provides the validation infrastructure that prior agent work lacked.

How This Paper Positions Itself

GEARS positions itself at the intersection of three research threads — uplift modeling, adaptive experimentation, and LLM-based agents — while explicitly arguing that existing approaches in each thread are individually insufficient. The paper's contribution is not a new optimization algorithm, a new bandit method, or a new prompting technique. Rather, it is a system architecture that composes these elements into a coherent workflow where:

• Uplift modeling (specifically, the GAS framework; Wu et al., 2025) serves as the candidate generation engine, efficiently exploring the combinatorial policy space and approximating the Pareto frontier.
• LLM-based reasoning (through Claude Sonnet) serves as the orchestration layer, interpreting high-level product intent, selecting appropriate analytical skills, and generating interpretable policy recommendations.
• Deterministic governance (the novel contribution) serves as the safety layer, enforcing stability criteria that neither uplift models nor LLMs can reliably assess on their own.

The paper frames this composition as a paradigm shift from "ranking optimization as static model selection" to "ranking optimization as autonomous discovery within a programmable experimentation environment." This is not merely rhetorical: the shift is substantive because it changes what the system optimizes for. Traditional approaches optimize for expected metric lift given historical data. GEARS optimizes for deployable metric lift given stability constraints and operational context — a strictly harder problem that requires reasoning about factors invisible in the training data.

The paper's novelty claim rests on three specific, named innovations that differentiate GEARS from prior work:

1. Specialized Agent Skills: Unlike general-purpose tool descriptions or prompt templates, these are modular, filesystem-based resources that encapsulate expert ranking knowledge into executable procedures with three components (metadata for routing, structured instructions, internal artifact references). This is distinct from prior work on tool-augmented LLMs because the skills encode domain-specific analytical workflows (trade-off diagnosis, feature interpretation) rather than generic tool interfaces.

2. Deterministic Lifecycle Governance: Unlike LLM-based self-critique or generic output evaluation, this introduces validation hooks that enforce reproducible, quantitative stability criteria — specifically, feature stability over a 6-month window measured by user-cohort shift ratios, with explicit thresholds derived from empirical baselines. This is not a prompting technique; it is a software engineering pattern embedded in the agent's execution environment.

3. Vibe Optimization: A paradigm where operators steer ranking systems through high-level natural language intent (e.g., "prioritize long-term engagement while maintaining guardrails") rather than low-level parameter tuning. The agent handles translating qualitative "vibes" into explicit algorithmic constraints — a workflow that prior systems required human experts to perform manually.

The paper is careful to note that GEARS is a general framework, not a personalization-specific solution. Personalization is chosen as the illustrative application because it demonstrates the combinatorial complexity (cohorts × treatments × constraints) and deployment feasibility requirements that motivate the framework. But the architecture — intent translation, skill-based reasoning, deterministic governance — is intended to generalize across ranking optimization tasks.

Finally, the paper positions its empirical contribution as production validation, not benchmark evaluation. The offline experiments on 100 synthesized instructions demonstrate that GEARS outperforms prompting baselines on policy selection metrics. But the real evidence is in Table 3 and the Section 5 case study: statistically significant metric lifts across nine production surfaces, with the agent automating what was previously a multi-week expert-driven process. This production grounding distinguishes GEARS from agent frameworks that are evaluated solely on synthetic benchmarks and addresses the practical concern that agent-driven optimization might discover statistically appealing but operationally invalid policies — a concern that the paper's governance mechanism is specifically designed to prevent.

3. Technical Approach

This is a systems architecture paper whose core idea is that ranking optimization should be reframed not as a static model selection task, but as an autonomous discovery process where an LLM-based agent navigates a programmable experimentation environment, generating and validating policies through a pipeline that composes candidate generation (via heterogeneous treatment effect estimation), insight-driven selection (via specialized agent skills that encode expert ranking knowledge), and deterministic lifecycle governance (via validation hooks that audit stability before deployment).

3.1 Reader Orientation

GEARS is a framework that wraps an LLM agent around Meta's existing experimentation infrastructure to automate the end-to-end workflow of discovering, validating, and recommending ranking policies from A/B experiment data. It solves the problem that statistically optimal policies identified by uplift models are frequently undeployable because they ignore the "engineering context" — feature stability, infrastructure constraints, and qualitative business goals — by inserting deterministic stability checks and domain-expert reasoning into the policy selection pipeline, so the agent can reject brittle candidates that purely statistical methods would promote.

3.2 Big-Picture Architecture (Diagram in Words)

The GEARS system has five major components arranged as a pipeline:

1. Intent Translation Module — takes a natural-language description of desired trade-offs (e.g., "prioritize long-term engagement while maintaining guardrails") and converts it into a structured search specification with objectives, cohort constraints, and experiment parameters. This is the interface between human operators and the autonomous agent.

2. Candidate Generation Engine (GAS) — an existing heterogeneous treatment effect (HTE) framework that efficiently explores the combinatorial space of user segments × treatments, producing hundreds of candidate policies with metric measurements and confidence intervals. GEARS extends GAS with a tolerance-based frontier expansion mechanism that admits near-Pareto candidates, not just strictly optimal ones.

3. Insight Generator (Agent + Skills) — an LLM-based reasoning layer (Claude Sonnet) that orchestrates domain-specific analytical operations by activating modular Specialized Agent Skills. Each skill is a filesystem-based resource encoding structured analytical instructions, operational procedures, and references to internal artifacts (SQL scripts, codebases). The agent also draws on a Domain Knowledge Brain — a curated repository of historical experimental learnings — to ground its reasoning in past empirical evidence. A progressive disclosure strategy prevents context-window bloat by loading skill metadata first, full instructions only upon activation, and internal scripts only during execution.

4. Deterministic Lifecycle Governance — a set of validation hooks that audit every policy candidate against reproducible stability criteria before it can be recommended. These hooks check feature stability over a 6-month window (using user-cohort shift ratios), statistical reliability, and cohort consistency. Policies that fail these checks are rejected with structured feedback to the agent, triggering refinement.

5. Output: Production-Ready Configuration — the final deliverable is a ranked list of validated, deployable policies with accompanying interpretability analysis, including the rationale for why each policy was selected and how it navigates the multi-objective trade-off space.

Information flows sequentially: a human operator provides high-level intent → the Intent Translation module produces a search specification → GAS generates a high-recall set of candidate policies → the agent activates relevant Skills to analyze candidates, consulting the Domain Knowledge Brain for historical context → Governance hooks filter out unstable or unreliable policies → the agent produces a ranked recommendation with justification. If Governance rejects a candidate, structured feedback loops back to the agent for refinement.

3.3 Roadmap for the Deep Dive

• First, the Intent-Conditioned Personalization module (Section 3.1) — the entry point where natural-language "vibes" become executable search specifications and where GAS generates candidate policies, including the Tolerance-Based Frontier Expansion algorithm that broadens the candidate set beyond convex Pareto-optimal solutions.
• Second, the Insight-Driven Policy Selection module (Section 3.2) — the agent's reasoning core, covering Specialized Agent Skills (their three-part structure and how they encode expert knowledge), the Domain Knowledge Brain (how it grounds the LLM in historical evidence), and the progressive disclosure strategy (how GEARS manages context-window constraints during long-horizon reasoning).
• Third, Deterministic Lifecycle Governance (Section 3.3) — the validation infrastructure that enforces long-term stability, including the exact hook mechanisms, the stability criteria (6-month window, user-cohort shift ratios, explicit empirical thresholds from Table 2), and the feedback loop that triggers policy refinement.
• Fourth, the GAS algorithm and Pareto frontier approximation (Appendix B) — the mathematical foundation for candidate generation, including the scalarized weight-search procedure and the quantile-based segmentation strategies that define the combinatorial policy space.
• Fifth, the Tolerance-Based Frontier Expansion algorithm (Appendix C) — the specific two-stage procedure that admits near-Pareto candidates, with the tolerance-dominance condition and its dependence on per-metric uncertainty estimates.

This order follows the system's information flow: from human intent → candidate generation → agent reasoning → validation → output. The appendices on GAS and tolerance expansion are included last because they provide the mathematical grounding for components introduced earlier in the pipeline.

3.4 Detailed, Sentence-Based Technical Breakdown

Intent-Conditioned Personalization: Translating "Vibes" into Executable Search

The first stage of the GEARS pipeline is the Intent-Conditioned Personalization module (Section 3.1), which bridges the semantic gap between a human operator's high-level qualitative goals and the structured numerical optimization that the downstream system can execute. The paper describes this module as focusing on "structured exploration rather than final policy selection" — meaning its job is to generate a high-recall set of candidate policies that the subsequent agent reasoning and governance layers can filter and refine.

The input to this module is a natural-language intent describing desired trade-offs. The paper gives the example: "prioritizing long-term engagement while maintaining guardrails." This is deliberately vague — the operator does not specify which features to use, what cohort definitions to consider, what statistical thresholds to apply, or how to weight competing objectives. The module's responsibility is to convert this qualitative "vibe" into a search specification with three concrete components:

1. Objectives: which metrics to optimize (primary objectives) and which metrics to protect (guardrail metrics).
2. Cohort constraints: which user segmentation strategies to explore, derived from the available feature space. The paper's GAS framework (detailed in Appendix B) uses quantile-based segmentation over user features, including both individual splits (dividing a feature's distribution into N equal quantile bins) and binary splits (thresholding at a specific quantile to create two segments).
3. Experiment parameters: bounds on the search space, such as which treatments to consider and what statistical significance thresholds to apply.

The translation process itself is not described in exhaustive algorithmic detail in the main paper, but the architecture is clear: the LLM agent interprets the natural-language intent and constructs the corresponding search specification by activating appropriate Skills (e.g., a skill for parsing multi-objective trade-off language, a skill for identifying relevant features from the experiment metadata). This specification is then passed to the GAS (Generalized Adaptive Segmentation) framework (Wu et al., 2025), which executes the actual candidate generation.

GAS: Candidate Generation via Weighted Scalarization

GAS is a large-scale Heterogeneous Treatment Effect (HTE) framework that matches optimal interventions to specific user segments across conflicting objectives. The paper uses GAS as a black-box candidate generator within GEARS, but provides sufficient mathematical detail (in Appendix B) to understand what it computes.

Policy definition. A policy in GAS is a mapping from user segments to treatment actions. Formally, given a set of M actions (treatments) A = {a₁, ..., a_M} plus a control action a₀, a policy P is represented as a set of B (segment, action) pairs:

$P = \{(S_1, a_1), \ldots, (S_B, a_B)\}$

where the segments S₁, ..., S_B form a complete partition of the user population (every user belongs to exactly one segment), and each segment is assigned exactly one treatment action.

The "combinatorial" nature of the policy space that the paper emphasizes comes from this definition: with D features, N quantile bins per feature, and both individual and binary split strategies, the number of possible segmentations grows combinatorially. For each segmentation, there are M^B possible treatment assignments (each of B segments can receive one of M treatments). This is the space that GAS must efficiently search.

Quantile-based segmentation. GAS defines candidate segments through two quantile-based strategies over user features. Let Q(X, p) denote the 100p-th percentile of feature X in the user population, with Q(X, 0) = -∞ by convention, and let N be the number of quantile bins (a hyperparameter).

For an individual split, a feature X is divided into N segments:

$S^{\text{ind}}_i(X) = \left\{ u \;\middle|\; Q\left(X, \frac{i-1}{N}\right) < X(u) \leq Q\left(X, \frac{i}{N}\right) \right\}$

where u denotes a user and X(u) is that user's feature value. This creates N non-overlapping segments that partition the full range of X.

For a binary split at threshold index i₀ (where i₀ ∈ {1, ..., N-1}), the feature is divided into two segments — users below or at the threshold, and users above it:

$S^{\text{bin}}_{1,i_0}(X) = \left\{ u \;\middle|\; Q(X, 0) < X(u) \leq Q\left(X, \frac{i₀}{N}\right) \right\}$

$S^{\text{bin}}_{2,i_0}(X) = \left\{ u \;\middle|\; Q\left(X, \frac{i₀}{N}\right) < X(u) \leq Q(X, 1) \right\}$

The binary split strategy captures the intuition that treatment effects often differ between "high" and "low" users on a dimension (e.g., highly active vs. less active users), while the individual split strategy captures finer-grained heterogeneity when treatment effects vary more continuously.

Segment-level treatment effects. For any segment S (a subset of users), treatment a_j, and metric δ_k, GAS computes the segment-level heterogeneous treatment effect as the average difference in outcomes between users in S who received treatment a_j and users in S who received the control a₀:

$\text{HTE}(S, a_j, \delta_k) = \frac{1}{|S|} \sum_{u_i \in S} \left( \delta_k(u_i, a_j) - \delta_k(u_i, a_0) \right)$

where |S| is the number of users in the segment, δ_k(u_i, a_j) is the observed outcome of user u_i under action a_j on metric δ_k, and δ_k(u_i, a₀) is the outcome under control.

What it computes: for a given segment, treatment, and metric, this is the average lift (or reduction) that users in that segment experience from the treatment compared to what they would have experienced under control. A positive value means the treatment improves the metric for that segment; a negative value means it harms the metric. This is the fundamental signal that policies are built on.

Why this form: segment-level aggregation is necessary because individual-level treatment effects are unobservable — we never see the same user simultaneously under treatment and control. The segment-level HTE approximates the average individual treatment effect within the segment by leveraging the randomized assignment: users in treatment and control groups within S are, in expectation, comparable on all dimensions except the treatment received. Computing this per-segment (rather than globally as an Average Treatment Effect) is what enables personalization — different segments may respond differently to the same treatment.

Pareto frontier via random weight search. Because GAS must optimize over K potentially conflicting metrics simultaneously, it uses a scalarization approach: convert the multi-objective problem into a set of single-objective problems by taking weighted sums of the per-metric lifts, then solve each single-objective problem to get a point on the Pareto frontier.

For a policy P = {(S₁, a₁), ..., (S_B, a_B)} and a weight vector w = (w₁, ..., w_K) ∈ W ⊂ ℝ^K (where W is a set of sampled weight vectors from the (K-1)-dimensional simplex — meaning all weights are non-negative and sum to 1), the scalarized objective is:

$\sum_{b=1}^{B} \sum_{k=1}^{K} \frac{w_k}{|S_b|} \sum_{u_i \in S_b} \delta_k(u_i, a_b)$

where |S_b| is the number of users in segment b, and a_b is the treatment assigned to that segment.

What it computes: for a given weight vector, this is the total weighted sum of metric lifts across all segments in the policy. The inner sum ∑_{u_i ∈ S_b} δ_k(u_i, a_b) aggregates the raw outcomes (not lifts) for all users in segment b under their assigned treatment on metric k. The 1/|S_b| normalizes by segment size so that larger segments don't dominate. The outer sums aggregate across metrics (with weights w_k) and segments. Essentially, this is the policy's "score" under a particular set of trade-off preferences — if w heavily weights metric 1, policies that excel on metric 1 will score highly.

Why this form: scalarization is a standard technique for multi-objective optimization because it reduces a vector-valued optimization to a scalar-valued one. By sampling many weight vectors w and solving for each, GAS approximates the Pareto frontier — the set of policies where improving one metric necessarily degrades another. The paper notes that "standard scalarization approaches used to approximate the Pareto frontier tend to emphasize convex regions of the trade-off space," which motivates the tolerance-based expansion described next.

The resulting Pareto policy set is:

$P_{\text{pareto}} := \left\{ \arg\max_a \sum_{b=1}^{B} \sum_{k=1}^{K} \frac{w_k}{|S_b|} \sum_{u_i \in S_b} \delta_k(u_i, a_b) \;\middle|\; w \in W \right\}$

where arg max_a means selecting the treatment assignment a = (a₁, ..., a_B) that maximizes the scalarized objective for the given weight vector w.

In practice, GAS solves this approximately: it samples W weight vectors from the simplex, computes segment-level HTEs for all candidate segments and treatments, evaluates the scalarized objective for each candidate policy, and retains the top-performing policies per weight vector. The paper references this procedure as the basis for the candidate pool that downstream GEARS components analyze.

Tolerance-Based Frontier Expansion: Admitting Near-Pareto Candidates

A key limitation of standard Pareto frontier construction is that it only includes policies that are strictly non-dominated — meaning no other policy is better on at least one metric while being at least as good on all others. In real ranking systems, policies that are slightly worse on one metric but substantially more stable or operationally robust may be preferable. The paper formalizes this insight through a Tolerance-Based Frontier Expansion algorithm (Section 3.1, with full pseudocode in Appendix C).

The algorithm has two stages:

Stage 1: Candidate collection via random weight search. Exactly as in standard GAS, the algorithm samples W weight vectors from the (M-1)-dimensional simplex (where M is the number of metrics), computes the scalarized objective score for every policy in P, and retains the top-K policies per weight vector (where K is a hyperparameter controlling recall). All retained policies are added to a candidate set C. This is essentially a high-recall broadening of the standard Pareto procedure: instead of keeping only the single best policy per weight vector, keep the top-K, which may include policies that are slightly suboptimal on the scalarized objective but strong on individual metrics.

Stage 2: Tolerance-based Pareto filtering. Standard Pareto dominance uses a strict criterion: policy q dominates policy p if q is at least as good as p on all metrics AND strictly better on at least one metric. The tolerance-based variant relaxes this by introducing a tolerance margin ε_m(p) for each policy p on each metric m, defined as:

$\epsilon_m(p) = \tau \cdot \sigma_m(p)$

where τ ≥ 0 is a tolerance hyperparameter (controlling how much "slack" to allow) and σ_m(p) is the estimated uncertainty (standard error) of policy p's mean on metric m.

What it computes: the tolerance margin is proportional to the statistical uncertainty of the policy's metric estimate. A policy with high uncertainty on a metric gets a wider tolerance band — meaning it's harder for another policy to strictly dominate it on that metric, because the dominant policy must clear the uncertainty band, not just the point estimate.

Why this form: uncertainty-proportional tolerances prevent the algorithm from prematurely discarding policies whose apparent suboptimality might be due to estimation noise rather than genuine inferiority. A policy that appears slightly worse on a metric but has high variance might actually be equivalent in expectation; the tolerance band accounts for this. Setting τ = 0 recovers standard strict Pareto dominance; larger τ values admit more near-Pareto candidates.

Under tolerance-based dominance, policy q tolerance-dominates policy p (written q ≻_τ p, assuming all metrics are to be maximized) if:

$(\forall m,\; \mu_m(q) \geq \mu_m(p) - \epsilon_m(p)) \;\wedge\; (\exists m,\; \mu_m(q) > \mu_m(p) + \epsilon_m(p))$

where μ_m(p) and μ_m(q) are the estimated mean metric values for policies p and q on metric m. In words: q tolerance-dominates p if q is at least as good as p on every metric within p's tolerance band (first condition), AND q is strictly better than p on at least one metric even after accounting for p's tolerance band (second condition).

The final output is the set C_τ of all policies in the candidate pool C that are not tolerance-dominated by any other policy in C. This set includes both strict Pareto-optimal policies and near-Pareto policies that fall within tolerance margins, expanding the candidate frontier to include operationally robust but slightly suboptimal configurations.

The algorithm's pseudocode (Appendix C, Algorithm 1) clarifies the implementation: for each candidate policy p, the algorithm checks all other candidates q to see if any tolerance-dominates p. If none does, p is admitted to C_τ. This is O(|C|²) in the size of the candidate set but is applied after the candidate collection stage, so |C| is bounded by W × K (number of weights times top-K per weight).

Insight-Driven Policy Selection: Agent Reasoning with Specialized Skills

Once the Candidate Generation engine produces a high-recall set of policies (and the Tolerance-Based Frontier Expansion broadens it to include near-Pareto candidates), the next stage is Insight-Driven Policy Selection (Section 3.2). This is where the LLM agent, powered by Specialized Agent Skills and grounded by the Domain Knowledge Brain, analyzes the candidate pool to select and rank deployable policies.

The paper frames this stage as addressing a critical gap: "candidates are frequently undeployable in production systems due to unobserved factors such as feature instability, infrastructure constraints, or misalignment with specific business guardrails." These factors are "unobserved" in the sense that they are not encoded in the candidate generation's objective function — GAS optimizes for metric lifts, not feature stability or infrastructure compatibility. The agent's job is to surface these hidden constraints and filter accordingly.

Specialized Agent Skills: Encapsulating Expert Knowledge into Executable Procedures

The core architectural innovation in this stage is the Specialized Agent Skill — a modular, filesystem-based resource that externalizes expert ranking knowledge into reusable, executable computational capabilities. The paper is explicit that these are "distinct from conventional static prompt templates." Rather than injecting all domain knowledge into a single prompt (which the paper argues leads to "context rot" — performance degradation from overly long prompts), Skills are loaded dynamically and executed procedurally.

Each Skill is structured as a three-component tool:

1. Lightweight metadata for efficient agent routing. This is a compact description of what the skill does, when it should be activated, and what inputs it requires. The metadata is exposed to the agent at all times (via the progressive disclosure strategy described below) so the agent can decide which skills to activate for a given reasoning task without loading the full skill contents into context.

2. Structured step-by-step analytical instructions. When the agent activates a skill, the full instructions are loaded. These instructions encode the procedural knowledge that a human ranking expert would apply: what analyses to run, in what order, what statistical checks to perform, what thresholds to apply, and how to interpret the results. For example, a "trade-off diagnosis" skill might specify: (a) identify the Pareto frontier from candidate metrics, (b) compute the slope of the trade-off curve (how much of metric A must be sacrificed for a unit gain in metric B), (c) check whether any candidates achieve both objectives within acceptable bounds, (d) flag candidates where the trade-off is favorable vs. unfavorable. These instructions are domain-specific — they encode ranking expertise, not generic analytical reasoning.

3. Direct references to internal artifacts such as SQL scripts and codebases. During execution, the skill can access and invoke actual production code — not just describe what it would do. For example, a "feature stability audit" skill might reference a specific SQL script that queries feature value distributions over a 6-month window and computes shift metrics. The agent doesn't hallucinate or approximate the stability analysis; it executes the same scripts a human engineer would run.

The paper's examples of skill capabilities include feature explanation (interpreting what a segmentation feature means in the ranking context and whether it's appropriate for the current product surface), feature stability audits (assessing whether a feature's distribution shifts over time, which would cause a cohort-based policy to degrade), and guardrail interpretation (determining whether a policy violates implicit or explicit business constraints, such as not degrading a secondary metric beyond a specified threshold).

What makes Skills different from standard LLM tool-use. In typical tool-augmented LLM frameworks, tools are described through natural-language function signatures — the model sees a description like "search_database(query: str) -> results" and can invoke it. GEARS Skills go beyond this in two ways. First, they encode procedural workflows, not just atomic operations. A Skill isn't "run this SQL query"; it's "run this sequence of analyses in this specific order, with these interpretation guidelines, to produce a deployability assessment." Second, they are filesystem-based and modular — they exist as persistent, version-controlled resources that can be updated independently of the agent and shared across different GEARS deployments. This means ranking experts can author and iterate on Skills without modifying the agent architecture.

Domain Knowledge Brain: Grounding the Agent in Historical Evidence

The execution of Skills alone is insufficient because the agent lacks "platform-specific context (e.g., past experience)." A feature stability audit might show that a feature is technically stable, but a human expert would also know that this feature was deprecated in a related surface six months ago or that similar policies targeting this feature failed to generalize in previous experiments due to infrastructure changes. The paper addresses this through the Domain Knowledge Brain — a curated reasoning backbone that supplies the LLM with "high-fidelity historical data and domain expert learnings."

The Knowledge Brain is described as "curated" — it is not a raw database of all past experiments, but a structured repository of distilled learnings: which features proved stable across multiple experiments, which cohort definitions consistently produced generalizable policies, which treatment types tend to interact with specific product surfaces in predictable ways. The paper describes its role as providing the "critical 'contextual evidence' behind past experiments" and systematizing innovation by ensuring the agent's recommendations are "empirically valid and deeply integrated with complex system dynamics."

This is distinct from a simple retrieval-augmented generation (RAG) approach because the Knowledge Brain is actively curated and maintained by domain experts — it's a knowledge base, not just a document store. The agent doesn't retrieve raw experiment reports; it accesses structured evidence summaries that have been validated and distilled by human experts. This curation step is what allows the agent to avoid the hallucination and grounding problems that plague generic RAG-based agents when applied to highly technical domains.

Progressive Disclosure: Preventing Context Rot

A critical implementation challenge for long-horizon LLM reasoning in data-rich environments is context rot: as the agent accumulates information (experiment metrics, feature metadata, historical evidence, skill instructions, intermediate analyses), the context window fills with content that is "semantically plausible but operationally brittle" — the model loses track of which information is reliable, which constraints are active, and what the current reasoning step requires.

GEARS addresses this through progressive disclosure, a three-stage information loading strategy:

1. Stage 1 — Compact metadata exposure for skill retrieval. At all times, the agent sees only lightweight metadata for each available Skill (the first component of the skill structure described above). This is sufficient for the agent to decide which Skills are relevant to the current reasoning step without consuming context with full instructions or artifact references.

2. Stage 2 — Structured instructions injected upon skill activation. When the agent decides to activate a Skill (e.g., "run feature stability audit on Feature 3"), the full step-by-step analytical instructions are loaded into context. The agent now has the procedural knowledge needed to execute the analysis. But the internal artifacts (SQL scripts, code references) remain unloaded.

3. Stage 3 — Specific internal scripts accessed purely during execution, with subtasks delegated to subagents. Only when the agent actually needs to execute a script (e.g., to query the feature shift database) does it access the artifact. Furthermore, the paper notes that "subtasks delegated to subagents" — meaning that long-running or complex executions may be offloaded to separate agent instances with their own isolated context windows. The main agent receives only the results, not the full execution trace.

This staged design "maintains a clean context window, externalizes expertise efficiently, and prevents reasoning drift during long-horizon policy exploration." The paper doesn't provide exact context-window budgets or quantifiable metrics on how much context rot is reduced, but the architectural pattern is clear: only load information when it's needed, delegate heavy computation to subagents, and keep the main agent's context focused on the current reasoning step.

Agent Workflow for Policy Selection

The paper describes the agent's overall workflow in Section 3.2 as "autonomously executing complex, domain-specific tasks, such as trade-off diagnosis and feature interpretation." While the exact execution trace is not specified, we can reconstruct the likely flow:

1. Receive the candidate pool (output from GAS + Tolerance-Based Frontier Expansion) and the original search specification (from Intent Translation). The candidate pool contains, for each policy: the segmentation definition (which features, which quantile cutoffs), the assigned treatments per segment, the estimated metric lifts with confidence intervals, and any metadata flags (e.g., feature set version, experiment ID).

2. Activate diagnostic Skills based on the instruction type. For a "maximize with constraint" instruction, the agent might activate a constraint-checking skill that verifies the guardrail metric is within acceptable bounds for each candidate. For a "trade-off analysis" instruction, it might activate a Pareto-frontier visualization skill and a slope-analysis skill.

3. Consult the Domain Knowledge Brain for historical context on the features, treatments, and product surface involved. Has a similar cohort definition been used before? Did the feature prove stable across time? Were there infrastructure issues with deploying this treatment type?

4. Execute stability and robustness analyses by activating validation-oriented Skills. These may include the feature stability audit (which queries internal databases for feature distribution shifts), cohort consistency checks (do the treatment effects hold across temporal slices?), and guardrail interpretation (does this policy violate any implicit constraints from the intent specification?).

5. Synthesize findings into a ranked recommendation. The agent produces a ranked list of recommended policies, with justification for each ranking decision and interpretability analysis — explaining not just which policies are best, but why, in terms that a human operator can understand and trust.

The output of this stage is not yet the final recommendation — it passes through the Deterministic Lifecycle Governance layer (Section 3.3) before being delivered.

Deterministic Lifecycle Governance: Enforcing Long-Term Stability

The final and most novel component of the GEARS pipeline is Deterministic Lifecycle Governance (Section 3.3). While the agent reasoning stage applies domain expertise to interpret and evaluate policies, it still relies on LLM-based judgment, which can be inconsistent, hallucinatory, or insufficiently rigorous for production deployment decisions. The Governance layer introduces non-negotiable, reproducible stability criteria that every policy candidate must satisfy, enforced through deterministic validation hooks embedded in the agent's execution environment.

The paper motivates this component by citing Cemri et al. (2025), who showed that "multi-agent systems often produce outputs that appear locally reasonable yet fail to generalize due to insufficient verification." In ranking environments, failures of generalization typically arise from three sources:

• Overfitting transient signals: A policy identifies a cohort × treatment combination that shows strong lifts during the experiment window, but the lift is driven by temporary factors (seasonal effects, a concurrent product launch, sampling noise) that don't persist.
• Reliance on unstable features: A policy segments users based on a behavioral feature whose distribution shifts substantially over time. The cohort definition that was optimal during the experiment may target entirely different users six months later.
• Cohort-specific artifacts: A treatment effect appears strong in one temporal slice of the data but reverses or disappears in another, indicating that the apparent effect is not a genuine causal relationship.

The Governance layer addresses these failure modes through validation hooks — deterministic checks that audit every intermediate and final decision against reproducible stability criteria. The paper distinguishes these from LLM-based evaluation (like self-critique): hooks execute fixed, auditable procedures (SQL queries, statistical tests) that produce the same result every time, independent of the LLM's sampling or reasoning. The paper explicitly frames this as a "stability-first design" that "grounds agent-driven exploration in long-term reliability rather than short-term gains."

Governance Mechanisms: What the Hooks Check

The paper specifies three categories of validation that the governance hooks enforce:

1. Statistical reliability: ensuring that metric improvements are not driven by noisy or transient features. The paper does not specify the exact statistical tests used, but from context (the emphasis on feature stability), these likely include checks on the width of confidence intervals for metric lifts (policies with very wide intervals are flagged as unreliable), consistency of treatment effects across bootstrap resamples, and minimum sample-size requirements per segment (very small segments produce noisy estimates).

2. Feature consistency: ensuring that features used for segmentation remain stable over a 6-month window. This is the most quantitatively detailed hook, described in Section 4.3 with a benchmarking methodology.

3. Performance persistence: ensuring that policies maintain their metric lifts across temporal slices. The paper validates this through backtesting (Figure 4), showing that the selected policy's metric improvement "remains consistent over a period of one month."

Feature Stability Benchmark: User-Cohort Shift Ratio

The paper establishes a quantitative benchmark for feature stability (Section 4.3) that serves as the primary governance hook mechanism. The core metric is the User-Cohort Shift Ratio (R_shift): the percentage of users who migrate from their assigned cohort (bucket) to a different one over a 6-month window.

What it measures: consider a feature X used to define cohorts (e.g., a user engagement score). At time t = 0 (the experiment start), each user is assigned to a cohort based on their value of X — for example, "high engagement" if X > p75, "low engagement" otherwise. At time t = 6 months, the feature is re-measured. Some users who were "high engagement" at t = 0 are now "low engagement" at t = 6 months (and vice versa). The shift ratio R_shift is the fraction of users whose cohort membership changed. A high R_shift means the feature's distribution is unstable — the cohorts defined at experiment time no longer correspond to the same users at deployment time, so any policy targeting those cohorts will degrade.

Why 6 months: the paper uses a 6-month window because this corresponds to a typical timeframe for shipping a ranking policy from experiment to production. If the feature shifts substantially within this window, the policy will be stale before it even launches.

The paper benchmarks this metric using two cohort definitions:

• Quantile Cuts: divide users into 4 equal-sized buckets based on feature value. This is more sensitive to distribution shifts because any change in a user's relative position within the distribution can cause a bucket migration.
• Binary Cuts: divide users at the p25/p75 thresholds, creating "low" (bottom 25%), "middle" (25-75%), and "high" (top 25%) groups (though the paper primarily discusses binary cuts as a two-group split for simplicity). This is less sensitive because only users crossing the threshold boundary migrate; users who shift within their group don't count.

The paper establishes an empirical baseline by measuring R_shift for a "perceived stable" feature set (called Feature Set S) that domain experts consider reliable. As shown in Table 2, even this baseline exhibits drift: 6% shift under quantile cuts, 2% under binary cuts. This is the lower bound for unavoidable natural drift in the system — even the most stable features experience some user migration over six months.

Product features show much higher volatility. The paper documents two examples in Table 2:

• Feature 2 (stable): 16% shift under quantile cuts, 4% under binary cuts. This feature is considered deployable — its shift is above the baseline but within acceptable bounds.
• Feature 3 (stable under binary cuts): 30% shift under quantile cuts, 10-12% under binary cuts. This feature is flagged as stable only under the binary cut definition — under quantile cuts, its shift is too high. This illustrates why the governance layer checks multiple cohort definitions: a feature may be acceptable for binary targeting but not for fine-grained quantile targeting.
• Feature 4 (unstable): approximately 50% shift under quantile cuts, approximately 20% under binary cuts. This feature is categorically rejected for policy targeting — a 50% shift means half of the targeted users will have changed cohorts by the time the policy ships.
• Feature 5 (unstable): shift data not available under quantile cuts (possibly because the feature's distribution is too irregular for quantile cuts), approximately 30% under binary cuts. Also rejected.

Pre-Search Filter: Automatic Feature Disqualification

Based on these benchmarks, GEARS implements a pre-search filter as a governance hook:

• Features must exhibit R_shift ≤ 15% under binary cuts OR R_shift ≤ 45% under quantile cuts to enter the candidate generation search space.

What this means operationally: before GAS even generates candidate policies, the governance layer audits all available features for stability. Features exceeding these thresholds are removed from the feature pool, so no candidate policy can be generated that targets those features. This is a "pre-search" filter because it operates before the computationally expensive candidate generation stage — it prevents the system from wasting computation exploring policies that are guaranteed to be undeployable due to feature instability.

Why these thresholds: the paper doesn't explicitly justify the specific numbers, but they can be inferred from the benchmarking data. The binary cut threshold (R_shift ≤ 15%) is set between Feature 3's 10-12% (stable) and Feature 4's ~20% (unstable), creating a conservative boundary that admits Feature 3 while rejecting Feature 4. The quantile cut threshold (R_shift ≤ 45%) similarly falls between Feature 3's 30% (flagged as stable only under binary) and Feature 4's ~50% (unstable), though the exact rationale is less clear from the data presented. The asymmetry in thresholds (15% for binary, 45% for quantile) reflects that quantile cuts inherently produce higher migration rates (more granular bucketing means more opportunities to cross boundaries), so the threshold must be correspondingly higher.

Feedback Loop: Rejection Triggers Refinement

The Governance layer is not a one-pass filter. When a candidate policy fails a validation hook, "the system rejects the policy and provides structured feedback to the agent, triggering a refinement loop." The paper doesn't provide extensive detail on what this feedback looks like, but from the system architecture, we can infer: the agent receives information about which hook failed (e.g., "Policy #47 rejected: Feature 4 R_shift = 52% under quantile cuts, exceeds threshold of 45%") and can adjust its reasoning — for example, exploring alternative feature combinations, relaxing cohort definitions, or flagging to the human operator that no stable policy exists for the current constraints.

This feedback loop is what makes GEARS agentic rather than merely an automated pipeline: the agent can iterate, learn from governance rejections, and refine its recommendations, similar to how a human engineer would respond to a deployment review flagging stability concerns. The paper's framing of "grating autonomy to agents" acknowledges that this autonomy introduces risk — the agent might explore unstable regions of the policy space — and positions Governance as the counterbalancing force that ensures exploration stays within safe bounds.

Backtesting Validation (Figure 4)

As a final validation, the paper demonstrates through backtesting (Figure 4) that the policy selected by GEARS under governance constraints maintains its metric improvement over a one-month period. The plot shows "the metrics improvement achieved by the selected policy remains consistent over a period of one month." This is presented as evidence that the governance layer successfully filters out policies that would have degraded — the selected policy's stability is not assumed but empirically validated against historical data.

The controlled experiment described in Figure 3 (with four policy candidates) shows only one surviving the governance filter ("Best for Metric 2"), and that survivor demonstrates consistent performance in the backtest. This microcosm illustrates the governance layer's function: of four statistically promising candidates, three are filtered out (presumably due to feature instability or other violations), and only the stable one reaches deployment recommendation.

Summary of Design Choices and Their Justifications

• GAS as candidate generator over training a custom optimization algorithm: GAS is a proven, large-scale HTE framework already deployed at Meta (Wu et al., 2025). Using it as a black-box candidate generator allows GEARS to benefit from GAS's efficient search over combinatorial policy spaces while focusing its own contribution on the reasoning and governance layers that GAS lacks.

• Tolerance-based Pareto expansion over strict Pareto filtering: strict Pareto optimality excludes near-optimal policies that may be substantially more stable or operationally robust. By admitting candidates within uncertainty-proportional tolerance bands, GEARS creates a larger candidate pool that downstream reasoning can filter based on deployability criteria, not just metric optimality.

• Specialized Agent Skills with three-part structure over monolithic prompting or simple tool descriptions: the three-part structure (metadata, instructions, artifacts) enables progressive disclosure, which prevents context rot and allows the agent to scale to complex, multi-step analytical workflows without degrading reasoning quality. Simple tool descriptions (like function signatures) don't encode the procedural knowledge that ranking experts apply; monolithic prompts that include everything upfront would overwhelm the context window.

• Domain Knowledge Brain as curated evidence repository over raw retrieval: raw retrieval over past experiment documents would surface noisy, unvalidated information that could confuse the agent or reinforce spurious patterns. A curated knowledge base ensures the agent grounds its reasoning in high-quality, expert-validated evidence.

• Deterministic hooks over LLM-based evaluation for stability checking: LLM-based critics (like Self-Refine) can hallucinate evaluations, miss statistical subtlety, or produce inconsistent judgments across runs. Deterministic hooks execute fixed, auditable procedures (SQL queries, statistical tests) that produce reproducible results — essential for production deployment decisions where consistency and auditability matter.

• Pre-search feature filtering over post-hoc stability checking: filtering unstable features before candidate generation prevents the system from wasting computation on policies that are guaranteed to be undeployable. Post-hoc checking would be less efficient and might still allow unstable features to influence the optimization (e.g., GAS might overweight an unstable feature because it produces spuriously high lifts in the experiment window).

• User-Cohort Shift Ratio (R_shift) over alternative stability metrics: R_shift directly measures what matters for policy deployment — whether the users targeted by a cohort definition remain the same users over time. Alternative metrics like feature variance or distribution distance might capture instability but don't directly translate to policy degradation in the same way.

• 6-month stability window over shorter or longer horizons: 6 months corresponds to a typical ship cycle from experiment to production deployment at Meta's scale. A shorter window might miss seasonal or gradual shifts; a longer window might be overly conservative and exclude features that are stable enough for practical use.

4. Key Insights and Innovations

Innovation 1: Reframing Ranking Optimization as Autonomous Discovery Rather Than Static Model Selection

The paper's most fundamental conceptual move is not a particular algorithm but a reframing of what ranking optimization means as a problem. Prior to GEARS, the dominant paradigm across uplift modeling (Künzel et al., 2019; Zhao et al., 2017), heterogeneous treatment effect estimation (Wei et al., 2024), and adaptive experimentation (Olson et al., 2025; Wu et al., 2022) treated ranking optimization as a static model selection task: given historical experiment data, find the policy that maximizes expected metric lifts. The objective is defined entirely within the training distribution; deployment feasibility is someone else's problem.

GEARS argues that this formulation is fundamentally incomplete for production ranking systems, where the true objective — finding policies that are simultaneously high-performing, stable under temporal shift, compatible with infrastructure constraints, and aligned with qualitative product intent — cannot be reduced to a scalar optimization over historical data alone. The paper's reframing treats ranking optimization as an autonomous discovery process within a programmable experimentation environment, where the agent navigates not just a metric space but an entire ecosystem of executable operations, validation constraints, and contextual evidence.

This is a fundamental shift, not an incremental refinement, because it changes what the optimizer has access to. A static model selection approach only sees the training data and an objective function; it cannot query whether a segmentation feature is deprecated, whether a cohort definition produces stable targeting over six months, or whether a metric trade-off violates an unspoken product principle. GEARS's agent, by contrast, can actively probe these questions by activating Skills, consulting the Domain Knowledge Brain, and triggering Governance hooks — operations that are definitionally outside the scope of model selection.

The significance of this reframing extends beyond immediate performance gains. It establishes a new category of optimization problem — one where the search space includes not just model parameters but environmental interactions (tool execution, evidence retrieval, constraint checking) — and provides an architectural blueprint for solving it. The paper's claim that this addresses the engineering context constraint is specifically this: previous systems bottlenecked on the manual translation of product intent into executable hypotheses, and treating experimentation as an interactive environment rather than a static dataset is what breaks that bottleneck.

The evidence for this reframing's value is not a single ablation but the architecture as a whole. The fact that removing the bash-based filtering stage (GEARS w/o Bash) causes "substantial degradation" in Top-1 accuracy and ranking quality (Table 1) directly supports the claim that static optimization over candidate metrics is insufficient — the environment interaction (pre-filtering suboptimal candidates via deterministic checks) is what stabilizes downstream reasoning, and purely statistical approaches that skip this step perform worse.

Innovation 2: Externalizing Ranking Expertise into Executable, Modular Skills (Not Prompt Templates)

The concept of giving LLMs access to tools is well-established in the agent literature (Qian et al., 2025; Mialon et al., 2023; Dong et al., 2025). The concept of prompt engineering to inject domain knowledge is even older (Mei et al., 2025; Amatriain, 2024). GEARS's Specialized Agent Skills are distinctive because they occupy a middle ground that neither approach captures: they are not generic tool descriptions (function signatures with natural-language docstrings) and not static prompt templates (blocks of text injected at inference time), but modular, filesystem-based resources that encode expert procedural knowledge into executable workflows with three structured components (metadata, step-by-step analytical instructions, and direct references to internal artifacts).

What makes this a conceptual innovation rather than an implementation detail is the procedural knowledge externalization it enables. Ranking expertise — knowing that a feature stability audit should precede cohort selection, that trade-off slope analysis should inform guardrail thresholds, that historical evidence from past experiments should constrain current policy search — is tacit, context-dependent, and difficult to encode in a single optimization objective or a fixed prompt. Skills make this knowledge modular and reusable across experiments, surfaces, and product teams. A feature stability audit Skill authored by one ranking expert can be activated by GEARS agents across entirely different product surfaces without modification, because the Skill encodes the analytical procedure, not the specific features or metrics.

The paper's progressive disclosure strategy — loading metadata first, full instructions only upon activation, internal artifacts only during execution — is not merely an engineering optimization for context-window management. It is the architectural mechanism that makes Skills composable without overwhelming the agent. Prior work on LLM tool-use (Wang et al., 2024) typically exposes all available tools to the agent at once, which scales poorly as the tool library grows. GEARS's staged loading allows the skill library to expand indefinitely without degrading the agent's reasoning on any individual task — the agent only sees what it needs when it needs it. This is a qualitative difference from existing tool-augmented LLM architectures and addresses a fundamental scaling challenge for domain-specific agent systems.

The evidence that Skills contribute beyond simple filtering is the ablation in Table 1: removing Skills (GEARS w/o Skill) causes a "moderate but consistent drop in performance" across all metrics, with NDCG@1 falling from 0.94 to 0.87 and Top-1 Accuracy dropping from 0.86 to 0.77. The fact that GEARS w/o Skill still outperforms the strongest baseline (Code-as-Action at NDCG@1 of 0.77) indicates that the Governance layer alone provides substantial gains, but the full 0.94 NDCG@1 requires Skills — the expert procedural knowledge they encode adds value beyond what deterministic filtering can achieve.

Innovation 3: Deterministic Governance as a Separation of Concerns Between Optimization and Stability

The paper's most architecturally novel contribution is Deterministic Lifecycle Governance — a set of validation hooks that enforce reproducible stability criteria throughout the agent's decision process. What makes this an innovation rather than simply "checking for overfitting" is the separation of concerns it establishes between two fundamentally different types of policy evaluation that prior systems conflated.

In traditional uplift modeling and adaptive experimentation workflows, optimization and validation happen in the same statistical framework: a policy is selected because it maximizes some objective (expected lift, upper confidence bound, posterior mean), and its deployment fitness is implicitly assumed to follow from statistical significance. The paper argues — and demonstrates through its feature stability benchmarks (Table 2) — that this conflation is dangerous. A policy can be statistically significant in the experiment window (high lift, tight confidence intervals) while being operationally invalid because the features used to define its cohorts shift 50% over six months. Statistical optimization cannot detect this failure mode because it operates entirely on the experiment data distribution; the failure mode exists in the relationship between the experiment distribution and the deployment distribution, which is outside the optimizer's scope.

Deterministic Governance separates these concerns by introducing non-negotiable, reproducible criteria that operate in a different evaluative framework entirely. The Governance hooks do not ask "is this policy statistically optimal?" — they ask "is the feature used to define this policy's cohorts stable over a 6-month window?", "does this policy's performance persist across temporal slices?", and "is this policy's metric improvement driven by noise rather than signal?" These questions require different data (feature shift measurements over time, backtesting windows) and different reasoning (threshold checks against empirical baselines) than the candidate generation optimizer provides.

The specific mechanism — the User-Cohort Shift Ratio (R_shift) measured over a 6-month window with empirically derived thresholds from a benchmarked stable feature set — is what grounds this separation in quantitative rigor rather than hand-waving about "robustness." The paper establishes that even a "perceived stable" feature set exhibits 6% shift under quantile cuts (Table 2), creating a calibrated baseline for unavoidable natural drift. Features exceeding 45% quantile shift or 15% binary shift are automatically disqualified — these thresholds are not arbitrary but derived from empirical measurement of what constitutes "too unstable to deploy."

This is a fundamental conceptual contribution, not an incremental refinement, because it establishes a new design principle for agent-driven optimization systems: optimization and governance should operate in distinct evaluative frameworks with separate data requirements and failure detection capabilities. The paper's citation of Cemri et al. (2025) — that multi-agent systems "often produce outputs that appear locally reasonable yet fail to generalize due to insufficient verification" — positions Governance as the solution to a known failure mode that prior agent architectures lacked the vocabulary to address.

The evidence is in Figure 4 (backtesting shows the selected policy's metric improvement "remains consistent over a period of one month") and Table 2 (the Governance pre-search filter automatically disqualifies high-lift but unstable features that "baseline methods would have erroneously selected"). The ablation in Table 1 — GEARS w/o Bash causes "substantial degradation" — further supports that Governance is not optional but load-bearing: removing it breaks the system.

Innovation 4: Tolerance-Based Frontier Expansion as a Diagnostic Concept for Operational Robustness

The Tolerance-Based Frontier Expansion algorithm (Section 3.1, Appendix C) seems at first glance like a minor technical modification to Pareto filtering — add a tolerance band proportional to uncertainty, admit near-Pareto candidates. But its deeper significance is as a diagnostic concept: it formalizes the insight that in production ranking systems, strict Pareto optimality is not just computationally expensive but conceptually wrong as a selection criterion, because it systematically excludes policies that are slightly suboptimal on point estimates but substantially more robust in deployment.

Standard multi-objective optimization frameworks treat the Pareto frontier as the gold standard: any policy not on the frontier is dominated and should be discarded. This makes sense when the objective function fully captures deployment desiderata. The paper's key observation is that in ranking systems, it doesn't — stability, interpretability, and operational constraints are not encoded in the per-metric lift estimates that define the Pareto frontier. A policy that is 2% worse on metric A but uses features with 4% shift rather than 50% shift is clearly preferable for deployment, but strict Pareto filtering would discard it before the stability analysis even runs.

The tolerance mechanism is the architectural fix for this timing problem. By admitting near-Pareto candidates into the candidate pool before the Governance layer applies stability checks, GEARS ensures that operationally robust but slightly suboptimal policies survive long enough to be evaluated on their true deployment fitness. The tolerance band's proportionality to uncertainty (ε_m(p) = τ · σ_m(p)) is the mathematical expression of a deeper principle: policies whose apparent suboptimality falls within estimation noise should not be prematurely eliminated, because the noise band may be concealing genuine operational advantages.

This is a conceptual innovation because it changes the role of the Pareto frontier from an endpoint (the final set of optimal policies) to an intermediate filter (an expanded candidate set that preserves optionality for downstream reasoning). Prior work on multi-objective optimization — both in ranking specifically (Wu et al., 2025) and in the broader HTE literature — treats Pareto optimality as the terminal selection criterion. GEARS demotes it to a candidate generation heuristic, with the real selection happening in the Governance layer based on criteria invisible to the optimizer.

The evidence is structural rather than ablative: the paper doesn't report a "GEARS without Tolerance Expansion" variant because the expansion is upstream of the agent reasoning, but the Figure 3 Pareto efficiency plot — showing the Pareto frontier (dark blue line) alongside annotated "key policies of interest" — illustrates the concept visually. The candidates that survive to Governance evaluation include both frontier and near-frontier policies, and the final recommendation ("Best for Metric 2") is selected from this expanded set. Without tolerance expansion, policies in the near-frontier region would have been discarded, potentially including the ultimately selected policy.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The offline evaluation uses a benchmark dataset constructed from 20 internal Meta A/B experiments. For each experiment, the GAS algorithm (Wu et al., 2025) was first run to generate hundreds of policy candidates with associated metric measurements and confidence intervals. Then, five distinct instruction types representing common policy selection scenarios were automatically synthesized for each experiment: (1) Maximize Both — find policies that jointly optimize two metrics, (2) Maximize with Constraint — optimize a primary metric while ensuring a secondary metric does not regress, (3) Tradeoff Analysis — identify Pareto-optimal policies representing different trade-off points, (4) Efficiency Optimization — select policies with the highest composite efficiency score, and (5) Single Metric — maximize a single target metric regardless of others. This process yielded a total of 100 instructions (20 experiments × 5 instruction types). For each instruction, the ground-truth was computed as the top-5 policies according to the specified optimization criteria.

• Base model(s). All experiments use Claude Sonnet (the paper cites it as ant) as the backbone LLM for the agent reasoning layer. The paper does not specify the exact model version or parameter count, and does not justify the choice of Claude Sonnet beyond its role as the reasoning engine — presumably selected for its strong instruction-following and analytical reasoning capabilities. The candidate generation engine (GAS) and governance hooks operate independently of the LLM choice; the LLM is only involved in the Insight-Driven Policy Selection stage.

• Metrics. The paper evaluates policy selection performance using six metric families drawn from information retrieval and recommender systems evaluation, all defined with respect to the ground-truth top-5 policies for each instruction:

  • Precision@K: the fraction of recommended policies within the top-K that are in the ground-truth set. Reported for K ∈ {1, 3, 5}.
  • Recall@K: the proportion of ground-truth policies that appear within the top-K predictions. Reported for K ∈ {1, 3, 5}.
  • NDCG@K (Normalized Discounted Cumulative Gain): evaluates ranking quality by rewarding correct policies placed at higher ranks. Defined as NDCG@K = DCG@K / IDCG@K, where DCG@K = Σᵢ₌₁ᴷ (2^{relᵢ} - 1) / log₂(i + 1), with relᵢ ∈ {0, 1} indicating whether the policy at position i belongs to the ground-truth set. IDCG@K is the ideal DCG (all ground-truth policies at the top). Reported for K ∈ {1, 3, 5}.
  • Top-1 Accuracy: whether the highest-ranked prediction exactly matches the single best ground-truth policy.
  • Top-1 in GT: a relaxed version reporting whether the top-ranked prediction belongs to the ground-truth set (any of the top-5).
  • Ranking Correlation: Spearman's ρ between predicted and ground-truth rankings, capturing global ordering fidelity.

  Together, these metrics capture complementary dimensions: correctness (Top-1 Accuracy), coverage (Recall@K), the precision-coverage trade-off (Precision@K), ranking quality (NDCG@K), and global ordering (Ranking Correlation).

• Baselines. The paper benchmarks against five established prompting strategies, all using the same backbone LLM (Claude Sonnet):

  • Naive Prompting: directly queries the LLM with the task instruction and candidate data without additional reasoning guidance. No chain-of-thought, no self-critique, no tool use.
  • Chain-of-Thought (CoT) (Wei et al., 2022): encourages step-by-step reasoning by prompting the model to first understand the objective, analyze the data, apply selection criteria, and then provide recommendations.
  • Self-Consistency (Wang et al., 2022): samples 5 reasoning paths per instruction with temperature 0.7 and aggregates predictions via Borda count voting to improve robustness over a single CoT sample.
  • Self-Refine (Madaan et al., 2023): a two-stage approach where the model first generates initial recommendations, then critically reviews and refines its own output in a single iteration to correct potential errors.
  • Code-as-Action (Wang et al., 2024): instead of generating only text, the LLM generates and executes code to solve the task. This makes outputs verifiable and reproducible, reducing errors and hallucinations in computation- or data-driven settings.

  The paper does not include ablation baselines that combine Code-as-Action with Self-Consistency or Self-Refine, which would have tested whether the gains from GEARS are additive to the strongest individual prompting technique.

• Generation budget / compute accounting. For the offline policy selection experiments, all methods operate on the same tabular experiment records (the candidate policies with their metrics and confidence intervals) and must output a ranked list. The paper does not measure or constrain compute in terms of LLM inference cost, token counts, or wall-clock time. For Self-Consistency, the budget is 5 sampled reasoning paths per instruction with temperature 0.7, while all other methods (including GEARS) use a single inference pass (though GEARS may involve multiple internal steps through skill activation and governance hooks). The paper does not normalize for total inference compute across methods — a GEARS run with multiple skill activations, Knowledge Brain queries, and governance checks likely consumes more tokens than a single CoT prompt, but this cost is not reported or controlled for.

• Cross-validation / statistical protocol. The paper does not describe any cross-validation or statistical significance testing protocol for the offline policy selection experiments in Table 1. The 100 instructions are treated as a fixed evaluation set. For the production experiments in Section 4.4 and Section 5.1, the paper reports statistical significance (mean ± standard error) for metric lifts in A/B experiments, which implicitly handles statistical reliability through the experiment design (randomized controlled trials with control groups). The feature stability benchmark (Section 4.3) uses empirical measurements over a 6-month window with explicit thresholds, but the thresholds themselves are derived from a single baseline feature set (Feature Set S) rather than through cross-validated threshold selection.

Main Quantitative Results

Structured Policy Selection: GEARS vs. Prompting Baselines (Table 1, Figure 5)

The headline result: GEARS achieves an NDCG@1 of 0.94 and Top-1 Accuracy of 0.86, substantially outperforming every baseline. The strongest baseline — Code-as-Action — achieves NDCG@1 of 0.77 and Top-1 Accuracy of 0.68, meaning GEARS provides a relative improvement of approximately 22% on NDCG@1 and 26% on Top-1 Accuracy. The full performance table (Table 1) tells a consistent story across all metrics:

Ranking Quality (NDCG): GEARS at NDCG@1 = 0.94, NDCG@3 = 0.96, NDCG@5 = 0.96. The strongest baseline (Code-as-Action) achieves 0.77, 0.87, 0.87 respectively. The gap narrows at higher K values — at NDCG@5, GEARS leads by roughly 0.09 — but remains substantial. Naive Prompting sits at the bottom with 0.57/0.70/0.74, illustrating that raw LLM reasoning without structure is severely limited on this task.

Precision: GEARS at Prec@1 = 0.94, Prec@3 = 0.60, Prec@5 = 0.37. Code-as-Action achieves 0.77/0.52/0.33. The pattern across all methods is that Prec@1 is high, Prec@3 drops substantially, and Prec@5 drops further — this reflects the ground-truth contains only 5 policies out of hundreds of candidates, so even perfect ranking would have Prec@5 = 5/K = lower numbers at higher K. GEARS's Prec@1 of 0.94 means its top recommendation is almost always in the ground-truth set. The drop to Prec@3 = 0.60 means that among the top-3 recommendations, on average only 1.8 are ground-truth — room for improvement at broader recall.

Recall: GEARS at Rec@1 = 0.56, Rec@3 = 0.94, Rec@5 = 0.95. Code-as-Action achieves Rec@1 = 0.45, Rec@3 = 0.84, Rec@5 = 0.88. The Rec@3 of 0.94 is striking: GEARS retrieves 94% of the ground-truth top-5 policies within its own top-3 recommendations. This means the agent is highly effective at surfacing the relevant candidates even when it doesn't rank them in perfect ground-truth order. Code-as-Action's Rec@5 of 0.88 means it misses 12% of ground-truth policies entirely — a meaningful coverage gap.

Global Ranking Correlation: GEARS achieves 0.82 Spearman's ρ, compared to Code-as-Action's 0.59 and Naive Prompting's 0.20. This metric captures whether the predicted ordering of all candidates matches the ground-truth ordering. GEARS's 0.82 indicates strong global agreement, while the 0.59 for Code-as-Action suggests its rankings are only moderately correlated with ground truth. Self-Refine (0.34) and CoT (0.31) show that iterative refinement alone does not recover ranking structure.

Top-1 Performance: GEARS at Top-1 Accuracy = 0.86, Top-1 in GT = 0.94. Code-as-Action achieves 0.68 and 0.77 respectively. The 0.94 Top-1 in GT means that in 94% of instructions, GEARS's top-ranked policy is somewhere in the ground-truth top-5 — a high bar for practical utility, since a human operator reviewing the top recommendation would almost always see a valid policy. The 0.86 Top-1 Accuracy (strict match with the single best policy) is lower but still strong, indicating that GEARS usually identifies the best policy but sometimes selects a different ground-truth policy as its top recommendation — a failure mode that may be acceptable if the recommended policy is still in the top-5.

The GEARS w/o Bash ablation tells a dramatic story: NDCG@1 drops to 0.40, Top-1 Accuracy to 0.26, and Ranking Correlation to 0.80. The NDCG collapse (0.94 → 0.40) and Top-1 Accuracy collapse (0.86 → 0.26) indicate that without deterministic pre-filtering, the agent's reasoning becomes severely degraded — it cannot reliably identify good policies from the raw candidate pool. Interestingly, Ranking Correlation only drops to 0.80, suggesting the agent without bash filtering still has some coarse sense of candidate ordering but loses the precision to rank the very best candidates at the top. The paper interprets this as evidence that "deterministic pre-filtering of suboptimal candidates is critical for stabilizing downstream reasoning over tabular decision spaces."

The GEARS w/o Skill ablation shows a moderate but consistent drop: NDCG@1 falls from 0.94 to 0.87, Top-1 Accuracy from 0.86 to 0.77, and Ranking Correlation from 0.82 to 0.72. This is important because GEARS w/o Skill still substantially outperforms Code-as-Action (NDCG@1 0.87 vs. 0.77), indicating that the Governance and Intent Translation components alone provide strong performance — but Skills add the final margin that pushes GEARS into the 0.94 NDCG@1 regime. The paper interprets this as evidence that "structured, reusable Skills provide additional gains beyond simple filtering by improving the model's interpretation of selection rules and ranking criteria."

Feature Stability Benchmark and Governance Effectiveness (Table 2, Figure 4)

The feature stability benchmark in Table 2 establishes the empirical foundation for the Governance layer's pre-search filter. The key numbers:

• Baseline Feature Set S (perceived as stable by domain experts): 6% shift under quantile cuts, 2% under binary cuts over a 6-month window. This is the lower bound for unavoidable natural drift.
• Feature 2 (engagement-related): 16% quantile shift, 4% binary shift. Flagged as stable.
• Feature 3 (engagement-related): 30% quantile shift, 10-12% binary shift. Flagged as "Stable (Binary)" — acceptable under binary cuts but not under fine-grained quantile targeting.
• Feature 4 (product-related): ~50% quantile shift, ~20% binary shift. Flagged as unstable — rejected by the pre-search filter.
• Feature 5 (product-related): quantile shift not available (possibly due to irregular distribution), ~30% binary shift. Flagged as unstable.

The thresholds derived from this benchmark — R_shift ≤ 15% (binary) or ≤ 45% (quantile) — are conservative: Feature 3 at 10-12% binary passes, while Feature 4 at ~20% binary fails. The asymmetry (15% for binary, 45% for quantile) reflects the inherently higher migration rates under finer-grained quantile bucketing.

The paper claims this benchmark "allowed GEARS to automatically disqualify high-lift but unstable features (e.g., Feature 4) that baseline methods would have erroneously selected." This is a comparative claim — traditional uplift models operating purely on experiment-window data would see Feature 4's metric lifts without observing its 50% cohort migration rate, and would therefore promote policies targeting Feature 4 that would degrade before deployment. The Governance layer prevents this by rejecting the feature at the pre-search stage.

Figure 4 provides deployment validation: a backtest showing the selected policy's metric improvement "remains consistent over a period of one month." The paper does not report the exact numerical stability of the improvement (e.g., variance of the lift over the backtest window), but the visual in Figure 4 shows a roughly flat line, supporting the claim that the Governance layer successfully filtered out policies that would have degraded.

Production Deployment Results Across Surfaces (Table 3)

Table 3 reports statistically significant metric lifts across nine production surfaces, covering multiple domains:

Several patterns stand out. First, every surface with a reported metric shows a positive lift — there are no reported degradations. Second, the magnitude of lifts varies substantially, from 0.011% (Surface 6, Metric 1) to 0.37% (Surface 8, Metric 2), reflecting diversity in surface characteristics and optimization headroom. Third, surfaces with multiple reported metrics (Surface 3, Surface 6, Surface 8, Surface 9) demonstrate GEARS's ability to simultaneously improve multiple objectives — Surface 8 achieves 0.13% on Metric 1 and 0.37% on Metric 2 concurrently.

The paper does not report:

• Statistical significance thresholds or p-values for any of these lifts.
• The baseline against which these lifts are measured (presumably the existing production ranking configuration before GEARS optimization, but this is not stated).
• Whether any metrics degraded on these surfaces (only positive lifts are reported; guardrail metrics that were "maintained" or showed neutral results may have been omitted).
• The number of experiments or iterations required to achieve these lifts (each reported lift may represent the best result from multiple attempts).
• The specific policy configurations that produced these lifts (cohort definitions, treatment types, feature sets).

The Section 5.1 case study provides one detailed example: in a large-scale recommendation setting with two competing treatments (Treatment 1 improves Metric 2 by +0.282% ± 0.074 but hurts Metric 1 by -0.049% ± 0.043; Treatment 2 improves Metric 1 by +0.036% ± 0.034 but hurts Metric 2 by -0.289% ± 0.073), GEARS discovered cohort-specific policies where "very active" users receive Treatment 1 (which improves the key metric with minimal impact on the competing metric) while "less active" users receive Treatment 2 (which helps them become more engaged). The paper claims this deployment "yields a statistically significant lift on the prioritized metric while maintaining neutrality on the competing metric," though it does not report the exact numerical lifts of the cohort-targeted policy or whether the competing metric remained within the claimed neutrality bounds.

Ablation Studies and Robustness Checks

GEARS without bash-based filtering (GEARS w/o Bash): Removing the deterministic pre-filtering of suboptimal candidates causes catastrophic degradation. Table 1 shows NDCG@1 collapses from 0.94 to 0.40, Top-1 Accuracy from 0.86 to 0.26, Precision@1 from 0.94 to 0.40, and Recall@1 from 0.56 to 0.24. The only metric that partially survives is Ranking Correlation (0.80 vs. 0.82 for full GEARS), suggesting the agent retains coarse-grained candidate discrimination but loses precise ranking capability. The paper interprets this as evidence that "deterministic pre-filtering of suboptimal candidates is critical for stabilizing downstream reasoning over tabular decision spaces." An alternative interpretation is that the bash script is doing heavy lifting — it may be filtering the candidate pool to a small set of obviously good policies, making the agent's ranking task trivially easy. The paper does not report how many candidates survive the bash filter on average, which would clarify whether the agent is ranking 5 candidates or 500.

GEARS without Specialized Agent Skills (GEARS w/o Skill): Removing the Skills module causes a moderate but consistent degradation across all metrics. Table 1: NDCG@1 drops from 0.94 to 0.87, Top-1 Accuracy from 0.86 to 0.77, Precision@1 from 0.94 to 0.87, Recall@1 from 0.56 to 0.53, and Ranking Correlation from 0.82 to 0.72. The drop is relatively uniform — roughly 5-10% relative on most metrics. Importantly, GEARS w/o Skill still substantially outperforms Code-as-Action (NDCG@1 0.87 vs. 0.77), indicating that Governance and intent translation provide the majority of the performance gain, with Skills contributing the final increment. The paper interprets this as Skills "providing additional gains beyond simple filtering by improving the model's interpretation of selection rules and ranking criteria." A missing ablation: the paper does not test whether Skills benefit is from the procedural knowledge they encode or simply from additional LLM reasoning steps (i.e., is a CoT prompt that manually describes the same analytical procedure as effective as activating a Skill?).

Governance pre-search filter effectiveness (Section 4.3): The feature stability benchmark (Table 2) demonstrates that the governance hooks reject features with high cohort migration. The paper does not ablate the threshold values themselves — e.g., testing whether a stricter threshold (10% binary) or a looser threshold (20% binary) changes policy selection quality or production performance. The claim that baseline methods "would have erroneously selected" unstable features like Feature 4 is not directly tested — no baseline is run with and without the governance filter on the same experiment data to demonstrate that governance prevents a specific bad selection.

Progressive disclosure strategy: The paper describes the three-stage information loading strategy (Section 3.2) as preventing "context rot" and "reasoning drift," but provides no ablation comparing GEARS with progressive disclosure to a variant that loads all information upfront. This would be a natural ablation for a system paper — run the same agent on the same tasks with a single large prompt vs. progressive disclosure — but it is absent. The claim that progressive disclosure is necessary for long-horizon reasoning is therefore asserted architecturally but not empirically validated.

Domain Knowledge Brain: The paper describes the Knowledge Brain as supplying "critical 'contextual evidence' behind past experiments," but provides no ablation testing the agent's performance with vs. without access to this curated repository. It is unclear whether the historical learnings materially improve policy selection, or whether the agent could achieve similar performance through the Skills and Governance alone.

Self-Consistency sampling: The paper uses 5 samples for Self-Consistency. No sensitivity analysis is reported for this choice — would 10 or 20 samples improve Self-Consistency's performance, potentially closing the gap with GEARS at higher compute budgets? Since GEARS may use more inference compute than Self-Consistency (through multiple skill activations and governance checks), a compute-matched comparison would require increasing Self-Consistency's sample count.

Backtesting window: Figure 4 shows backtesting over a one-month period, demonstrating that the selected policy's improvement persists. The governance benchmark (Section 4.3) uses a 6-month window for feature stability. There is a mismatch between the validation window (1 month) and the stability threshold window (6 months) — a policy that survives a 1-month backtest might still degrade over months 2-6. The paper does not address this gap.

Negative result from ReST$^{EM}$: The paper does not report any negative results from GEARS itself. All reported metrics in Table 3 are positive lifts, all ablation results show degradation when components are removed (confirming they are beneficial), and the Section 5.1 case study reports a successful deployment. The absence of documented failure cases — surfaces where GEARS did not find improvements, policies that passed governance but degraded in production, instructions where GEARS performed worse than baselines — limits the ability to assess the framework's robustness and failure modes.

Critical Assessment

Claim 1: "GEARS consistently outperforms all baselines across most metrics" (Section 4.2, Table 1)

What was tested: Table 1 compares GEARS against five prompting baselines on 100 synthesized policy selection instructions over 20 internal experiments. The evaluation measures how well each method ranks policy candidates relative to a ground-truth top-5.

Does the evidence support the claim? Yes, for the specific setting tested. GEARS's NDCG@1 of 0.94 substantially exceeds the strongest baseline (Code-as-Action at 0.77). The pattern holds across all metric families. The ablation results (GEARS w/o Bash, GEARS w/o Skill) demonstrate that the performance is attributable to the named components rather than the LLM backbone alone.

What the evidence does NOT demonstrate: The evaluation uses only 100 instructions over 20 experiments — a relatively small sample. The ground-truth is constructed from the top-5 policies according to the specified optimization criteria, but the paper does not describe how the optimization criteria are operationalized into a ranking (e.g., for "Tradeoff Analysis," how is the ground-truth ranking determined?). The baselines are all single-pass prompting methods; no baseline combines Code-as-Action with Self-Consistency (multiple code executions with voting), which would likely be stronger than Code-as-Action alone. The compute budget is not controlled — GEARS likely uses more inference compute through skill activations, Knowledge Brain queries, and governance checks than a single CoT or Code-as-Action pass. A compute-matched comparison (e.g., Code-as-Action with 5× more code execution time to match GEARS's inference cost) is absent.

Conditional boundaries: The claim holds for the specific task of ranking pre-computed policy candidates given tabular experiment data. It does not demonstrate that GEARS can generate better policies from scratch — candidate generation is done by GAS, not GEARS. The claim does not extend to open-ended policy discovery without pre-generated candidates, nor to domains outside ranking optimization.

Claim 2: "Deterministic Lifecycle Governance enforces long-term stability and automatically filters out brittle policies" (Section 3.3, Section 4.3)

What was tested: The feature stability benchmark (Table 2) measures User-Cohort Shift Ratios for five features over a 6-month window and establishes thresholds for pre-search filtering. Figure 3 shows Pareto efficiency of candidate policies, with one surviving governance and being selected. Figure 4 shows a 1-month backtest of the selected policy's performance. The GEARS w/o Bash ablation (Table 1) demonstrates that removing governance causes catastrophic performance degradation on policy selection.

Does the evidence support the claim? Partially. The stability benchmark (Table 2) convincingly demonstrates that feature instability is a real phenomenon — Feature 4 shows 50% cohort migration over 6 months, and even the baseline feature set shows 6% drift. The pre-search filter based on explicit thresholds (≤15% binary, ≤45% quantile) is a concrete mechanism for rejecting unstable features. The GEARS w/o Bash ablation shows that removing governance substantially degrades policy selection, consistent with governance playing a critical role.

What the evidence does NOT demonstrate: The claim that governance "automatically filters out brittle policies that baseline methods would have erroneously selected" is asserted but not directly tested. The paper does not run a baseline uplift model on the same experiment data and show that it selects Feature 4-based policies while GEARS correctly rejects them. The backtesting validation (Figure 4) covers only a 1-month window, while the stability benchmark uses a 6-month window — the paper doesn't demonstrate that policies selected by GEARS remain stable over the full 6-month period. The claim that governance addresses "overfitting transient signals" and "reliance on unstable features" is supported for feature instability specifically, but the paper does not demonstrate governance effectiveness against other failure modes like temporal overfitting (policies that exploit experiment-window dynamics that don't persist) or cohort-specific artifacts (treatment effects that work on one temporal slice but not others). The governance hooks are described as auditing "statistical reliability, feature consistency, and robustness over a 6-month window" (Section 3.3), but the statistical reliability checks are not specified — the paper only details the feature stability hook.

The GEARS w/o Bash ablation requires careful interpretation. The catastrophic collapse in NDCG@1 (0.94 to 0.40) could indicate either: (a) governance is essential for stabilizing agent reasoning, as the paper claims, or (b) the bash script is performing most of the actual policy selection work by aggressively filtering the candidate pool, making the agent's ranking task trivially easy. The paper does not report the filter rate — if the bash script reduces 500 candidates to 5, and the agent simply reranks those 5, the agent's NDCG@1 of 0.94 is less impressive than if it's ranking 200 surviving candidates. Without this information, the ablation's interpretation is ambiguous.

Claim 3: "GEARS delivers improvements across various experimental surfaces" (Section 4.4, Table 3)

What was tested: Table 3 reports metric lifts across nine production surfaces, with values ranging from 0.011% to 0.37%. The Section 5.1 case study describes a specific deployment where GEARS discovered cohort-specific policies that improved one metric while maintaining neutrality on a competing metric.

Does the evidence support the claim? The evidence is suggestive but incomplete. Nine surfaces with positive lifts is a meaningful breadth of deployment, and the case study provides a concrete example of balancing a persistent trade-off. The lifts are all positive — there are no reported degradations.

What the evidence does NOT demonstrate: Critical information is missing. The paper does not report: (a) statistical significance for any of the lifts in Table 3 (the case study reports standard errors for baseline treatments but not for the GEARS-discovered policy), (b) whether any guardrail metrics degraded on these surfaces (only positive lifts are shown), (c) the number of attempts or iterations required to achieve these lifts (were these the first deployment, or selected after multiple failed attempts?), (d) the baseline configuration against which lifts are measured, (e) whether the lifts persisted beyond the initial deployment window, or (f) the human engineering effort saved — the paper claims GEARS automates "what was previously a multi-week, expert-driven discovery process" (Section 5.1) but provides no quantitative comparison of time-to-deployment with vs. without GEARS.

The reporting is selective: the paper shows only positive results across nine surfaces and one detailed case study. Without disclosure of surfaces where GEARS was attempted but failed to find improvements, or policies that were recommended but rejected by human operators, it's impossible to assess the framework's success rate. The absence of negative results is itself a limitation — in real-world ranking optimization, not every experiment yields deployable improvements.

Claim 4: "GEARS achieves near-Pareto-efficient policies" (Abstract, Section 3.1)

What was tested: GAS with Tolerance-Based Frontier Expansion generates candidate policies that include both strict Pareto-optimal and near-Pareto solutions (Figure 3). The Governance layer filters these candidates. The final selected policy ("Best for Metric 2") sits on or near the Pareto frontier.

Does the evidence support the claim? Weakly. Figure 3 shows a Pareto frontier and annotated policies, but does not quantify how close GEARS's selected policies are to the true Pareto frontier. The paper doesn't compare GEARS's selected policies against a ground-truth Pareto frontier computed by exhaustive search (which may be computationally infeasible) or against policies selected by alternative methods. "Near-Pareto-efficient" is a qualitative descriptor, not a quantitatively validated claim.

What would strengthen this claim: A comparison showing that policies selected by GEARS achieve metric trade-offs that are (a) closer to the true Pareto frontier than policies selected by baseline methods, (b) not achievable by random search over the same candidate pool, or (c) confirmed as Pareto-efficient through offline evaluation against held-out data.

Missing Experiments

Several experiments would substantially strengthen the paper:

1. Compute-matched comparison with baselines: Run Code-as-Action with the same inference compute budget as GEARS (e.g., multiple code executions with voting, or longer code with more analytical steps). This would distinguish whether GEARS's gains come from better architecture or simply more computation.

2. Sensitivity to backbone LLM: Run the same experiments with a different LLM (GPT-4, Gemini) to determine whether GEARS's performance is tied to Claude Sonnet's specific capabilities or generalizes across model families.

3. Ablation of progressive disclosure: Compare GEARS with staged information loading against a variant that loads all skill instructions, Knowledge Brain entries, and governance criteria into a single prompt. This would validate the "context rot" claim.

4. Direct comparison of governance vs. no-governance on real deployment stability: For a set of experiments, deploy both the GEARS-selected policy and the policy that would have been selected without governance (highest statistical lift, ignoring feature stability), and measure degradation of the ungoverned policy over 6 months. This would provide direct causal evidence for the governance layer's deployment impact.

5. Full negative results disclosure: Report the number of experiments where GEARS was attempted but did not find deployable improvements, the rate of governance rejection (what fraction of candidates are filtered), and any cases where deployed policies degraded after launch.

6. Human baseline: Compare GEARS's policy selections against those made by human ranking experts given the same tabular experiment data and time constraints. The claim of reducing "multi-week" expert workflows is not validated without a human comparison.

7. Sensitivity to ground-truth construction: The ground-truth is defined as top-5 policies according to optimization criteria derived from the synthesized instructions. Different operationalizations of the criteria (e.g., different weighting of trade-offs) would produce different ground-truths. Testing GEARS's robustness to ground-truth variation would validate that the performance gains are not fragile to the specific ground-truth construction method.

Summary of Experimental Strength and Weaknesses

The experiments convincingly demonstrate that GEARS outperforms prompting-based baselines on the specific task of ranking pre-computed policy candidates from tabular experiment data, and that removing governance or skills degrades performance (Table 1). The feature stability benchmark (Table 2) establishes that feature instability is a real phenomenon in Meta's ranking infrastructure, providing a quantitative foundation for the governance mechanism.

However, several central claims remain empirically under-supported: the claim that governance prevents selection of policies that baselines would erroneously promote is not directly tested; the claim of deployment stability is validated only over a 1-month backtest, not the 6-month window used for governance thresholds; the claim of reduced engineering overhead is asserted without quantitative comparison; and the production results (Table 3) lack the statistical detail, failure disclosure, and contextual information needed to assess their reliability and generalizability. The paper's strengths lie in the architecture and the problem formulation; the experiments provide proof-of-concept validation but fall short of the rigorous production evaluation that the ambitious claims require.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted For and Potentially Dominant

The assumption or constraint. GEARS's entire compute-optimal workflow — from intent translation through candidate generation to governance-filtered policy recommendation — depends on having access to pre-computed policy candidates with metric measurements and confidence intervals, generated by the GAS framework (Wu et al., 2025) from completed A/B experiments. Section 3.1 states that GAS produces "hundreds of policy candidates with their corresponding metric measurements and confidence intervals" for each experiment. The offline evaluation (Section 4.1) uses a benchmark dataset where "we initially ran the GAS algorithm to generate hundreds of policy candidates." The production results (Section 4.4, Table 3) and case study (Section 5.1) similarly depend on GAS-generated candidates from randomized experiments.

What is never accounted for is the cost of running these experiments and generating these candidates. A/B experiments at Meta's scale involve exposing millions of users to treatment and control arms over days or weeks. The GAS algorithm then searches over a combinatorial space of quantile-based segmentations × treatment assignments (Appendix B), computing segment-level heterogeneous treatment effects for each candidate. This is not a lightweight computation — it requires aggregating outcomes across all users in all treatment groups for every candidate segmentation. The paper provides no measurement of this cost in terms of experiment duration, compute resources, or data requirements.

The Governance layer adds further unaccounted costs. The feature stability benchmark (Section 4.3) requires computing the User-Cohort Shift Ratio (R_shift) over a 6-month window for every candidate feature. This means the system needs 6 months of historical feature data before it can even evaluate whether a feature is stable enough to enter the search space. The Domain Knowledge Brain (Section 3.2) requires ongoing expert curation — "a curated reasoning backbone that grounds the LLM in high-fidelity historical data and domain expert learnings." The cost of building and maintaining this knowledge base (expert time, data infrastructure, quality assurance) is not estimated.

The consequence. A practitioner evaluating GEARS for adoption faces a fundamental uncertainty: the headline performance numbers (NDCG@1 of 0.94, 0.14%–0.37% lifts across surfaces) are costs measured after the expensive infrastructure is in place — experiment data collected, GAS candidates generated, feature stability benchmarks computed, Knowledge Brain curated. The total cost of operating GEARS includes all of these upstream dependencies, and the paper provides no estimate of whether they dominate the downstream agent computation.

In a setting where a team is considering whether to adopt GEARS for a new product surface, the upfront investment is substantial: they must (a) run randomized experiments with sufficient statistical power to detect heterogeneous treatment effects across fine-grained segments, (b) deploy and maintain the GAS candidate generation pipeline, (c) accumulate 6 months of feature history to enable stability benchmarking, and (d) invest expert time in curating the Domain Knowledge Brain for the new surface. None of these costs appear in the paper's evaluation. The claimed efficiency gain — "automating what was previously a multi-week, expert-driven discovery process" (Section 5.1) — measures only the final analysis step, not the total time from experiment launch to deployed policy.

There is a second-order consequence: the approach is only applicable to surfaces where A/B experiments are already running at sufficient scale. For new product surfaces, smaller experiments, or contexts where randomized experimentation is infeasible (privacy-sensitive domains, low-traffic features), the entire GEARS pipeline is unavailable because its input — GAS-generated candidates from randomized experiments — does not exist.

What evidence exists in the paper. The paper provides no measurement of upstream costs anywhere. Section 4.1 describes the benchmark construction ("we initially ran the GAS algorithm to generate hundreds of policy candidates") without reporting the computational or time cost of doing so. Section 4.3 establishes feature stability benchmarks over a 6-month window without reporting the data infrastructure cost of computing shift ratios across all features. The Domain Knowledge Brain is described qualitatively (Section 3.2) without any quantification of curation effort.

The paper does not acknowledge this cost gap as a limitation. There is no caveat in Section 4 or Section 5 that the reported lifts should be interpreted net of upstream infrastructure costs, no sensitivity analysis of how total time-to-deployment varies with experiment duration or feature history requirements, and no comparison of the full end-to-end cost of GEARS-assisted optimization vs. the traditional manual workflow it replaces.

Mitigation status. Not addressed. The paper does not flag upstream cost as a limitation, does not propose methods for reducing experiment requirements or feature history needs, and does not suggest cost amortization strategies (e.g., reusing GAS candidates or stability benchmarks across related surfaces). The "future work" discussion (Section 6) focuses on extending the framework rather than reducing its operational cost. A practitioner adopting GEARS must independently estimate and budget for these costs.

---

6.2 Single Backbone LLM with No Model-Sensitivity Analysis

The assumption or constraint. All experiments use Claude Sonnet (cited as ant) as the backbone LLM for agent reasoning (Section 4.1: "All experiments use Claude Sonnet as the backbone LLM"). The entire GEARS architecture — intent translation, skill activation, candidate analysis, policy ranking — depends on this specific model's capabilities: its instruction-following, its analytical reasoning, its ability to interpret structured experiment data, and its robustness to long-context reasoning with progressive disclosure.

The paper provides no evidence that GEARS's performance generalizes across LLM families. It does not run the same experiments with GPT-4, Gemini, or open-source models. It does not investigate whether GEARS's gains over baselines (NDCG@1 of 0.94 vs. Code-as-Action at 0.77) are specific to Claude Sonnet's particular strengths (e.g., superior instruction following, better tabular data reasoning) or would replicate with other models.

This is consequential because Claude Sonnet is a commercial, API-accessed model with associated costs, latency, and availability constraints. An organization with an existing investment in a different LLM infrastructure (or with privacy requirements that preclude external API calls) cannot assume GEARS will work with their preferred model. Moreover, model behavior changes over time as providers update their models — Claude Sonnet's performance on GEARS tasks today may not match its performance six months from now, but the paper provides no characterization of this sensitivity.

The consequence. The paper's central performance claims are model-contingent: they hold for Claude Sonnet at the time of evaluation, but there is no evidence they hold for any other model, any other version of Claude, or even the same model in six months. For a practitioner deciding whether to adopt GEARS, this introduces two risks:

First, vendor lock-in: adopting GEARS means committing to Claude Sonnet (or at minimum, to a model with similar capabilities on this specific task), because there is no evidence the framework works with alternatives. If Anthropic changes its pricing, deprecates the model version, or modifies its behavior, the GEARS deployment may break or degrade in ways that are difficult to predict.

Second, reproducibility: the paper's results cannot be independently reproduced by researchers or practitioners without access to the same proprietary model. The ablation results (GEARS w/o Bash, GEARS w/o Skill) demonstrate component contributions conditional on Claude Sonnet, but the absolute performance levels and the relative ranking of baselines may differ with a different LLM. A team attempting to replicate GEARS with GPT-4 or Llama 3 would have no guidance on expected performance or failure modes.

There is a more subtle consequence for the Skills architecture specifically. Specialized Agent Skills encode ranking expertise into structured analytical instructions (Section 3.2). The effectiveness of these instructions depends on the LLM's ability to follow them reliably across diverse experimental contexts. If a different LLM interprets the same skill instructions differently — applying thresholds at different stages, weighting evidence differently, producing different structured outputs — the skill's behavior changes. The paper provides no characterization of this instruction-following robustness, making Skills implicitly coupled to Claude Sonnet's specific instruction-following behavior.

What evidence exists in the paper. None. The paper does not report experiments with any LLM other than Claude Sonnet, does not discuss model selection criteria, and does not acknowledge model-sensitivity as a limitation. The Implementation Details (Section 4.1) mention the model choice in a single sentence without justification.

Mitigation status. Not addressed at all. The paper does not suggest model-robustness experiments, does not propose model-agnostic skill designs (e.g., formal verification of skill execution rather than relying on instruction following), and does not discuss the vendor-lock-in implications of the single-model evaluation. A practitioner would need to conduct their own model-sensitivity analysis before adopting GEARS, and would need to establish monitoring for model behavior drift over time.

---

6.3 Production Results Are Reported Without Failure Cases, Statistical Detail, or Negative Outcomes

The assumption or constraint. Section 4.4 reports metric lifts across nine production surfaces (Table 3), and Section 5.1 provides a detailed case study of a successful deployment where GEARS discovered cohort-specific policies balancing two competing engagement metrics. The paper presents these results as evidence of GEARS's effectiveness in real-world environments, claiming it "delivers improvements across various experimental surfaces" (Section 4.4).

However, the production results suffer from a critical reporting asymmetry: only positive outcomes are shown. Every metric in Table 3 is a positive lift. The paper does not report: (a) the number of surfaces or experiments where GEARS was attempted but did not find statistically significant improvements, (b) policies that were recommended by GEARS but rejected by human operators after expert review, (c) metric degradations on guardrail metrics that were being "maintained" (the paper reports only the metrics being optimized, not whether any secondary metrics regressed), (d) policies that passed GEARS's governance checks but subsequently degraded in production (i.e., false negatives from the governance layer), or (e) the statistical significance of the reported lifts (p-values, confidence intervals, or false discovery rate corrections for the nine surfaces). The case study (Section 5.1) does report standard errors for the baseline treatments (e.g., Treatment 1: -0.049% ± 0.043 for Metric 1, +0.282% ± 0.074 for Metric 2) but does not report standard errors for the GEARS-discovered cohort-targeted policy, making it impossible to assess whether the final lifts are statistically distinguishable from the baselines.

The consequence. The production results in their current form are uninterpretable for decision-making. A practitioner considering GEARS needs to know not just that it can produce lifts (the existence proof in Table 3), but what its success rate and failure modes are: how often does it find improvements vs. return no recommendation vs. recommend a policy that degrades? Under what conditions does it fail? Are the lifts reported in Table 3 the best results from multiple attempts (cherry-picked) or the typical outcome?

The governance layer (Section 3.3) is specifically designed to prevent deployment of unstable policies. The paper should therefore be particularly interested in measuring its false positive rate (policies that passed governance but degraded) and false negative rate (policies that governance rejected but would have performed well). Neither is reported. Without these, the claim that governance "ensures that recommended configurations are durable and deployable" (Section 3.3) is an assertion, not an empirically validated property.

The missing statistical detail is especially problematic for the smaller lifts. Table 3 reports lifts as small as 0.011% (Surface 6, Metric 1) and 0.02% (Surface 9, Metric 3). With billions of users, even small relative lifts can translate to meaningful absolute impact, but only if they are statistically significant and not driven by noise. Without p-values, confidence intervals, or multiplicity corrections, these numbers could reflect sampling variation rather than genuine improvements. The case study's baseline treatments have standard errors of ±0.034 to ±0.074 (Table 4), suggesting that lifts of 0.01-0.02% may not be statistically distinguishable from zero — but the paper doesn't provide the corresponding standard errors for the GEARS-discovered policies to assess this.

What evidence exists in the paper. The paper provides exactly the data shown in Table 3 and the case study in Table 4. There are no error bars, no significance tests, no disclosure of failed attempts, no discussion of governance false positive/negative rates, and no sensitivity analysis showing that lifts persist over time beyond the 1-month backtest in Figure 4. The backtest itself (Figure 4) shows a flat line for the selected policy, but doesn't show what happened to the policies that governance rejected — which would provide direct evidence that governance is filtering correctly.

Mitigation status. The paper does not acknowledge the reporting asymmetry as a limitation. There is no caveat that the reported lifts may represent a selected subset of attempts, no discussion of the need for comprehensive failure analysis in production evaluation, and no suggestion that future work should measure governance error rates. The paper's framing as "production validation" (Section 1 contribution list: "We demonstrate the effectiveness of GEARS in real-world environments") implies a level of empirical rigor that the reported data do not support.

---

6.4 No Human Baseline — The Claim of Reduced Engineering Overhead Is Unvalidated

The assumption or constraint. A central claim of the paper is that GEARS reduces human engineering overhead. The abstract states the framework achieves gains "while significantly reducing human engineering overhead." Section 1 frames the problem as the "arduous process of translating ambiguous product intent into reasonable, executable, verifiable hypotheses" and positions GEARS as automating this process. Section 5.1 claims that GEARS "automated what was previously a multi-week, expert-driven discovery process." Section 6 claims GEARS "drastically accelerates the experimental feedback loop."

These claims compare GEARS against a human baseline — the traditional manual workflow of domain experts analyzing experiment data, identifying deployable policies, and writing recommendations. But the paper provides no measurement of the human baseline whatsoever. There is no experiment that measures how long human experts take to complete the same policy selection tasks, no comparison of human-selected policies against GEARS-selected policies on the same experiment data, no measurement of whether human experts would have discovered the same policies (or better ones) given equivalent time, and no quantification of the "engineering overhead" being reduced.

This is not a minor omission — it is the absence of the comparison that the paper's motivation and claimed impact depend on. The entire framing of Section 1 and Section 2 is that manual workflows are the bottleneck: "manual dependency creates a scalability barrier, leaving high-value policies undiscovered." To validate this claim, one must show that (a) manual workflows leave policies undiscovered that GEARS finds, and (b) GEARS finds them faster or with less human effort. Neither is shown.

The consequence. The claim of reduced engineering overhead is unsubstantiated. A skeptical reader could argue that GEARS does not reduce engineering overhead at all — it shifts it from manual policy analysis to infrastructure maintenance (curating the Domain Knowledge Brain, maintaining GAS pipelines, updating governance hooks, authoring and debugging Skills). The paper provides no accounting of total engineering effort in either the pre-GEARS or post-GEARS regime.

Furthermore, the absence of a human baseline makes it impossible to assess whether GEARS's policy selections are actually better than what a skilled human expert would produce. The offline evaluation (Table 1) compares GEARS against prompting baselines (all using the same Claude Sonnet backbone), not against human experts. The production results (Table 3) compare GEARS-discovered policies against production baselines, not against what human experts would have recommended given the same experiment data. It is possible that human experts would have discovered the same or better policies, and that GEARS's primary value is not in discovering better policies but in automating the discovery process to save time — but the paper doesn't measure time savings.

The case study (Section 5.1) illustrates this gap. The paper claims GEARS discovered that "very active" users benefit more from Treatment 1 while "less active" users benefit more from Treatment 2. A domain expert looking at the same experiment data might have immediately recognized this pattern — it is a relatively standard finding in recommendation systems that heavy and light users respond differently to content mix changes. Without a human comparison, there is no evidence that GEARS's discovery represents a capability beyond what an experienced data scientist would produce, or that the claimed "multi-week" timeline would have been necessary for a human to reach the same conclusion.

What evidence exists in the paper. Zero. There is no human baseline experiment, no measurement of engineering time, no comparison of GEARS vs. expert recommendations on the same data, and no survey or case study documenting the pre-GEARS workflow duration. The "multi-week" claim in Section 5.1 is an anecdote, not a measurement.

Mitigation status. Not addressed. The paper does not acknowledge the absence of a human baseline as a limitation, does not propose future work to measure engineering overhead reductions, and does not qualify its claims about efficiency. The "reduced human engineering overhead" claim appears in the abstract and conclusions as an established finding, not as a hypothesis requiring validation.

---

6.5 Single Task Domain and Evaluation Format — No Evidence of Generalization Beyond Tabular Policy Selection on Internal Meta Experiments

The assumption or constraint. All evaluation — both offline (Section 4.2, Table 1) and production (Section 4.4, Table 3) — is conducted on internal Meta A/B experiments for ranking optimization. The offline benchmark uses 20 internal experiments with 100 synthesized instructions (Section 4.1). The production results cover 9 product surfaces (Table 3). The case study (Section 5.1) is a large-scale recommendation setting.

All experiments share the same structure: tabular experiment records with candidate policies, associated metric measurements, and confidence intervals, from which the agent must select and rank deployable policies. The task is always policy selection from pre-computed candidates, not policy generation, experiment design, or open-ended optimization. The domain is always ranking personalization, not other ranking tasks (e.g., relevance tuning, diversity optimization, latency-performance trade-offs) or other domains where agent-driven optimization might apply (e.g., notification timing, ad placement, content moderation thresholds).

The paper frames GEARS as a "general framework" (Section 1: "While GEARS is designed as a general framework, personalization represents a particularly illustrative application"). But there is zero evidence for generality beyond the specific task, domain, and evaluation format tested.

The consequence. The paper's claims of generality are unsupported, and the practical scope of applicability is unknown. A practitioner evaluating GEARS for a related but distinct task — for example, optimizing notification send times, tuning content diversity parameters, or balancing ad load against user experience — cannot assume the reported performance will transfer. Several aspects of the evaluation may not generalize:

First, the tabular policy selection format assumes candidates are pre-generated with clearly defined metrics and confidence intervals. In many optimization tasks, the "candidates" are not pre-computed policy configurations but continuous parameters or combinatorial choices that must be explored online. GEARS's architecture (GAS candidate generation + agent selection) is tightly coupled to the pre-computed candidate paradigm; it's unclear how the framework would adapt to settings where candidate generation and evaluation are interleaved.

Second, the ground-truth construction (top-5 policies according to optimization criteria) assumes a well-defined notion of "optimal" that can be computed from experiment data. For tasks where the objective is harder to formalize (e.g., "make the feed feel more diverse" without a quantitative diversity metric), the ground-truth ranking is undefined, and the evaluation methodology used in Table 1 does not apply.

Third, the Governance layer's primary hook — feature stability benchmarking via User-Cohort Shift Ratio over 6 months — is specific to ranking personalization, where policies target user cohorts defined by behavioral features. For optimization tasks that don't involve user segmentation (e.g., global parameter tuning), this governance mechanism is irrelevant, and the paper provides no alternative.

Fourth, all experiments use Meta's internal infrastructure (GAS, internal A/B experiment data, internal feature stores for stability benchmarking, internal Domain Knowledge Brain). An external practitioner cannot replicate GEARS without building equivalent infrastructure — GAS is a proprietary Meta system (Wu et al., 2025), the feature stability benchmarks require Meta's feature stores, and the Domain Knowledge Brain is a Meta-curated resource.

What evidence exists in the paper. Only evidence from Meta-internal ranking personalization experiments. There are no experiments on public benchmarks, no cross-domain evaluations (e.g., applying GEARS to a non-ranking optimization task), and no discussion of what adaptation would be required for different domains.

Mitigation status. The paper acknowledges the scope limitation implicitly by focusing on personalization as "an illustrative setting" (Section 3.1), but does not discuss the generalizability gap, does not propose experiments to test generality, and does not identify which components of GEARS are domain-specific vs. domain-agnostic. The conclusion (Section 6) claims GEARS "establishes a new standard for AI-driven ranking infrastructure" without qualification, suggesting broader applicability than the evidence supports.

---

6.6 Governance Validation Window Mismatch — 1-Month Backtest Cannot Validate 6-Month Stability Claims

The assumption or constraint. The Governance layer (Section 3.3) is the paper's primary mechanism for ensuring deployment stability. It enforces feature stability over a 6-month window using User-Cohort Shift Ratios (R_shift), with explicit thresholds derived from empirical benchmarks (Section 4.3, Table 2): features must exhibit R_shift ≤ 15% (binary cuts) or ≤ 45% (quantile cuts) over 6 months to enter the search space. A feature with 50% shift (like Feature 4) is rejected. The claim is that governance "ensures that recommended configurations are durable and deployable" by filtering policies that would degrade under temporal shifts.

However, the deployment validation (Figure 4) shows only a 1-month backtest: "The backtest results indicate that the metrics improvement achieved by the selected policy remains consistent over a period of one month." There is a 5-month gap between the governance stability window (6 months) and the validation window (1 month). A policy that survives a 1-month backtest could still degrade in months 2-6 due to exactly the feature instability that the governance benchmarks measure over the 6-month horizon.

The consequence. The core claim of the Governance layer — that it ensures long-term stability — is not empirically validated over the relevant time horizon. The 1-month backtest demonstrates short-term consistency, which is a necessary condition for stability but not sufficient to establish the 6-month durability that the governance thresholds target. A skeptical interpretation: the features flagged as unstable in Table 2 (Feature 4 at ~50% shift over 6 months) might appear stable over a 1-month window — a 50% shift over 6 months corresponds to roughly 8% shift per month if linear, which might not produce visible degradation in a single-month backtest. The 1-month validation could therefore produce false confidence in policies that are actually unstable over the deployment lifetime.

The mismatch is particularly concerning for the paper's claim that governance prevents selection of policies that "baseline methods would have erroneously selected" (Section 4.3). To validate this claim, one would need to: (a) identify policies that baselines would select but governance rejects (e.g., policies targeting Feature 4), (b) deploy those rejected policies alongside the governance-approved policy, and (c) measure degradation of the rejected policies over the full 6-month window. The 1-month backtest of only the approved policy provides none of this evidence.

More broadly, the paper provides no evidence that the 6-month stability thresholds (R_shift ≤ 15% binary, ≤ 45% quantile) are correctly calibrated to predict deployment degradation. These thresholds are derived from a single baseline feature set (Feature Set S), but their predictive validity — whether features below these thresholds actually produce stable policies and features above them actually produce degrading policies — is not tested. The thresholds could be too conservative (rejecting features that would have produced stable policies, reducing the optimization headroom) or too permissive (admitting features that still produce degradation despite being below the threshold). Without a calibration study — measuring actual policy degradation as a function of feature R_shift — the thresholds are heuristics, not validated stability guarantees.

What evidence exists in the paper. The 1-month backtest in Figure 4. The feature stability benchmarks in Table 2 establish that features vary in stability but do not link R_shift to actual policy degradation. There is no validation over the full 6-month window, no degradation measurement for rejected policies, and no calibration study for the stability thresholds.

Mitigation status. Not addressed. The paper does not acknowledge the validation window mismatch, does not discuss the need for longer-term backtesting, and does not flag the threshold calibration as an open problem. The governance layer is presented as a complete solution ("ensures that recommended configurations are durable and deployable") without qualification about the validation horizon. A practitioner relying on the 1-month backtest as evidence of 6-month stability would be making an inference the data do not support — and the paper provides no warning about this gap.

7. Implications and Future Directions

How This Work Changes the Landscape

GEARS represents a category-defining architecture for a problem that prior work addressed only in fragments. It does not introduce a new optimization algorithm, a new prompting technique, or a new bandit method. Rather, it introduces a system decomposition — separating candidate generation (statistical optimization), insight-driven selection (domain-expert reasoning), and deterministic governance (stability enforcement) into distinct pipeline stages with different evaluative frameworks — that makes the ranking optimization problem tractable in a way no single-method approach could achieve.

The magnitude of this shift is best understood by what it makes obsolete as a standalone approach. Prior to GEARS, the implicit assumption across uplift modeling (Künzel et al., 2019; Zhao et al., 2017; Wei et al., 2024), adaptive experimentation (Olson et al., 2025; Wu et al., 2022), and LLM-agent methods (Wang et al., 2024; Madaan et al., 2023) was that a single evaluative framework — statistical optimization over experiment data — could produce deployable ranking policies. GEARS demonstrates that this assumption is false for production systems: statistically optimal policies are frequently undeployable because the criteria that determine deployment fitness (feature stability over 6-month windows, cohort consistency, alignment with qualitative product intent) are invisible in the experiment data distribution. The paper's feature stability benchmark (Table 2) provides concrete evidence: a feature showing 50% cohort migration over 6 months (Feature 4) would be eagerly selected by a pure statistical optimizer that observes only its experiment-window lifts, yet would produce a policy that degrades before it could ship. This is not a hypothetical concern — it is a measured property of Meta's production feature infrastructure.

By establishing this separation of concerns, GEARS changes what "solving ranking optimization" means as a research goal. Prior work asked: "Given experiment data, what policy maximizes expected lift?" GEARS asks: "Given an experimentation environment — including feature stores, historical benchmarks, infrastructure constraints, and qualitative product intent — what policies are simultaneously high-performing and deployment-stable?" This is a strictly harder problem that cannot be reduced to optimization over a fixed dataset, because the evidence needed to assess deployment fitness (6-month feature histories, backtesting windows, expert-curated knowledge) is not a subset of the experiment data. Future work on ranking optimization that ignores this distinction — that continues to treat optimization as purely a statistical estimation problem — will produce solutions that look strong on paper but fail the deployment test that GEARS's governance layer is designed to catch.

The paper also resolves a latent tension in the LLM-agent literature. Recent work (Cemri et al., 2025) documented that multi-agent LLM systems "often produce outputs that appear locally reasonable yet fail to generalize due to insufficient verification." Prior to GEARS, the response to this finding was unclear — should we build better prompting techniques to improve LLM self-verification? Should we limit agent autonomy? GEARS provides a structural answer: the verification mechanism should not be LLM-based at all. Deterministic governance hooks — executing fixed, reproducible procedures (SQL queries, statistical tests) against explicit, empirically calibrated thresholds — provide verification that is immune to the hallucination, inconsistency, and context-rot failures that plague LLM-based critics. The paper's evidence for this structural claim is the GEARS w/o Bash ablation (Table 1): removing deterministic governance causes NDCG@1 to collapse from 0.94 to 0.40, a far larger degradation than removing Skills (0.94 to 0.87). The governance layer is not an optional quality-of-life improvement; it is load-bearing for the system's core capability.

More subtly, GEARS reframes the role of domain experts in AI-driven optimization. Rather than being replaced by automation, experts are externalized into the infrastructure: their knowledge becomes Specialized Agent Skills (modular, filesystem-based procedural workflows) and the Domain Knowledge Brain (curated historical evidence). This shifts the expert's role from per-experiment manual analysis to ongoing skill authoring and knowledge curation — a higher-leverage activity where one expert's encoded knowledge can be reused across hundreds of experiments and multiple product surfaces. The paper does not quantify this leverage multiplier (see Section 6.4 of the prior analysis for the missing human baseline), but the architecture makes it structurally possible in a way that per-experiment manual workflows cannot match.

The work makes certain research directions more attractive and others less so. Verifier robustness — already identified as a bottleneck in the broader LLM reasoning literature — becomes central: GEARS's governance hooks are effectively verifiers that operate on policy stability rather than solution correctness, and the paper demonstrates that their quality is the dominant factor in system performance (via the GEARS w/o Bash ablation). Research on adaptive experimentation, meta-learning for HTE, and purely statistical uplift modeling become less attractive as standalone solutions for production ranking, because they address only the candidate generation stage while ignoring the deployment feasibility bottleneck that GEARS identifies as the primary constraint. They remain valuable as components within a GEARS-like architecture, but the paper makes clear that component-level optimization without system-level governance is insufficient.

Follow-Up Research This Work Enables

Governance threshold calibration via a degradation prediction study. The paper establishes feature stability thresholds (R_shift ≤ 15% binary, ≤ 45% quantile over 6 months) from a single baseline feature set (Feature Set S, Table 2), but provides no evidence that these thresholds predict actual policy degradation. A direct follow-up would: for a set of 50-100 experiments where both stable and unstable features were used for policy targeting, measure actual policy degradation over a 6-month deployment window as a function of the targeting feature's R_shift. Plot degradation rate (percentage of initial lift lost per month) against R_shift. This would produce a calibration curve showing whether the 15%/45% thresholds are conservative (rejecting features that would have been stable), permissive (admitting features that subsequently degrade), or well-calibrated. The study would also reveal the functional form — is degradation linear in R_shift, or is there a threshold effect where policies suddenly collapse above some critical value? This is newly tractable because GEARS's governance layer provides the infrastructure to systematically measure and log feature R_shift for every candidate policy, enabling a retrospective analysis that prior manual workflows could not support at scale.

Cross-model skill transfer and instruction-following robustness. The paper evaluates GEARS exclusively with Claude Sonnet. A critical open question is whether Specialized Agent Skills — authored and tested against one LLM's instruction-following behavior — transfer to other models without degradation. A concrete experiment: take the exact Skills used in the paper (or a representative subset, such as the feature stability audit skill and the trade-off diagnosis skill), run the same 100-instruction benchmark from Section 4.1 using GPT-4, Gemini 2.0, and Llama 3 (70B and 405B), and measure the drop in NDCG@1 relative to Claude Sonnet. If the drop is small (NDCG@1 remains above 0.85 across models), Skills are model-agnostic reasoning scaffolds — promising for open-source deployment. If the drop is large and model-specific (some models achieve 0.90+ while others collapse below 0.50), Skills must be calibrated per-model, which would make GEARS's deployment economics substantially more expensive and vendor-locked. This study would also reveal which skill components are most sensitive to model choice: is it the step-by-step analytical instructions, the artifact references, or the metadata routing? The paper makes this experiment tractable by providing a reproducible benchmark (100 instructions over 20 experiments, with ground-truth top-5 policies) and a clear evaluation methodology.

Dynamic governance: online stability assessment during candidate generation. The current governance architecture operates as a pre-search filter: unstable features are removed before candidate generation, and the remaining candidates are evaluated once. A more ambitious extension would make governance online and adaptive: during the GAS candidate generation stage, as each candidate policy is proposed, the governance layer computes a stability score (based on constituent features' R_shift values, cohort size, and treatment effect variance) and uses this score to guide the search — upweighting stable regions of the policy space, downweighting unstable ones, and terminating exploration early when all remaining candidates fail stability thresholds. This would transform governance from a binary gatekeeper (accept/reject) to a continuous optimization signal. Concretely: modify the GAS scalarized objective (Appendix B) to include a stability penalty term, Σ_k λ · (R_shift(f_k) / R_max), where f_k are the features used in a candidate policy's segmentation, R_shift is measured from the 6-month benchmark, and λ controls the stability-performance trade-off. Evaluate on the same 20 experiments with the 100 instructions, measuring whether stability-guided search produces policies with lower R_shift at equivalent metric lifts compared to post-hoc governance filtering. This is newly tractable because the paper provides the R_shift measurement methodology (Section 4.3) and the GAS candidate generation pipeline (Appendix B), making integration straightforward.

Failure mode catalog: systematic characterization of when GEARS governance fails. The paper reports only successes — nine surfaces with positive lifts (Table 3) and one detailed case study (Section 5.1) — but does not document failure modes. A rigorous follow-up would: deploy GEARS on 50 consecutive new experiments across multiple product surfaces, log every governance decision (accept/reject with structured reasons), track whether rejected policies would have degraded (by deploying a small subset of rejected candidates in a holdout validation arm), and track whether accepted policies maintained their lifts over a full 6-month deployment window. Measure: (a) the governance false positive rate (policies that passed governance but degraded by more than 50% of their initial lift within 6 months), (b) the governance false negative rate (policies that governance rejected but that, when deployed in the validation arm, maintained their lifts — these are lost opportunities), and (c) the agent's recommendation rejection rate (how often does the agent produce no recommendation because all candidates fail governance?). This would produce a confusion matrix for the governance layer that the current paper entirely lacks, and would identify specific failure modes — are false positives driven by features that appear stable on the 6-month R_shift metric but interact with other unstable features in deployment? Are false negatives concentrated in specific feature classes (engagement vs. product features in Table 2)? This experiment is large-scale and production-intensive, but it's the minimum requirement for validating GEARS's central claim that governance ensures deployment stability — a claim that the paper currently supports only with a 1-month backtest of a single policy (Figure 4).

GEARS for non-personalization ranking tasks: extending beyond cohort-treatment optimization. The paper evaluates GEARS exclusively on personalization — selecting cohort-treatment pairs from HTE experiments. But ranking systems involve many optimization decisions that are not cohort-level personalization: global parameter tuning (e.g., setting the weight of a diversity regularizer), latency-performance trade-offs (e.g., choosing model capacity given inference budget constraints), and multi-surface coordination (e.g., allocating impression budgets across recommendation surfaces to avoid cannibalization). A natural extension would adapt GEARS to one of these tasks and measure whether the architecture's components (Skills, Governance) transfer. Concretely: for a latency-performance trade-off task where the candidate "policies" are model configurations (different architectures, quantization levels, feature subsets) with measured latency and quality metrics, what replaces the feature stability governance hook? A natural candidate is a latency variance hook — measure whether a configuration's latency is stable under load variation over a multi-week window, analogous to the R_shift metric for feature stability. This would test whether the GEARS architecture (candidate generation + skill-based reasoning + deterministic governance) generalizes beyond its original domain, or whether the specific hooks and skills are so tightly coupled to personalization that substantial re-engineering is required per task.

Skill authoring cost and knowledge brain maintenance: an operational economics study. The paper claims GEARS reduces human engineering overhead but provides no measurement. A practical follow-up would track the full operational cost of running GEARS on a product surface for 6 months: (a) the number of new Skills authored (and expert hours per Skill), (b) the number of Knowledge Brain entries added or updated (and curator hours), (c) the number of governance threshold updates (as new feature stability benchmarks accumulate), (d) the number of agent runs that required human intervention (skill execution failures, governance false positives caught by human review, ambiguous recommendations requiring expert override), and (e) the total time from experiment launch to deployed policy recommendation with GEARS vs. a retrospective sample of pre-GEARS manual workflows on similar experiments. This would produce a total cost of ownership estimate that the paper currently lacks. If the per-experiment operational cost (skill authoring amortized, knowledge curation, human oversight) exceeds the cost of the manual workflow it replaces, GEARS's value proposition shifts from "reducing engineering overhead" to "enabling scalability" — it may cost more per experiment but handle a volume of experiments that would be impossible with manual workflows alone. Understanding which regime GEARS operates in is essential for adoption decisions, and the paper provides none of the data needed to distinguish them.

Practical Applications and Downstream Use Cases

Multi-surface ranking coordination at Meta scale. The paper demonstrates GEARS across nine product surfaces (Table 3) with lifts from 0.011% to 0.37%. The immediate practical application is expanding GEARS to coordinate policies across these surfaces jointly. Currently, each surface is optimized independently — a policy that improves Surface 3 by 0.10% on Metric 1 and 0.089% on Metric 2 might cannibalize engagement from Surface 1. A multi-surface GEARS deployment would use the same Skills architecture (with cross-surface guardrail skills added) and Governance layer (with cross-surface stability checks) to discover policies that are Pareto-efficient across surfaces, not just within a single surface. The 0.37% lift on Surface 8 (Metric 2) represents substantial headroom at Meta's billion-user scale — even a fraction of that lift, sustained across surfaces without cannibalization, translates to meaningful engagement gains. The paper's architectural modularity (Skills are filesystem-based and reusable across deployments, Section 3.2) makes cross-surface coordination a configuration change rather than an architecture change — new Skills for cross-surface guardrail checking, new Governance hooks for cannibalization detection, but the same pipeline structure.

Automated experiment analysis for teams without ranking expertise. The paper's offline benchmark (100 instructions over 20 experiments) demonstrates that GEARS can reliably select deployable policies from tabular experiment data with NDCG@1 of 0.94. This capability is directly applicable to product teams that run A/B experiments but lack dedicated ranking expertise. In a typical organization, a growth team might run experiments on notification timing, content ranking, or UI layout, but lack the specialized knowledge to interpret multi-metric trade-offs, audit feature stability, or assess deployment risk. GEARS could be deployed as an internal tool: the team provides an experiment link and a natural-language intent ("find policies that improve click-through without hurting time-on-page"), and GEARS returns a ranked list of deployable policies with interpretability analysis and stability guarantees. The paper's ablation (GEARS w/o Skill at NDCG@1 of 0.87) suggests that even without domain-specific Skills, the Governance layer alone provides substantial value — meaning a minimal deployment (intent translation + governance) could be operationalized quickly, with Skills added incrementally as domain expertise is encoded. The key limitation (Section 6.1) is the upstream cost of running GAS and accumulating 6-month feature histories, which may be prohibitive for smaller teams — a practical deployment would need to assess whether this infrastructure investment is justified by experiment volume.

Self-improving ranking infrastructure through automated policy distillation. Section 6 of the paper briefly mentions "self-optimizing systems" as a long-term vision. A concrete near-term application: use GEARS to generate high-quality deployable policies from past experiments, then distill those policies into the base ranking model as training data — a form of offline policy distillation. Specifically, for each completed experiment, GEARS selects the governance-approved recommended policy (e.g., "Target users in Q4 of engagement score with Treatment 2, users in Q1-Q3 with Treatment 1"). This policy specification — the cohort definitions, treatment assignments, and measured lifts — becomes a training example for a model that directly predicts optimal treatment assignments from user features, bypassing the need for per-experiment agent analysis. Over hundreds of experiments across multiple surfaces, this would produce a large dataset of GEARS-validated (governance-approved, backtested) treatment assignments, which could be used to train a single "policy recommendation" model that approximates GEARS's output without requiring LLM inference at deployment time. The paper's governance layer is critical here: only governance-approved policies enter the distillation dataset, ensuring the distilled model learns stable, deployable patterns rather than the brittle statistical optima that uplift models would surface.

When to Prefer This Method

The paper does not explicitly position GEARS against named alternative architectures or provide a decision framework for when to adopt it vs. alternative approaches. The comparison is primarily against manual expert workflows (which GEARS aims to automate) and prompting-only baselines (which GEARS outperforms on policy selection). There is no head-to-head comparison against alternative automated frameworks for ranking optimization (e.g., end-to-end RL-based policy search, Bayesian optimization with deployment constraints, or simpler automated reporting tools). As a result, a structured "Prefer GEARS when X, prefer alternative Y when Z" decision rule would be an extrapolation beyond the paper's evidence. The conditions where GEARS is likely beneficial can be inferred from its architecture — environments with pre-existing A/B experiment infrastructure, heterogeneous treatment effects, feature stability concerns, and multi-objective trade-offs — but the paper does not provide the comparative evidence against named alternatives that a decision matrix would require.