Scaling Agentic Capabilities, Not Context: Efficient Reinforcement Finetuning for Large Toolspaces

ArXiv: 2603.06713

🎯 Pitch

Small language models don't fail on large tool tasks because they lack reasoning ability—they fail because they can't manage context. ATLAS shows that a 4B model can approach frontier-agent performance simply by training it to load tools only when needed and to orchestrate execution with code instead of JSON, using rubric-based rewards to guide this structured behavior.

1. Executive Summary

This paper introduces ATLAS, a reinforcement finetuning framework that enables small language models (SLMs) to operate efficiently in large-scale MCP tool environments by learning how to acquire context and how to execute actions. Across MCP-Bench and a custom withheld benchmark using Qwen2.5-7B and Qwen3-4B models, ATLAS combines iterative tool loading (deferring full tool schemas until needed) with programmatic tool orchestration (expressing multi-step workflows as executable Python programs rather than turn-by-turn JSON interactions) and rubric-based reinforcement finetuning (decomposing task success into structured, task-aligned criteria scored by SLM judges rather than generic outcome rewards). The strongest SLM configuration—a 4B model with ATLAS—achieves a task fulfillment score of 4.15/10, approaching the frontier Kimi-K2 Thinking baseline at 4.38/10 despite operating under orders-of-magnitude tighter parameter and context budgets, establishing that structured execution combined with rubric-conditioned reinforcement finetuning closes the gap to frontier agents only when the SLM is explicitly trained to manage context acquisition and execution structure as learnable decisions.

2. Context and Motivation

The Core Problem: Small Language Models Break in Large Tool Environments

The fundamental challenge this paper addresses is straightforward to state but difficult to solve: small language models (SLMs) cannot reliably operate in modern agentic environments with hundreds of tools spread across dozens of servers. This is not because SLMs lack reasoning ability—the paper explicitly argues the limitation is not reduced reasoning capacity—but because the default architectural assumptions of frontier agent systems create a lethal mismatch with the operating constraints of efficient models. When you give a 4B-parameter model a task requiring coordination across multiple MCP servers, several things go wrong simultaneously, and they compound.

The most visible failure mode is context saturation. In a standard MCP setup, the agent receives a system prompt containing the full JSON schemas of every available tool across every server. For 28 servers exposing 257 tools (the scale of MCPBench used in this paper), this can consume tens of thousands of tokens before the agent has even seen the user's request. When the underlying model has a 32K context window—as the Qwen models in this paper do—the tool definitions alone can occupy most of the available budget, leaving minimal room for multi-turn reasoning, intermediate results, and the long execution traces that complex tasks demand. The paper quantifies this: under eager loading, Kimi-K2 Thinking (a 1T-parameter frontier model with 80K context) consumes roughly 23,768 tokens per trajectory on average (Table 1, Row 1). For SLMs operating at 32K context, this headroom is essentially nonexistent.

But context saturation is only the proximal symptom. The deeper problem is that SLMs lack the mechanisms to decide what context matters and when. When every tool definition is dumped into the prompt at the start, the model must reason over heterogeneous, verbose schemas from unrelated domains—weather APIs alongside financial data alongside scientific computing libraries—to select the right tool for a single sub-task. This is not merely wasteful; it is actively harmful. The model's attention is diluted across irrelevant information, making tool selection noisier and increasing the probability of misselection. A misselected tool early in a long-horizon workflow produces invalid outputs that propagate downstream, and with limited context remaining, the model struggles to recover.

The second compounding failure is execution brittleness in long-horizon tasks. In conventional JSON-based tool calling (the dominant paradigm inherited from ReAct-style agents), each tool invocation is a separate turn: the model generates a thought, calls a tool, receives the output, generates another thought, calls another tool, and so on. Control flow must be re-derived at each step from natural language reasoning. For a task requiring coordination across 2–3 servers with 4–6 distinct requirements (the "medium" to "hard" difficulty tiers in this paper), this turn-by-turn structure means that a small parsing error or a slightly incorrect argument in step 3 can invalidate everything downstream, and the model may not detect the error until step 12, at which point the context is too crowded to backtrack effectively.

The third and most subtle failure mode concerns supervision. MCP tasks rarely have a single verifiable ground-truth answer. A user request like "find the best-rated restaurant near the Metropolitan Museum that serves vegetarian food and check if there's a national park within driving distance" admits multiple valid execution trajectories. Did the agent use the right search radius? Were the tool outputs correctly interpreted? Was the parameter for "vegetarian" passed correctly? Outcome-based rewards—did the final answer look approximately right—cannot distinguish between an agent that executed correctly and one that got lucky with wrong intermediate steps. For SLMs trained with reinforcement learning, this makes credit assignment nearly impossible: the model receives a sparse, noisy signal at the end of a 20-turn trajectory and must somehow determine which of hundreds of decisions (server selection, tool selection, argument construction, output interpretation) was responsible for the outcome.

These three failure modes—context saturation, execution brittleness, and weak supervision—are not independent. Context saturation increases execution brittleness because the model cannot track state effectively. Execution brittleness amplifies the credit assignment problem because errors compound in ways that make it impossible to identify their source from a final outcome signal. Together, they create a regime where SLMs perform near chance on complex MCP tasks, even though the same models perform competently on isolated subtasks. The paper's cold-start baselines make this starkly visible: Qwen3-4B with iterative server loading achieves a task fulfillment score of just 2.73/10 (Table 1, Row 9), and Qwen2.5-7B fares worse at 2.33/10 (Row 4), despite both models being competent instruction-followers in simpler settings.

Why This Problem Matters: The Deployment Gap

This is not an academic concern. The paper is motivated by a concrete deployment reality: frontier models are too expensive, too slow, or too privacy-sensitive for many real-world agentic deployments, yet SLMs cannot fill the gap with current architectures.

The economic case is straightforward. Running a 1T-parameter model like Kimi-K2 Thinking for complex multi-turn agentic tasks costs orders of magnitude more than running a 4B model—in compute, in latency, and in energy. For enterprise deployments where an agent might handle thousands of requests per hour across customer support, internal tool orchestration, or data pipeline automation, the cost differential is prohibitive. Similarly, latency-sensitive applications (real-time assistants, interactive debugging, live data dashboards) cannot tolerate the multi-second inference times of frontier models, regardless of cost. And for privacy-constrained deployments—healthcare, legal, on-device personal assistants—sending user data to a cloud-hosted frontier model is simply not permissible.

Yet current SLMs cannot serve as drop-in replacements. The paper cites Kim et al. (2025) on the "science of scaling agent systems" to establish that naively adopting frontier agent designs for small models exposes structural failure modes. Figure 1 in the paper illustrates this concretely: traditional MCP agents incur context costs that grow linearly with the number of tools, while ATLAS's iterative loading and programmatic orchestration bound this growth. The implication is that without architectural changes, deploying an SLM in a large-toolspace environment means accepting either catastrophic failure rates or restricting the tool ecosystem to a handful of well-known APIs—defeating the purpose of MCP's universal connectivity.

The paper frames this as a capability gap that is architectural, not fundamental. SLMs can reason about tools, plan multi-step workflows, and interpret structured outputs—the evidence from simpler benchmarks is clear. What they lack is the scaffolding to do these things efficiently when the search space of possible actions is large and the context window is small. This reframing is important because it shifts the research question from "can we make SLMs smarter?" to "can we give SLMs better mechanisms for managing their own context and execution state?"—a fundamentally different optimization target.

Where Prior Approaches Fall Short

The paper identifies four lines of prior work, each addressing a piece of the problem but none providing an integrated solution for the SLM-in-large-toolspace regime.

Dynamic tool discovery works for frontier models, not SLMs. Frontier agent architectures (Anthropic's programmatic tool calling, Claude's dynamic tool search, described in Wu et al., 2025) elegantly solve the context saturation problem by retrieving tool schemas on-demand rather than preloading everything. The agent queries a tool registry at each step, loads only the relevant server, and executes. This works because frontier models have three properties SLMs lack: (1) large enough context windows that even "on-demand" loading doesn't strain the budget, (2) strong enough in-context reasoning to select tools from restricted views without seeing the full ecosystem, and (3) robust enough code synthesis to handle programmatic orchestration. The paper demonstrates this empirically: Kimi-K2 Thinking performs well with all tools eagerly loaded (TF 4.38, Row 1), but when switched to iterative tool loading—which restricts the tool view to JSON-limited schemas—its performance drops to 3.62 (Row 3). The very mechanism that saves context for SLMs hurts a frontier model because it removes information the model uses for reasoning. The lesson is that dynamic discovery alone is not a universal solution; its effectiveness is model-scale-dependent.

Learned tool invocation addresses tool selection but not execution structure. Work like Toolformer (Schick et al., 2023) and AutoTool (Jia and Li, 2025) trains models to decide which tool to call, often through self-supervised or RL-based approaches. These methods improve tool selection accuracy but treat execution as an afterthought—once the right tool is selected, the model still relies on standard JSON-based invocation and turn-by-turn interaction. In large-toolspace environments, correct tool selection is necessary but insufficient. An agent that selects the right weather API but passes a malformed location parameter, or correctly calls a mapping service but fails to parse the JSON response, still fails the task. Tool selection methods do not address the execution robustness problem that dominates failure modes in long-horizon settings.

Reinforcement learning for agents uses weak supervision signals. Recent work on RL for tool-using agents—ReTool (Feng et al., 2025a), ARTIST (Singh et al., 2025b), MUA-RL (Zhao et al., 2025)—trains models to interleave reasoning and tool calls using outcome-based rewards. These approaches typically rely on an LLM judge (often a frontier model like GPT-4o) that produces a single scalar score per trajectory, possibly with brief free-text justification. The paper identifies two fundamental problems with this approach when applied to the SLM-in-large-toolspace setting:

First, scalar rewards are too coarse for long-horizon, non-verifiable tasks. A single number cannot distinguish between an agent that executed four out of five requirements correctly and an agent that executed three correctly but with better parameter precision. When used in GRPO-style training (Guo et al., 2025), where advantage estimation depends on relative comparisons between rollouts for the same task, this coarseness introduces noise: two trajectories that differ in important structural ways receive similar scores because the judge's internal reasoning is inconsistent across evaluations. The paper explicitly states that "variability in the judge's internal reasoning and strictness can lead to inconsistent relative rankings, injecting noise into the learning signal and undermining stable policy optimization."

Second, frontier judges are expensive and don't scale. Every training iteration requires evaluating multiple rollouts per task with a frontier LLM. For 300 training tasks with 4 rollouts each (the paper's setup: batch size 16, rollout samples n=4), a single training step involves 1,200 frontier model evaluations. Over hundreds of training steps, the judging cost dominates the total compute budget. This creates a scalability ceiling: you can't train for many steps because the judge is too expensive, but you can't train for few steps because the learning signal is too noisy. It's a double bind.

Programmatic execution exists but isn't integrated with learning. Work on executable code actions (Wang et al., 2024b) and systems like mcp-cli (Schmid, 2026) demonstrate that representing tool interactions as code rather than natural language turns reduces context overhead and enables explicit control flow. But these are engineering solutions applied at inference time to already-capable models. They do not address the training problem: how do you teach an SLM to reliably generate correct programs using tool libraries it has never seen, when its coding knowledge is out-of-distribution relative to MCP tool APIs? The paper's Appendix A details the scaffolding required to make this work—input schema normalization, dynamic function allocation, informative error messages—and notes that even with this scaffolding, untrained SLMs produce "incorrect function names, mismatched arguments, or improper use of tool outputs." Programmatic execution without programmatic training is brittle.

The critical gap across all these approaches is the absence of mechanisms that explicitly regulate context growth and execution structure as learnable decisions. Frontier architectures assume the model is smart enough to manage context implicitly. Tool selection methods treat execution as a solved sub-problem. RL methods provide weak supervision that doesn't distinguish good process from lucky outcomes. Programmatic execution is a deployment trick, not a training objective. None of these lines of work ask: can we train an SLM to decide what context to acquire, when to acquire it, and how to represent execution compactly, using reward signals that capture the structured nature of agentic task success?

How This Paper Positions Itself

ATLAS is positioned as a framework that fills the gap between frontier agent architectures and SLM constraints by treating three things as first-class optimization targets: context acquisition, execution representation, and supervision structure.

The paper explicitly states (Section 1) that ATLAS is "complementary to frontier agent architectures in that it adopts the same high-level principles of on-demand tool access and structured execution, while explicitly targeting the efficiency-constrained regime where context, computation, and supervision are scarce." This is a careful positioning move. ATLAS does not claim to invent iterative tool loading or programmatic orchestration—these ideas exist in frontier systems. The novelty is in (1) making these mechanisms learnable through reinforcement finetuning rather than assuming the model can use them out of the box, and (2) introducing a supervision structure (rubric-based rewards) that makes this learning stable and scalable.

The reframing of agentic reasoning is crucial to understanding the paper's contribution. Rather than treating the problem as "improve the model's reasoning so it handles tool complexity better," ATLAS reframes it as "deciding what context to acquire, when to acquire it, and how to represent execution compactly." This is a metareasoning problem: the agent must learn a policy over its own information-gathering and representation choices, not just a policy over tool invocations. Iterative server loading is learning when to query the tool registry. Iterative tool loading is learning which tool schemas are worth the context cost. Programmatic orchestration is learning to represent execution state in code rather than natural language. These are decisions about the agent's own cognitive resource allocation, and the paper's central claim is that they must be learned—they cannot be scripted—because the optimal choices depend on task structure in ways that are too nuanced for hand-coded heuristics.

The rubric-based supervision mechanism is similarly positioned as solving a structural problem with RL for agents. Existing work treats the LLM judge as a black-box oracle that produces a reward. ATLAS decomposes the judging problem into rubric generation (done once per task, offline, with a frontier model) and rubric scoring (done many times per task, online, with an SLM). This separation has two benefits. First, it makes the reward signal consistent across rollouts—the rubric is fixed, so variability comes only from the trajectory, not from the judge's mood. Second, it makes the training loop scalable—once rubrics exist, evaluation can be performed by a 30B SLM rather than a frontier model, reducing judging cost by orders of magnitude per training step. The paper's finding that the SLM judge outperforms GPT-4o under rubric-based evaluation (Qwen3-4B: 3.87 vs. 3.43 TF, Table 1 Rows 13 vs. 12) is a striking validation of this decomposition: structured evaluation criteria reduce the judge's task from holistic assessment to criterion-specific checking, which SLMs can do reliably.

The paper also positions itself relative to the broader scaling narrative. The dominant story in the field, reinforced by scaling laws work (Hoffmann et al., 2022) and the success of ever-larger models, is that capability improvements come from scaling pretraining compute—bigger models, more data, longer training. The paper challenges this narrative not by denying the value of scale but by suggesting that for a given model scale, the architecture of context acquisition and execution structure dominates performance. A 4B model with ATLAS achieves 4.15/10 TF, approaching a 1T model's 4.38/10. The gap from 2.36 (cold-start ITL, Row 14) to 4.15 (ATLAS, Row 19) is 1.79 points—nearly matching the 1.62-point gap from the cold-start SLM to the frontier model. In other words, architectural and training improvements recover more performance than the scale difference between 4B and 1T parameters. This is a provocative finding that reframes the research agenda from "how do we train bigger models?" to "how do we make small models use their context and compute more effectively?"

Finally, the paper connects to the Model Context Protocol (MCP) ecosystem specifically, not just tool use generally. MCP standardizes how agents connect to external services, which is precisely what enables the tool-space explosion the paper addresses. As enterprises adopt MCP and connect agents to dozens or hundreds of servers, the context management problem becomes universal—not just for SLMs, but eventually for frontier models as well, as tool ecosystems outgrow even 80K context windows. ATLAS's mechanisms (server-level scoping, tool-level lazy loading, programmatic state management) are designed to scale sublinearly with the number of servers, making them relevant even as context windows grow. The paper is thus positioned not just as an SLM efficiency technique but as a direction for agent architecture more broadly—one where scaling agentic capabilities means scaling structure, not just context length.

3. Technical Approach

3.1 Reader Orientation

This paper presents ATLAS, a reinforcement finetuning framework that trains small language models to act as capable agents in environments with hundreds of tools spread across dozens of servers. The system solves the problem of SLMs breaking down in large-toolspace settings—not by making the model smarter, but by teaching it to make good decisions about what information to load into its limited context window and how to represent complex multi-step workflows compactly as executable code rather than verbose natural language turns, using structured, task-specific evaluation rubrics to provide stable learning signals in settings where no single ground-truth answer exists.

3.2 Big-Picture Architecture

The system has five major components that operate in a training loop:

1. The Base SLM (Qwen2.5-7B or Qwen3-4B) — the policy being trained. It receives a task description and makes sequential decisions about which servers to load, which tool schemas to materialize, and (in programmatic mode) what Python code to execute.

2. The MCP Tool Ecosystem (28–39 servers, 257–314+ tools) — the environment. Servers expose tools with schemas; the agent must discover, invoke, and compose them to complete multi-requirement tasks.

3. The Execution Scaffolding (ISL, ITL, PTC) — three progressive mechanisms that define how the agent interacts with tools. ISL lets the agent choose which server to load next; ITL lets it defer full tool schemas until needed; PTC replaces turn-by-turn JSON calls with a persistent Python interpreter that executes tool calls as function invocations.

4. The Rubric Generator (GPT-5, offline) — produces task-specific evaluation rubrics once per task before training begins. Each rubric is a set of weighted criteria across four categories: Task Fulfillment, Tool Appropriateness, Tool Grounding, and Parameter Accuracy.

5. The SLM Judge (Qwen3-30B-Instruct, online) — evaluates agent trajectories during training by scoring each trajectory against the fixed rubrics, producing a composite reward used by GRPO for policy updates.

Information flows as follows: Before training, GPT-5 generates rubrics for each of the ~300 training tasks (one-time offline cost). During training, the SLM receives a task, interacts with MCP servers through the execution scaffolding to produce a trajectory, the SLM judge scores that trajectory against the pre-generated rubric to produce a composite reward, and GRPO uses this reward to update the SLM's policy. A separate evaluation judge (o4-mini) assesses held-out performance on MCPBench and ATLAS-Test tasks without access to the training rubrics.

3.3 Roadmap for the Deep Dive

• First, the three execution mechanisms (ISL, ITL, PTC) and their Python scaffolding—these define the action space the SLM learns to navigate.
• Second, the rubric generation pipeline and the mathematical structure of rubric-based rewards—this is the supervision mechanism that makes learning possible in non-verifiable settings.
• Third, the training setup—GRPO, hyperparameters, and the cold-start RL regime—to understand how policy optimization interacts with the execution scaffolding and reward structure.
• Fourth, the evaluation framework—how performance is measured and why the evaluation judge is separated from the training judge.
• Fifth, the key design choices and their justifications—why each piece exists and what alternative would have been worse.

3.4 Detailed Sentence-Based Technical Breakdown

This paper is primarily a systems and training methodology paper whose core idea is that efficient agentic behavior in large toolspaces is a learned skill—specifically, learning when to acquire context, which tools to inspect, and how to structure execution—and that rubric-conditioned reinforcement finetuning provides the right supervision signal to teach this skill to small models that cannot fall back on raw scale.

Iterative Server Loading

Iterative Server Loading is the coarsest context-control mechanism. Instead of preloading the schemas of all available MCP servers into the system prompt at the start of an episode, the agent is given a compact index—essentially a list of server names and brief descriptions—and a meta-operation that retrieves the detailed tool schemas for a server when explicitly requested.

The mechanics work as follows. At episode start, the agent sees the task description and the compact server index. The agent's first decision is: which server do I need right now? It selects a server, issues the meta-operation to materialize that server's tools, receives the detailed schemas for only that server's tools, and executes the required tool calls within that server's scope. Only after completing work with the current server—or determining it needs capabilities from another server—does the agent select another server and repeat the process. This staged exposure means that at any given moment, the agent is reasoning over tools from exactly one server rather than tools from 28 servers simultaneously.

The critical property is that server selection is an explicit decision embedded in the agent's action space. The agent is not following a pre-scripted sequence of server loads; it must learn to choose servers based on the current task state and what it has already observed. This is a non-trivial decision: load the wrong server first, and you waste context on irrelevant tool schemas while the correct tools remain inaccessible until you switch. Load servers in the wrong order, and you might need to revisit a server you already left, incurring redundant context costs.

The paper quantifies the context savings. For Qwen3-4B under ISL, average tokens per trajectory is 9,152 (Table 1, Row 9), compared to Kimi-K2 with all tools loaded at 23,768 (Row 1)—though this comparison is confounded by model scale, it demonstrates the architectural principle that server-level scoping bounds context growth. ISL alone, however, is insufficient for task success: Qwen3-4B achieves only 2.73/10 TF under ISL without learning (Row 9). Server-level scoping prevents context explosion but doesn't help the model choose within a server's tool set, nor does it provide any execution structure beyond standard JSON tool calling.

Iterative Tool Loading

Iterative Tool Loading refines ISL by adding a second level of lazy materialization. Even within a single server, eager loading of all tool schemas can be prohibitive—MCP servers often expose "hundreds of tools with verbose schemas" (Section 2.2). ITL separates high-level planning from detailed tool grounding by introducing a two-phase loading process within each server.

When the agent loads a server under ITL, it initially observes only a compact list of tool names—not their full JSON schemas. This lightweight capability overview allows the agent to reason about which tools might be relevant and construct a rough plan without committing context to verbose schema definitions. As execution proceeds, when the agent determines it needs a specific tool, it explicitly requests the detailed schema for that tool, materializing the full function signature, parameter descriptions, and output format only at the point of use.

The mechanism is implemented through meta-operations that the agent can invoke: one to list tool names for a loaded server, and another to fetch the full schema for a named tool. The agent must learn to interleave these meta-operations with its task-oriented reasoning—deciding when it has enough information from tool names to proceed versus when it needs the full schema to construct correct arguments.

The paper demonstrates that ITL reduces token usage compared to ISL (Qwen3-4B: 9,152 → 9,045 tokens, Rows 9 → 14 in Table 1), but this comes with a performance tradeoff in the cold-start setting. Task fulfillment actually drops from 2.73 to 2.36. Why? Because the model now faces a harder reasoning problem: it must decide which tools merit full schema loading based only on names, and if it makes the wrong decision—loading an irrelevant tool or failing to load a necessary one—it has fewer opportunities to correct the error. The paper explicitly notes that "models not explicitly trained to reason over JSON-limited tool views struggle to fully exploit ITL under cold-start conditions." This is the same pattern observed with the frontier Kimi-K2 model, where ITL causes a performance drop (4.38 → 3.62, Rows 1 → 3) because the model was not trained to handle restricted schema views.

The implication is that ITL is not a free lunch—it trades context savings for increased reasoning difficulty, and the tradeoff only becomes favorable when the model is explicitly trained to navigate restricted tool views. This is why ATLAS treats ITL as a learned behavior through reinforcement finetuning, not merely an architectural choice.

Programmatic Tool Calling and the Python Scaffolding

Programmatic Tool Calling replaces the dominant JSON-based turn-by-turn interaction pattern with a unified code-based execution model. Under PTC, the agent does not generate individual tool calls separated by natural language reasoning. Instead, it writes executable Python code that invokes tools as functions, stores intermediate results in variables, implements explicit control flow, and produces final output—all within a single persistent interpreter session.

The core insight is that execution state should live in program memory, not in the prompt. In JSON-based tool calling, every intermediate result—a weather report, a list of restaurants, a map of national parks—is injected back into the context window for the model to read and reason about on the next turn. For a task with 5 sequential tool calls, each returning a JSON blob of several hundred tokens, the context accumulates these blobs and the model must parse them anew at each step. PTC eliminates this: the Python interpreter receives tool outputs, the code stores them in variables, and subsequent tool calls access those variables directly. The model never sees the intermediate results as raw text; it only sees the final output that the code chooses to expose.

This is not merely an efficiency gain—it fundamentally changes the nature of execution errors. In JSON-based calling, if the model misparses a tool output at turn 3, it may not discover the error until turn 8, at which point the context is cluttered with irrelevant information and recovery is difficult. In PTC, if the code encounters an error—an incorrect function name, a type mismatch, an attribute access on the wrong data structure—the Python interpreter provides an immediate error message with a traceback. The agent can then edit the specific line of code responsible, rather than replaying the entire reasoning trace.

The challenge is that MCP tool libraries are out-of-distribution for the model's coding knowledge. The model has never seen the function signatures, argument names, or output structures of these specific MCP tools during pretraining. Without additional structure, the model generates code that would be correct for a standard library but fails against the actual MCP API. The paper's Appendix A describes a multi-layer Python scaffolding that bridges this gap.

Scaffolding Layer 1: Input Schema Normalization. Different MCP servers describe their tools using inconsistent JSON schemas—different key names for similar properties, different levels of detail, different conventions for optional vs. required parameters. The scaffold runs an LLM script offline to map all tool input schemas into a common, clean schema and convert them to Python function signatures. For example, a tool might be normalized to:

home_manager_search(query: str (required), limit: int (optional, default=20, max=100))

These normalized signatures are stored offline and programmatically verified against the original JSON. This standardization means the model sees a consistent interface across servers—all function signatures follow the same naming and typing conventions regardless of the underlying API's idiosyncrasies.

Scaffolding Layer 2: MCPServer Class Abstraction and Dynamic Function Allocation. The scaffold provides a uniform MCPServer class instantiated per server (e.g., time_mcp = MCPServer("Time MCP")). When the model writes time_mcp.get_current_time(timezone='America/New_York'), the class dynamically maps this to the underlying MCP JSON API call. The class's attributes are populated at instantiation based on the server name, using the pre-computed schema mapping from Layer 1. This means the model does not need to know whether get_current_time is invoked via REST, WebSocket, or stdio—the abstraction handles the transport.

Scaffolding Layer 3: Server Output Conversion. MCP server responses arrive as serialized text (JSON strings). The scaffold converts these into appropriate Python native types using ast.literal_eval() with a top-down parsing approach that moves from structural types (lists, tuples, dicts) to primitives (int, float, str). This is essential because the model's code needs to index into dictionaries, iterate over lists, and perform arithmetic on numeric values—operations that fail on raw strings.

Scaffolding Layer 4: Output Schema and Example Fetching. Since most MCP servers do not define output schemas, the scaffold generates them proactively. An LLM endpoint is prompted to call each tool with realistic arguments (guided by the tool's input schema), stores the successful output format and an example for each tool, and makes this available through a get_tools_info([]) function. This function returns full schemas and examples only for specifically requested tools, and the representations are Python-native, making them significantly more token-efficient than raw JSON schemas.

Scaffolding Layer 5: Informative Error Logging. Standard Python error messages are unhelpful for MCP-specific failures. The scaffold augments them:

• For incorrect function names, instead of "object of class MCPServer has no attribute 'incorrect_func_name'", it produces: "MCP Server 'server_name' doesn't have the tool 'incorrect_func_name'. Available tools: [list]. Did you mean [closest tool]?"
• For argument errors, it passes through the server's own feedback on parameter mistakes.
• For incorrect output access, if the model tries to index a string as a dictionary, it produces: "You have tried to access a string as a dict, please check the output logs or use get_tools_details[]" along with the output schema.

These informative errors are the mechanism that enables the model to self-correct: instead of hitting a cryptic error and restarting from scratch, the model receives specific guidance about what went wrong and can make a targeted edit to the offending code line.

The paper reports that PTC improves cold-start performance even without learning: Qwen3-4B ITL with PTC achieves 2.94/10 TF versus 2.36 without PTC (Table 1, Rows 15 vs. 14). This is a ~25% relative improvement, attributed to "programmatic control flow reduces execution failures and stabilizes long-horizon behavior." However, token usage increases (9,045 → 13,462 average tokens, Rows 14 → 15) because executable code is more verbose than JSON tool calls—the added context cost of code is offset by reduced interaction turns (20 → 18) and improved task success.

Rubric-Based Reinforcement Finetuning

This is the paper's core training innovation. The problem it solves is that agentic MCP tasks are non-verifiable: there is no single ground-truth answer, and multiple execution trajectories can be valid. Outcome-based rewards (a single scalar score from an LLM judge) provide weak, noisy supervision because they collapse all dimensions of performance into one number and the judge's internal reasoning varies across evaluations.

ATLAS replaces scalar judging with rubric-conditioned judging. The key structural insight is separating rubric generation (done once per task, offline, with a frontier model) from rubric scoring (done many times per task during training, with an SLM).

The conceptual framework formalizes rubric-based rewards through a multi-level aggregation:

Level 1: Criterion-level scoring. For a given task $x$, the rubric generator produces a set of $N$ criteria:

$\mathcal{R}(x) = \{(C_i, D_i, W_i)\}_{i=1}^N$

where $C_i$ is the criterion name (e.g., "Weather data retrieved for correct city"), $D_i$ is a natural language description of what constitutes satisfactory performance on this criterion, and $W_i$ is a numerical weight indicating the criterion's importance.

Each criterion belongs to one of four categories: Task Fulfillment (TF)—whether core task requirements are met; Tool Appropriateness (TA)—whether selected tools are relevant and necessary; Tool Grounding (TG)—whether tool outputs are used faithfully and correctly; and Parameter Accuracy (PA)—the correctness and precision of tool arguments.

For a trajectory $\tau$, the judge assigns a score $d_i(\tau) \in [0, 1]$ to each criterion $C_i$, where 1 indicates full satisfaction and 0 indicates complete failure.

Level 2: Category-level aggregation. For each category $R \in \{\text{TF}, \text{TA}, \text{TG}, \text{PA}\}$ containing $N_R$ criteria, the category score is a weighted average:

$S_R(\tau) = \frac{\sum_{i=1}^{N_R} W_i \, d_i(\tau)}{\sum_{i=1}^{N_R} W_i}$

where $W_i$ is the weight of criterion $i$, and $d_i(\tau)$ is the judge-assigned score for that criterion on trajectory $\tau$.

What it computes: For each category, this produces a normalized score in $[0, 1]$ that summarizes performance across all criteria in that category, weighted by importance. If task fulfillment has three criteria with weights [3, 2, 1] and the agent scores [0.8, 0.6, 0.0], the category score is $(3 \times 0.8 + 2 \times 0.6 + 1 \times 0.0) / (3+2+1) = 3.6/6 = 0.6$.

Why this form: Weighted averaging (rather than unweighted averaging or minimum) allows the rubric designer to express that some criteria matter more than others—getting the main answer right is more important than using a perfectly precise parameter. The normalization by sum of weights ensures the score stays in $[0, 1]$ regardless of how many criteria exist per category. Alternative aggregation methods like taking the minimum would make the score dominated by the single worst criterion, which is inappropriate when criteria have independent failure modes.

Level 3: Composite reward. The final trajectory reward aggregates category scores with fixed category weights:

$R(\tau) = \sum_{R \in \{\text{TF}, \text{TA}, \text{TG}, \text{PA}\}} \alpha_R \, S_R(\tau)$

where $\alpha_R$ is a fixed weight per category, shared across all tasks, with higher weight assigned to task fulfillment.

What it computes: a single scalar reward in $[0, \sum \alpha_R]$ (with normalized weights, typically $[0, 1]$) that the agent receives at the end of one trajectory. This scalar is what GRPO uses for advantage estimation and policy updates.

Why this form: Explicitly separating categories and weighting them forces the judge to evaluate each dimension independently before combining. This is fundamentally different from asking a judge to produce a single score directly: the rubric constrains the judge's reasoning to specific, observable criteria rather than allowing holistic impressionistic evaluation. The fixed category weights ($\alpha_R$) ensure that task fulfillment dominates the reward regardless of how many criteria exist in each category for a particular task, preventing tasks with many grounding criteria from implicitly downweighting task success.

Automated Rubric Generation. The paper explicitly acknowledges that "manual rubric design is not scalable in MCP settings due to task diversity and heterogeneous tool usage." Rubrics are therefore generated automatically once per task using GPT-5 in an offline process.

The rubric generation prompt (detailed in Appendix F.1) includes only the task specification and available tool context—it is explicitly independent of any particular agent trajectory. This is crucial: if the rubric were generated from the frontier model's own trajectory, it would encode that model's biases and strategy preferences, penalizing valid alternative approaches that look different.

The rubric is constrained by three design principles:

1. Observability: criteria must be evaluable from the execution trace (the sequence of tool calls, arguments, outputs, and final answer). Criteria like "the agent had good intentions" or "the plan was clever" are excluded.
2. Non-overlapping: criteria within a category should capture distinct aspects of performance to avoid double-counting errors.
3. Functional alignment: criteria must reflect functional task requirements rather than surface-level language quality—whether the final response is well-written matters less than whether it's factually grounded in tool outputs.

The paper provides examples in Appendix E showing what these rubrics look like in practice. For a task involving weather lookup and restaurant recommendation, rubrics might include: "Correct city identified from user query" (TF, weight 3), "Weather data retrieved for the correct date range" (TA, weight 2), "Restaurant recommendations based on actual weather conditions" (TG, weight 2), and "API parameters correctly formatted for the weather service" (PA, weight 1).

Why GPT-5 for generation? Rubric generation requires strong task understanding and the ability to decompose a complex request into evaluable criteria—capabilities that current SLMs lack. However, this cost is paid once per task before training begins, not per trajectory. For ~300 training tasks, this is a manageable one-time expense.

SLM Judge for Scoring. The key scalability argument is that once rubrics exist, trajectory evaluation becomes a criterion-checking task rather than a holistic assessment task. The judge does not need to decide whether the trajectory was "good"—it needs to check whether specific, well-defined criteria were satisfied.

The paper demonstrates this empirically (Table 1): Under rubric-based evaluation, Qwen3-30B-Instruct as judge produces better training outcomes than GPT-4o as judge (Qwen3-4B: 3.87 TF with SLM rubrics vs. 3.43 with GPT-4o rubrics, Rows 13 vs. 12). Since the evaluation judge (o4-mini) is held fixed across conditions, this difference cannot be attributed to evaluation bias—the SLM-trained policy genuinely performs better.

The explanation offered is that rubric-conditioned evaluation reduces variance in reward signals. Without rubrics, a frontier judge like GPT-4o has latitude to weigh different aspects of performance differently across evaluations—sometimes prioritizing task completion, sometimes prioritizing tool correctness—leading to inconsistent relative rankings between trajectories for the same task. The rubric constrains this variance by forcing the judge to evaluate the same criteria in the same way every time. An SLM can do this reliably; a frontier model, with its greater capacity for nuanced judgment, may actually be less consistent because it perceives more dimensions of variability that the rubric doesn't ask it to evaluate.

Training Setup: GRPO with Cold-Start RL

ATLAS uses a cold-start reinforcement finetuning regime: there is no supervised fine-tuning phase with expert demonstrations before RL begins. The base instruction-tuned model (Qwen2.5-7B-Instruct or Qwen3-4B-Instruct) enters RL directly, receiving all supervision from rubric-based rewards. This is an important design choice: it tests whether the reward structure alone can teach the model effective agentic behavior, without requiring expensive trajectory-level annotations for supervised warm-start.

The optimization algorithm is Group Relative Policy Optimization (GRPO), as described in Guo et al. (2025). GRPO works by sampling multiple rollouts (trajectories) for the same task, computing the advantage of each rollout relative to the group mean, and updating the policy to increase the probability of above-average rollouts and decrease the probability of below-average ones.

The core GRPO advantage computation for a trajectory $\tau$ from a group of $n$ trajectories for the same task is:

$A(\tau) = \frac{R(\tau) - \mu_R}{\sigma_R}$

where $R(\tau)$ is the rubric-based reward for trajectory $\tau$, $\mu_R = \frac{1}{n}\sum_{j=1}^n R(\tau_j)$ is the mean reward across the group, and $\sigma_R$ is the standard deviation of rewards across the group.

What it computes: a standardized advantage score indicating how much better (positive) or worse (negative) this trajectory's reward is compared to the average reward of all trajectories sampled for this task. A trajectory with reward 0.8 in a group where the mean is 0.6 and standard deviation is 0.1 gets an advantage of +2.0—it's two standard deviations above average.

Why this form: Group-relative normalization is essential because tasks vary in difficulty and maximum achievable reward. Without normalization, a moderately successful trajectory on an easy task would receive a higher absolute reward than a brilliantly successful trajectory on a hard task, and the optimizer would overweight easy tasks. By normalizing within each task's group, GRPO ensures that the policy update is driven by relative quality within each task type, not by cross-task differences in reward scale. The division by standard deviation ($\sigma_R$) rather than a fixed constant makes the advantage scale adaptively—when all trajectories for a task are similar (low variance), small differences get amplified; when they vary widely, the advantage is appropriately conservative.

This dependence on consistent relative rankings is precisely why noisy, high-variance rewards are so damaging. If the judge's internal variability causes two trajectories that are genuinely similar in quality to receive substantially different scores, the apparent advantage signal is dominated by judge noise rather than genuine policy differences. The rubric-based approach mitigates this by making the judge's evaluation more consistent, which directly improves the signal-to-noise ratio of the GRPO advantage estimates.

Training hyperparameters (from Table 2):

• Train batch size: 16
• PPO mini-batch size: 4
• Max context window: 31,000 tokens
• Rollout samples per task ($n$): 4
• Rollout temperature: 1.0
• Advantage estimator: GRPO
• Learning rate: $1 \times 10^{-6}$
• Optimizer: AdamW
• LR schedule: Flat (no decay)
• Precision: bfloat16
• Max tool calls per trajectory: 20
• Max tool response length: 4,000 tokens
• KL loss coefficient: 0.001

Gradient masking over tool outputs. An important implementation detail: during policy updates, gradients are masked over tokens that represent tool outputs. The model generates tool calls (which server to load, which tool to invoke, what arguments to pass), but the tool responses are determined by the environment, not the model. Masking gradients over these tokens ensures that policy updates only affect the model's decisions—planning, tool selection, argument construction, execution control, and termination—rather than attempting to optimize the (non-differentiable) environment responses.

Hardware and framework. All experiments run on machines with 8 NVIDIA B200 GPUs. The verl library serves as the RL framework, extended with custom support for MCP tool calling including a custom rollout structure, truncation of overlong tool responses (capped at 4,000 tokens per the hyperparameters), and integration of LLM-based judge evaluation into the training loop. Reward computation and policy updates occur online during training—the model generates rollouts, the judge scores them, and GRPO updates the policy, all within the same training step.

Evaluation Framework

The paper separates the training judge from the evaluation judge to avoid self-evaluation bias. During training, rewards come from either a frontier judge (GPT-4o) or an SLM judge (Qwen3-30B-Instruct) conditioned on rubrics. During evaluation, a fixed external judge (o4-mini) scores all models using the standard MCPBench evaluation prompt (Appendix F.3), which assesses trajectories across the same four categories (TF, TA, TG, PA) but without access to the task-specific rubrics used in training.

This separation is methodologically important. If the training judge and evaluation judge were the same model, improvements in training reward might reflect the judge's own biases rather than genuine task performance improvements. By holding the evaluation judge fixed and distinct from the training judge, the paper ensures that reported performance gains reflect actual capability improvements visible to an independent assessor.

The primary evaluation metric is Task Fulfillment (TF) on a 0–10 scale, directly measuring end-to-end success on compositional, long-horizon MCP tasks. Secondary metrics include Grounding on Tool Results, Tool Appropriateness, Parameter Accuracy, average turns per trajectory, and average tokens per trajectory (reported in Appendix C, Tables 3–4).

Training data filtering. The training set of 304 tasks was constructed by generating over 1,000 candidate tasks using MCPBench's synthetic task generation pipeline (with o4-mini), then filtering through three stages:

1. Automated quality filters: tasks must receive solvability > 9 and utility > 8 from an LLM judge (o4-mini).
2. Frontier model solvability check: remaining tasks are executed with Kimi-K2 Thinking (1T parameters); only tasks achieving TF > 5 are retained. This ensures training tasks are actually solvable with the available tools, preventing the model from being trained on impossible tasks that would provide no useful learning signal.
3. Difficulty stratification: tasks are explicitly stratified into Easy (2–3 distinct requirements), Medium (4–5), and Hard (6+), with the distribution shown in Appendix Table 5.

Test sets. Two held-out evaluation sets are used:

• MCPBench (in-distribution): 104 unseen tasks from the same 28 servers used in training, measuring task-level generalization.
• ATLAS-Test (out-of-distribution): 100 tasks from 11 new servers not seen during training, measuring server-level and tool-level generalization under distribution shift.

Summary of Design Choices and Justifications

• Cold-start RL (no supervised warm-up): Tests whether the reward structure alone suffices; avoids the cost and potential bias of collecting expert demonstrations for ~300 diverse tasks. If the method works, it proves the reward signal is rich enough to teach the skill from scratch.

• GRPO over PPO with value function: GRPO's group-relative normalization is better suited to tasks with varying difficulty and reward scales—a value function would need to learn to predict expected reward across heterogeneous tasks, which is itself a hard learning problem in this setting. Group-relative advantage requires only that rewards are consistent within a task group, not comparable across tasks.

• Rubric generation with GPT-5 (offline) but scoring with SLM (online): Separates the hard reasoning step (decomposing a task into evaluable criteria) from the cheaper checking step (verifying whether each criterion is satisfied in a trajectory). The hard step is done once per task; the cheap step is done thousands of times during training.

• Three-stage training data filtering (quality, solvability, difficulty): Prevents training on impossible tasks (which would produce only negative examples), ensures tasks represent realistic agentic workflows (utility filter), and provides curriculum structure through difficulty stratification.

• Gradient masking over tool outputs: Prevents the strange optimization behavior that would occur if the model were rewarded or penalized for the content of tool responses it cannot control. This focuses learning entirely on the model's own decisions.

• Separate evaluation judge (o4-mini) distinct from any training judge: Eliminates the possibility that performance improvements are artifacts of judge bias rather than genuine capability gains. This is standard practice in RL for agents, but the paper's careful documentation of this separation strengthens the credibility of the results.

• Four-category rubric structure (TF, TA, TG, PA): Captures the major dimensions of agentic tool use identified in prior work (MCPBench evaluation framework). Task Fulfillment measures whether the user's request is satisfied; Tool Appropriateness measures whether the agent chose sensible tools rather than over- or under-using the tool ecosystem; Tool Grounding measures whether the agent faithfully used tool outputs rather than hallucinating; and Parameter Accuracy measures the precision of argument construction. Together they provide a complete picture of agentic competence.

• ast.literal_eval() with top-down parsing for output conversion: Takes advantage of Python's built-in safe evaluation that handles nested structures correctly. The top-down approach (testing structural types before primitives) prevents misclassification of strings that happen to contain dictionaries from being incorrectly parsed.

• Informative error messages with server-specific hints rather than raw Python tracebacks: Recognizes that the model's debugging ability depends on the quality of error signals. A message like "did you mean [closest tool]?" gives the model a direct path to correction that stack traces don't provide.

4. Key Insights and Innovations

Innovation 1: Treating Context Acquisition and Execution Structure as Learnable Decisions, Not Fixed Architecture

The most conceptually distinctive move in this paper is the reframing of what makes agentic behavior efficient in large toolspaces. Prior work—both in frontier agent architectures (Wu et al., 2025; Anthropic, 2025) and in learned tool invocation (Schick et al., 2023; Jia and Li, 2025)—treats mechanisms like dynamic tool discovery and programmatic execution as architectural features: you design them into the system, and a sufficiently capable model uses them correctly out of the box. The implicit assumption is that if the scaffolding exists, the model's general reasoning ability will handle the rest.

ATLAS challenges this assumption with a diagnostic finding: the mechanisms that save context for SLMs actively hurt untrained models, including frontier ones. The paper demonstrates this through a clean empirical dissociation. Kimi-K2 Thinking, a 1T-parameter frontier model, achieves 4.38/10 TF with all tools eagerly loaded (Table 1, Row 1). When switched to iterative tool loading—a mechanism designed to reduce context consumption—its performance drops to 3.62 (Row 3). The same pattern holds for cold-start SLMs: Qwen3-4B drops from 2.73 under ISL to 2.36 under ITL (Rows 9 → 14). The mechanism that should help actually hurts because it makes the reasoning problem harder—the model must now decide which tools to materialize based on limited information, a skill it hasn't acquired.

This is not an implementation failure; it is a conceptual finding. It means that the effectiveness of context-control mechanisms is not inherent to the mechanism but depends on whether the model has learned to use it. You cannot simply bolt iterative tool loading onto an SLM and expect gains. The skill of deciding what context to acquire, when to acquire it, and how to represent execution compactly must be explicitly trained—it is a policy learning problem, not an engineering problem.

This reframing distinguishes ATLAS from both ends of the prior work spectrum. On the frontier-architecture end, the assumption is that scale eliminates the need for explicit training on context management; the paper shows this assumption fails for SLMs and even partially for frontier models under restricted information. On the learned-tool-invocation end (Toolformer, AutoTool), the focus is on which tool to call, not how to structure the information flow around tool calling. ATLAS expands the learning target from tool selection to a broader set of meta-decisions: server scoping, tool schema materialization, and execution representation.

The significance of this reframing extends beyond the specific mechanisms. It suggests that as tool ecosystems scale, the bottleneck shifts from reasoning about what to do to reasoning about how to manage the cognitive resources needed to figure out what to do. This is a metareasoning problem—the agent must learn a policy over its own information-gathering actions—and it is fundamentally different from learning a policy over task-oriented actions. Prior work largely conflated these; ATLAS separates them, making context management a first-class optimization target rather than an assumed capability.

Evidence for the reframing comes from the paper's central interaction finding (Section 5.4): structured execution without learning yields limited gains, while learning without structured execution saturates early. The best performance emerges only when both are present—the ITL+PTC+RFT configuration achieves 4.15/10 TF (Row 19), substantially exceeding either component alone. This positive interaction is the empirical signature of the reframing: execution structure provides the representational capacity for efficient behavior, but learning provides the policy to exploit it.

Innovation 2: Rubric-Based Supervision as a Mechanism for Stable RL in Non-Verifiable Domains

The paper's second conceptual contribution is the diagnosis that the primary bottleneck in RL for agentic tool use is not the optimization algorithm or the model architecture, but the structure of the supervision signal. Prior work on RL for tool-using agents—ReTool (Feng et al., 2025a), ARTIST (Singh et al., 2025b), MUA-RL (Zhao et al., 2025)—uniformly uses scalar outcome rewards from LLM judges. The implicit model is: a powerful judge produces a score, and the RL algorithm uses that score as a reward. The paper identifies two failure modes in this model that are particularly acute for SLMs in large-toolspace settings.

Failure mode 1: Coarseness. A single scalar cannot distinguish between trajectories that succeed and fail in structurally different ways. Two trajectories might both receive a score of 6/10 from a judge, but one satisfied four out of five task requirements while the other satisfied three with better parameter precision. In GRPO, where advantage estimation depends on relative comparisons between rollouts for the same task (see Section 3.4), this coarseness means that genuinely different behaviors can receive similar rewards, flattening the advantage landscape and providing no gradient toward improving specific failure modes. The paper explicitly connects this to judge variability: "variability in the judge's internal reasoning and strictness can lead to inconsistent relative rankings, injecting noise into the learning signal."

Failure mode 2: Scalability ceiling. Frontier judges are expensive, and evaluating thousands of trajectories per training run with GPT-4o imposes a cost that dominates the training budget. This creates a double bind: you cannot train for many steps because the judge is too costly, but you cannot train for few steps because the reward signal is too noisy to converge quickly.

The innovation is not "use rubrics"—rubric-based evaluation is well-established in education and has been applied in NLP evaluation (Yu et al., 2025). The innovation is the separation of rubric generation from rubric scoring, which simultaneously addresses both failure modes. By generating rubrics once per task offline with a frontier model (GPT-5) and scoring trajectories online with an SLM (Qwen3-30B), the approach: (a) provides structured, multi-dimensional feedback that decomposes task success into specific, evaluable criteria, giving the RL optimizer gradient information about which aspects of behavior to improve; and (b) makes the per-step judging cost tractable, enabling longer training with more consistent evaluation.

The most striking empirical finding supporting this innovation is that the SLM judge under rubric-based evaluation outperforms the frontier judge (Qwen3-4B: 3.87 TF with SLM rubrics vs. 3.43 with GPT-4o rubrics, Table 1 Rows 13 vs. 12). This is counterintuitive: a weaker model produces better training outcomes as a judge. The paper's explanation—that rubric-conditioned evaluation reduces variance by constraining the judge to specific, well-defined criteria rather than holistic assessment—is a diagnostic insight about the nature of LLM judging. Frontier models, with their greater capacity for nuanced reasoning, may actually be less consistent judges because they perceive and weigh dimensions of trajectory quality that the rubric doesn't specify, introducing variance that harms GRPO's relative comparisons. The SLM, constrained to check specific criteria, produces more stable rankings even if its absolute judgments are less sophisticated.

This finding has implications beyond this paper. It suggests that the standard approach of "use the strongest available model as judge" may be suboptimal for RL training, where consistency of relative rankings matters more than accuracy of absolute scores. It also enables a practical scaling path for RL in agentic domains: invest in high-quality rubric generation once per task, then use cheap models for repeated evaluation. This is a fundamental shift from the current paradigm where judging cost scales with training steps.

Innovation 3: Empirical Evidence That Structure-Plus-Learning, Not Scale, Is the Efficient Path to Agentic Capability

The paper's third contribution is an empirical finding with implications for how the field thinks about scaling agentic systems. The dominant narrative in LLM research, reinforced by scaling laws (Hoffmann et al., 2022), is that capability improvements come primarily from scaling model size and training compute. The paper provides a concrete counterexample: in large-toolspace agentic tasks, the gap between a 4B SLM and a 1T frontier model can be almost entirely closed through architectural and training improvements, without changing model scale.

The numbers are worth examining closely because they reveal the magnitude and boundaries of this claim. The cold-start SLM baseline (Qwen3-4B with ITL) achieves 2.36/10 TF (Table 1, Row 14). The frontier baseline (Kimi-K2 Thinking with all tools loaded) achieves 4.38/10 (Row 1). The gap is 2.02 points. The best ATLAS configuration (Qwen3-4B with ITL, PTC, and rubric-based RFT) achieves 4.15/10 (Row 19). The gain from ATLAS is 1.79 points—it recovers roughly 89% of the gap to the frontier model. In other words, architectural and training improvements contribute more to closing the frontier gap than the 250× difference in parameter count.

This is not simply a "small models can be good" finding—that claim exists in prior work (Lu et al., 2024; Belcak et al., 2025). The distinctive contribution is the decomposition of where the gains come from. The paper's ablation structure (Table 1, organized by execution strategy and learning configuration) allows the reader to attribute improvements to specific components:

• Moving from ISL to ITL without learning: −0.37 points (2.73 → 2.36, Rows 9 → 14). Context control alone hurts.
• Adding PTC without learning: +0.58 points (2.36 → 2.94, Rows 14 → 15). Execution structure provides modest gains.
• Adding generic RFT to ITL: +1.12 points (2.36 → 3.48, Rows 14 → 16). Learning dominates, but saturates.
• Adding rubric-based RFT to ITL: +1.67 points (2.36 → 4.03, Rows 14 → 18). Structured supervision pushes further.
• Combining ITL, PTC, and rubric-based RFT: +1.79 points (2.36 → 4.15, Rows 14 → 19). Structure plus learning is synergistic.

The decomposition reveals that the gains are not additive—they are interactive. ITL creates a harder reasoning problem that only pays off when the model is trained to handle it. PTC provides execution robustness that amplifies the benefits of learning by reducing the failure rate of correctly-planned trajectories. Rubric-based rewards provide the credit assignment precision needed to learn both context management and execution structure simultaneously. No single component dominates; the system-level integration is the source of the gain.

This finding matters because it challenges the implicit assumption in much of the agentic systems literature that frontier-scale models are the only viable path to competent agentic behavior. The paper does not claim SLMs can match frontier models on all tasks—the hardest difficulty tier (6+ requirements) remains challenging—but it demonstrates that for a substantial fraction of realistic MCP tasks, the performance ceiling is set by architecture and training methodology, not by parameter count. This reframes the research agenda from "how do we train bigger models for agentic tasks?" to "how do we design architectures and training procedures that make efficient use of the model capacity we already have?"

The paper is appropriately cautious about overclaiming. The frontier model still leads (4.38 vs. 4.15), the evaluation is on synthetic MCP tasks, and the training data filtering ensures tasks are within the SLM's potential reach. But the core empirical result—that structure-plus-learning recovers most of the scale gap—is robust within these bounds and provides a concrete counterpoint to scale-centric narratives.

Innovation 4: The Diagnostic Finding That ITL Creates a Reasoning Problem That Must Be Learned, Not Engineered Away

A subtler but important contribution is a specific diagnostic finding that emerged from the paper's systematic ablation of loading strategies. As noted under Innovation 1, iterative tool loading reduces context usage but degrades cold-start performance. The paper could have interpreted this as a failure of ITL for SLMs and abandoned it in favor of eager loading or server-level scoping alone. Instead, the paper interprets it as evidence that restricted information views create a distinct reasoning skill that must be acquired through training, not engineered around.

This is a non-obvious claim because the natural engineering instinct is to simplify the agent's task: if the model struggles with limited tool views, give it more information. The paper's counterargument is that giving the model more information is precisely what causes context saturation in large toolspaces, and that the long-term solution is not to avoid restricted views but to train the model to operate effectively under them. The evidence for this interpretation comes from the interaction between ITL and RFT: ITL without learning underperforms ISL, but ITL with rubric-based RFT substantially outperforms ISL with the same RFT (4.03 vs. 3.87, Rows 18 vs. 13, both with Qwen3-30B rubrics). The mechanism that initially hurt becomes advantageous once the model learns to use it.

This finding has implications for how the field designs tool-use interfaces for language models. The dominant paradigm—providing full tool schemas in the system prompt—is optimized for models with large context windows and strong in-context reasoning. As tool ecosystems scale and as smaller models are deployed in agentic roles, this paradigm breaks. The paper's diagnostic finding suggests that the solution is not to keep expanding context windows (a hardware/architecture solution) but to develop models that can actively manage their own information intake (a capability solution). This is a different kind of research target: rather than building better tool registries, build models that are better at using tool registries adaptively.

The paper does not fully solve the problem of learning to operate under restricted views—the 4.15 TF ceiling is below the 4.38 frontier baseline with all tools loaded—but it establishes the problem as learnable and provides a training methodology (rubric-based RFT with structured execution) that makes learning tractable. This opens a research direction that prior work on tool selection and dynamic discovery had not fully articulated: the training problem for information-restricted agentic reasoning, as distinct from the inference-time mechanism design problem.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The training set comprises 304 tasks spanning 28 MCP servers, generated using MCPBench's synthetic task generation pipeline with o4-mini and filtered through three stages: automated quality filters (solvability > 9, utility > 8), frontier model solvability verification (tasks must achieve TF > 5 when executed by Kimi-K2 Thinking, 1T parameters), and difficulty stratification into Easy (2–3 requirements), Medium (4–5), and Hard (6+). For evaluation, two held-out test sets are used: MCPBench (104 unseen tasks from the same 28 servers, measuring in-distribution generalization) and ATLAS-Test (100 tasks drawn from 11 new servers not seen during training plus the original 28 servers, measuring out-of-distribution generalization to novel servers and tool combinations). The task distribution across server counts and difficulty tiers is provided in Appendix Table 5.

• Base model(s). All SLM experiments use Qwen2.5-7B-Instruct (7B parameters) and Qwen3-4B-Instruct-2507 (4B parameters), both with 32K context length. Qwen2.5-7B is chosen for strong instruction-following and suitability for reinforcement finetuning; Qwen3-4B is selected as a smaller model with native tool-calling support, representing a more challenging efficiency-constrained regime. As a frontier baseline, Kimi-K2 Thinking (1T parameters, 80K context) is evaluated with traditional MCP execution (all tools eagerly loaded) and with iterative loading variants to isolate the impact of context control independent of model scale. An additional variant with Qwen3-30B-Instruct is reported in Appendix C.

• Metrics. The primary metric is Task Fulfillment (TF) on a 0–10 scale, evaluated by an LLM-as-judge framework using o4-mini as the fixed evaluation judge. The judge scores complete agent trajectories across four categories following the MCPBench evaluation protocol: Task Fulfillment (whether core task requirements are satisfied), Tool Appropriateness (whether selected tools are relevant and necessary), Tool Grounding (whether tool outputs are used faithfully and correctly), and Parameter Accuracy (correctness and precision of tool arguments). The full evaluation prompt is provided in Appendix F.3. The training judge (which produces rewards for RL) and the evaluation judge are explicitly separated: training rewards come from either GPT-4o (frontier judge) or Qwen3-30B-Instruct (SLM judge) conditioned on task-specific rubrics, while evaluation scores always come from o4-mini without access to training rubrics.

• Baselines. The paper compares agent variants differing only in execution and learning mechanisms: (a) traditional MCP execution with all tools eagerly loaded (frontier baseline only), (b) iterative server loading (ISL) without learning, (c) iterative server and tool loading (ITL) without learning, (d) ISL with reinforcement finetuning using a generic GPT-4o judge producing scalar rewards, (e) ISL with RFT using a generic Qwen3-30B judge producing scalar rewards (following the LLM-as-judge paradigm from prior work such as ARTIST, Singh et al. 2025b, and DeepSeek-R1, Guo et al. 2025), (f) ISL with RFT using rubric-based rewards and a GPT-4o judge, (g) ISL with RFT using rubric-based rewards and a Qwen3-30B judge, (h) ITL with generic RFT (Qwen3-30B judge), (i) ITL with PTC (no learning), (j) ITL with PTC and generic RFT (Qwen3-30B judge), and (k) ITL with PTC and rubric-based RFT (Qwen3-30B judge). The frontier baseline Kimi-K2 is evaluated under three execution modes: all tools eagerly loaded, ISL, and ITL.

• Generation budget / compute accounting. The paper measures compute indirectly through average tokens per trajectory and average interaction turns, reported alongside TF scores in Table 1 and Appendix Tables 3–4. All SLM models share identical training hyperparameters (Table 2): 4 rollout samples per task (n=4), train batch size 16, PPO mini-batch size 4, maximum 20 tool calls per trajectory, and tool responses truncated at 4,000 tokens. All experiments use 8× NVIDIA B200 GPUs with bfloat16 precision. The rubric generation cost (GPT-5, offline, once per task) is explicitly separated from the per-step judging cost (SLM, online), making the training loop's marginal cost dominated by the SLM judge rather than a frontier model.

• Cross-validation / statistical protocol. The paper does not employ cross-validation in the traditional sense. Instead, task generalization is assessed through two held-out test sets: MCPBench (in-distribution, 104 tasks from seen servers) and ATLAS-Test (out-of-distribution, 100 tasks including 11 unseen servers). The rubric generator (GPT-5) is used offline before training begins; the training judge and evaluation judge are distinct models (Qwen3-30B or GPT-4o for training rewards, o4-mini for evaluation), preventing self-evaluation bias. Training curves (Figure 3) track composite training rewards and validation task fulfillment across steps, with validation performed on held-out MCPBench tasks.

Main Quantitative Results

5.1 Overall Performance on MCPBench

Table 1 reports the primary results across all model families, execution strategies, and RFT configurations. The headline finding is that the strongest SLM configuration—Qwen3-4B with ITL, PTC, and rubric-based RFT with a Qwen3-30B judge—achieves 4.15/10 TF, approaching the frontier Kimi-K2 Thinking baseline at 4.38/10 TF (Row 19 vs. Row 1), despite operating with 250× fewer parameters and substantially tighter context budgets.

The cold-start baselines establish the severity of the problem. Without any learning, both SLMs perform poorly: Qwen2.5-7B under ISL achieves 2.33/10 TF (Row 4), Qwen3-4B under ISL achieves 2.73/10 (Row 9), and Qwen3-4B under ITL achieves 2.36/10 (Row 14). These scores in the 2–3 range reflect near-random performance on compositional, multi-requirement MCP tasks. In contrast, the frontier Kimi-K2 model achieves 4.38/10 with all tools eagerly loaded (Row 1), demonstrating that raw scale can partially compensate for inefficient context management.

The paper's central interaction finding is that structured execution amplifies the gains from reinforcement finetuning. The performance progression for Qwen3-4B tells this story clearly: cold-start ITL at 2.36 (Row 14), adding generic RFT at 3.48 (Row 16, a gain of +1.12), adding rubric-based RFT at 4.03 (Row 18, a further gain of +0.55), and combining ITL + PTC + rubric-based RFT at 4.15 (Row 19, the best result). Each component contributes, but the combination produces more than the sum of individual gains: ITL provides context efficiency, PTC provides execution robustness, and rubric-based RFT provides the credit assignment precision needed to learn both simultaneously.

The frontier model shows an important negative pattern: Kimi-K2's performance degrades under iterative loading. Moving from all tools loaded (4.38, Row 1) to ISL (4.11, Row 2) to ITL (3.62, Row 3), the frontier model loses 0.76 TF points—a substantial drop for a model of this scale. The paper interprets this as evidence that "models not explicitly trained to reason over JSON-limited tool views struggle to fully exploit ITL," establishing that the skill of operating under restricted information views must be learned, not assumed.

5.2 Effect of Execution Structure Under Cold-Start Conditions

Holding learning fixed (no RFT), the paper isolates the impact of each execution mechanism.

Iterative Server Loading (ISL). Both SLMs exhibit low TF under ISL without learning: Qwen2.5-7B at 2.33 (Row 4) and Qwen3-4B at 2.73 (Row 9). The paper concludes that "server-level context control alone is insufficient for non-verifiable MCP tasks." ISL reduces context by scoping tool exposure to one server at a time, but does not help the model select tools within a server or structure execution across multiple tool calls.

Iterative Tool Loading (ITL). Moving from ISL to ITL reduces token usage (Qwen3-4B: 9,152 → 9,045 average tokens, Rows 9 → 14) but yields a small performance decrease from 2.73 to 2.36 TF. This mirrors the Kimi-K2 pattern where ITL degrades performance (4.11 → 3.62, Rows 2 → 3). The paper emphasizes this as a critical diagnostic: "models not explicitly trained to reason over JSON-limited tool views struggle to fully exploit ITL under cold-start conditions." The context savings are real but come at the cost of increased reasoning difficulty—the model must decide which tools to materialize based only on tool names, and untrained models make poor decisions.

Programmatic Tool Calling (PTC). Adding PTC to ITL improves cold-start performance from 2.36 to 2.94 (Rows 14 → 15), a gain of 0.58 TF points (~25% relative improvement). However, token usage increases from 9,045 to 13,462 (approximately 49% more tokens) because executable Python code is more verbose than JSON tool calls. The paper attributes the TF gain to reduced execution failures: "programmatic control flow reduces execution failures and stabilizes long-horizon behavior." The increased token cost is offset by fewer interaction turns (20 → 18) and, critically, by providing a stronger substrate for subsequent learning.

5.3 Impact of Reinforcement Finetuning

Holding execution structure fixed, the paper examines how RFT transforms performance.

ISL + RFT. Under ISL, RFT produces large gains across both model scales. For Qwen2.5-7B, generic RFT with a GPT-4o judge improves TF from 2.33 (Row 4) to 3.02 (Row 5, gain of +0.69). Switching to rubric-based RFT with an SLM judge further improves to 3.18 (Row 8, gain of +0.85 over the cold-start baseline). Qwen3-4B shows the same pattern but with larger magnitude: generic RFT with GPT-4o reaches 3.25 (Row 10, gain of +0.52 over baseline 2.73), while rubric-based RFT with Qwen3-30B reaches 3.87 (Row 13, gain of +1.14 over baseline). The gains are substantial (39–42% relative improvement) but appear to saturate under ISL—the execution structure limits how much learning can improve performance because the model is still constrained by eager tool loading within each server and turn-by-turn JSON interaction.

ITL + RFT (without PTC). Under ITL, generic RFT improves Qwen3-4B from 2.36 (Row 14) to 3.48 (Row 16, gain of +1.12). However, as the paper notes, "JSON-based ITL remains less effective than its PTC counterpart, even after learning." The model learns to handle restricted tool views, but the JSON-based execution still suffers from state tracking failures and context accumulation over long trajectories.

ITL + PTC + RFT (best-performing SLM regime). This is where the full ATLAS system realizes its potential. For Qwen3-4B, generic RFT with ITL+PTC reaches 3.91 (Row 17), already exceeding the best ISL-based result (3.87, Row 13). Adding rubric-based rewards pushes performance to 4.15 (Row 19), the highest SLM score in the table. The gain from adding PTC to the ITL + RFT pipeline is visible by comparing Rows 18 (ITL + rubric RFT, no PTC: 4.03) and 19 (ITL + PTC + rubric RFT: 4.15)—PTC adds 0.12 TF points even after strong RFT, indicating complementary benefits rather than redundancy.

An interesting detail: the paper notes that rubric-based rewards were not directly applied to PTC during training ("we do not yet apply rubric-based rewards to PTC, due to the challenge of defining concrete rubrics over executable code"). The reported configuration (Row 19) represents the best combination available given this constraint.

Figure 3 provides training dynamics, showing composite training rewards (left) and validation TF (right) across training steps for ISL, ITL, and ITL+PTC configurations. The paper highlights a notable dissociation: PTC and ITL achieve slightly lower training rewards across steps yet outperform other variants on validation TF at test time. The interpretation is that "this divergence indicates stronger generalization to unseen tasks and suggests that these methods produce trajectories that are less susceptible to reward hacking the training-time judge, prioritizing true task success over optimizing the train reward."

5.4 Rubric-Based Rewards vs. Generic Rewards

The paper presents a systematic comparison of reward design choices.

Rubrics vs. generic rewards. Across both model scales and execution strategies, rubric-based RFT consistently outperforms generic scalar rewards. For Qwen2.5-7B under ISL, generic RFT with GPT-4o achieves 3.02 (Row 5) while rubric-based RFT with GPT-4o achieves 3.13 (Row 7, gain of +0.11). The gains are larger for Qwen3-4B: under ISL, generic RFT with GPT-4o reaches 3.25 (Row 10) while rubric-based RFT with GPT-4o reaches 3.43 (Row 12, gain of +0.18). Under ITL, the generic RFT baseline reaches 3.48 (Row 16) while rubric-based RFT reaches 4.03 (Row 18, gain of +0.55). These results are consistent across judge types: the rubric advantage holds whether the judge is GPT-4o or Qwen3-30B.

The paper attributes this to "the importance of task-aligned credit assignment in non-verifiable MCP tasks." Rubrics decompose task success into specific, evaluable criteria, providing the RL optimizer with gradient information about which aspects of behavior to improve, rather than a single scalar that collapses all dimensions.

Frontier vs. SLM judges. The most striking judge comparison is in Table 1, Rows 12–13: under rubric-based evaluation, the SLM judge (Qwen3-30B) produces better training outcomes than the frontier judge (GPT-4o) for Qwen3-4B (3.87 vs. 3.43). This result is robust across model scales: Qwen2.5-7B shows the same pattern with rubric-based judges (SLM: 3.18, Row 8; GPT-4o: 3.13, Row 7), though the gap is smaller. Under generic rewards, GPT-4o and Qwen3-30B perform comparably (Qwen2.5-7B: 3.02 vs. 2.84, Rows 5–6; Qwen3-4B: 3.25 vs. 3.26, Rows 10–11), with no consistent advantage for the frontier judge.

Since all models are evaluated by a fixed external judge (o4-mini), these training-time judge differences cannot be attributed to evaluation bias. The paper interprets the SLM judge's advantage under rubrics as evidence that rubric-conditioned evaluation reduces variance in reward signals: the SLM, constrained to check specific criteria, produces more consistent relative rankings than the frontier model, which may perceive and weigh dimensions of trajectory quality not specified in the rubric. In GRPO, where advantage estimation depends on relative comparisons between rollouts for the same task, this consistency directly improves the signal-to-noise ratio of policy updates.

5.5 Closing the Gap to Frontier Models

The paper quantifies the frontier gap and the extent to which ATLAS closes it. The frontier Kimi-K2 Thinking baseline achieves 4.38/10 TF with all tools eagerly loaded (Row 1). The best SLM configuration (Qwen3-4B ITL+PTC with rubric-based RFT) achieves 4.15/10 (Row 19), representing 94.7% of the frontier score. The cold-start SLM baseline (Qwen3-4B ITL, no RFT) achieves 2.36/10 (Row 14), meaning ATLAS recovers (4.15 − 2.36) / (4.38 − 2.36) ≈ 88.6% of the gap between the untrained SLM and the frontier model.

The paper characterizes this as the SLM "approaching frontier-agent performance under far tighter parameter and context budgets." The 4B SLM uses roughly 0.4% of the frontier model's parameters (4B vs. 1T), operates within a 32K context window versus 80K, and consumes substantially fewer tokens per trajectory on average (13,400 tokens under ATLAS ITL+PTC, Row 19, vs. 23,768 for the frontier model with all tools loaded, Row 1). The paper explicitly notes that performance improvements from ATLAS components are "complementary, yielding consistent additive improvements when applied together."

Diagnostic metrics beyond TF are reported in Appendix C, Tables 3 and 4. The detailed breakdowns show parallel improvements across all four evaluation categories. For Qwen3-4B ITL+PTC with rubric-based RFT on MCPBench (Table 3, the equivalent of Table 1 Row 19): TF is 4.15, Grounding is not directly reported for this row in the main table but appears in Appendix C. On ATLAS-Test (Table 4), the same configuration achieves TF 3.79, demonstrating that gains transfer to out-of-distribution servers, though with some degradation relative to in-distribution performance.

The paper also reports a result with Qwen3-30B-Instruct (Table 3, bottom row): ITL+PTC with rubric-based RFT achieves 4.44/10 TF on MCPBench, actually exceeding the Kimi-K2 frontier baseline (4.38, Row 1). This appears in the appendix rather than the main table, but it provides additional evidence that the ATLAS methodology scales with model size—a 30B model with ATLAS can match or surpass a 1T model without ATLAS, reinforcing the claim that architecture and training methodology can substitute for raw parameter count.

Additional Diagnostic Metrics

Beyond TF, Table 1 reports average turns and average tokens. Several patterns emerge:

• Token efficiency of ITL: Under cold-start conditions, ITL reduces tokens relative to ISL (Qwen3-4B: 9,152 → 9,045 tokens, Rows 9 → 14), and ITL produces the lowest token usage among all configurations for the frontier model (Kimi-K2: 23,768 → 21,746 → 18,290 tokens across eager/ISL/ITL, Rows 1–3).

• Token cost of PTC: PTC consistently increases token usage due to executable code representations (Qwen3-4B ITL: 9,045 tokens, Row 14; ITL+PTC: 13,462 tokens, Row 15). However, this added cost is offset by improved TF once learning is applied. The paper frames this as a favorable tradeoff: "the added cost reflects explicit execution structure and is offset by improved task success."

• Turn reduction with RFT: Reinforcement finetuning generally reduces the average number of interaction turns. For Qwen3-4B under ISL, RFT reduces turns from 24 (Row 9) to 17–19 (Rows 10–13). For ITL+PTC, RFT reduces turns from 18 (Row 15) to 18–20 (Rows 17, 19). The frontier model shows the opposite pattern under ITL: turns increase from 20 (all tools loaded, Row 1) to 28 (ITL, Row 3), consistent with the interpretation that the frontier model struggles with restricted views and requires more interactions.

• Training dynamics (Figure 3): The left panel shows composite training rewards across steps for ISL, ITL, and ITL+PTC configurations. The right panel shows validation TF on MCPBench. The key observation is the dissociation mentioned earlier: PTC and ITL have slightly lower training rewards but higher validation TF, suggesting reduced overfitting to the training judge. The curves also show that learning is stable—rewards and TF increase monotonically and appear to approach asymptotes rather than degrading, indicating that the rubric-based reward structure does not suffer from the reward hacking or over-optimization issues that plague scalar reward approaches in other RL-for-agents settings.

Ablation Studies and Robustness Checks

Execution structure without learning: Table 1 shows that ISL (Row 9: 2.73 TF), ITL (Row 14: 2.36), and ITL+PTC (Row 15: 2.94) without RFT all produce TF scores well below 3/10. The paper correctly concludes that "structured execution alone is insufficient for small models." Cold-start performance is poor regardless of execution mechanism, establishing that execution structure provides the capacity for improvement but not the improvement itself.

Generic RFT vs. rubric-based RFT with matched judges: For Qwen3-4B under ISL, generic RFT with GPT-4o achieves 3.25 (Row 10) while rubric-based RFT with GPT-4o achieves 3.43 (Row 12), a gain of +0.18. With the SLM judge, generic RFT achieves 3.26 (Row 11) while rubric-based RFT achieves 3.87 (Row 13), a gain of +0.61. The rubric advantage is larger with the SLM judge, suggesting that rubrics are particularly beneficial when the judge has limited evaluation capacity—the structure compensates for the SLM's weaker holistic assessment ability.

Frontier judge vs. SLM judge under generic rewards: Table 1 Rows 10–11 show that GPT-4o and Qwen3-30B produce similar training outcomes under generic rewards for Qwen3-4B (3.25 vs. 3.26). For Qwen2.5-7B, GPT-4o has a slight advantage (3.02 vs. 2.84, Rows 5–6). The overall pattern is no consistent frontier judge advantage under generic rewards, which is itself informative: the standard assumption that stronger models make better judges does not hold in this RL training context.

Frontier judge vs. SLM judge under rubric-based rewards: This is the key robustness check for the paper's central claim about scalable judging. Table 1 Rows 12–13 show the SLM judge (3.87) outperforming the GPT-4o judge (3.43) under rubric-based rewards for Qwen3-4B. The same pattern holds for Qwen2.5-7B (SLM: 3.18 vs. GPT-4o: 3.13, Rows 8 vs. 7). This result is robust across model scales and contradicts the intuitive expectation that stronger judges produce better training outcomes.

ITL without vs. with RFT: Table 1 Rows 14 vs. 16 show that generic RFT improves ITL performance from 2.36 to 3.48 (+1.12, a 47% relative gain). Rows 14 vs. 18 show that rubric-based RFT improves ITL to 4.03 (+1.67, a 71% relative gain). The cold-start ITL result (2.36) being worse than ISL (2.73) confirms the paper's claim that ITL creates a harder reasoning problem that only pays off after training.

PTC without vs. with RFT: Rows 15 vs. 17 show that adding generic RFT to ITL+PTC improves TF from 2.94 to 3.91 (+0.97). Rows 15 vs. 19 show that adding rubric-based RFT to ITL+PTC improves TF to 4.15 (+1.21). PTC plus learning is the strongest configuration, but PTC alone provides only modest gains—the learning is doing the heavy lifting.

Generalization to unseen servers (ATLAS-Test): Appendix Table 4 provides a critical robustness check. The best SLM configuration (Qwen3-4B ITL+PTC with rubric-based RFT) achieves 3.79/10 TF on ATLAS-Test, compared to 4.15 on in-distribution MCPBench (Table 3). The degradation of 0.36 points is substantial but not catastrophic—the model transfers partially to novel servers and tool combinations. The frontier Kimi-K2 baseline drops from 4.38 (MCPBench, Table 3) to 4.36 (ATLAS-Test, Table 4) with all tools loaded, showing essentially no degradation, which is expected for a model of its scale operating without learning.

Scaling to 30B parameters: Appendix C Table 3 reports Qwen3-30B-Instruct with ITL+PTC and rubric-based RFT achieving 4.44/10 TF on MCPBench, exceeding the Kimi-K2 baseline (4.38). This result demonstrates that ATLAS benefits scale with model size—the methodology is not specific to very small models but provides gains across the parameter spectrum. However, this configuration was not evaluated with the same breadth of ablations as the 4B and 7B models, limiting direct comparisons.

Training stability (Figure 3): The training and validation curves in Figure 3 show monotonic improvement without degradation, across ISL, ITL, and ITL+PTC configurations. This is a meaningful robustness check because RL for language models often exhibits reward hacking or performance collapse, particularly when using learned reward models. The stability of ATLAS training—with composite rewards increasing smoothly and validation TF tracking training improvements—suggests that the rubric-based reward structure provides a well-behaved optimization landscape.

Solvability filtering of training data: The paper's training data pipeline includes a critical filtering step: only tasks where Kimi-K2 Thinking achieves TF > 5 are retained. This is described in Section 4.1 and Appendix D.2. While not an ablation in the traditional sense, this is a methodological choice with significant implications. It means the training set excludes tasks that are genuinely impossible or extraordinarily difficult, focusing learning on tasks where the SLM has some plausible path to success. The paper does not report what fraction of generated tasks were filtered out, nor does it evaluate SLM performance on the filtered-out tasks. This is a boundary condition on the reported results: the SLM was not trained on impossible tasks, and its failure modes on such tasks are unknown.

Critical Assessment

The experiments provide strong support for the paper's central empirical claims, but several important qualifications apply.

Claim: "Structured execution combined with rubric-conditioned reinforcement finetuning closes the gap to frontier agents." Supported, but partially. The best SLM configuration (4.15 TF) indeed approaches the frontier baseline (4.38 TF), recovering ~89% of the cold-start-to-frontier gap. However, this claim requires qualification about what "closes the gap" means operationally. The frontier model still leads by 0.23 TF points, and the frontier model achieves its score without any task-specific training, rubric generation, or execution scaffolding beyond standard tool loading. A fairer comparison might give the frontier model the same execution scaffolding and training—but the paper shows that the frontier model's performance degrades under iterative loading, suggesting this wouldn't help. The more precise claim is that ATLAS enables a 4B SLM to achieve performance that approaches (but does not match or exceed) an untrained 1T frontier model in a specific task distribution.

The Appendix C result with Qwen3-30B achieving 4.44 TF (exceeding Kimi-K2's 4.38) is notable but appears only in the appendix and with less ablation coverage. This result would strengthen the paper's thesis if it were in the main results table with matched experimental detail.

Claim: "Rubric-based rewards enable SLMs to serve as effective judges, outperforming GPT-4o-based generic judging." Strongly supported. The SLM judge (Qwen3-30B) under rubric-based rewards produces better training outcomes than GPT-4o under both generic and rubric-based rewards, across both model scales (Qwen2.5-7B: 3.18 SLM rubrics vs. 3.02 GPT-4o generic; Qwen3-4B: 3.87 SLM rubrics vs. 3.25 GPT-4o generic). The result is robust and the mechanism (reduced variance in relative rankings) is well-motivated. However, the comparison is between different judge models using different reward structures—it is not a pure ablation of rubric vs. no-rubric with the same judge. The finding that SLMs outperform frontier models specifically under rubric-based evaluation is the clean result; the comparison to generic GPT-4o is confounded by the simultaneous change of both judge model and reward structure.

Claim: "Execution structure amplifies the benefits of reinforcement finetuning." Supported by the interaction pattern in Table 1: ITL+PTC+RFT (4.15) substantially exceeds ITL+RFT without PTC (4.03) and ISL+RFT (3.87), while ITL+PTC without RFT (2.94) only modestly exceeds ITL without RFT (2.36) and ISL without RFT (2.73). The gains from structure are small in isolation but large in combination with learning. The paper could have strengthened this claim by running the full ablation matrix—RFT with all combinations of ISL, ITL, PTC under matched judge conditions—rather than having some combinations only with SLM judges and others only with generic rewards.

Missing experiments that would strengthen the paper:

• Pure ablation of rubric vs. generic reward with the same judge model. The comparison of "GPT-4o generic" vs. "GPT-4o rubrics" is confounded because the rubric-based judge has access to structured criteria that the generic judge does not. A cleaner ablation would give both judges the same information (rubric text) but ask one for a single score and the other for per-criterion scores, isolating the effect of structured output from the effect of structured input.

• Scaling the number of rubric criteria. The paper uses four categories with task-specific criteria but does not ablate the granularity of the rubric. Does performance improve with more fine-grained criteria or saturate? Is there a point where too many criteria make the judge's task harder again?

• Direct comparison of ATLAS-trained SLM vs. ATLAS-trained frontier model. The paper shows that Kimi-K2's cold-start performance degrades under ITL, but does not apply RFT to Kimi-K2. This leaves open the question: does ATLAS training benefit frontier models as much as SLMs, or is the benefit specific to the capacity-constrained regime?

• Ablation of training data filtering thresholds. The solvability > 9, utility > 8, and Kimi-K2 TF > 5 filters remove an unspecified fraction of generated tasks. How does performance change if these thresholds are relaxed? Would training on harder (but still solvable) tasks improve robustness? Would including some impossible tasks (TF < 5) hurt or help through exposure to failure modes?

• Ablation of rubric generation model quality. Rubrics are generated by GPT-5. How would performance change if rubrics were generated by a weaker model (GPT-4o, Qwen3-30B, or the training SLM itself)? The paper's thesis about scalable judging depends on the claim that rubric generation requires frontier capability while rubric scoring does not—but this claim is not directly tested.

Genuine weaknesses in the experimental design:

1. Single benchmark ecosystem (MCPBench). All results are on synthetically generated MCP tasks from the MCPBench pipeline. While the paper evaluates on both in-distribution and out-of-distribution server sets, the task structure (multi-requirement, tool-mediated, non-verifiable) is uniform. It is unknown whether ATLAS's benefits transfer to different agentic task formats—code generation, dialogue-based assistance, open-ended research—or to non-MCP tool ecosystems.

2. Training data is filtered by frontier model solvability. The filtering step that retains only tasks where Kimi-K2 achieves TF > 5 means the training distribution is explicitly biased toward tasks that are solvable by a very large model. This likely excludes tasks requiring capabilities genuinely outside the SLM's reach. The paper is transparent about this (Appendix D.2), but it means the reported results apply to tasks that are at least potentially within the SLM's capability envelope. Performance on genuinely out-of-capability tasks is unmeasured.

3. Small test sets. With 104 MCPBench test tasks and 100 ATLAS-Test tasks, the evaluation sample size is modest. Stratified by difficulty and server count (Appendix Table 5), some cells have very few tasks—for example, 3-server Easy tasks have 0 test instances, and 3-server Medium tasks have 21 ATLAS-Test instances. This makes difficulty-stratified performance comparisons unreliable at the finest granularity.

4. No statistical significance reporting. The paper reports point estimates (TF scores) without confidence intervals, standard errors, or significance tests. Given the small test set sizes, the reported differences between configurations (e.g., 4.15 vs. 4.03, a 0.12 gap between the two best SLM variants) may not be statistically significant. This is a common limitation in LLM agent evaluation but worth noting when comparing closely-spaced results.

5. Single training run per configuration. The paper does not report whether results are averaged over multiple random seeds or training runs. RL training is stochastic, and single-run results can overstate differences between configurations. The reported gains from rubric-based rewards over generic rewards (e.g., +0.55 from Row 16 to Row 18) could be partially attributable to random variation in training dynamics.

6. Unclear compute budget for training. The paper reports training hyperparameters (Table 2) but not total training FLOPs, wall-clock time, or number of training steps. This makes it difficult to assess the practical cost of ATLAS training relative to alternatives (e.g., supervised fine-tuning on expert trajectories, or simply using a larger model without training).

7. The PTC + rubric RFT result is incomplete. The paper notes that "we do not yet apply rubric-based rewards to PTC, due to the challenge of defining concrete rubrics over executable code." The best reported SLM result (4.15, Row 19) uses PTC for execution but it's unclear from the text whether the rubric-based RFT in this configuration actually evaluates the code-based execution or some hybrid. This is a significant caveat on the headline number.

Conditional boundaries on the claims:

• The paper's claims hold for MCPBench-style tasks with 1–3 servers and 2–6+ requirements, where the available tools are sufficient to solve the task (verified by frontier model solvability filtering). Claims should not be extrapolated to tasks requiring capabilities genuinely absent from the base model.

• The claim that SLM judges outperform frontier judges holds specifically under rubric-based evaluation. Under generic rewards, SLM and frontier judges perform comparably (Table 1 Rows 10–11).

• The claim of closing the frontier gap holds for in-distribution server sets (4.15 vs. 4.38 on MCPBench). On out-of-distribution servers (ATLAS-Test), the gap is larger (3.79 vs. 4.36 from Appendix Table 4), suggesting that generalization to entirely novel tools remains a challenge.

• The efficiency claims are measured in tokens per trajectory and interaction turns, not in FLOPs or wall-clock time. The training cost (GPT-5 rubric generation, SLM judge evaluation, RL policy updates) is not included in per-trajectory efficiency metrics, and the latency implications of serial vs. parallel tool calls are not discussed.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Unaccounted for in the Headline Efficiency Gains

The assumption or constraint. ATLAS's rubric-based reinforcement finetuning depends on task-specific rubrics generated once per task by a frontier model (GPT-5) before training begins. The paper explicitly separates this as an offline cost: "Manual rubric design is not scalable in MCP settings due to task diversity and heterogeneous tool usage. ATLAS therefore generates rubrics automatically once per task using a frontier LLM (GPT-5) offline" (Section 3.1). For the ~300 training tasks, this means 300 GPT-5 inference calls, each producing a detailed, multi-criteria rubric. The paper does not report the token cost, latency, or monetary cost of this generation step, nor does it amortize this cost into any reported efficiency metric.

The consequence. The headline claim that ATLAS enables "scalable and cost-efficient reinforcement finetuning" (Section 3.1) by replacing frontier judges with SLM judges during training is accurate for the per-step judging cost, but the fixed upfront cost of rubric generation with GPT-5 remains unaccounted for. For a practitioner deploying ATLAS on a new set of tasks, the pipeline requires: (1) generating ~300+ rubrics with GPT-5, (2) filtering training tasks through a frontier model solvability check (Kimi-K2 Thinking at 1T parameters, Section 4.1), and (3) running the RL training loop. The paper only optimizes step (3). In a production setting where tasks change frequently—new tool integrations, updated APIs, evolving user requirements—this upfront cost recurs with each task distribution shift. The "scalable" claim therefore applies to the marginal cost of training iterations given fixed rubrics, not to the total cost of deploying ATLAS on a new task distribution. A practitioner who needs to generate rubrics for thousands of tasks rather than hundreds would face a GPT-5 inference bill that could rival or exceed the RL training cost itself.

What evidence exists in the paper. The paper provides no ablation or measurement of rubric generation cost. Section 3.1 describes the process qualitatively, Appendix F.1 provides the rubric generation prompt, and Appendix E shows example outputs, but there is no reporting of tokens-per-rubric, total generation time, or estimated API cost. The solvability filtering step (Section 4.1), which requires executing all candidate tasks with Kimi-K2 Thinking and retaining only those with TF > 5, represents an additional unquantified frontier model inference cost that occurs before training. The paper acknowledges that the initial task generation produced "over 1,000 multi-server tasks" (Section 4.1), of which only 304 survived the three-stage filtering pipeline (quality, solvability, difficulty). The cost of generating and filtering the ~700 discarded tasks—including running Kimi-K2 on all of them—is not reported.

Mitigation status. The paper partially acknowledges the cost separation by describing rubric generation as "once per task" and "offline" (Section 3.1), framing it as a one-time investment. However, it does not quantify this investment, propose methods to reduce it (e.g., rubric reuse across similar tasks, few-shot rubric generation, smaller rubric generators), or amortize it into any reported metric. Section 7 briefly gestures toward future work on scaling but does not specifically address rubric generation cost. The paper's central scalability argument—that SLM judges can replace frontier judges during training—implicitly assumes that the upfront rubric generation cost is negligible relative to the per-step judging cost it eliminates, but this assumption is untested and task-count-dependent. For a practitioner with 300 tasks, it may hold; for 3,000 tasks, it may not; the paper provides no data to determine the crossover point.

Performance Collapses on the Hardest Problems; Test-Time Compute Cannot Create Capability the Base Model Lacks

The assumption or constraint. ATLAS trains SLMs to manage context and structure execution, but it does not expand the fundamental reasoning or knowledge capabilities of the base model. The paper's training data filtering procedure embeds an implicit assumption about the capability ceiling: tasks are retained for training only if Kimi-K2 Thinking (1T parameters) can achieve TF > 5 on them (Section 4.1, Appendix D.2). This means the training distribution is explicitly restricted to tasks that are at least potentially solvable by a model with SLM-scale reasoning capacity, since if a 1T model can solve a task, the task's reasoning requirements are within the reach of language models in general. Tasks requiring capabilities genuinely outside the SLM's pretraining distribution—specialized domain knowledge, complex multi-step deduction beyond its reasoning depth, or tool interactions it cannot conceptualize—are filtered out before training begins.

The consequence. ATLAS cannot help an SLM solve problems that are fundamentally beyond its capability envelope. This is visible in the paper's difficulty stratification (Appendix Table 5): hard tasks (6+ distinct requirements) are present in the training set (12 single-server, 29 two-server, 12 three-server), but the paper never reports performance stratified by difficulty tier. The aggregate TF of 4.15/10 for the best SLM configuration (Table 1, Row 19) blends easy, medium, and hard tasks into a single number, obscuring whether the gains are concentrated on easier tasks while hard tasks remain near the cold-start baseline. The paper's own framing acknowledges this boundary: "the primary limitation of SLMs is the absence of mechanisms that explicitly regulate context growth and execution structure" (Section 1), implying that if context management were the only limitation, ATLAS would solve the problem. But the filtering pipeline reveals a second limitation—fundamental capability—that ATLAS does not address. The paper's decision to exclude tasks where Kimi-K2 scores TF ≤ 5 means the evaluation cannot observe ATLAS's failure mode on genuinely hard problems; those problems were never in the training or test distribution.

A practitioner facing a task distribution that includes genuinely novel or complex problems—requiring reasoning depth the SLM lacks, or domain knowledge outside its pretraining—should expect ATLAS to provide minimal benefit. The paper demonstrates that ATLAS amplifies existing capability (recovering ~89% of the frontier gap on the filtered task set), but provides no evidence that it creates capability where none exists. For deployment scenarios where the task difficulty distribution is unknown or skews hard, this is a critical blind spot.

What evidence exists in the paper. The solvability filtering is described in Section 4.1 and Appendix D.2. The Kimi-K2 TF > 5 threshold is stated explicitly. However, the paper does not report: (a) what fraction of the original 1,000+ generated tasks were filtered out at the solvability stage, (b) the difficulty distribution of filtered-out vs. retained tasks, (c) performance of ATLAS-trained SLMs on the filtered-out tasks, or (d) TF scores stratified by difficulty tier (Easy/Medium/Hard) on MCPBench or ATLAS-Test. The difficulty stratification exists in the training data description (Appendix Table 5) but is never used as an analysis dimension in the results. This makes it impossible to determine whether the 4.15/10 TF is driven by near-ceiling performance on easy tasks masking near-floor performance on hard ones, or whether ATLAS provides roughly uniform gains across difficulty levels.

Mitigation status. The paper does not address this limitation explicitly. The filtering choice is presented as a data quality measure ("to ensure that retained tasks are genuinely solvable," Appendix D.2) rather than as a limitation on the scope of the claims. The absence of difficulty-stratified results means the reader cannot assess where ATLAS's benefits are concentrated. The paper does not propose methods to extend ATLAS to harder problems (e.g., curriculum learning starting from easier tasks, or combining ATLAS with retrieval-augmented generation to supply missing domain knowledge), nor does it discuss the boundary between "context management problem" and "capability problem" as a dimension for future work.

The Frontier Model Baseline Is Not Given the Same Execution Scaffolding or Training as the SLM, Making the Gap-Closing Claim Asymmetric

The assumption or constraint. The paper's central empirical claim—that ATLAS enables a 4B SLM to "approach frontier-agent performance" (Section 5.5)—compares the ATLAS-trained SLM against a frontier model (Kimi-K2 Thinking) using standard eager tool loading without any task-specific training. The paper is transparent about this design: Kimi-K2 is evaluated under three execution modes (all tools loaded, ISL, ITL) but never receives RFT, rubric-based rewards, or PTC scaffolding. The rationale is partially tested: the paper shows that Kimi-K2's performance degrades under ITL (4.38 → 3.62, Table 1 Rows 1 → 3), suggesting that giving the frontier model ATLAS-style execution without ATLAS-style training would hurt rather than help. But this leaves open a critical question: what would happen if the frontier model were trained with ATLAS?

The consequence. The "closing the gap" narrative is asymmetric in a way that systematically advantages the SLM. The SLM receives: (1) training on 304 in-distribution tasks with structured rewards, (2) PTC scaffolding with custom error handling and schema normalization, and (3) iterative loading mechanisms. The frontier model receives none of these; it is evaluated zero-shot with its standard tool-calling interface. The paper demonstrates that for SLMs, these components are essential—without them, SLM performance is in the 2–3 TF range. But it does not test whether these same components would also improve the frontier model, potentially expanding the gap rather than closing it.

The consequence is that the 4.38 vs. 4.15 comparison understates the frontier model's potential performance in this task setting. If Kimi-K2 were given the same 304-task training set with rubric-based RFT and PTC scaffolding, it might achieve substantially higher TF—widening the absolute gap even if the relative improvement from ATLAS is smaller for the larger model. The Appendix C result with Qwen3-30B—which achieves 4.44 TF with ATLAS, actually exceeding Kimi-K2's 4.38—provides a partial counterargument: if a 30B ATLAS-trained model can beat a 1T zero-shot model, perhaps scale-plus-ATLAS eventually asymptotes. But this is a single data point without the full ablation matrix (30B with vs. without ATLAS, 1T with vs. without ATLAS), so it does not resolve the asymmetry.

A practitioner choosing between "deploy a 4B model with ATLAS" and "deploy a frontier model" needs to know whether the frontier model's zero-shot performance represents its ceiling or its floor. The paper provides evidence that it's closer to the ceiling for SLMs (ATLAS provides large gains) but no evidence either way for frontier models. If frontier models also benefit substantially from ATLAS-style training—which is plausible, since they too struggle with execution errors and context management at scale—then the absolute performance ceiling is higher than the paper's baseline suggests, and the 4B SLM may not be "approaching" frontier performance in any operationally meaningful sense; it may simply be approaching a weak baseline.

What evidence exists in the paper. Table 1 provides Kimi-K2's performance under three execution modes (Rows 1–3), showing that ITL degrades performance. Appendix C Table 3 provides additional metrics (Grounding, Tool Appropriateness, Parameter Accuracy) for Kimi-K2 under these modes. No experiment trains Kimi-K2 with RFT, rubric-based rewards, or PTC. The paper does not discuss this asymmetry as a limitation. The Qwen3-30B result (Appendix C Table 3, achieving 4.44 TF with ITL+PTC+RFT) is the closest the paper comes to testing ATLAS at larger scales, but it is evaluated only on MCPBench (not ATLAS-Test), not compared against a 30B zero-shot baseline without ATLAS, and not compared against an ATLAS-trained frontier model.

Mitigation status. Not addressed. The paper presents the Kimi-K2 comparison as the "frontier baseline" without discussing the training asymmetry. The statement that ATLAS is "complementary to frontier agent architectures" (Section 1) suggests that ATLAS could be applied to frontier models but the paper does not test this. A fairer comparison would either: (a) give the frontier model the same training data and reward structure as the SLM, testing whether ATLAS benefits scale with model size; or (b) give the SLM the same zero-shot evaluation conditions as the frontier model (no training, no PTC), isolating the pure architectural contribution. The current design conflates "model scale" with "training status," making it impossible to attribute the gap closure to architecture vs. training vs. an interaction.

Programmatic Tool Calling Increases Token Usage and the Rubric-Based RFT + PTC Combination Is Incompletely Evaluated

The assumption or constraint. PTC replaces turn-by-turn JSON tool calling with executable Python code, which the paper acknowledges increases token usage: under ITL+PTC (cold start), average tokens rise from 9,045 to 13,462 (Table 1, Rows 14 → 15), a ~49% increase. The paper frames this as an acceptable tradeoff because PTC reduces interaction turns and improves task success once learning is applied. However, the paper also acknowledges that rubric-based rewards were not directly applied to PTC during training: "we do not yet apply rubric-based rewards to PTC, due to the challenge of defining concrete rubrics over executable code" (Section 5.3 discussion of Row 19). This means the best reported SLM configuration—ITL+PTC with rubric-based RFT (4.15 TF, Row 19)—uses a training setup where the reward signal was not designed to evaluate code-based execution.

The consequence. There are two separate concerns here. First, the token efficiency tradeoff of PTC is not fully characterized. The 49% increase in tokens per trajectory under cold-start conditions is reported, but the paper does not report token counts for the PTC+RFT configurations in a way that allows clean comparison. Table 1 Row 19 (ITL+PTC with rubric-based RFT) shows 13,400 average tokens, while Row 18 (ITL with rubric-based RFT, no PTC) shows 11,151 tokens—a 20% increase. But Row 16 (ITL with generic RFT) shows 12,815 tokens without PTC, suggesting that RFT itself increases token usage (longer trajectories as the model becomes more thorough), and the marginal token cost of PTC after RFT may be smaller than the cold-start comparison suggests. The paper does not disentangle these effects. For a practitioner deploying ATLAS in a latency- or context-sensitive setting, the token budget required for PTC-based execution is material and the paper does not provide a clear accounting.

Second and more fundamentally, the rubric-based reward signal was not designed for code-based execution. The paper explicitly states that rubrics were not applied to PTC trajectories "due to the challenge of defining concrete rubrics over executable code." But Table 1 Row 19 is labeled "PTC + RL w/ Qwen3-30B J. (Rubrics)," implying that rubrics were used as rewards for PTC training. If the rubrics were not designed to evaluate code quality, correctness, or execution structure, what exactly was the SLM judge scoring? The paper does not clarify whether the rubric-based RFT in Row 19 evaluated the final task output only (ignoring the code-based execution), used a modified rubric that included code-level criteria, or applied the standard rubrics to the code-bearing trajectory in some approximation. This ambiguity undermines the headline result: the 4.15 TF score is the paper's best SLM number, but the training procedure that produced it is incompletely specified.

A practitioner attempting to replicate the PTC + rubric RFT result would face an underdefined training setup. Should they generate rubrics that include criteria about code correctness? About execution efficiency? About error handling? The paper provides no guidance because it acknowledges the challenge without resolving it.

What evidence exists in the paper. Table 1 reports token counts alongside TF scores. The cold-start PTC token increase is visible (Rows 14 → 15). The paper's discussion of PTC in Section 5.2 notes that "PTC increases token usage due to executable representations, but reduces interaction turns and enables higher TF once learning is applied. The added cost reflects explicit execution structure and is offset by improved task success." This is a qualitative tradeoff argument without quantitative cost-benefit analysis. The rubric-PTC interaction is discussed in the Section 5.3 text accompanying Rows 18–19, where the paper states the limitation explicitly. Appendix A provides detailed PTC scaffolding design but does not discuss how it interacts with rubric-based evaluation.

Mitigation status. The paper partially mitigates the token concern by arguing the tradeoff is favorable and providing token numbers that allow readers to make their own assessment. The rubric-PTC interaction is acknowledged but not mitigated—the paper states the limitation without offering a solution, making the Row 19 result provisional. No future work is proposed for rubric design over code-based execution, despite this being identified as a current challenge. A complete mitigation would require either: (a) developing rubrics that evaluate code-based trajectories (checking for correct function calls, proper error handling, efficient control flow) and demonstrating that they improve training outcomes; or (b) reporting PTC results with generic (non-rubric) rewards as the primary result and treating rubric-based PTC as preliminary. The paper does neither.

All Experiments Are on Synthetic MCP Tasks with a Single Model Family; Cross-Domain and Cross-Model Generalization Is Untested

The assumption or constraint. All experiments use synthetically generated tasks from the MCPBench pipeline, evaluated on two model families (Qwen2.5 and Qwen3) with a single frontier baseline (Kimi-K2). The paper explicitly acknowledges this scope: the training set is "synthetic but realistic MCP tasks constructed using live MCP servers" (Section 4.1), and the models are described as "open-weight language models spanning different parameter scales and levels of agentic capability" (Section 4.2). The tasks all share a common structure: multi-requirement queries requiring tool discovery, invocation across 1–3 MCP servers, and synthesis of results. There is no evaluation on non-MCP tool ecosystems (e.g., REST APIs without MCP standardization, code execution environments without server abstractions, embodied agent tasks), on non-synthetic tasks (e.g., real user queries from production systems), or on tasks with fundamentally different structure (e.g., open-ended dialogue, creative generation, tasks requiring negotiation or multi-agent coordination).

The consequence. The paper's findings may be specific to the MCP task format, the synthetic generation pipeline, or the Qwen model family's particular strengths and weaknesses. Several aspects of ATLAS could fail to transfer:

• Rubric generation quality: GPT-5 may produce effective rubrics for MCPBench-style tasks because these tasks have clear, enumerable requirements (find weather, get restaurants, check parks). For more open-ended agentic tasks—"help me plan a vacation," "debug this codebase," "negotiate with this API"—generating comprehensive, non-overlapping, observable rubrics may be substantially harder. The paper's rubric design principles (observability, non-overlapping, functional alignment) assume tasks decompose cleanly into independent criteria, which may not hold for fuzzier objectives.

• PTC scaffolding: The Python scaffolding described in Appendix A (schema normalization, MCPServer class, output conversion, informative errors) is specifically designed for MCP's JSON-based server protocol. Deploying ATLAS in a different tool ecosystem (e.g., REST APIs with inconsistent documentation, proprietary enterprise software with binary interfaces, physical robotics APIs) would require re-engineering the entire scaffolding layer, with no guarantee that the resulting interface would be as learnable for SLMs.

• Model family dependence: Qwen models may have particular properties—instruction-following strength, code generation capability, tool-calling training—that make them well-suited to ATLAS's approach. The paper notes that Qwen3-4B has "native tool-calling support" (Section 4.2), which may give it an advantage in learning ITL and PTC behaviors that other model families (e.g., Llama, Gemma, Mistral) lack. Without experiments on multiple model families, it's unclear whether ATLAS's gains are specific to Qwen's training recipe.

• Task diversity: All tasks involve 1–3 servers and 2–6+ distinct requirements. The paper does not test whether ATLAS scales to tasks requiring 10+ servers or 20+ sequential tool calls, which would stress-test the context management mechanisms. The finding that ITL reduces context but increases reasoning difficulty (Section 5.2) suggests there may be a crossover point where even trained models cannot effectively manage the restricted information views ITL imposes—this crossover is not characterized.

What evidence exists in the paper. The task generation process is described in Section 4.1 and Appendix D. The server list (28 MCPBench servers + 11 ATLAS extensions) is provided. The model selection rationale is given in Section 4.2, citing Qwen's instruction-following and tool-calling capabilities. The ATLAS-Test evaluation (100 tasks from 11 unseen servers) provides a partial generalization test, showing that gains transfer to novel tools but with some degradation (TF drops from 4.15 on MCPBench to 3.79 on ATLAS-Test for the best configuration, Appendix Tables 3–4). However, this tests generalization across servers within the same task format and synthetic generation pipeline—not across task formats, model families, or tool ecosystems.

Mitigation status. The paper does not claim cross-domain or cross-model generalization; the scope is explicitly MCP tasks with Qwen models. The ATLAS-Test evaluation partially addresses server-level generalization but not task-format or model-family generalization. No future work is proposed on extending ATLAS to other domains, other model families, or non-synthetic tasks. For a practitioner using non-Qwen models or operating in non-MCP tool environments, the paper provides no evidence that ATLAS's benefits will transfer, and the substantial engineering investment required to replicate the scaffolding (Appendix A) represents an unquantified risk.

Training Stability and Result Reliability Are Assessed from Single Runs on Small Test Sets Without Statistical Reporting

The assumption or constraint. The paper reports point estimates for all metrics (TF, Grounding, Tool Appropriateness, Parameter Accuracy, turns, tokens) without confidence intervals, standard errors, statistical significance tests, or indication of whether results are averaged over multiple training runs with different random seeds. The test sets are small: 104 tasks for MCPBench and 100 tasks for ATLAS-Test. When stratified by server count and difficulty (Appendix Table 5), some cells contain very few instances—for example, 3-server Hard tasks have only 18 instances in MCPBench-Test. The training dynamics curves (Figure 3) show validation TF increasing over steps but do not include error bars or shaded regions indicating variance across seeds.

The consequence. Several of the paper's key comparisons involve small absolute differences between configurations where statistical reliability matters:

• The best SLM result (4.15 TF, Row 19) vs. the second-best (4.03 TF, Row 18) is a gap of 0.12 on a 0–10 scale. On a test set of 104 tasks, this corresponds to roughly 1.2 additional tasks fully satisfied out of 104—a difference that could easily arise from sampling variation, judge stochasticity, or single-seed training noise.
• The claim that rubric-based rewards outperform generic rewards (e.g., 4.03 vs. 3.48 for ITL with SLM judge, Rows 18 vs. 16, a gap of 0.55) is larger and more likely robust, but still unaccompanied by variance estimates.
• The claim that the SLM judge outperforms the GPT-4o judge under rubrics (3.87 vs. 3.43, Rows 13 vs. 12, gap of 0.44) is central to the paper's scalability argument, but the comparison involves two different training runs with different judge models—and potentially different random seeds, different training trajectories, and different final checkpoints. Without seed averaging, it's impossible to determine whether the observed difference reflects a genuine judge quality effect or random variation in RL training dynamics.

The training curves in Figure 3 show that validation TF is not perfectly monotonic—there are fluctuations across steps—which means the final reported TF depends on when training was stopped and which checkpoint was evaluated. The paper does not specify an early stopping criterion, whether the best checkpoint on validation TF was selected, or whether the reported numbers are from the final checkpoint. If the best validation checkpoint was selected, the reported numbers may be optimistic relative to a fixed-budget training run.

What evidence exists in the paper. The paper reports point estimates in Table 1 and Appendix Tables 3–4. Figure 3 shows training curves for composite reward (left) and validation TF (right) across steps for ISL, ITL, and ITL+PTC configurations, with a single line per configuration—no error bars, no multiple seeds. The hyperparameters table (Table 2) does not list a random seed. The training data description (Appendix D.2) does not mention train/validation splits within the 304 training tasks; the validation TF in Figure 3 is presumably computed on the MCPBench test set (104 tasks), meaning test performance was monitored during training. If the best validation checkpoint was selected, this introduces a subtle form of test-set leakage: the reported TF may reflect a model selected because it performed well on the test set, overstating generalization.

Mitigation status. Not addressed. The paper follows common practice in LLM agent evaluation—point estimates without statistical reporting—but this practice is increasingly recognized as problematic when comparing closely-spaced results. For a paper whose central claims involve relative comparisons between configurations (rubrics vs. generic, SLM judge vs. frontier judge, PTC vs. no PTC), the absence of variance estimates weakens the inferential strength of those comparisons. The paper does not discuss seed sensitivity, checkpoint selection, or statistical testing as limitations, and does not propose future work on more rigorous evaluation protocols. A practitioner deciding between ATLAS configurations based on reported TF differences of 0.1–0.5 points has no way to assess whether those differences would replicate on a different random seed, a different task sample, or a different training run.

7. Implications and Future Directions

How This Work Changes the Landscape

ATLAS makes a reframing argument that shifts how the field should think about agentic capability in large toolspaces. The dominant narrative—reinforced by scaling laws work and the successive releases of ever-larger frontier models—has been that competent agentic behavior requires scale: bigger models, longer context windows, more pretraining compute. ATLAS provides a concrete counterexample: on MCPBench, a 4B SLM with ATLAS achieves 4.15/10 TF, approaching a 1T frontier model's 4.38/10 (Table 1), recovering ~89% of the cold-start-to-frontier gap while using 0.4% of the parameters and a 32K context window versus 80K. This is not a claim that SLMs universally match frontier models—the paper is explicit that hard problems remain challenging and that the frontier model still leads—but it demonstrates that for a substantial class of realistic agentic tasks, the performance ceiling is set by architecture and training methodology rather than by parameter count.

The reframing operates at a conceptual level that distinguishes ATLAS from prior work on both tool use and RL for agents. Prior work on dynamic tool discovery (Wu et al., 2025; Anthropic, 2025) treats mechanisms like iterative loading and programmatic execution as architectural features—you design them into the system, and a sufficiently capable model uses them out of the box. Prior work on learned tool invocation (Schick et al., 2023; Jia and Li, 2025) focuses on which tool to call, treating execution as a solved sub-problem. ATLAS argues that context acquisition and execution structure are themselves learnable skills that must be explicitly trained, not assumed capabilities. The diagnostic finding that makes this argument convincing is the counterintuitive pattern in Table 1: iterative tool loading degrades cold-start performance for both SLMs (Qwen3-4B: 2.73 → 2.36, Rows 9 → 14) and the frontier model (Kimi-K2: 4.11 → 3.62, Rows 2 → 3), but improves performance substantially after reinforcement finetuning (Qwen3-4B ITL with rubric RFT: 4.03, Row 18). If ITL were merely an architectural feature, it would either help or hurt uniformly—it wouldn't switch from harmful to helpful based on training status. The fact that it does switch is the empirical signature that context management is a learned policy, not an engineering choice.

This reframing reconciles a tension in the prior literature. On one hand, dynamic tool discovery and programmatic execution have been demonstrated to work well for frontier models in production systems (Anthropic's Claude platform). On the other hand, attempts to deploy SLMs in similar settings produce brittle failures (Belcak et al., 2025; Kim et al., 2025). The ATLAS diagnosis is that both observations are correct and non-contradictory: dynamic discovery does work for frontier models because their scale provides enough reasoning capacity to handle restricted information views without explicit training, but it doesn't work for SLMs because they lack this capacity and must acquire it through reinforcement finetuning. The prior literature's conflicting findings reflect an unacknowledged model-scale confound: methods tested on large models succeed, the same methods tested on small models fail, and the field had not recognized that the mechanism's effectiveness is scale-dependent and trainable.

The paper also shifts the conversation around supervision for agentic RL. The standard approach—using a frontier LLM judge to produce a single scalar reward per trajectory—has two known problems: coarseness (a single number cannot distinguish structurally different failure modes) and cost (frontier judge inference dominates the training budget). ATLAS's solution is to separate rubric generation (done once per task offline with GPT-5) from rubric scoring (done many times per task online with a 30B SLM). The finding that the SLM judge under rubric-based evaluation outperforms GPT-4o (Qwen3-4B: 3.87 TF with SLM rubrics vs. 3.43 with GPT-4o rubrics, Rows 13 vs. 12) is a striking validation of this decomposition. It challenges the default assumption that stronger models make better judges for RL training, and suggests instead that consistency of relative rankings matters more than absolute judgment quality for GRPO-style advantage estimation. A frontier model, with its greater capacity for nuanced reasoning, may perceive dimensions of trajectory quality that the rubric doesn't specify, introducing variance that harms the relative comparisons GRPO depends on. The SLM, constrained to check specific, well-defined criteria, produces more stable rankings even if its absolute judgments are less sophisticated.

This finding has implications beyond ATLAS. It suggests that the standard RL-for-agents pipeline—generate rollouts, ask GPT-4o for a score, update the policy—is suboptimal not just because it's expensive, but because the reward signal is inherently noisy in ways that hurt optimization. The field should invest in structured supervision (task-specific rubrics, multi-dimensional rewards, criterion-level scoring) and use cheaper, more consistent judges, rather than pursuing ever-larger judge models. This redirects research attention from judge model scaling to supervision structure design.

Finally, the paper provides a concrete diagnostic about what makes tool-use environments hard for SLMs. It is not that SLMs lack reasoning ability in general—the paper explicitly argues the limitation is "not reduced reasoning ability" (Section 1). It is that SLMs lack the metareasoning capacity to manage their own context and execution state when the action space is large. This diagnosis suggests that the research agenda for agentic SLMs should focus on resource management architectures—mechanisms for deciding what information to load, when to load it, and how to represent it compactly—rather than on improving raw reasoning through scale. It makes the case that efficient agentic behavior is a distinct capability from general reasoning, and that it can be acquired through targeted training even when scale is unavailable.

Follow-Up Research This Work Enables

Stress-test rubric-conditioned judging on non-MCP tasks with fuzzier success criteria. The paper's rubric approach assumes tasks decompose cleanly into observable, non-overlapping, functionally-aligned criteria. This holds for MCPBench's structured multi-requirement queries but may break down for more open-ended agentic tasks—customer support conversations, creative tool use, open-ended research assistance—where success is multidimensional and criteria interact. A strong follow-up would take ATLAS's rubric generation and scoring pipeline, apply it to a benchmark like WebArena or SWE-bench (where task success has ground-truth verification but process quality is multi-faceted), and measure whether rubric-based rewards outperform scalar rewards for training. The key measurement: does the rubric advantage in Table 1 (e.g., +0.55 TF for ITL with rubric vs. generic RFT for Qwen3-4B, Rows 18 vs. 16) replicate in settings where the rubric generator must define criteria for tasks with less clearly enumerable requirements? If the rubric advantage shrinks or reverses, it would establish a boundary condition: rubric-based supervision works when tasks have clean decomposition, but not when success criteria interact.

Apply ATLAS training to a frontier model and measure whether benefits are scale-dependent. The paper shows that ATLAS provides large gains for 4B and 7B models but never trains Kimi-K2 (1T) with the same pipeline. The Qwen3-30B result in Appendix C (achieving 4.44 TF with ATLAS, exceeding Kimi-K2's 4.38) hints that ATLAS benefits scale with model size, but the ablation is incomplete. A direct experiment would train Kimi-K2 (or another frontier model, e.g., Llama-3-70B or Mixtral-8x22B) on the same 304 training tasks with rubric-based RFT and PTC scaffolding, then measure TF on MCPBench and ATLAS-Test. If the frontier model's gain from ATLAS is proportionally similar to the SLM's gain (~70% relative improvement from cold-start to ATLAS-trained), it would validate ATLAS as a universal training methodology. If the gain is smaller, it would suggest ATLAS specifically addresses capacity limitations that frontier models overcome through scale alone, establishing a boundary condition. Either outcome is informative: the first strengthens the paper's scaling claims, the second clarifies where scale substitutes for structure and where it doesn't.

Develop and evaluate rubrics that evaluate code-based execution quality. The paper's headline SLM result (4.15 TF, Table 1 Row 19) uses PTC for execution but acknowledges that "we do not yet apply rubric-based rewards to PTC, due to the challenge of defining concrete rubrics over executable code" (Section 5.3). This is a clear gap. A follow-up would design rubric criteria specifically for code-based trajectories: correctness of function calls against the MCPServer API, proper error handling (does the code catch and recover from tool errors?), efficiency of control flow (does the code avoid redundant tool calls?), and state management (are intermediate results stored and reused correctly?). These rubrics would be generated alongside the existing TF/TA/TG/PA rubrics and used to score PTC trajectories during training. The key measurement: does PTC-specific rubric conditioning improve training outcomes over using task-level rubrics alone (the current Row 19 setup) or over generic rewards with PTC (Row 17, 3.91 TF)? This would close the paper's own identified gap and potentially push the SLM frontier-gap closure above 95%.

Measure whether rubric-based RL reduces reward hacking compared to scalar RL, using a held-out verifier. The paper's Figure 3 shows that PTC and ITL achieve lower training rewards but higher validation TF than ISL—a dissociation interpreted as reduced reward hacking. This is a post-hoc observation, not a controlled experiment. A direct test would train three variants of the same model on the same tasks: (a) scalar reward from GPT-4o (standard approach), (b) rubric-based reward from GPT-4o (same judge, structured output), and (c) rubric-based reward from Qwen3-30B (ATLAS approach). For each variant, measure the correlation between training reward and validation TF across training steps. If rubric-based rewards produce higher correlation (training reward better predicts held-out performance), it would directly support the paper's claim that structured rewards improve signal quality. If the SLM judge under rubrics produces the highest correlation, it would validate the specific ATLAS claim that SLM judging under rubrics is superior. The experiment would also assess whether reward hacking manifests as a divergence between training and validation curves—which the paper observes for ISL but not for ITL+PTC—and whether rubric conditioning systematically reduces this divergence.

Ablate the training data filtering pipeline to establish boundary conditions on ATLAS's effectiveness. The paper filters training tasks through three stages: quality filters (solvability > 9, utility > 8), Kimi-K2 solvability verification (TF > 5), and difficulty stratification. It never reports SLM performance on the filtered-out tasks, nor what fraction of generated tasks were discarded at each stage. A follow-up would take the discarded tasks—particularly those filtered at the solvability stage (Kimi-K2 TF ≤ 5)—and evaluate ATLAS-trained SLMs on them zero-shot. This would answer a critical open question: does ATLAS provide any benefit on tasks that are genuinely difficult for language models, or is its benefit strictly limited to tasks within the base model's capability envelope? If ATLAS-trained SLMs show zero improvement on filtered-out tasks (TF remaining at cold-start levels), it would establish that ATLAS amplifies existing capability but cannot create it—a boundary condition the paper currently cannot characterize because the filtered tasks are excluded from evaluation. If there is partial transfer, it would suggest ATLAS teaches generalizable context management skills that help even on out-of-capability problems. Either outcome refines the paper's contribution from "ATLAS closes the frontier gap" to "ATLAS closes the frontier gap on tasks where the SLM has some latent capability, which excludes X% of naturally occurring MCP tasks."

Test ATLAS on a non-Qwen model family to assess model-family dependence. All experiments use Qwen2.5-7B and Qwen3-4B, both from the same model family with shared pretraining and instruction-tuning recipes. The Qwen3-4B's "native tool-calling support" (Section 4.2) may give it an advantage in learning ITL and PTC behaviors that other model families—Llama, Gemma, Mistral—lack. A replication with, for example, Llama-3.1-8B-Instruct and Gemma-2-9B-IT would test whether ATLAS's gains are specific to Qwen's capabilities. The key measurement: does the ITL cold-start degradation (performance dropping from ISL to ITL) replicate across model families? Does the rubric-based RFT advantage over generic rewards hold? If ATLAS benefits transfer, it strengthens the paper's claim that the methodology addresses a general SLM limitation. If they don't—if Llama models, for instance, cannot learn to navigate restricted tool views even with rubric-based training—it would suggest that ATLAS depends on specific pretraining properties (code generation strength, instruction-following, tool-calling training) that not all SLMs possess, narrowing the paper's applicability claims.

Practical Applications and Downstream Use Cases

Cost-efficient on-premise agent deployments for enterprises with privacy constraints. The paper's core efficiency result—a 4B model with ATLAS achieving 4.15/10 TF versus a 1T frontier model at 4.38/10—directly enables deployment scenarios where sending data to cloud-hosted frontier models is impermissible due to privacy regulations (healthcare, legal, financial services) or prohibitive due to API costs at scale. An enterprise running 10,000 agentic queries per day on internal MCP-connected tools (e.g., querying internal databases, orchestrating CRM and ERP systems, generating compliance reports) could deploy a single 8×B200 GPU server running a Qwen3-4B model with ATLAS, process all queries in-house, and achieve task success rates approaching what a cloud frontier model would provide, while eliminating per-query API costs and data exfiltration risk. The paper's token efficiency data supports this: ATLAS-trained SLM trajectories consume ~13,400 tokens on average (Table 1 Row 19) versus ~23,768 for the frontier baseline (Row 1), meaning the on-premise deployment also reduces inference compute per query. The rubric generation cost (GPT-5, ~300 calls) and solvability filtering (Kimi-K2, ~1,000 task executions) are one-time upfront investments amortized over the deployment lifetime.

Latency-sensitive interactive agent applications. The paper doesn't directly measure latency, but the interaction turn counts in Table 1 are a proxy: ATLAS-trained Qwen3-4B with PTC completes tasks in roughly 18 turns on average (Rows 15, 19) versus 20+ for untrained variants and cold-start configurations. More importantly, PTC's code-based execution means that multiple tool calls within a single code block execute without model inference between them—the Python interpreter runs the code, making sequential tool calls and storing intermediate results, and the model only generates the initial program and any error-correction edits. For a task requiring 5 sequential tool calls with dependencies between them, JSON-based execution requires 5 model inference steps (one per tool call) plus reasoning turns; PTC requires 1–2 inference steps (program generation plus potential error correction). For real-time applications—voice assistants that need to execute multi-step tool workflows while the user waits, or automated trading systems where latency directly costs money—this reduction in inference steps could make the difference between a usable and unusable SLM-based agent, even if total FLOPs are comparable. The paper provides the architectural blueprint (Appendix A) for the Python scaffolding that makes this work.

Scalable self-improvement loops for tool-using agents. The paper's rubric-based RFT pipeline—generate rubrics once, score trajectories cheaply with an SLM, update the policy with GRPO—provides a practical recipe for continuous improvement of deployed agents without human annotation or expensive frontier model judging. An organization deploying an ATLAS-trained SLM could collect trajectories from production usage, periodically generate new rubrics for novel tasks using GPT-5 (or, as SLM capabilities improve, using the trained SLM itself with a rubric generation prompt), score those trajectories with the SLM judge, and run additional GRPO training steps to adapt the policy to distribution shift. Because the per-step judging cost is dominated by a 30B SLM rather than a frontier model, this loop is economically viable to run weekly or even daily. The paper's training stability result (Figure 3 showing monotonic improvement without degradation) suggests this loop would not collapse from reward hacking, though the paper's single-seed reporting limits confidence in this claim. The automated rubric generation pipeline (Appendix F.1) is a concrete template: send task specification + available tool context to a frontier model, receive structured criteria with weights, deploy to the SLM judge. This is an operationalized self-improvement workflow that the paper validates at ~300-task scale.

Tool ecosystem scaling without context window expansion. The paper's ITL mechanism bounds context growth sublinearly with the number of tools: rather than loading all tool schemas eagerly (context grows linearly with tool count), ITL loads only tool names initially and materializes full schemas on demand. For enterprises connecting agents to expanding MCP ecosystems—starting with 20 servers and growing to 200 as internal teams expose new APIs—ITL means the context cost of tool information does not increase proportionally. The paper quantifies this: under eager loading, Kimi-K2 consumes 23,768 tokens on average (Table 1 Row 1); under ITL, 18,290 tokens (Row 3), a ~23% reduction. For SLMs with 32K context windows, this reduction is the difference between having headroom for multi-turn reasoning and saturating the context with tool definitions before the task even begins. The practical implication is that organizations can standardize on MCP for internal tool connectivity without being forced to upgrade model scale or context length every time a new server is added—ATLAS provides a mechanism for scaling tool ecosystems independently of model capacity, decoupling infrastructure growth from model requirements.

When to Prefer This Method

The paper positions ATLAS specifically for the efficiency-constrained regime where context, computation, and supervision are scarce (Section 1). This framing articulates clear boundary conditions:

Prefer ATLAS (rubric-based RFT with iterative loading and programmatic orchestration) when:

• You are deploying an SLM (4B–30B parameters) rather than a frontier model (100B+). The paper demonstrates gains of 39–71% relative improvement over cold-start baselines for 4B and 7B models (Table 1, e.g., Qwen3-4B ITL: 2.36 → 4.03 with rubric RFT, Row 14 → Row 18), and the 30B result in Appendix C shows the methodology scales. If you have access to a 1T-parameter model with 80K+ context, the paper's evidence suggests you may not need ATLAS—the frontier model's zero-shot performance (4.38 TF, Row 1) exceeds the best ATLAS-trained SLM (4.15 TF, Row 19), though the Appendix C 30B result at 4.44 TF complicates this picture.

• Your task distribution includes multi-requirement, tool-mediated queries with 2–6+ distinct requirements spanning 1–3 MCP servers, where success depends on correct tool selection, parameter construction, and output grounding rather than on domain expertise or creative reasoning. The paper's gains are demonstrated on MCPBench-style tasks; there is no evidence for or against transfer to dialogue-based assistance, open-ended research, or embodied agent tasks.

• Your tool ecosystem is large enough that eager loading saturates context (dozens of servers, hundreds of tools). The paper shows token savings from ITL even for the frontier model (23,768 → 18,290 tokens, Table 1 Rows 1 → 3), and these savings are proportionally more important for models with smaller context windows.

• You can afford a one-time upfront investment in rubric generation (GPT-5 inference on ~300 tasks) and solvability filtering (frontier model execution on ~1,000 candidate tasks) to construct the training pipeline. The per-step training cost is dominated by the SLM judge, making ongoing training scalable.

• Your tasks have clearly enumerable success criteria that can be captured in observability, non-overlapping, functionally-aligned rubrics. The paper's rubric design principles assume tasks decompose cleanly; tasks with fuzzy, interacting, or subjective success criteria may not benefit from rubric-based supervision.

Prefer standard approaches (scalar reward RL with frontier judge, or deployment without training) when:

• You are using a frontier model (100B+ parameters) with a large context window (80K+ tokens). The paper does not test ATLAS on frontier models, and the frontier model's zero-shot performance already exceeds the best ATLAS-trained SLM on MCPBench (4.38 vs. 4.15). The cost of rubric generation, solver filtering, and RL training may not be justified if zero-shot performance is already adequate.

• Your task distribution includes genuinely hard problems that are outside the base SLM's capability envelope. The paper's training data is explicitly filtered to include only tasks where Kimi-K2 achieves TF > 5, and it provides no evidence that ATLAS helps on tasks filtered out at this stage. For task distributions skewed toward very hard problems (e.g., 6+ requirements requiring specialized domain knowledge), the paper offers no path to improvement.

• Your inference volume is extremely high (millions of queries per day) and the one-time rubric generation cost amortizes but still represents a significant engineering investment. The paper does not characterize the crossover point where ATLAS's training cost exceeds the savings from using a smaller model—this calculation is deployment-specific and untested.

• Latency requirements forbid multi-turn interaction entirely. Even with PTC's reduced turn counts, ATLAS trajectories average 18–20 interaction turns (Table 1). Some real-time applications cannot tolerate more than 1–2 model inferences per query, regardless of architecture.