Towards a Science of Scaling Agent Systems

🎯 Pitch

Adding more AI agents to a task can backfire spectacularly: multi-agent systems sometimes slash performance by 70% compared to a single agent, yet the paper shows when and why—finding that coordination helps on decomposable financial reasoning but cripples sequential planning, and that an agent’s errors can be amplified up to 17 times more under the wrong architecture versus just 4 times under a centralized one.

1. Executive Summary

This paper studies how the performance of language-model-based agent systems changes when they are scaled across coordination architecture, model capability, and task properties, using 260 controlled configurations spanning five canonical agent architectures (Single-Agent, Independent, Centralized, Decentralized, Hybrid), three LLM families (OpenAI, Google, Anthropic), and six agentic benchmarks (BrowseComp-Plus, Finance-Agent, PlanCraft, WorkBench, SWE-bench Verified, Terminal-Bench). The core contribution is a quantitative scaling principle—a mixed-effects regression model using empirically measured coordination metrics including efficiency (success per relative turn count), trace-level error amplification (how many times baseline errors propagate through agent interactions—ranging from 4.4× under centralized coordination to 17.2× under independent), message density, and redundancy—that achieves cross-validated R² = 0.373 across all six benchmarks (R² = 0.413 with a task-grounded capability metric), and correctly identifies the best-performing architecture for 87% of held-out configurations. Performance relative to single-agent baselines spans from +80.8% improvement on decomposable financial reasoning under centralized coordination to −70.0% degradation on sequential planning under independent coordination, establishing that multi-agent coordination yields diminishing returns once single-agent baselines exceed approximately 45% accuracy—a capability-saturation effect that persists as the most robust finding across cluster-robust inference (p = 0.004) and multiple-comparison correction (p_Holm = 0.018).

2. Context and Motivation

The Core Problem: We Don't Know When Multi-Agent Systems Actually Help

The fundamental question this paper tackles is deceptively straightforward: if you can deploy multiple language model agents instead of one, when does it actually improve performance? This matters because multi-agent systems (MAS) are being rapidly adopted across domains—code generation, web browsing, medical decision-making, financial analysis, and scientific discovery—under an implicit assumption that more agents, collaborating through structured communication, will outperform a single agent working alone. Yet this assumption rests on surprisingly fragile empirical foundations.

The paper identifies a critical gap that separates practice from principle. Despite the proliferation of multi-agent architectures—peer-to-peer debate, hierarchical orchestration, ensemble-style voting, hybrid combinations—there exists no quantitative framework for predicting when adding agents improves outcomes and when it instead introduces coordination costs that degrade them. As the authors state in Section 1:

"concurrent works question whether multi-agent coordination outperforms single-agent systems (SAS), leaving the conditions under which MAS provides genuine benefits remain underexplored."

This is not a minor caveat. It is a foundational uncertainty at the heart of an entire subfield. Without a principled understanding, practitioners rely on heuristics ("more agents must be better," "debate improves reasoning," "centralized coordination reduces errors") that may be correct in some settings and catastrophic in others. The paper's central claim is that these heuristics are not merely imprecise—they are systematically misleading when applied across task types without accounting for difficulty, decomposability, and tool complexity.

Why This Problem Matters: Beyond Academic Curiosity

The gap is consequential for several practical and theoretical reasons that the paper develops across Sections 1-2:

Deployment economics are directly affected. Multi-agent systems consume substantially more tokens than single-agent baselines. The paper reports that coordination overhead ranges from 58% (Independent architecture) to 515% (Hybrid architecture) relative to the single-agent token budget (Table 5). If this additional compute doesn't yield proportional performance gains—and the paper shows it often doesn't—then organizations are burning inference budget on coordination overhead rather than reasoning. For production systems handling millions of queries, the difference between deploying a single strong model versus a multi-agent ensemble with 5× the latency and cost (as Anthropic [44] report: 15× token consumption) represents a massive resource allocation decision made without empirical guidance.

The capability landscape is shifting under our feet. As frontier models gain extended context windows, sophisticated tool use, and improved self-reflection capabilities, the value proposition of multi-agent systems becomes increasingly unclear. Section 1 articulates this tension precisely:

"As base LLMs gain extended context windows, sophisticated tool use, and improved self-reflection, the unique value proposition of multi-agent collaboration becomes unclear. The answer likely depends on task characteristics and architectural choices that remain to be systematically quantified."

A single GPT-5 or Gemini-2.5-Pro might now outperform a team of weaker agents on tasks that previously required collaboration. But a team of strong agents might still outperform any individual. The paper provides the first systematic characterization of where the crossover point lies (the 45% single-agent baseline threshold) and which architectures perform best in which regimes.

The distinction between agentic and non-agentic tasks matters. Section 2 introduces a crucial taxonomy: agentic tasks require (i) sustained multi-step environmental interactions, (ii) iterative information gathering under partial observability, and (iii) adaptive strategy refinement from feedback. Non-agentic tasks—GSM8K, MMLU, HumanEval—evaluate single-shot reasoning. The literature has a known bias: on non-agentic benchmarks, multi-agent systems show monotonic improvement through ensemble effects (e.g., 89% on HumanEval with five agents) because voting corrects independent errors without sequential compounding. But the paper observes that:

"tasks showing MAS advantages in single-shot settings often exhibit opposite patterns under genuine interaction."

This means the existing multi-agent evaluation literature—conducted predominantly on non-agentic benchmarks—may be providing systematically misleading guidance about when collaboration provides value in real-world deployments, where sustained environmental interaction is the norm.

Conflicting Prior Evidence and Where Existing Approaches Fall Short

The paper is motivated by a genuine tension in the literature that prior work has not resolved. This tension operates at multiple levels.

The "More Agents Is All You Need" vs. "Single Agents Are Enough" Debates

On one side, a body of work suggests that scaling the number of agents produces consistent benefits. Li et al. [41] demonstrate performance improvements with increased team size on reasoning benchmarks. Qian et al. [42] propose scaling laws for multi-agent collaboration, though the paper notes they found "no significant universal pattern." Du et al. [27] show that multi-agent debate improves factuality and reasoning—but with the important caveat (noted by Smit et al. [58]) that debate does not reliably outperform single-agent strategies like self-consistency.

On the other side, a growing body of work challenges the assumption that MAS provides genuine value over simpler alternatives. Huang et al. [3] and Neubig [17] argue that single-agent systems with strong tool use often match or exceed multi-agent configurations. Gao et al. [16] demonstrate that benefits diminish as base models improve, with frontier models often outperforming teams of weaker agents. Cemri et al. [15] catalogue 14 distinct failure modes in multi-agent systems (Cohen's Kappa = 0.88), providing the failure taxonomy (MAST) that this paper builds upon in Section 4.4. Cognition AI [18] takes the strongest position: "Don't build multi-agents."

These conflicting findings are not fundamentally contradictory—they reflect different methods being applied to different task types under different evaluation protocols—but the field lacked a framework for reconciling them. This paper's central insight is that the effectiveness of multi-agent coordination is governed by quantifiable trade-offs between architectural properties and task characteristics, and that ignoring this heterogeneity makes the question "do multi-agent systems work?" unanswerable without specifying the task, architecture, and capability regime.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior work along several axes:

Confounded evaluations prevent causal attribution. Section 1 argues that "existing MAS evaluations compare architectures using different prompts, tools, or computational budgets, conflating architectural effects with implementation choices and precluding clean causal attribution." If one paper's decentralized system uses better prompts or more tool budget than another's centralized system, performance differences tell us nothing about the architectures themselves. This is not a minor methodological concern—it is a barrier to scientific progress. You cannot derive scaling principles from confounded comparisons.

Evaluations focus on outcomes, not process. Beyond final accuracy, the paper notes that prior work largely ignores "process dynamics such as coordination overhead, error propagation, and information flow that determine whether collaboration succeeds or fails." This is analogous to evaluating a drug without measuring the mechanism of action: you might observe that it works (or doesn't), but you cannot predict when it will work or why it might fail in new settings. The paper's empirical coordination metrics—efficiency, trace-level error amplification, message density, redundancy—are designed to fill precisely this gap.

No systematic exploration of the architecture-task interaction. While prior work has studied individual architectures (debate [27], hierarchical orchestration [28], ensemble methods) in specific domains, no prior study has systematically varied both architecture type and task domain to characterize their interaction. The paper's design explicitly tests whether "coordination benefits arise from matching communication topology to task structure, not from scaling the number of agents" (Section 2). This framing shifts the research question from "which architecture is best?" to "under what conditions does each architecture excel?"

Scaling laws are absent or incomplete for agentic systems. While neural scaling laws for pretraining are well-established (Kaplan et al. [54]), the paper notes that "collaborative scaling exhibits logistic growth patterns emerging at substantially smaller scales" with "no significant universal pattern" (citing Qian et al. [42]). This means the standard toolkit of scaling laws—power-law relationships between compute and performance—doesn't directly transfer to multi-agent systems, which have coordination dynamics that are fundamentally different from parameter scaling. The paper's contribution is not to propose universal scaling laws but to characterize the conditions under which scaling produces benefits versus costs.

No quantitative architecture selection guidance exists. Section 1 frames this as the practical gap: "practitioners rely on heuristics, hindering both the emergence of a science of agent systems and, critically for real-world deployment, the ability to determine when multi-agent coordination provides genuine value over simpler single-agent alternatives." The paper's regression model (Equation 1) and the derived 45% baseline threshold directly address this gap by providing quantitative rules: if your single-agent baseline exceeds ~45% accuracy, coordination is unlikely to help; if it's below, centralized or decentralized architectures provide the best returns depending on task structure.

Theoretical Framing: Information Fragmentation and the Coordination Tax

Section 1 provides a theoretical framework that explains why agentic tasks exhibit fundamentally different multi-agent scaling behavior than non-agentic tasks. The core mechanism is information fragmentation:

"Single-agent systems maximize context integration by maintaining a unified memory stream in which all reasoning steps share full access to prior history, enabling effectively constant-time access to global context. In contrast, multi-agent systems impose intrinsic information fragmentation: while parallel agents enable diverse exploration, they incur an unavoidable coordination tax in which the global context must be compressed into inter-agent messages."

This framing draws on Malone and Crowston's [29] concept of coordination costs and connects to the broader organizational behavior literature on team effectiveness (McGrath [30], Lencioni [31]). The insight is that multi-agent systems face a fundamental trade-off that single-agent systems avoid: diversity of perspective vs. completeness of context. Parallel agents can explore different solution strategies simultaneously, but each agent operates with only partial information about what others have discovered, and synchronizing that information requires costly communication.

On agentic tasks, this trade-off is particularly sharp because "coordination overhead scales with interaction depth, agents operate on progressively divergent world states, and errors cascade through execution chains rather than being corrected through voting." The paper reports that after 10 interactions, agent world states diverge to only 34% overlap (Section 2), meaning agents are sampling from increasingly different "realities." This explains why voting-based error correction—effective on static benchmarks—fails in agentic settings: there is no single ground-truth output to vote on, only a sequence of interdependent decisions under partial information.

How This Paper Positions Itself

The paper positions itself not as proposing a new multi-agent architecture, but as providing the first systematic empirical characterization of the conditions under which coordination provides value. This is framed as a scientific contribution rather than an engineering contribution:

"We introduce quantitative scaling principles for agent systems as a predictive model, capturing how performance varies with coordination, model capability, and measurable system and task factors."

Several aspects of this positioning are distinctive:

Controlled experimentation as scientific method. The paper explicitly controls for implementation confounds that plague prior comparisons—identical prompts, tools, and computational budgets across all architectures—to isolate architectural effects. This mirrors the controlled experiment methodology of the natural sciences and contrasts with the benchmark-driven approach common in ML, where confounded comparisons are the norm.

Unified analytical framework rather than architectural advocacy. The paper's five architectures (SAS, Independent, Decentralized, Centralized, Hybrid) are not presented as competitors to be ranked but as a "structural ablation of coordination mechanisms" (Section 3.1), where Independent isolates parallelization, Decentralized introduces peer communication without hierarchy, Centralized introduces hierarchical verification, and Hybrid combines both. This design enables attribution of effects to specific coordination mechanisms rather than generic "multi-agent" labels.

Predictive model as scientific output. Rather than simply reporting which architecture performed best on which benchmark, the paper derives a regression model (Equation 1) that predicts performance from measurable properties. The cross-validated R² = 0.373 (R² = 0.413 with ACI) represents a specific, falsifiable claim about how agent-system performance emerges from task properties, model capability, and coordination structure. The paper acknowledges the limits of this model—it works for relative architecture selection (87% accuracy on held-out configurations) but struggles with absolute cross-domain prediction—which is the appropriate level of epistemic modesty for a first quantitative framework.

Connection to broader scaling paradigm. The paper explicitly connects to the scaling laws tradition (Kaplan et al. [54]) while distinguishing agentic scaling from neural scaling: "collaborative scaling exhibits logistic growth patterns" with super-linear turn-count scaling (exponent 1.724, compared to neural scaling exponents of ~0.76). This positioning suggests that agent systems represent a new scaling regime requiring new analytical tools, not a straightforward extension of existing scaling laws.

Summary of the Gap and the Position

The paper addresses a fundamental gap: the absence of quantitative, empirically grounded principles for predicting when multi-agent coordination improves performance, when it degrades it, and by how much. This gap matters because multi-agent systems are being deployed with significant computational cost based on heuristics ("more agents is all you need") that are contradicted by a body of conflicting evidence. Prior work has evaluated architectures in confounded comparisons, without controlling for implementation differences, and without measuring the coordination dynamics that determine success or failure.

The paper's position is that architecture-task alignment, not team size, determines collaborative success, and that this alignment can be characterized through measurable coordination metrics (efficiency, error amplification, message density, redundancy) that explain performance variance across 260 configurations. The 45% single-agent baseline threshold, the tool-coordination trade-off, and the 4.4× vs. 17.2× error amplification difference between centralized and independent architectures are not merely interesting observations—they are the first quantitative principles for designing agent systems, analogous to (but distinct from) the pretraining scaling laws that transformed how we allocate training compute.

3. Technical Approach

3.1 Reader Orientation

The authors are building a predictive framework—not a new agent architecture, but a quantitative model that takes measurable properties of a task, a language model, and a coordination structure as inputs and outputs an expected performance score. The problem it solves is that practitioners currently choose between single-agent and multi-agent systems based on heuristics ("more agents is all you need," "centralized coordination reduces errors") with no principled way to predict when those heuristics hold. The shape of the solution is a regression equation whose coefficients are estimated from 260 controlled experiments, where each experiment varies exactly one dimension (model capability, architecture, benchmark) while holding prompts, tools, and compute budgets identical to isolate the causal effect of coordination structure on performance.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components:

Agent System Configurations (the independent variables) — Five canonical architectures (Single-Agent, Independent MAS, Centralized MAS, Decentralized MAS, Hybrid MAS) instantiated across nine models from three LLM families. Each architecture specifies a communication topology (who talks to whom), an orchestration policy (how outputs are aggregated), and a computational budget (matched total reasoning tokens across all configurations).
Six Agentic Benchmarks (the task environment) — BrowseComp-Plus (web browsing), Finance-Agent (financial reasoning), PlanCraft (Minecraft planning), WorkBench (business tool use), SWE-bench Verified (software engineering), and Terminal-Bench (CLI tasks). Each benchmark requires sustained multi-step environmental interaction, partial observability, and adaptive strategy refinement.
Empirical Coordination Metrics (the measured intermediates) — Six quantities extracted from execution traces: coordination overhead ( $O\%$ ), message density ( $c$ ), redundancy rate ( $R$ ), coordination efficiency ( $E_c$ ), trace-level error amplification ( $A_{e}^{\text{trace}}$ ), and success rate ( $S$ ). These capture how coordination succeeds or fails, not just whether it does.
Scaling Principle Regression Model (the predictive output) — A mixed-effects regression (Equation 1) with 20 parameters that maps model capability, task properties, architecture type, and coordination metrics onto expected task accuracy. The model is fit on 260 configurations and achieves cross-validated $R^2 = 0.373$ .

Information flows as follows: a model (with Intelligence Index $I$ ) is deployed on a benchmark (with tool count $T$ and single-agent baseline $P_{\text{SA}}$ ) under a specific architecture → the system runs and produces execution traces → empirical coordination metrics are extracted from those traces → the regression model predicts performance from those metrics plus the original inputs → the predicted performance can be compared across architectures to select the optimal one for a given task.

3.3 Roadmap for the Deep Dive

First, the agent system formalism (Section 3.1)—the mathematical definitions of single-agent and multi-agent systems, their components, and the four multi-agent architectures. This is the vocabulary needed to understand everything that follows.
Second, the agentic task definition and benchmark selection (Section 3.2)—what makes a task "agentic" vs. non-agentic, why this distinction matters for multi-agent evaluation, and the six benchmarks used. This explains what is being measured and why those choices prevent confounded comparisons.
Third, the experimental design principles (Section 4.1)—the controlled evaluation framework, including the five architectures, the Intelligence Index capability metric, the matched-compute budget strategy, and the configuration matrix ( $N = 260$ experiments). This is the "how" of the controlled experiments.
Fourth, the empirical coordination metrics (Section 4.4)—the six quantities extracted from execution traces (overhead, message density, redundancy, efficiency, error amplification, turn count). These are the mechanistic lenses through which the paper interprets why architectures succeed or fail.
Fifth, the scaling principle regression model (Section 4.3)—the full predictive equation (Equation 1), including which terms are included and why, the mechanistic hypotheses behind each interaction, and the cross-validation procedure. This is the quantitative synthesis of everything the experiments measure.
Sixth, the robustness checks (Section 4.5)—cluster-robust standard errors, multiple-comparison correction, capability metric sensitivity, and out-of-sample validation. These address threats to statistical validity and establish which findings are most reliable.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical measurement paper whose core idea is that the performance difference between single-agent and multi-agent systems on agentic tasks is governed by quantifiable coordination properties—and that these properties can be predicted from task structure, model capability, and architecture type using a regression model with no dataset-specific parameters.

Agent System Formalism: What Constitutes an Agent System

The paper grounds its analysis in a formal definition of agent systems (Section 3.1) that draws on the multi-agent system survey literature [14, 19]. This formalism serves two purposes: it provides precise vocabulary for describing the five architectures, and it enables systematic control of what varies between experiments.

Definition. An agent system is a 4-tuple:

$S = (A, E, C, \Omega)$

where $A = \{a_1, \ldots, a_n\}$ is the set of agents (with $n \geq 1$ ), $E$ is the shared environment, $C$ is the communication topology defining which agents can send messages to which other agents, and $\Omega$ is the orchestration policy determining how agent outputs are aggregated and when the system terminates.

What it represents: $S$ is the complete specification of how a set of LLM instances will coordinate to solve a task. The agent set $A$ determines how many independent reasoning loci exist; the environment $E$ provides the tool APIs and observation feedback; the communication topology $C$ constrains information flow; and the orchestration policy $\Omega$ defines the decision-making procedure that converts multiple agent outputs into a single final answer.

Why this decomposition: separating communication topology ( $C$ ) from orchestration policy ( $\Omega$ ) is critical because they represent distinct coordination dimensions. An Independent architecture and a Centralized architecture both use a star topology, but they differ in orchestration: Independent merely concatenates outputs (aggregation without verification), while Centralized routes through an orchestrator that cross-checks and synthesizes. This decomposition enables the paper's structural ablation (Section 3.1): Independent isolates the effect of parallelization without communication, Decentralized isolates the effect of peer communication without hierarchy, Centralized isolates the effect of hierarchical verification, and Hybrid combines both mechanisms.

Individual agent definition. Each agent $a_i$ is further decomposed:

$a_i = (\Phi_i, \mathcal{A}_i, M_i, \pi_i)$

where $\Phi_i$ is the reasoning policy (the LLM itself), $\mathcal{A}_i = \{\text{ToolCall}(t, \theta) : t \in \mathcal{T}, \theta \in \Theta_t\}$ is the action space consisting of tool invocations parameterized by available tools $\mathcal{T}$ and valid parameter configurations $\Theta_t$ , $M_i$ is the agent's internal memory, and $\pi_i : \mathcal{H} \rightarrow \mathcal{A}_i$ is the decision function that maps observation histories to actions.

What $\pi_i$ computes: given the agent's current observation history $h_{i,t}$ (a sequence of previous action-observation pairs), the LLM $\Phi_i$ generates a reasoning trace and selects the next tool call $\alpha_{i, t+1}$ . This is the core agent loop:

undefined