Why Do Multi-Agent LLM Systems Fail?

ArXiv: 2503.13657

🎯 Pitch

Multi-agent LLM systems are failing at staggering rates—between 41% and 86.7% across state-of-the-art frameworks—and the root cause is not just weak models but fundamentally flawed system designs that lead to agents ignoring each other, prematurely ending tasks, or repeating steps in endless loops. This paper introduces the first comprehensive taxonomy of 14 distinct MAS failure modes and a dataset of 1,600+ annotated traces, revealing that even with identical LLMs, well-designed systems can slash specification and coordination failures by up to 68%.

---

1. Executive Summary

This paper systematically investigates why multi-agent LLM systems (MAS) fail, introducing MAST-Data, a dataset of over 1,600 annotated execution traces collected across 7 popular MAS frameworks on coding, math, and general agent benchmarks, and MAST (Multi-Agent System Failure Taxonomy), a structured framework identifying 14 distinct failure modes clustered into 3 categories—system design issues (e.g., step repetition, disobeying task specifications), inter-agent misalignment (e.g., reasoning-action mismatches, information withholding), and task verification (e.g., premature termination, incorrect verification). The paper finds that 41% to 86.7% of MAS executions fail across state-of-the-art frameworks, and that failures are not merely artifacts of underlying LLM limitations but often stem from system design deficiencies—establishing that simple interventions like prompt improvements and topology redesigns yield only modest gains (up to +15.6% success rate increase through verification step additions), while deeper structural redesigns are required for robust reliability. The taxonomy enables a compute-optimal failure diagnosis framework, showing that well-designed MAS can outperform poorly-designed ones by 60–68% in reducing specification and coordination failures even when using the same underlying model, and that verification remains the most pervasive unresolved challenge, appearing even in successful traces.

2. Context and Motivation

The Core Problem: We Don't Understand Why Multi-Agent LLM Systems Fail

Multi-agent LLM systems (MAS) have become one of the most enthusiastically pursued directions in agentic AI research. The premise is compelling: rather than relying on a single monolithic LLM, decompose complex tasks across specialized agents that collaborate through structured communication — one agent designs, another codes, a third reviews, a fourth tests, with a manager orchestrating the workflow. This mirrors how human organizations tackle complex problems, and in principle, it should offer several advantages: task decomposition into manageable subtasks, parallelization of independent work streams, context isolation so each agent works within a focused scope, specialized model ensembling where different agents can use different LLMs optimized for their roles, and diverse reasoning through multi-perspective discussion.

The paper's opening empirical finding is stark: 41% to 86.7% of MAS executions fail across 7 popular open-source frameworks (ChatDev, MetaGPT, HyperAgent, AppWorld, AG2, Magentic-One, OpenManus) on their respective benchmarks (Figure 5 in Appendix B). These are not obscure research prototypes — they are widely-cited, actively-maintained frameworks that represent the state of the art in MAS design. Yet their failure rates are high enough that, as the authors note, "their performance gains often remain minimal compared to single-agent frameworks or simple baselines like best-of-N sampling" (Section 1).

This gap between enthusiasm and reliability motivates the paper's central question: Why do MAS fail?

The question is deceptively simple but practically urgent. Without a systematic understanding of failure mechanisms, MAS development proceeds through trial and error — tweak a prompt here, restructure agent roles there, run the benchmark again, see if the number goes up. This is not engineering; it is alchemy. The field lacks the diagnostic vocabulary, the empirical grounding, and the analytical framework to ask even basic questions like: "Is this failure due to bad system design or a bad underlying model?" or "Did this intervention actually fix the root cause, or just mask a symptom?"

The paper argues that addressing this gap requires three things that did not previously exist: (1) a comprehensive dataset of annotated MAS failures across diverse systems and tasks, (2) a standardized taxonomy defining failure modes with enough precision to enable consistent annotation and cross-system comparison, and (3) a scalable annotation methodology that makes collecting and analyzing failure data practical at scale. The paper's contributions — MAST-Data, MAST, and the LLM annotator pipeline — are designed to fill each of these gaps respectively.

Why This Problem Matters: Real-World Stakes

The paper's motivation is not purely academic. Several practical considerations make MAS reliability a pressing concern:

MAS adoption is accelerating without reliability guarantees. Frameworks like ChatDev, MetaGPT, and AutoGen (the predecessor to AG2) have accumulated thousands of GitHub stars and are being integrated into production software engineering pipelines. The paper notes that systems like MetaGPT simulate "the SOPs of different roles in Software Companies to create open-ended software applications" (Appendix B.1, Table 3), while ChatDev simulates "different Software Engineering phases like design, code, QA through simulated roles in a software engineering company." These are not toy demonstrations — they are being positioned as practical tools for real software development. If they fail 41–87% of the time, the cost of those failures (wasted compute, incorrect outputs, developer time spent debugging agent interactions rather than writing code directly) can easily erase any productivity gains.

MAS complexity makes debugging uniquely difficult. Unlike single-agent systems where an incorrect output can often be traced to a specific model failure (hallucination, poor reasoning, instruction misunderstanding), MAS failures involve convoluted multi-agent dynamics. Section 3 articulates this challenge precisely: "failures in MAS are frequently complex. They involve convoluted agent interactions and the compounding effects of individual model behaviors and overall system design." A bug might manifest as Agent A producing wrong code, but the root cause could be that Agent B withheld critical API documentation (Failure Mode 2.4: Information Withholding), or that Agent C prematurely terminated the conversation before verification was complete (Failure Mode 3.1: Premature Termination), or that the system's workflow allowed Agent A to proceed without the CEO's approval (Failure Mode 1.2: Disobey Role Specification). Without a taxonomy, developers are left reading through thousands of lines of agent conversation logs with no structured way to diagnose what went wrong.

The choice of MAS architecture is currently a black art. The paper's comparative analysis (Section 5.1, Appendix F) reveals that different MAS architectures have radically different failure profiles even when using the same underlying LLM on the same task. MetaGPT reduces FC1 (System Design Issues) failures by 39% compared to ChatGPT when both use GPT-4o, but simultaneously exhibits 1.56× more FC3 (Task Verification) failures. This is non-obvious and non-trivial tradeoff information that no developer could infer from aggregate success rates alone. Without the kind of fine-grained failure analysis that MAST enables, architecture choices are made based on intuition or convenience rather than evidence about which design patterns are robust against which failure modes.

Verification is the silent killer. One of the paper's most striking findings (Insight 3 in Section 4, FC3) is that verification-related failures (premature termination, incomplete verification, incorrect verification) appear even in successful traces (Appendix J.1, Table 7). In other words, MAS can complete tasks correctly while still exhibiting verification flaws — the system got lucky, but the verification process is unreliable. This is the multi-agent equivalent of a codebase that passes tests but has known race conditions: it works until it doesn't, and when it fails, it fails silently with no mechanism to catch the error. This finding has direct implications for anyone considering deploying MAS in safety-critical applications.

The problem space is growing faster than our understanding. The paper evaluates 7 frameworks, but the broader MAS ecosystem is expanding rapidly — the authors cite systems spanning software engineering, drug discovery, scientific simulations, and general-purpose agents (Section 1, references [5–11]). Each new framework introduces novel architectural patterns, communication protocols, and role specifications, creating an ever-expanding space of possible failure modes. Without a generalizable framework for understanding failures, each new system requires its own ad-hoc debugging effort, and lessons learned from one system do not transfer to others.

Where Prior Approaches Fall Short

The paper identifies several categories of prior work and explains why none of them adequately address the question "why do MAS fail?"

Top-down benchmark evaluations obscure failure dynamics. Numerous benchmarks exist for evaluating agentic systems — SWE-bench for software engineering, GAIA for general agent tasks, GSM-Plus for math reasoning — and these provide aggregate performance metrics (Section 2.1). The paper acknowledges their value but argues they offer only a "top-down perspective, focusing on aggregate performance or high-level objectives like trustworthiness and security." Knowing that MetaGPT scores 40% on ProgramDev tells you that it fails 60% of the time, but tells you nothing about why it fails. Is it generating syntactically correct but semantically wrong code? Is it failing to coordinate agent roles? Is it completing tasks but verifying incorrectly? Aggregate metrics collapse all failure modes into a single number, making them useless for targeted improvement.

Existing challenge surveys are high-level and not empirically grounded. Several surveys highlight challenges and risks in MAS at a broad level (Han et al., 2024; Hammond et al., 2025, cited as [25, 26]). The paper notes these "meaningfully contribute towards understanding specific issues or providing high-level overviews" but "do not offer a fine-grained, empirically grounded taxonomy of why MAS fail across diverse systems and tasks." A survey saying "coordination is hard in MAS" is not actionable; a taxonomy saying "Failure Mode 2.4: Information Withholding occurs when an agent possesses critical data and fails to share it, as illustrated by the Phone Agent not telling the Supervisor Agent that the API requires a phone-number-formatted username" (Figure 3) is directly actionable.

Single-agent design principles don't directly transfer. Works like Anthropic's "Building Effective Agents" blog post and Kapoor et al.'s "AI Agents That Matter" (2024) provide design guidance focused on single-agent systems — emphasizing modularity, simplicity, and avoiding over-engineering (Section 2.2). The paper acknowledges that these insights are valuable but argues they don't address the unique failure modes that emerge from multi-agent interaction. A single agent can't exhibit "Information Withholding" (FM-2.4) because there's no other agent to withhold information from. It can't experience "Conversation Reset" (FM-2.1) because there's no inter-agent conversation to reset. The failure modes in FC2 (Inter-Agent Misalignment) are fundamentally multi-agent phenomena that require their own taxonomy and diagnostic framework.

Prior failure analysis datasets focus on different problems. The paper explicitly distinguishes MAST-Data from several related datasets (Section 2.3):

• Bansal et al. (2024) catalog challenges in human-agent interaction, not autonomous agent-agent interaction. The failure modes when a human collaborates with an AI (misunderstandings, misaligned expectations, over-trust) are different from those when two AI agents collaborate (withholding information, ignoring input, conversation reset).
• MT-Bench-101 evaluates LLMs in multi-turn human-LLM dialogues, not multi-agent conversations. The evaluation criteria (coherence, engagement, safety) are designed for chatbot quality, not task-completion reliability.
• Code generation error taxonomies (Da et al., 2023) focus on single-model code output errors (syntax errors, logic errors, missing edge cases), not on failures arising from agent coordination breakdowns.

Attribution-focused tools don't capture failure patterns. The paper mentions two recent tools that come closest to its goals but still fall short:

• AGDebugger (Epperson et al., 2025) provides an interactive interface for inspecting and editing message histories in agent teams. This is a debugging tool for individual traces — it helps a developer figure out what happened in one specific failed run. It does not provide a taxonomy of failure modes or a systematic way to aggregate failure patterns across runs.

• Who&When (Zhang et al., 2025) attributes task failures to specific agents and error steps. This is useful for assigning blame ("Agent X caused this failure at step Y") but the paper argues it focuses on "summarizing failures for specific task items by attributing them to particular agents and error steps" rather than identifying recurring failure patterns. Knowing that Agent X failed at step Y doesn't tell you whether the underlying pattern is "Reasoning-Action Mismatch" (FM-2.6) that will recur across many tasks, or a one-off hallucination.

No unifying failure framework exists. This is the paper's central contention. Prior work has identified instances of MAS failures and proposed ad-hoc fixes, but there is no shared vocabulary, no standardized classification scheme, and no empirical foundation for understanding which failures are common, which are system-specific, and which are fundamental to the multi-agent paradigm. The result is a fragmented literature where researchers study failures in their own systems using their own terminology, making it impossible to compare findings, aggregate knowledge, or build on each other's work.

How This Paper Positions Itself

The paper's positioning is explicit and multi-layered:

First, it frames itself as a foundational empirical contribution. The abstract states: "We introduce MAST-Data, the first comprehensive dataset of 1600+ annotated traces collected across 7 popular MAS frameworks. MAST-Data is the first multi-agent system dataset to outline the failure dynamics in MAS for guiding the development of better future systems." The emphasis on "first" is deliberate — the paper sees itself as establishing the empirical groundwork for a new research subfield focused on MAS reliability, analogous to how early software bug databases enabled systematic research on software defects.

Second, it positions the taxonomy as a pragmatic engineering tool, not a theoretical claim. The authors explicitly state: "While MAST serves as a foundational first step towards unifying the understanding of MAS failures, we do not claim it covers every potential failure pattern" (Section 1). The taxonomy is derived empirically through Grounded Theory from 150 traces and validated through inter-annotator agreement (κ = 0.88), but it is presented as a working framework that should evolve with new systems and findings, not as a complete theoretical model of MAS failure.

Third, it stakes a strong position on the primacy of system design over model capability. This is perhaps the paper's most important intellectual commitment. Section 5.3 argues: "While one could simply attribute failures in MAST-Data to limitations of present-day LLM (e.g., hallucinations, misalignment), we conjecture that improvements in the base model capabilities will be insufficient to address the full MAST." The analogy to organizational theory is explicit: "even organizations of sophisticated individuals can fail catastrophically if the organization structure is flawed." The paper thus positions MAS failures as primarily systems engineering problems, not ML model problems — a position with significant implications for where research investment should be directed.

Fourth, it defines success in terms of enabling future work rather than solving the problem. The contributions are framed as infrastructure for the research community: the dataset (MAST-Data) provides empirical grounding, the taxonomy (MAST) provides a shared language, and the LLM annotator provides a scalable methodology. The paper's intervention studies (Appendix H) deliberately demonstrate that simple fixes are insufficient, arguing by example that the taxonomy reveals problems requiring "more than superficial fixes" and "structural MAS redesigns." This is a meta-contribution: the paper provides the diagnostic tools while arguing that the treatment requires community-wide effort.

Fifth, it connects to the broader "AI engineering" movement. The reference to specifications as "the missing link to making the development of LLM systems an engineering discipline" (Stoica et al., 2024, reference [36]) and the emphasis on rigorous methodology (Grounded Theory, inter-annotator agreement, cross-validation on unseen systems) position this work within a growing push to treat LLM system development as an engineering discipline with testable, falsifiable claims, rather than an artisanal craft of prompt tweaking and architecture guessing.

In summary, the paper addresses a gap that is simultaneously practical (MAS fail too often to be reliable), methodological (no shared framework exists for diagnosing those failures), and intellectual (we lack a theory of why multi-agent coordination breaks down in LLM-based systems). Its response is to build empirical infrastructure — data, taxonomy, tools — that enables the community to collectively address these challenges rather than proposing a single solution.

3. Technical Approach

3.1 Reader Orientation

What is being built: A systematic methodology for identifying, classifying, and analyzing why multi-agent LLM systems (MAS) fail during task execution, consisting of a failure taxonomy (MAST), a dataset of annotated failure traces (MAST-Data), and an automated annotation pipeline that labels new traces with taxonomy failure modes.

What problem this solves and the "shape" of the solution: When a MAS fails—producing incorrect code, getting stuck in loops, or terminating prematurely—developers currently have no structured way to diagnose what went wrong beyond reading through thousands of lines of agent conversation logs. The solution has three layers: first, empirically derive a standardized vocabulary of 14 failure modes through expert analysis of real MAS traces; second, use that vocabulary to annotate a large-scale dataset of failures across diverse systems, creating an empirical map of which failures occur where; third, build an LLM-based annotation pipeline that scales this diagnosis to new systems without requiring hundreds of expert hours per framework. The output is both a diagnostic instrument (MAST) that developers can apply to their own systems and a body of empirical evidence (MAST-Data) about which design patterns cause which failure modes.

3.2 Big-Picture Architecture (Diagram in Words)

The overall methodology has four major components arranged in a pipeline from empirical observation to automated diagnosis:

1. Trace Collection Engine — Gathers raw execution traces from 7 MAS frameworks (ChatDev, MetaGPT, HyperAgent, AppWorld, AG2, Magentic-One, OpenManus) across coding, math, and general agent benchmarks. Each trace is a complete log of all agent-to-agent conversations, tool calls, code outputs, and final task outcomes. This component produces unlabeled raw text logs plus ground-truth task success/failure labels determined through human evaluation of task completion.

2. Taxonomy Derivation Process (MAST) — Six expert human annotators apply Grounded Theory methodology to 150 traces from 5 frameworks, iteratively coding failure behaviors, refining definitions through inter-annotator agreement studies (3 rounds, κ = 0.88), and organizing the resulting 14 failure modes into 3 categories mapped to execution stages. This component produces MAST: a structured classification scheme with precise definitions, examples, and prevalence statistics for each failure mode.

3. LLM-as-Judge Annotation Pipeline — An automated system that takes the finalized MAST taxonomy (definitions + few-shot examples), prompts OpenAI's o1 model with a complete MAS execution trace, and outputs binary labels for each of the 14 failure modes plus textual justifications. This pipeline is calibrated against human expert annotations on a held-out set (accuracy 94%, κ = 0.77) and validated on two previously unseen MAS frameworks (κ = 0.79).

4. MAST-Data Construction — The LLM annotator is applied at scale to 1,642 traces spanning all 7 frameworks, multiple underlying LLMs (GPT-4o, GPT-4, Claude-3.7-Sonnet, Qwen2.5-Coder-32B-Instruct, CodeLlama-7b-Instruct-hf), and multiple benchmarks (ProgramDev, SWE-Bench Lite, AppWorld Test-C, GSM-Plus, OlympiadBench, MMLU, GAIA). The resulting dataset maps each trace to its failure mode profile, enabling cross-system comparative analysis.

Information flows sequentially: raw traces → expert GT analysis → MAST taxonomy → LLM annotator calibration → large-scale annotation → MAST-Data. The taxonomy and LLM annotator are then reusable: a developer with a new MAS can run the LLM annotator on their traces using the published MAST definitions without repeating the expert analysis phase.

3.3 Roadmap for the Deep Dive

• First, the Grounded Theory methodology for deriving MAST from raw traces: how 150 traces were selected, what open coding and constant comparative analysis involve, how theoretical saturation was determined, and why this qualitative approach was necessary given the absence of pre-existing failure definitions.

• Second, the inter-annotator agreement process that transformed preliminary failure observations into a standardized taxonomy: the iterative refinement procedure (3 rounds, 5 traces per round, 3 annotators), how disagreements were resolved, how Cohen's Kappa was computed and interpreted, and what changes were made to definitions between rounds.

• Third, the MAST taxonomy structure itself: the 14 failure modes organized into 3 categories, their mapping to execution stages (Pre-Execution, Execution, Post-Execution), the prevalence statistics from the annotated corpus, and the key insights derived from each category.

• Fourth, the LLM-as-Judge annotation pipeline: the few-shot prompt structure, the calibration process against human annotations, the evaluation metrics (accuracy, recall, precision, F1, Cohen's κ), and the validation on unseen MAS frameworks to establish generalizability.

• Fifth, the MAST-Data construction at scale: the trace collection configurations (which frameworks with which models on which benchmarks), the annotation protocol, and the resulting dataset statistics (1,642 traces, 210 human-annotated traces in MAST-Data-human, cost analysis per framework).

• Sixth, the intervention case studies that demonstrate MAST's practical utility: how failure mode breakdowns guided specific system modifications in AG2/MathChat and ChatDev, the before-and-after failure distribution analyses, and what these results imply about the sufficiency of tactical fixes.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology and dataset paper whose core idea is that systematically understanding MAS failures requires three interdependent artifacts: a taxonomy derived empirically from expert analysis of real traces, a large-scale annotated dataset that maps failures across diverse systems, and a scalable annotation mechanism that makes the taxonomy practically applicable.

---

Grounded Theory Analysis for Taxonomy Derivation

The paper's starting challenge is that no pre-existing framework defines what counts as a MAS failure, let alone categorizes different types of failures. Prior work either evaluated aggregate performance (benchmarks like SWE-bench, GAIA) or identified challenges at a high level without precise operational definitions. To build a taxonomy from scratch without imposing arbitrary categories, the authors adopt Grounded Theory (GT) , a qualitative research methodology from the social sciences that builds theoretical frameworks inductively from empirical data rather than testing pre-formulated hypotheses.

Trace selection for GT analysis. The initial corpus consists of 150 MAS execution traces collected from 5 frameworks: HyperAgent, AppWorld, AG2, ChatDev, and MetaGPT. These were chosen through theoretical sampling — a GT technique where data sources are selected to maximize diversity across system objectives and interaction patterns rather than randomly. The 5 frameworks span three architectural paradigms:

• Hierarchical workflow (ChatDev, HyperAgent): agents organized in manager-subordinate relationships with predefined phase transitions (design → coding → testing → review).
• Assembly line (MetaGPT): agents pass outputs sequentially following Standard Operating Procedures encoded in prompts.
• Star topology (AppWorld): a central supervisor agent communicates with specialized service-specific agents in one-on-one conversations.
• Flexible conversation (AG2): an agentic framework without fixed topology, supporting customizable conversation patterns.

The task domains cover programming (ChatDev, MetaGPT, HyperAgent, AppWorld) and math problem-solving (AG2 MathChat). By spanning different architectures and task types, the sampling ensures that observed failure patterns are not artifacts of a single system design.

The four GT techniques applied. The six expert human annotators applied GT through an iterative cycle of four operations:

1. Open coding: Each annotator reads through complete execution traces — averaging over 15,000 lines of text each — and labels specific segments with codes describing the failure behavior observed. For example, an annotator reading a ChatDev trace where the CPO agent terminates discussion without the CEO's approval might apply an initial code like "agent_oversteps_role" or "premature_conversation_end." These codes are descriptive labels attached to concrete trace evidence.

2. Constant comparative analysis: As annotators code more traces, they continuously compare new observations against previously coded instances. When AppWorld exhibits a pattern where the Spotify Agent fails to tell the Supervisor Agent that an API requires a phone-number-formatted username (trace segment illustrated in Figure 3), an annotator compares this against similar patterns in MetaGPT where an agent doesn't share API documentation — both involve an agent possessing critical information and not communicating it. Through comparison, the abstract pattern "agent possesses information that another agent needs but does not share it" emerges across concrete instances in different systems.

3. Memoing: Annotators document their evolving understanding of failure patterns as they analyze traces. A memo might note: "Failure pattern X appears in both ChatDev and HyperAgent but manifests differently — in ChatDev it's role-based (CEO/CPO dynamic), while in HyperAgent it's related to the Navigator/Planner message format. The underlying pattern is about information flow obstruction, but the mechanism differs by architecture."

4. Theorizing: The emerging patterns are organized into a structured taxonomy with categories, subcategories, and definitions. This is not a single pass but an iterative refinement: initial patterns are proposed, tested against more traces, refined or split when counterexamples are found, and consolidated when distinct codes are discovered to capture the same underlying phenomenon.

Theoretical saturation. The GT process continues until theoretical saturation is reached: the point where analyzing additional traces yields no new failure mode insights — new traces only produce instances of already-identified patterns. The paper does not specify exactly how many traces were analyzed before saturation, but the process required "over 20 hours of annotation per expert for these 150 traces" (Section 3.1), indicating that all 150 were coded through multiple iterations. The substantial expert effort (6 experts × 20+ hours each = 120+ person-hours minimum) underscores the methodological rigor: this is not a quick labeling exercise but a careful qualitative analysis.

Why GT rather than a top-down theoretical framework? The paper does not explicitly justify GT, but the choice follows from the problem structure. If the authors had predefined failure categories (e.g., "communication failure," "reasoning failure," "tool misuse"), they would risk imposing categories that don't match how MAS actually fail. GT lets the failure modes emerge from the data, ensuring the taxonomy reflects empirical reality rather than the authors' preconceptions. This is particularly important because MAS failures often involve emergent behaviors that are not obvious from individual system components — the whole point of the paper is that failure dynamics are not predictable from knowledge of the underlying LLM or the system architecture in isolation.

---

Inter-Annotator Agreement and Taxonomy Standardization

The initial GT analysis produces a preliminary set of failure mode definitions, but these are the output of individual expert reasoning and may not be consistently applicable by different annotators. To transform these into a standardized taxonomy suitable for creating consistent labels across MAST-Data's 1,642 traces, the paper conducts a rigorous Inter-Annotator Agreement (IAA) process that iteratively refines definitions until independent annotators reach high consensus.

The IAA iterative refinement procedure. The process operates in rounds, with each round following a fixed protocol:

1. Three expert annotators independently label the same set of 5 randomly selected traces from the initial 150-trace corpus, using the current version of the taxonomy.
2. For each trace, they mark which failure modes are present according to their understanding of the current definitions.
3. The annotators then convene to discuss disagreements — not just to decide which label is "correct," but to understand why the definitions led different annotators to different conclusions about the same trace.
4. Based on these discussions, the authors refine the taxonomy: adjusting failure mode definitions to remove ambiguity, adding new modes if distinct patterns were being conflated, removing modes that proved indistinguishable in practice, or merging modes that capture the same underlying phenomenon.
5. The refined taxonomy becomes the instrument for the next round.

The paper conducted three such rounds, totaling "about 10 hours solely for resolving disagreements, not including the annotation time itself" (Section 3.2). The three-round design allows for progressive refinement: Round 1 typically reveals major ambiguities in definitions, Round 2 tests whether refinements resolved the ambiguities, and Round 3 confirms that the final definitions are consistently applicable.

Quantifying agreement: Cohen's Kappa. Inter-annotator agreement is measured using Cohen's Kappa coefficient, defined as:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ is the observed proportion of agreement among annotators (how often they assign the same labels to the same trace) and $p_e$ is the expected proportion of agreement by random chance (what agreement would look like if annotators were guessing based on the overall label distribution).

What it computes: Kappa adjusts raw agreement by subtracting the agreement that would occur by chance. If two annotators agree 90% of the time but the labels are imbalanced (e.g., almost all traces get "no failure" for most modes), the chance-corrected agreement might be much lower. The denominator normalizes so that $\kappa = 1$ means perfect agreement, $\kappa = 0$ means agreement no better than chance, and negative values mean agreement worse than chance (systematic disagreement).

Why this form: Simple percentage agreement is misleading when some categories are rare. If a failure mode occurs in only 2% of traces, two annotators who always say "no" will agree 98% of the time without any real diagnostic skill. Kappa penalizes this by estimating how often chance agreement would occur given the observed marginal distributions and subtracting it. The benchmark interpretation (κ > 0.8 = "strong agreement," κ > 0.6 = "substantial agreement") is well-established in the annotation literature.

Achieved agreement levels. The paper reports "a strong average of κ = 0.88 in the final rounds" of the IAA process on the initial MAS frameworks (Section 3.2). This is a high value — it means the refined MAST definitions enable independent annotators to diagnose failure modes with very high consistency. For context, κ > 0.8 is typically considered the threshold for "almost perfect agreement" in fields like medical diagnosis and legal document classification.

Additionally, after the taxonomy was finalized, the authors conducted a generalization validation round on two previously unseen MAS frameworks (OpenManus and Magentic-One) with new benchmarks (MMLU and GAIA). This additional round achieved κ = 0.79, which falls in the "substantial agreement" range (0.61–0.80). The slight drop from 0.88 to 0.79 is expected when testing on out-of-domain systems — the taxonomy was refined on programming and math tasks, and GAIA introduces general-agent tasks with different interaction patterns. The fact that agreement remains at 0.79 without further definition refinement demonstrates MAST's generalizability.

What changed between preliminary and final MAST. The paper does not enumerate every definitional change made during IAA, but the iterative nature of the process means that ambiguities discovered in one round were resolved before the next. For example, if Round 1 revealed that annotators were conflating "Information Withholding" (FM-2.4: agent has information but doesn't share it) with "Ignored Other Agent's Input" (FM-2.5: agent receives information but doesn't act on it), the Round 2 definitions would add explicit distinctions — perhaps clarifying that FM-2.4 requires evidence the withholding agent possesses the information, while FM-2.5 requires evidence the agent acknowledged receipt of the information before disregarding it. These refinements are what drive κ from initial lower values to the final 0.88.

Why 5 traces per round? The paper doesn't explicitly justify the sample size, but 5 traces × 3 rounds = 15 traces total used for IAA, which is sufficient for iterative definition refinement in qualitative annotation studies. The goal is not statistical power but qualitative depth: each trace averages over 15,000 lines of text, and the discussion of disagreements is the primary mechanism for improving definitions, not the quantitative agreement score itself. The κ = 0.88 on the final round serves as a validation checkpoint confirming the refinements worked.

---

The MAST Taxonomy Structure and Failure Mode Definitions

The output of the GT and IAA processes is MAST, the Multi-Agent System Failure Taxonomy, illustrated in Figure 1. The taxonomy identifies 14 fine-grained failure modes organized into 3 overarching categories, with each mode mapped to the MAS execution stage where its root cause typically emerges.

The three failure categories (FCs):

FC1: System Design Issues (44.2% of failures). These failures "originate from system design decisions, and poor or ambiguous prompt specifications" (Section 4, FC1). The key insight is that these are not purely model failures — they reflect flaws in how the MAS is architected, how agent roles are specified, how conversation state is managed, or how termination conditions are defined. The paper explicitly argues that "a well-designed MAS can result in performance gain when using the same underlying model," meaning these failures are addressable through engineering improvements without requiring better base LLMs.

The five modes under FC1 are:

• FM-1.1 (11.8%): Disobey Task Specification — Failure to adhere to the specified constraints or requirements of a given task. This can stem from flaws in MAS design regarding agent roles and workflow, poor user prompt specifications, or limitations of the underlying LLM. The paper provides an example: ChatDev is tasked to create a Wordle game with "randomly select a new 5-letter word each day" but still produces code with a fixed word list, even with an explicit prompt — suggesting the failure stems from the MAS's design for interpreting specifications, not just prompt ambiguity.

• FM-1.2 (1.5%): Disobey Role Specification — Failure to adhere to the defined responsibilities and constraints of an assigned role, potentially leading to an agent behaving like another. The case study (Appendix H.2) documents instances where the CPO agent in ChatDev terminates conversation without the CEO agent's consensus.

• FM-1.3 (15.7%): Step Repetition — Unnecessary reiteration of previously completed steps, potentially causing delays or errors. This is the most frequent single failure mode across all categories. The prevalence (15.7%) suggests that MAS often get stuck in loops where agents re-execute work that has already been completed — a coordination failure where the system lacks mechanisms to track what has been accomplished.

• FM-1.4 (2.8%): Loss of Conversation History — Unexpected context truncation, disregarding recent interaction history and reverting to an antecedent conversational state. This is a state management failure: the system's memory mechanism (e.g., context window management) discards information that agents still need.

• FM-1.5 (12.4%): Unaware of Termination Conditions — Lack of recognition or understanding of the criteria that should trigger the termination of agents' interaction, potentially leading to unnecessary continuation. The paper provides an example (Appendix N.2) where in AG2 MathChat, the mathproxyagent keeps asking the Assistant to "Continue. Please keep solving the problem" even after the Assistant correctly identifies that the problem is unsolvable with the given information.

FC2: Inter-Agent Misalignment (32.3% of failures). These failures "arise from a breakdown in critical information flow from inter-agent interaction and coordination during execution" (Section 4, FC2). The key insight is that these failures involve agents failing to maintain accurate models of what other agents know, need, or intend — what the paper calls "the collapse of 'theory of mind'" (Section 4, Insight 2). The paper argues that solutions like standardized communication protocols (Model Context Protocol, Agent-to-Agent) address message format but not message content — agents still fail to communicate the right information at the right time because they lack the social reasoning to infer what other agents need.

The six modes under FC2 are:

• FM-2.1 (2.2%): Conversation Reset — Unexpected or unwarranted restarting of a dialogue, potentially losing context and progress made in the interaction. This is distinct from FM-1.4 (Loss of Conversation History) in that the reset is an agent-initiated action (an agent decides to restart) rather than a passive context management failure.

• FM-2.2 (6.8%): Fail to Ask for Clarification — Inability to request additional information when faced with unclear or incomplete data, potentially resulting in incorrect actions. The paper provides an example (Appendix N.4) where in AppWorld, the Supervisor Agent uses the password as a token instead of asking the Spotify Agent for clarification about authentication requirements.

• FM-2.3 (7.4%): Task Derailment — Deviation from the intended objective or focus of a given task, potentially resulting in irrelevant or unproductive actions. The paper provides an example (Appendix N.5/N.6) where in AG2 MathChat, agents correctly compute "how much did John spend" but then answer "how much money did John have left" — a different question entirely. The agents solved a related but wrong problem.

• FM-2.4 (0.85%): Information Withholding — Failure to share or communicate important data or insights that an agent possesses and could impact decision-making of other agents if shared. Figure 3 illustrates this with the Phone Agent failing to communicate that the API requires a phone-number-formatted username to the Supervisor Agent, who also fails to seek clarification, leading to repeated failed logins. This is the rarest failure mode in the taxonomy, suggesting it is a relatively unusual but high-impact coordination breakdown.

• FM-2.5 (1.9%): Ignored Other Agent's Input — Disregarding or failing to adequately consider input or recommendations provided by other agents. The paper provides an example (Appendix N.11) where an agent in a multi-agent peer review system acknowledges that another agent's solution is correct but then ignores this fact and presents its own (contradictory) solution without detecting the contradiction.

• FM-2.6 (13.2%): Reasoning-Action Mismatch — Discrepancy between the logical reasoning process and the actual actions taken by the agent, potentially resulting in unexpected or undesired behaviors. The paper provides an example (Appendix N.10) where in HyperAgent, the Navigator correctly identifies the root cause of a pylint bug and proposes a specific code fix internally, but then communicates completely different (and less useful) information to the Planner — the diagnosis and the communication diverge. At 13.2%, this is the second-most-prevalent failure mode overall, suggesting that agents frequently "think" correctly but "act" incorrectly in their interactions.

FC3: Task Verification (23.5% of failures). These failures "involve inadequate verification processes that fail to detect or correct errors, or premature termination of tasks" (Section 4, FC3). The key insight is that verification is treated as a final-stage checkbox rather than a multi-level process: "many existing verifiers perform only superficial checks, despite being prompted to perform thorough verification, such as checking if the code compiles or if there are leftover TODO comments" (Section 4). The paper draws an analogy to software engineering: "MAS development should take lessons from traditional software development where programmers test their code before committing."

The three modes under FC3 are:

• FM-3.1 (6.2%): Premature Termination — Ending a dialogue, interaction, or task before all necessary information has been exchanged or objectives have been met, potentially resulting in incomplete or incorrect outcomes. The paper provides an example (Appendix N.13) where in HyperAgent, the Editor claims to have completed a code modification but the Executor discovers the modification was never applied — the task terminates with an incorrect claim of completion.

• FM-3.2 (8.2%): No or Incomplete Verification — (Partial) omission of proper checking or confirmation of task outcomes or system outputs, potentially allowing errors or inconsistencies to propagate undetected. The paper provides an example (Appendix N.6) where a ChatDev-generated chess program passes superficial checks (code compilation) but contains runtime bugs because it fails to validate against actual game rules.

• FM-3.3 (9.1%): Incorrect Verification — Failure to adequately validate or cross-check crucial information or decisions during the iterations, potentially leading to errors or vulnerabilities. The paper provides an example (Appendix N.9) where MetaGPT's chess implementation includes movement tests for knights, bishops, rooks, queens, and kings, but the tests only verify that pieces can move to specific squares — they don't verify that the movement rules are correctly implemented (e.g., that pieces can't move through other pieces). The verification exists but checks the wrong things.

Execution stage mapping. The paper maps each failure mode to one or more stages in the MAS execution pipeline (shown in Figure 1):

• Pre-Execution: Failures whose root causes emerge from decisions made before execution begins — mainly FC1 modes related to prompt specification, role definition, and system architecture.
• Execution: Failures that manifest during active agent interaction — FC1 modes involving state management (FM-1.3, FM-1.4), all FC2 modes (inter-agent dynamics only occur during execution), and FM-3.3 (incorrect verification during iterations).
• Post-Execution: Failures related to final checking and termination — FM-1.5 (not recognizing when to stop), FM-3.1 (stopping too early), and FM-3.2 (not checking adequately before declaring completion).

A failure mode that spans multiple stages (indicated by spanning bars in Figure 1) means "the underlying issue can manifest or have implications across these different phases of operation." For example, FM-1.1 (Disobey Task Specification) can originate from poor pre-execution prompt design but only become apparent when execution produces output that violates constraints.

Prevalence statistics. The percentages in Figure 1 represent "the prevalence of each failure mode and category as observed in our analysis of 1642 MAS execution traces." Since a single trace can exhibit multiple failure modes, these are occurrence rates, not a partition — the percentages across modes sum to more than 100%. The category-level percentages (44.2%, 32.3%, 23.5%) sum to 100% of observed failure instances. The prevalence ordering reveals which problems are most common: Step Repetition (15.7%), Reasoning-Action Mismatch (13.2%), Unaware of Termination Conditions (12.4%), and Disobey Task Specification (11.8%) are the four most frequent individual failure modes, collectively accounting for over half of all observed failures.

What MAST intentionally does not cover. The paper is explicit about scope limitations: "While we recognize that prior works have noted some individual failure types and we do not claim MAST is exhaustive, it offers precise definitions for a structured approach to understanding why MAS fail" (Section 4). The authors also acknowledge that "some MAS failures can stem from fundamental limitations of current LLMs, such as hallucination or instruction following" but deliberately focus on "identifying failure patterns where improvements in system design, agent coordination, and verification can offer room to improve the reliability of MAS." The taxonomy is designed to capture actionable failure modes — those where engineering changes (not just better base models) can make a difference.

LLM-as-Judge Annotation Pipeline

Manually annotating 1,642 traces with the 14 MAST failure modes using human experts would be prohibitively expensive — recall that the initial GT analysis required over 20 hours per expert for just 150 traces. To scale annotation while maintaining consistency with MAST definitions, the paper develops an automated LLM-as-Judge pipeline that uses OpenAI's o1 model to classify traces.

The annotation task. Given a complete MAS execution trace (the full text of all agent-to-agent conversations, tool calls, and outputs) and the MAST taxonomy (definitions for all 14 failure modes), the LLM annotator must output, for each failure mode, a binary label indicating whether that mode is present in the trace, plus a textual justification explaining the evidence for each detected mode. The task is challenging because: (1) traces are long — averaging over 15,000 lines of text each; (2) failure modes can co-occur and interact, requiring holistic analysis; (3) the distinctions between some modes are subtle (e.g., FM-2.4 Information Withholding vs. FM-2.5 Ignored Other Agent's Input require distinguishing whether information was possessed-but-unshared versus received-but-disregarded); and (4) the same surface behavior (e.g., wrong output) can stem from different root causes depending on context.

Few-shot prompting strategy. The LLM annotator is prompted with:

1. The complete MAST taxonomy definitions — the full textual descriptions of all 14 failure modes with their defining characteristics and examples.
2. Few-shot examples from human-annotated data — selected traces from the IAA studies where human experts applied MAST labels with justifications, serving as demonstrations of the annotation task. The paper does not specify the exact number of few-shot examples, but they are drawn from the MAST-Data-human pool of 21 human-annotated traces.
3. The target trace — the complete execution trace to be annotated.

The prompt instructs the o1 model to analyze the trace, identify which failure modes are present according to the MAST definitions, and provide reasoning for each identification. The output format is structured to enable programmatic extraction of binary labels per failure mode.

Calibration and validation. The LLM annotator's performance is evaluated against human expert annotations on a held-out set from the IAA studies. The paper reports multiple metrics (Table 2):

Metric definitions:

• Accuracy (0.94): The proportion of all binary decisions (14 failure modes × N traces) where the LLM annotator's label matches the human expert consensus label. At 94%, the LLM annotator makes the same call as human experts on the vast majority of failure mode decisions.

• Recall (0.77): The proportion of failure modes that human experts identified that were also identified by the LLM annotator. A recall of 0.77 means the LLM annotator catches 77% of the failures that humans would flag — it misses about 23% of failures that humans detect.

• Precision (0.833): The proportion of failure modes flagged by the LLM annotator that human experts also flagged. A precision of 0.833 means that when the LLM annotator says a failure mode is present, human experts agree 83.3% of the time — about 17% of LLM-flagged failures are false positives.

• F1 (0.80): The harmonic mean of precision and recall, providing a single balanced metric. F1 penalizes models that sacrifice one for the other — high F1 requires both good detection and low false alarm rates.

• Cohen's κ (0.77): Chance-corrected agreement between LLM and human annotators, as defined in Section 3.2. A κ of 0.77 is in the "substantial agreement" range, confirming that the LLM annotator's agreement with humans is not just a byproduct of the label distribution.

Why the few-shot version substantially outperforms zero-shot. The zero-shot o1 (κ = 0.58) to few-shot o1 (κ = 0.77) improvement is large — a 19-point Kappa gain. This suggests that the MAST definitions alone are insufficient for consistent annotation; the model needs concrete examples of how the abstract definitions map to actual trace evidence. This is consistent with the human experience: the IAA process required 10 hours of discussion to align annotators' interpretations of definitions, and the few-shot examples serve a similar function for the LLM — they demonstrate the application of definitions to concrete cases.

Generalization validation. Before using the LLM annotator for large-scale MAST-Data construction, the paper validates its applicability on two MAS frameworks (OpenManus and Magentic-One) and two benchmarks (MMLU and GAIA) that were not part of the initial taxonomy development or few-shot example selection. An additional human IAA round on these out-of-domain traces using the finalized MAST yields κ = 0.79 — comparable to the κ = 0.77 achieved on the calibration set. This cross-validation is critical: it demonstrates that the LLM annotator generalizes to systems with different architectures (OpenManus is hierarchical, Magentic-One is star-topology) and task domains (GAIA involves general web-based tasks, not programming or math), supporting the claim that MAST and the annotation pipeline are broadly applicable.

Cost analysis (Appendix K). The paper reports the average API cost per trace annotation across different MAS frameworks (Table 9). Costs vary substantially by framework, primarily driven by trace length:

• AppWorld: $0.37 per trace (shortest traces — limited multi-turn conversations with service-specific agents)
• HyperAgent: $0.97 per trace
• AG2: $1.17 per trace
• ChatDev: $2.13 per trace
• MetaGPT: $2.45 per trace
• Magentic-One: $1.31 per trace
• OpenManus: $4.14 per trace (longest traces)

The average cost across all frameworks is $1.80 per trace. For the full 1,642-trace MAST-Data, this implies a total annotation cost of approximately$2,956 — a one-time cost that is trivial compared to the human expert effort that would be required for equivalent coverage (the initial 150-trace GT analysis alone consumed 120+ person-hours).

MAST-Data Construction at Scale

With the validated MAST taxonomy and LLM annotator in hand, the paper constructs MAST-Data through systematic large-scale data collection across diverse configurations.

Framework and model coverage. MAST-Data comprises 1,642 annotated traces across the configurations detailed in Table 1. The data collection strategy spans multiple axes of variation:

• 7 MAS frameworks: ChatDev, MetaGPT, HyperAgent, AppWorld, AG2 (MathChat), Magentic-One, OpenManus
• 4 model families: GPT-4 series (GPT-4, GPT-4o, GPT-4o-mini), Claude series (Claude-3.7-Sonnet), Qwen (Qwen2.5-Coder-32B-Instruct), CodeLlama (CodeLlama-7b-Instruct-hf)
• 6 benchmarks: ProgramDev/programming tasks (custom benchmarks of 30 and 100 tasks), SWE-Bench Lite (real-world GitHub issue resolution), AppWorld Test-C (cross-service API tasks), GSM-Plus and OlympiadBench (math reasoning), MMLU (general knowledge), GAIA (general agent tasks)

This combinatorial coverage enables the comparative analyses in Section 5.1 and Appendix F: comparing failure profiles across different underlying LLMs while holding the MAS framework constant (e.g., MetaGPT with GPT-4o vs. MetaGPT with Claude-3.7-Sonnet, both on ProgramDev-v2), and comparing failure profiles across different MAS architectures while holding the model constant (e.g., ChatDev vs. MetaGPT, both with GPT-4o on ProgramDev-v2).

Annotation protocol. For each trace, the LLM annotator (o1 with few-shot examples) produces binary labels for all 14 MAST failure modes plus textual justifications. The annotation also records whether the task was completed successfully (human-evaluated for 210 traces in MAST-Data-human, a mix of human-evaluated and LLM-evaluated for the remaining traces — the paper specifies "HE: Human Evaluated" and "LA: LLM Annotated" in Table 1).

MAST-Data-human subset. The paper releases a smaller dataset of 21 traces where "each trace is annotated by three human experts" during the IAA studies, providing ground-truth annotations for calibration and validation. For these traces, the annotation includes the specific MAST failure mode labels with textual justifications from human experts, serving as gold-standard examples for training or evaluating annotation pipelines.

Unique trace counts by configuration. Table 1 specifies the exact trace counts for each configuration:

• 30 traces each for the initial 7 MAS on their primary benchmarks (GPT-4o for most, Claude-3.7-Sonnet for HyperAgent, GPT-4 for AG2 MathChat) — these 210 traces form the human-evaluated core
• 100 traces each for ChatDev and MetaGPT on ProgramDev-v2 with GPT-4o, Claude-3.7-Sonnet, Qwen2.5-Coder-32B-Instruct, and CodeLlama-7b-Instruct-hf — these 800 traces enable the LLM comparison and architecture comparison analyses
• Additional traces for AG2 MathChat on OlympiadBench (206 with GPT-4o), GSMPlus (193 with Claude-3.7-Sonnet), and MMLU (168 with GPT-4o-mini); plus Magentic-One on GAIA (165 with GPT-4o) — these 732 traces extend coverage to math and general agent domains

A note on closed-source MAS exclusion. The paper explored including Manus, a popular closed-source AI agent platform, but found that it doesn't provide access to full agent execution traces — only final outputs are visible. Without access to internal agent conversations, "reliable, detailed failure annotation using MAST is infeasible" (Appendix B.3). The paper reports that Manus achieves a 60% success rate on ProgramDev through human evaluation of task correctness alone, but cannot be incorporated into MAST-Data's fine-grained failure analysis. This limitation highlights a broader challenge: the most capable MAS may be the least transparent, creating a tension between capability and debuggability.

Open-source release. The paper releases three artifacts:

1. MAST-Data: The full 1,642 annotated traces with LLM annotations (binary labels per failure mode plus justifications)
2. MAST-Data-human: The 21 traces with three independent human expert annotations each, serving as a gold-standard subset
3. The LLM annotator pipeline: Open-sourced code enabling other researchers to apply MAST annotation to their own MAS traces, packaged as agentdash (pip-installable Python library, described in Appendix C)

The pip-installable library provides a simple API: initialize with an OpenAI API key, call MASTAnnotator.produce_taxonomy(trace), and receive a dictionary mapping failure mode IDs to boolean values plus a summary. This lowers the barrier for other MAS developers to adopt MAST-based failure diagnosis.

Intervention Case Studies as Validation of MAST's Utility

The paper includes two intervention case studies (Appendix H) that serve a dual purpose: demonstrating that MAST-identified failures are real and actionable (not just annotation artifacts), and testing the hypothesis that tactical fixes are insufficient — that addressing MAS failures requires structural redesign.

Case Study 1: AG2 MathChat on GSM-Plus. The baseline configuration is a Student agent collaborating with an Assistant agent with Python code execution capability. MAST analysis of baseline traces reveals frequent failures in FC1 (specification issues) and FC3 (verification issues). Two interventions are tested:

1. Improved prompt with verification section (prompt-only change): The original unstructured prompt is replaced with a structured prompt (Appendix L.2) that: (a) explicitly separates the problem-solving approach into three cases (direct Python solution, reasoning-based solution, step-by-step with Python queries), (b) adds a verification section requiring the agent to "Verify your solution step by step, documenting each part of the verification process in a designated VERIFICATION section" before presenting the final answer, and (c) adds explicit instructions for handling missing data.

2. New topology with specialized roles (architectural change): The two-agent system is restructured into three specialized agents — Problem Solver (chain-of-thought without tools, Appendix L.3), Coder (writes and executes Python, Appendix L.4), and Verifier (critically evaluates solutions, only agent that can terminate conversation, Appendix L.5). The Verifier must receive solutions from both other agents before making decisions.

Results (Table 5) show that with GPT-4, the improved prompt yields a significant improvement (84.75% → 89.75%), but the new topology does not (85.50%, p = 0.4 by Wilcoxon test, not statistically significant). With GPT-4o, both interventions show statistically significant improvements (p = 0.03), with improved prompt reaching 89.00% and new topology reaching 88.83%. The key finding is inconsistency: the effectiveness of interventions depends on the underlying model, and even the best intervention leaves 10–15% failure rate.

Case Study 2: ChatDev on ProgramDev-v0 and HumanEval. The baseline ChatDev simulates a software company with CEO, CTO, CPO, Programmer, Reviewer, and Tester agents. MAST analysis reveals failures in FM-1.2 (Disobey Role Specification, particularly CPO terminating discussions without CEO approval) and FC3 (verification issues, superficial code checks). Two interventions:

1. Prompt modifications with hierarchy enforcement (prompt-only change): Modified system prompts (Appendix M.2) that explicitly state reporting relationships ("You report to the technical lead or CTO"), emphasize role-specific responsibilities, and require CEO approval for conversation termination. Verifier prompts are enhanced to "Check the edge cases and special cases and instances for the task we are doing. Do not miss any cases. Do not suffice with generic and superficial cases."

2. Topology change from DAG to cyclic graph with iterative refinement (architectural change): The framework's workflow is restructured from a directed acyclic graph to a cyclic graph where the process terminates only when the CTO agent confirms all reviews are properly satisfied, with a maximum iteration cutoff. This enables multiple review-fix cycles rather than a single linear pass.

Results (Table 5) show modest improvements: ProgramDev-v0 baseline 25.0% → improved prompt 34.4% → new topology 40.6%; HumanEval baseline 89.6% → improved prompt 90.3% → new topology 91.5%. The gains are real but limited — even after interventions, ChatDev fails on 59.4% of ProgramDev-v0 tasks and 8.5% of HumanEval tasks. A +15.6% improvement (ProgramDev-v0, baseline to new topology) is meaningful but doesn't approach reliable deployment.

MAST-based failure distribution analysis post-intervention. The paper uses MAST not just to guide interventions but to measure their effects at the failure-mode level (Appendix H.3, Figures 10 and 11). For AG2, the topology intervention reduces FC2 (Inter-Agent Misalignment) failures from 796 to 205 — a 74% reduction in coordination failures — but FC1 (System Design Issues) and FC3 (Verification) remain high. For ChatDev, the topology intervention reduces all three categories, but FC3 (Verification) stubbornly persists at 169 failures even in the best configuration. This granular breakdown is the paper's key value proposition: "while we can see aggregate improvements, breakdown of failure modes from MAST show that no intervention we tried resolves all failures... Proposing more complex solutions is required" (Section 5.3).

Why these results support the "structural redesign needed" argument. The paper's central claim is not that MAST fixes MAS — it's that MAST reveals problems that can't be fixed with simple prompt tweaks or role reassignments. The intervention studies provide evidence for this: despite targeted improvements informed by MAST analysis, task completion rates remain low, and failure modes persist or shift rather than disappear. The paper interprets this as evidence that "achieving high reliability may require combinatorial changes ranging from agent system organization to model level improvements" (Section 5.3), pointing toward the structural strategies discussed in Appendix G.2 (comprehensive verification, standardized communication protocols, uncertainty quantification, memory/state management) as necessary components of any effective solution.

4. Key Insights and Innovations

Innovation 1: The Problem Is System Design, Not Model Capability — An Organizational Theory of MAS Failure

The dominant assumption in the MAS community, prior to this paper, was that multi-agent failures primarily trace to the limitations of the underlying LLMs. If agents produce incorrect code, hallucinate facts, or fail to follow instructions, the natural diagnosis is that the base model isn't capable enough — wait for GPT-5, Claude-4, or a better fine-tuned model, and the coordination problems will dissolve. This assumption is implicit in how the field has invested effort: improving model reasoning through scaling and post-training, developing better single-agent prompting techniques, and treating MAS architecture as a secondary optimization on top of a model capability foundation.

This paper provides the first systematic empirical argument that this assumption is wrong — or at least dramatically incomplete. The key evidence comes from the comparative analysis in Appendix F (Figures 8 and 9). When the same underlying model (GPT-4o) is used in two different MAS architectures (MetaGPT vs. ChatDev) on the same benchmark (ProgramDev-v2), MetaGPT exhibits 60–68% fewer failures in System Design Issues (FC1) and Inter-Agent Misalignment (FC2). If failures were primarily model-driven, the architecture shouldn't matter this much — both systems use GPT-4o, so both should fail at similar rates. The fact that architecture choice produces a 60–68% difference in failure rates on the same model implies that system design is not a marginal optimization but a first-order determinant of reliability.

This is a fundamental conceptual reframing, not an incremental finding. Prior work in the "AI agents that matter" vein (Kapoor et al., 2024) and Anthropic's design guidance argued that complexity can hinder practical adoption — that is a pragmatic warning about over-engineering. This paper goes further: it provides an empirical taxonomy showing which specific design choices cause which specific failure patterns, and argues that the failures follow patterns recognizable from organizational theory. The authors explicitly invoke Perrow's "Normal Accidents" theory of high-risk organizations and Roberts' work on high-reliability organizations (Section 5.3) to argue that MAS failures are organizational failures — problems of structure, role definition, communication protocol, and verification process — not primarily problems of individual agent competence.

The strength of this argument is that the paper demonstrates it rather than merely stating it. The intervention studies (Appendix H, Table 5) show that architecture-level changes (workflow restructuring, role enforcement, adding verification cycles) produce measurable improvements (+15.6% on ProgramDev-v0 for ChatDev topology changes) without touching the underlying model at all. Conversely, the paper shows that model substitution matters but doesn't eliminate structural failure patterns: when MetaGPT switches from GPT-4o to Claude-3.7-Sonnet (Figure 8), the overall failure count changes but the distribution of failure types remains similar — suggesting the architecture imposes a failure profile that model quality modulates but doesn't fundamentally alter.

The implication is a reorientation of the MAS research agenda. If the dominant failure modes are organizational, then the path to reliability runs through better system engineering — formal specification of agent responsibilities, structured communication protocols with explicit information requirements, multi-level verification integrated into workflows — rather than through better base models alone. This doesn't mean models don't matter (the paper shows they do — GPT-4o outperforms Claude-3.7-Sonnet in FC1 and FC2), but it does mean that waiting for better models while ignoring architecture is a losing strategy. The paper's core intellectual move is turning MAS failure from a model evaluation problem into a systems engineering problem, with MAST providing the diagnostic vocabulary that makes systematic engineering possible.

Innovation 2: The Failure Taxonomy as a Diagnostic Instrument — Moving from Aggregate Metrics to Structured Failure Profiles

Prior to this paper, evaluating a MAS meant running it on a benchmark and reporting a single number: accuracy, success rate, pass@k. The paper's Section 2 explicitly catalogs this landscape — SWE-bench, GAIA, GSM-Plus, MMLU, and others all operate in this "top-down perspective, focusing on aggregate performance" mode. These numbers tell you that a system failed but provide zero signal about why. A developer seeing "ChatDev: 40% success on ProgramDev" learns nothing actionable — the failure could be due to specification violations, coordination breakdowns, verification gaps, or any combination thereof, each requiring a completely different intervention.

The paper introduces a fundamentally different evaluation paradigm: structured failure profiling through MAST's 14-mode taxonomy. Instead of asking "did the system succeed?", MAST asks "which of 14 known failure patterns appeared in this execution?" A system's evaluation is no longer a scalar but a 14-dimensional vector indicating which failure modes occur and at what frequency. This shift from scalar to vector evaluation is what enables the paper's core analytical contributions: comparing failure profiles across architectures (MetaGPT vs. ChatDev in Figure 9), across models (GPT-4o vs. Claude-3.7-Sonnet in Figure 8), and across interventions (before vs. after in Figures 10 and 11).

What makes this genuinely innovative rather than an obvious extension of prior work is the empirical grounding of the taxonomy categories. The 14 failure modes are not theoretical constructs — they emerged from a rigorous Grounded Theory analysis of 150 real MAS traces with 120+ person-hours of expert annotation. The IAA process (κ = 0.88) demonstrates that these categories are not subjective interpretations but consistently applicable diagnostic labels that independent experts converge on. This is what distinguishes MAST from prior survey papers that listed "challenges" at a high level (Han et al., 2024; Hammond et al., 2025) — those surveys said things like "coordination is difficult," which is true but useless for diagnosis. MAST says "Failure Mode 2.4: Information Withholding — the agent possesses critical information and fails to communicate it, as evidenced by the Phone Agent not telling the Supervisor Agent that the API requires a phone-number-formatted username" (trace segment in Figure 3). That level of specificity enables targeted interventions: fix the communication protocol to require explicit information handoff, or prompt agents to proactively list API requirements.

The paper's correlation analysis (Appendix E, Figures 6 and 7) provides additional validation that the taxonomy captures genuinely distinct phenomena rather than splitting the same underlying problem into artificially fine categories. The low inter-category correlations (0.17–0.32 between FC1/FC2/FC3) suggest the three categories measure different aspects of MAS behavior. The moderate within-category correlations for some mode pairs (max 0.63 between FM-1.4 Loss of Conversation History and FM-2.1 Conversation Reset) are actually informative — they show that related but distinct phenomena can be distinguished (these are different root causes with similar symptoms: one is passive context truncation, the other is active agent-initiated restart).

The significance beyond raw performance is that MAST enables a new type of research question that was previously unanswerable: "Does changing the underlying LLM reduce coordination failures or specification failures?" (Appendix F answers: GPT-4o reduces FC1/FC2 but FC3 remains stubborn), "Does adding a verifier agent shift failures from FC3 to FC2?" (the ChatDev intervention analysis shows tradeoffs between categories), "Are open-source models failing in the same ways as closed-source models, or differently?" (Appendix I, Table 6 answers: they fail more but with similar patterns, suggesting the failure taxonomy captures system-level rather than model-level phenomena). These are diagnostic questions that a scalar accuracy metric cannot address.

The paper is also transparent that this is a first-generation diagnostic instrument, not a complete theory. The authors explicitly note the taxonomy doesn't cover every failure pattern, the difficulty of distinguishing some modes even with high IAA, and the challenge of applying the framework to domains without clean correctness signals. This is a strength, not a weakness — it frames MAST as an evolving tool for a new subfield rather than a finished product, inviting community refinement while providing immediate practical utility.

Innovation 3: Verification Is the Pervasive Unresolved Challenge — It Fails Even When Systems Succeed

One of the paper's most striking and counterintuitive findings emerges not from aggregate statistics but from the fine-grained analysis in Appendix J.1 (Table 7): verification-related failure modes (FC3: FM-3.1 Premature Termination, FM-3.2 No or Incomplete Verification, FM-3.3 Incorrect Verification) appear frequently even in successful traces. In ChatDev, FM-3.2 occurs in 10% of successful traces and 10% of failed traces — verification is equally broken regardless of outcome. In MetaGPT, FM-3.3 (Incorrect Verification) occurs in 16.7% of successful traces versus 27.8% of failed traces — worse in failures, but still present in one of every six successful runs.

This is a genuinely novel empirical discovery with significant implications. Prior work on verification in LLM systems (e.g., self-verification in Chain-of-Thought prompting, verifier models for math reasoning like Cobbe et al., 2021) focused on whether verification improves accuracy. The implicit assumption was that verification is a reliable mechanism that sometimes fails — when it works, it catches errors; when it fails, errors slip through. This paper's data reveals a more troubling picture: verification is an unreliable mechanism that sometimes succeeds by coincidence. The system is not "verifying correctly" on successful traces — it is producing a correct output despite broken verification. This is the multi-agent equivalent of a codebase where all tests pass because the tests don't actually check the right properties, not because the code is correct.

The Insight 3 formulation in Section 4 captures this: "Current verifier implementations are often insufficient; sole reliance on final-stage, low-level checks is inadequate." The GT analysis of traces revealed that existing verifiers "perform only superficial checks, despite being prompted to perform thorough verification, such as checking if the code compiles or if there are leftover TODO comments" (Section 4). The ChatDev chess example (Appendix N.6) is illustrative: a chess program passes code compilation checks, has movement tests for pieces, but the tests only verify that pieces can move to specific squares — they don't check that the movement rules are correctly implemented (pawns can't move through pieces, castling requires specific conditions, en passant exists). The verification exists, it runs, it reports success — and the program is broken.

This finding has a direct practical implication that the paper draws out: "MAS development should take lessons from traditional software development where programmers test their code before committing." In software engineering, testing is not a checkbox at the end — it is integrated into the development process at multiple levels (unit tests, integration tests, system tests, acceptance tests) with explicit coverage criteria. MAS verification, by contrast, is typically a single agent at the end of a pipeline running a few checks and declaring success. The paper's proposed direction — "multi-level checks for both low-level correctness and high-level objectives" and "using external knowledge, collecting testing output throughout generation" — is essentially arguing that MAS need software-engineering-grade verification infrastructure, not just better verifier prompts.

The significance of this finding lies in what it says about the path to MAS reliability. If verification failures were simply correlated with task failures (i.e., verification works correctly on successful runs and breaks on failed runs), then improving task success rates would automatically improve verification rates — solve the underlying capability problem and verification takes care of itself. The fact that verification is decoupled from task success means it is an independent problem requiring independent solutions. You can improve the base model, refine agent prompts, restructure workflows, and still have a system that can't reliably tell whether its own output is correct. This is a fundamental architectural limitation that no amount of model scaling will fix — it requires rethinking how verification is integrated into MAS workflows, not just how it is prompted.

Innovation 4: The Scalable Annotation Pipeline as Research Infrastructure — Enabling Empirical MAS Science

The paper's methodological contribution — the LLM-as-Judge annotation pipeline — is easy to undervalue as "just engineering." But viewed in context of the field's current state, it represents a fundamental enabler for empirical MAS research. Prior to this work, anyone wanting to analyze MAS failures systematically faced an impossible bottleneck: manual annotation of long execution traces by expert humans. The paper reports that the initial GT analysis of 150 traces required over 20 hours per expert — extrapolating to the paper's full 1,642-trace dataset, manual annotation would require approximately 220 hours per annotator, or roughly 6 person-weeks of full-time work. At typical research lab costs, this is prohibitively expensive for a single paper, and completely infeasible as a standard practice for MAS development.

The LLM annotator pipeline (o1 with few-shot examples, κ = 0.77 against human experts, 94% accuracy, $1.80 average cost per trace) changes the economics of MAS failure analysis from person-weeks to dollars. More importantly, it changes the *nature* of what research questions can be asked. When annotation costs$1.80 per trace rather than 1.5 hours of expert time, it becomes feasible to:

• Run comparative analyses at scale: The paper annotates traces from 7 frameworks × 4 model families × multiple benchmarks, producing the cross-sectional comparisons in Section 5.1 and Appendix F. Without automated annotation, each such comparison would require a separate annotation effort.

• Perform iterative development with failure diagnostics: The intervention studies (Appendix H) use MAST to measure failure mode distributions before and after changes. In a development cycle, a team could annotate traces from their current system, identify the dominant failure modes, implement targeted fixes, re-annotate, and verify that the targeted modes decreased — all within hours rather than weeks.

• Study failure mode evolution: As LLMs improve (GPT-4 → GPT-4o → future models), do failure patterns shift? Do some failure modes disappear while others persist? Such longitudinal questions require annotating traces from multiple model versions — feasible at $1.80/trace, impossible at 1.5 hours/trace.

The paper validates the pipeline's generalizability through the κ = 0.79 agreement on previously unseen MAS frameworks (OpenManus, Magentic-One) with new benchmarks (MMLU, GAIA), demonstrating that the annotation quality doesn't degrade significantly when applied to systems outside the calibration set. This is crucial for the pipeline's role as shared infrastructure — if it only worked on the specific frameworks used in calibration, its value to the broader community would be limited.

What makes this contribution more than just "we used an LLM to label data" is the methodological rigor in calibration and validation. The paper doesn't just prompt an LLM and report accuracy — it conducts formal IAA studies comparing LLM annotations against human experts, reports multiple metrics (accuracy, recall, precision, F1, Cohen's κ), compares zero-shot against few-shot performance to quantify the value of demonstration examples, and validates on out-of-domain systems. This transforms the pipeline from an unvalidated heuristic ("the LLM probably gets it right most of the time") to a calibrated instrument with known performance characteristics ("the LLM achieves κ = 0.77, meaning substantial agreement with human experts, with recall of 0.77 and precision of 0.833").

The paper also demonstrates the pipeline's practical accessibility through the agentdash Python library (Appendix C), which reduces MAST annotation to a few lines of code (pip install agentdash, initialize with API key, call produce_taxonomy(trace), receive structured failure mode labels). This is infrastructure-thinking: the goal is not just to produce a paper but to build tools that other researchers and developers can adopt with minimal friction, accelerating the entire field's ability to do empirical MAS failure analysis.

The limitation, which the paper acknowledges implicitly through its cost analysis, is dependence on a specific commercial LLM (OpenAI's o1). If o1's pricing or availability changes, or if o1's annotation behavior drifts, the pipeline's calibration may need re-validation. The paper doesn't explore whether other models (Claude, open-source alternatives) could achieve comparable agreement at lower cost — this is natural future work, but it means the current pipeline is tied to a specific API provider in ways that could affect reproducibility.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses a custom-generated benchmark called ProgramDev (30 coding problems) for the initial 150-trace Grounded Theory analysis and human evaluation core, plus ProgramDev-v2 (100 programming problems) for the comparative analyses of LLMs and MAS architectures (Appendix D). Additional benchmarks include SWE-Bench Lite (real-world GitHub issues, used with HyperAgent), AppWorld Test-C (cross-service API tasks, used with the AppWorld MAS), GSM-Plus (augmented math reasoning, used with AG2 MathChat), OlympiadBench (harder math problems, used with AG2), MMLU (general knowledge, used with AG2), and GAIA (general web-based agent tasks, used with Magentic-One). ProgramDev tasks are "programming challenges, such as implementing Tic-Tac-Toe, Chess, or Sudoku, for which abundant solutions and descriptions are readily available online" and are "intended to be relatively straightforward for MAS, rather than exceptionally difficult, to better isolate specific failure dynamics" (Appendix D). The test set sizes vary by configuration: 30 traces for each of the 7 initial MAS frameworks on their primary benchmarks, 100-trace blocks for ChatDev/MetaGPT on ProgramDev-v2 with different models, and varying trace counts for the math and general agent evaluations (detailed in Table 1). All task outcomes in the 210-trace human-evaluated core are verified by human evaluators, while larger-scale annotations rely on LLM evaluation for task completion.

• Base model(s). The paper evaluates across four model families with multiple variants: GPT-4 series (GPT-4, GPT-4o, GPT-4o-mini), Claude series (Claude-3.7-Sonnet), and two open-source families—Qwen (Qwen2.5-Coder-32B-Instruct) and CodeLlama (CodeLlama-7b-Instruct-hf). The primary human-evaluated experiments use GPT-4o for most frameworks (ChatDev, MetaGPT, AppWorld, Magentic-One, OpenManus), Claude-3.7-Sonnet for HyperAgent, and GPT-4 for AG2 MathChat. The model choices are not explicitly justified with a "representativeness" claim in the style of scaling-law papers; rather, the diversity of models (closed-source frontier models, open-source models at different scales, models specialized for code) serves the paper's goal of demonstrating that "the choice of LLM... critically shape[s] the landscape of potential failures" (Appendix F) and that failure patterns are not idiosyncratic to a single model family.

• Metrics. The primary evaluation metric is task success rate — the fraction of traces where the MAS produces a correct final output as determined by human evaluation (for the 210-trace core) or automated grading against ground truth (for benchmarks like GSM-Plus, MMLU, GAIA where answer verification is straightforward). For failure analysis, the key metrics are failure mode occurrence rates (the fraction of traces exhibiting each of the 14 MAST failure modes) and failure category proportions (FC1: 44.2%, FC2: 32.3%, FC3: 23.5% across all annotated traces). The LLM annotator's performance is evaluated against human expert annotations using accuracy (0.94), recall (0.77), precision (0.833), F1 (0.80), and Cohen's Kappa (κ = 0.77 for few-shot o1, calibrated against the human IAA consensus on held-out traces; Table 2). These annotation quality metrics are reported separately from task success metrics — they measure how well the automated pipeline reproduces human failure mode judgments, not how well the MAS performs.

• Baselines. The paper's comparisons are primarily cross-sectional rather than against a fixed baseline method. Within each MAS framework, the paper compares: (1) different underlying LLMs in the same architecture (e.g., MetaGPT with GPT-4o vs. MetaGPT with Claude-3.7-Sonnet, Figure 8); (2) different MAS architectures with the same model (e.g., ChatDev vs. MetaGPT, both with GPT-4o, Figure 9); (3) different model scales within the same architecture and family (Qwen2.5-Coder-32B-Instruct vs. CodeLlama-7b-Instruct-hf, both in ChatDev and MetaGPT, Table 6 in Appendix I); and (4) intervention variants — original system vs. prompt-improved vs. topology-redesigned versions of AG2 and ChatDev (Table 5, Figures 10–11). There is no single "best-of-N sampling" or "single-agent with ReAct" baseline against which all MAS are compared, since the frameworks operate on different benchmarks with different task formats. Each framework's original implementation serves as its own baseline for within-framework interventions.

• Generation budget / compute accounting. The paper does not measure or control for "test-time compute" in the FLOPs-counting sense familiar from scaling-law papers. Instead, the unit of analysis is a complete MAS execution trace — one full run of the system from task input to termination. Traces vary substantially in length across frameworks: from shorter traces in AppWorld (average annotation cost $0.37, implying fewer conversation turns) to very long traces in OpenManus (average annotation cost$4.14, implying extensive multi-turn interactions; Table 9). The paper does not attempt to equalize or account for differences in the number of LLM calls, tokens generated, or wall-clock time between frameworks. This means that comparisons across frameworks (e.g., "MetaGPT has fewer FC1 failures than ChatDev") do not control for the amount of computation expended — MetaGPT's advantage could partly reflect more efficient use of compute rather than better architecture, or vice versa. Within-framework comparisons (same system, different models or interventions) implicitly control for compute since the architecture determines the number of agent interactions.

• Cross-validation / statistical protocol. The paper uses several validation strategies: (1) Inter-annotator agreement studies with three expert annotators across three rounds of iterative refinement on the initial 150-trace corpus, achieving κ = 0.88 in final rounds (Section 3.2) — this validates the taxonomy's consistency, not the MAS performance; (2) Out-of-domain generalization validation on two unseen MAS frameworks (OpenManus, Magentic-One) with new benchmarks (MMLU, GAIA), achieving κ = 0.79 (Section 3.4) — this validates that MAST and the LLM annotator transfer to systems not used in taxonomy development; (3) Few-shot vs. zero-shot comparison for the LLM annotator (Table 2), showing that few-shot examples improve κ from 0.58 to 0.77; (4) Wilcoxon signed-rank tests for the AG2 MathChat intervention comparisons, testing whether performance differences are statistically significant rather than sampling noise — the improved prompt with GPT-4 yields p < 0.05 (interpreted as significant), while the topology change with GPT-4 yields p = 0.4 (not significant; Section H.1); (5) Six repetitions of the AG2 MathChat experiments to assess performance consistency, with results reported as mean ± standard deviation (e.g., "84.75 ± 1.94" for the GPT-4 baseline; Table 5). No cross-validation is applied to the MAS performance evaluation itself — the paper does not split its ProgramDev or benchmark datasets into train/validation/test folds for system tuning, since the interventions are designed manually based on qualitative trace analysis rather than optimized against a validation metric.

Main Quantitative Results

The paper's quantitative results fall into four logical groupings: (1) the prevalence and distribution of failure modes across the full MAST-Data corpus, (2) comparative failure profiles across different LLMs within the same MAS architecture, (3) comparative failure profiles across different MAS architectures with the same LLM, and (4) intervention studies demonstrating that MAST-guided fixes produce measurable but limited improvements.

Overall Failure Prevalence and Distribution

The headline numbers from the 1,642-trace MAST-Data corpus, illustrated in Figure 1, reveal the relative frequency of each failure mode:

• System Design Issues (FC1) accounts for 44.2% of all failure instances. Within this category, FM-1.3 (Step Repetition) is the single most frequent failure mode at 15.7%, followed by FM-1.5 (Unaware of Termination Conditions) at 12.4%, FM-1.1 (Disobey Task Specification) at 11.8%, FM-1.4 (Loss of Conversation History) at 2.8%, and FM-1.2 (Disobey Role Specification) at 1.5%.

• Inter-Agent Misalignment (FC2) accounts for 32.3% of failures. FM-2.6 (Reasoning-Action Mismatch) is the second-most-prevalent failure mode overall at 13.2%, followed by FM-2.3 (Task Derailment) at 7.4%, FM-2.2 (Fail to Ask for Clarification) at 6.8%, FM-2.1 (Conversation Reset) at 2.2%, FM-2.5 (Ignored Other Agent's Input) at 1.9%, and FM-2.4 (Information Withholding) at 0.85% — the rarest failure mode in the entire taxonomy.

• Task Verification (FC3) accounts for 23.5% of failures. FM-3.3 (Incorrect Verification) at 9.1%, FM-3.2 (No or Incomplete Verification) at 8.2%, and FM-3.1 (Premature Termination) at 6.2%.

These percentages represent occurrence rates across traces, not a partition — a single trace commonly exhibits multiple failure modes. The category-level percentages (44.2%, 32.3%, 23.5%) sum to 100% of failure instances, but individual mode percentages sum to more than 100% due to co-occurrence.

Failure rates across frameworks (Figure 5, Appendix B). The paper reports task failure rates for six MAS frameworks on their respective benchmarks: Magentic-One on GAIA fails at 62.0%, AG2 on OlympiadBench at 41.0%, AppWorld on Test-C at 86.7%, HyperAgent on SWE-Bench Lite at 74.7%, ChatDev on ProgramDev at 66.7%, and MetaGPT on ProgramDev at 60.0%. These rates span a wide range (38–86.7%) and are not directly comparable across frameworks due to different benchmark difficulties. The paper emphasizes that the existence of high failure rates across all systems — even the best-performing (Magentic-One at 38% success) fails nearly two-fifths of the time — motivates the need for systematic failure analysis.

Failure distribution by framework (Figure 4). The 210-trace subset (30 traces per framework) reveals system-specific failure profiles:

• AppWorld shows a notably high count of FM-3.1 (Premature Termination), which the paper attributes to "its star topology and lack of a predefined workflow making termination conditions less obvious" (Section 5.1).
• OpenManus exhibits the highest count of FM-1.3 (Step Repetition) among all systems.
• HyperAgent shows elevated FM-1.3 (Step Repetition) and FM-3.3 (Incorrect Verification).
• ChatDev and MetaGPT show broadly distributed failures but with MetaGPT exhibiting fewer total failures in FC1 and FC2 than ChatDev — a pattern analyzed in detail in Appendix F.
• AG2 and Magentic-One show intermediate failure counts with reasonably balanced distributions.

The paper explicitly notes that "as the specific tasks and benchmarks may differ across the MAS configurations shown, these results are intended to illustrate system-specific failure profiles rather than to serve as a performance comparison across MAS" (Figure 4 caption). This is an important caveat: the per-framework distributions in Figure 4 reflect both the system's architecture and the benchmark's difficulty, and these effects are not separable without a common benchmark.

Comparative Analysis: Effect of Underlying LLM

The paper compares GPT-4o against Claude-3.7-Sonnet when both are used within the MetaGPT framework on the same ProgramDev-v2 benchmark (100 traces each, Figure 8 in Appendix F):

• Overall failure reduction: GPT-4o exhibits markedly fewer total failures than Claude-3.7-Sonnet: 38 FC1 failures vs. 63 (−39.7%), 47 FC2 failures vs. 58 (−19.0%), and 77 FC3 failures vs. 83 (−7.2%). The largest relative improvement is in System Design Issues, suggesting GPT-4o has "stronger capabilities in instruction following or aspects of 'social reasoning' for agentic collaboration" (Appendix F).

• Failure distribution shift: Despite the overall reduction, the distribution of failures across categories is similar between the two models. Both show FC3 (Task Verification) as the modal category, with GPT-4o at 77 FC3 failures and Claude at 83 — the gap narrows substantially here. The paper interprets this as evidence that "robust verification remains a significant challenge regardless of the LLM used, though GPT-4o has a marginally lower count here" (Appendix F).

• Per-mode differences: At the individual failure mode level, GPT-4o shows substantially lower FM-1.1 (Disobey Task Specification: GPT-4o 22 vs. Claude 36), FM-1.2 (Disobey Role Specification: 13 vs. 1 — note Claude has fewer here), FM-1.3 (Step Repetition: 1 vs. 36), FM-2.2 (Fail to Ask for Clarification: 6 vs. 16), and FM-2.3 (Task Derailment: 3 vs. 17). However, for FM-3.2 (No or Incomplete Verification), Claude shows lower counts (26 vs. 52), and for FM-3.3 (Incorrect Verification), the counts are similar (Claude 57 vs. GPT-4o 57). This per-mode breakdown reveals that model substitution affects some failure modes dramatically (Step Repetition drops from 36 to 1 — near-elimination) while leaving others essentially unchanged (Incorrect Verification at 57 for both models).

Comparative Analysis: Effect of MAS Architecture

The paper compares ChatDev against MetaGPT when both use GPT-4o on the same ProgramDev-v2 benchmark (100 traces each, Figure 9 in Appendix F):

• FC1 (System Design Issues): MetaGPT shows 38 FC1 failures vs. ChatDev's 118 — a 67.8% reduction. At the individual mode level, MetaGPT has dramatically fewer FM-1.1 (Disobey Task Specification: 22 vs. 89), FM-1.3 (Step Repetition: 1 vs. 24), and FM-1.5 (Unaware of Termination Conditions: 3 vs. 10). The paper attributes this to MetaGPT's use of Standard Operating Procedures (SOPs) encoded in agent prompts: "in MetaGPT, the adherence to task specifications and role specifications are done mostly through SOPs, demonstrating strong performance in FC1 especially."

• FC2 (Inter-Agent Misalignment): MetaGPT shows 47 FC2 failures vs. ChatDev's 116 — a 59.5% reduction. Significant per-mode reductions include FM-2.2 (Fail to Ask for Clarification: 7 vs. 22), FM-2.3 (Task Derailment: 10 vs. 60), and FM-2.6 (Reasoning-Action Mismatch: 17 vs. 57). The paper suggests MetaGPT's "architecture or operational flow is more effective at maintaining adherence to specifications and ensuring smoother agent coordination."

• FC3 (Task Verification): Critically, this pattern reverses. MetaGPT shows 77 FC3 failures vs. ChatDev's 30 — a 156.7% increase. At the individual mode level, MetaGPT has more FM-3.2 (No or Incomplete Verification: 46 vs. 10) and FM-3.3 (Incorrect Verification: 57 vs. 19). The paper explains this by noting that "ChatDev places a higher importance in verification as it is reflected by the specific testing and reviewing phases in ChatDev's architectural design, causing fewer verification issues," while MetaGPT's strength in specification adherence through SOPs is not matched by equally strong verification mechanisms.

This cross-architecture comparison is the paper's strongest evidence for its central claim that "both the choice of LLM and the specific design of the MAS architecture critically shape the landscape of potential failures, and improvements likely require a holistic approach considering both aspects" (Appendix F). The fact that the better-performing architecture on FC1/FC2 is worse on FC3 demonstrates that architecture choices involve genuine tradeoffs, not uniform improvements.

Opposite direction of effects: The model comparison (GPT-4o vs. Claude in MetaGPT) showed that GPT-4o reduces FC1/FC2 but makes minimal difference to FC3. The architecture comparison (MetaGPT vs. ChatDev with GPT-4o) shows that MetaGPT reduces FC1/FC2 but increases FC3. Together, these suggest that FC3 (verification) is the "hardest" failure category — resistant to both model improvements and architecture changes that benefit other categories, and potentially even exacerbated by architectures that are otherwise strong.

Failure Modes in Open-Source Models

The paper extends its analysis to open-source models (Qwen2.5-Coder-32B-Instruct and CodeLlama-7b-Instruct-hf) in both ChatDev and MetaGPT on ProgramDev-v2 (400 total traces, Table 6 in Appendix I):

• Massive performance gap between open-source models: Qwen2.5-Coder-32B-Instruct is "substantially more robust than CodeLlama-7b-Instruct-hf on these tasks, exhibiting far fewer failures overall." For example, in ChatDev, Qwen shows 96 FM-1.3 (Step Repetition) vs. CodeLlama's 97 — but for FM-1.1 (Disobey Task Specification), Qwen shows 35 vs. CodeLlama's 76, and for FM-2.3 (Task Derailment), Qwen shows 9 vs. CodeLlama's 76. The gap is larger for some modes than others, suggesting that model capability affects certain failure types more than others.

• Open-source models lag closed-source models: Comparing Qwen in MetaGPT (Table 6) against GPT-4o in MetaGPT (Figure 8), Qwen shows approximately 3× more FC1 failures (12 + 1 + 35 + 0 + 3 = 51 vs. GPT-4o's 38) and roughly comparable FC3 failures but substantially more FC2 failures (4 + 0 + 0 + 12 + 16 = 32 vs. GPT-4o's 47 — note Qwen shows fewer for some modes but the distribution differs). CodeLlama shows dramatically higher failures across all categories compared to both Qwen and the closed-source models.

• Architecture effects persist even with weak models: CodeLlama in MetaGPT vs. ChatDev shows that MetaGPT still reduces FC1 failures (94 + 12 + 99 + 23 + 76 = 304 in ChatDev vs. components summing to lower in MetaGPT) but the pattern is noisier due to the high baseline failure rate. The paper's conclusion is that "both open-source models show a higher frequency of failures compared to the leading closed-source models analyzed in our paper... This suggests a performance gap and highlights important areas for future improvement in open-source models for multi-agent tasks" (Appendix I).

Failure Mode Correlation with Task Success

The paper investigates an important question: are all failure modes equally indicative of eventual task failure, or are some modes "more fatal" than others? The analysis in Appendix J.1 (Table 7) compares failure mode occurrence rates between successful and unsuccessful traces for ChatDev and MetaGPT:

• Successful runs are not failure-free: In ChatDev successful traces, FM-1.1 (Disobey Task Specification) occurs at 20.0%, FM-1.3 (Step Repetition) at 20.0%, FM-3.2 (No or Incomplete Verification) at 10.0%, and FM-3.3 (Incorrect Verification) at 20.0%. The mere presence of a failure mode does not guarantee task failure — the system can succeed despite exhibiting failure patterns.

• Some failures are "more fatal" than others: FM-1.5 (Unaware of Termination Conditions) appears at 0.0% in ChatDev successful traces vs. 10.0% in failed traces — it occurs only in failures. FM-2.4 (Information Withholding) appears at 0.0% in successful vs. 5.0% in failed for ChatDev, and 0.0% vs. 5.6% for MetaGPT. These modes appear to be "critical bugs that are highly likely to derail the task" (Appendix J.1).

• Verification failures persist in success: FM-3.2 (No or Incomplete Verification) appears in 10.0% of ChatDev successful traces — the same rate as in ChatDev failure traces (10.0%). In MetaGPT, FM-3.2 appears in 16.7% of successful traces vs. 5.6% of failed traces — verification is broken more often in successful runs than in failed ones for this architecture. FM-3.3 (Incorrect Verification) shows a similar pattern: 20.0% in ChatDev successful vs. 25.0% in failed, and 16.7% in MetaGPT successful vs. 27.8% in failed. The paper interprets this as evidence that "while these systems can complete some tasks, their verification process still contains flaws. MAST identifies such systemic weaknesses, even when they do not cause an immediate task failure" (Appendix J.1).

Failure Mode Variation Across Benchmarks

Fixing the MAS framework (AG2) and LLM (GPT-4o), the paper compares failure category rates across three benchmarks with different difficulty levels (Table 8 in Appendix J.2):

• GSM (easiest): FC1 rate = 0.53 per trace, FC2 = 1.33, FC3 = 0.37
• MMLU (medium): FC1 = 1.06, FC2 = 1.01, FC3 = 0.60
• Olympiad (hardest): FC1 = 1.19, FC2 = 1.21, FC3 = 0.67

The trend is monotonic for FC1 and FC3 — harder benchmarks produce higher failure rates per trace in these categories. FC2 is non-monotonic (GSM 1.33 > Olympiad 1.21 > MMLU 1.01), which the paper doesn't explain but may reflect domain-specific coordination challenges in math problem-solving at the GSM difficulty level. The paper notes that "while the failure profiles for MMLU and Olympiad are similar, the GSM benchmark results in a much lower number of Inter-Agent Misalignment and Specification failures" — a finding that suggests benchmark difficulty and domain interact in determining which failure modes dominate.

Intervention Study Results

The paper presents quantitative results from two case studies (Appendix H, Table 5) where MAST-guided interventions were applied and task success rates measured before and after:

Case Study 1: AG2 MathChat on GSM-Plus (200 exercises, 6 repetitions):

With GPT-4, the improved prompt yields a statistically significant +5.0 percentage point improvement (84.75% → 89.75%). The new topology yields only +0.75 percentage points (85.50%), which a Wilcoxon test finds not statistically significant (p = 0.4). With GPT-4o, both interventions are significant (p = 0.03 for both), with improved prompt reaching 89.00% (+4.75 points) and new topology reaching 88.83% (+4.58 points). The paper interprets these results as showing that interventions "can reduce failures" but "are not universal, and their effectiveness varies based on factors such as the underlying LLM" (Appendix H.1).

Case Study 2: ChatDev on ProgramDev-v0 (32 tasks) and HumanEval:

On ProgramDev-v0, the prompt improvements yield +9.4 percentage points (25.0% → 34.4%), and the topology change (DAG to cyclic with iterative refinement) yields +15.6 points (25.0% → 40.6%). On HumanEval, the gains are more modest: +0.7 points from prompt changes and +1.9 points from topology changes. The paper notes that even the best configuration fails on 59.4% of ProgramDev-v0 tasks, emphasizing that "not all failure modes are resolved, and task completion rates still remain low, indicating that more substantial improvements are needed" (Appendix H.2).

MAST-based failure distribution analysis of interventions (Figures 10 and 11, Appendix H.3):

• AG2: The original system shows 625 FC1 failures, 692 FC2 failures, and 305 FC3 failures across all traces. The prompt intervention reduces these to 687, 796, and 335 respectively — surprisingly, all three categories increase slightly with the prompt change. The topology intervention dramatically reduces FC2 from 796 to 205 (74.2% reduction), reduces FC3 from 335 to 86 (74.3% reduction), and reduces FC1 from 687 to 171 (75.1% reduction). Per-mode analysis shows that FM-2.6 (Reasoning-Action Mismatch) drops from 168 to 46 with topology changes, FM-1.3 (Step Repetition) drops from 133 to 50, and FM-3.3 (Incorrect Verification) drops from 159 to 45.

• ChatDev: The original system shows 424 FC1, 447 FC2, and 209 FC3 failures. The prompt intervention reduces these to 372, 403, and 176 respectively (12.3%, 9.8%, 15.8% reductions). The topology intervention further reduces them to 355, 376, and 169 (additional 4.6%, 6.7%, 4.0% reductions beyond the prompt change). The reductions are more modest than in AG2 — the topology change doesn't achieve the dramatic FC2 elimination seen there — and FC3 (verification) proves stubborn, declining only from 209 to 169 (19.1% reduction) even with the most aggressive intervention.

The MAST-based analysis reveals that the aggregate accuracy improvements in Table 5 are accompanied by different patterns of failure mode reduction. In AG2, the topology change produces a massive reduction in FC2 (coordination failures) that the aggregate accuracy alone (only +0.75 points with GPT-4, statistically insignificant) fails to capture — the system is becoming more coordinated even if not more accurate, suggesting the accuracy bottleneck lies elsewhere. In ChatDev, the topology change helps but leaves FC3 verification failures largely untouched, which aligns with the paper's Insight 3 that verification requires more fundamental solutions than workflow restructuring.

Ablation Studies and Robustness Checks

LLM annotator: zero-shot vs. few-shot prompting: The few-shot o1 configuration achieves substantially better agreement with human experts than zero-shot across all metrics (Table 2). Accuracy improves from 0.89 to 0.94 (+5.6%), recall from 0.62 to 0.77 (+24.2%), precision from 0.68 to 0.833 (+22.5%), F1 from 0.64 to 0.80 (+25%), and Cohen's κ from 0.58 to 0.77 (+32.8%). The dramatic improvement in recall suggests that without few-shot examples, the LLM annotator is conservative — it misses many failures that humans identify (recall 0.62). The few-shot examples provide the model with concrete demonstrations of how abstract MAST definitions map to actual trace evidence, substantially reducing false negatives. The improvement in precision (fewer false positives) is also substantial, indicating that the examples help calibrate what constitutes sufficient evidence for a failure mode label.

LLM annotator generalization to unseen MAS: The LLM annotator calibrated on the initial 5 MAS frameworks is evaluated on two held-out systems — OpenManus and Magentic-One — with new benchmarks (MMLU, GAIA). Human IAA on these traces using finalized MAST achieves κ = 0.79, which is in the "substantial agreement" range and comparable to the LLM annotator's κ = 0.77 on the calibration set (Section 3.4). This demonstrates that neither the taxonomy definitions nor the LLM annotator's few-shot examples are overtuned to the specific systems used in development. The paper also reports a "correlation study" in Appendix E that validates the distinctiveness of failure categories — inter-category correlations are low (0.17–0.32, Figure 6), while within-category mode correlations are moderate (max 0.63 between FM-1.4 Loss of Conversation History and FM-2.1 Conversation Reset, Figure 7), confirming that the taxonomy captures genuinely distinct phenomena rather than artificial subdivisions.

Failure mode "fatality" analysis: The breakdown of failure mode occurrence in successful vs. failed traces (Appendix J.1, Table 7) serves as an implicit validation that MAST labels are not merely correlated with task failure in a trivial way. If all failure modes appeared exclusively in failed traces, the taxonomy would be redundant with the binary success/failure label. The finding that some modes (FM-3.2, FM-3.3) appear frequently in successful traces while others (FM-1.5, FM-2.4) appear almost exclusively in failures demonstrates that MAST captures variance in failure severity and type that a simple success/failure binary misses.

Architecture and model ablation through cross-sectional comparison: The comparative analyses in Appendix F (Figures 8 and 9) function as ablations: by varying one factor (LLM or architecture) while holding the other constant, the paper isolates the contribution of each to the failure profile. The key finding — that architecture matters more than model for FC1/FC2, but neither architecture nor model choice fully addresses FC3 — emerges from this design. However, these are not controlled experiments in the causal sense; they are observational comparisons where the "ablation" is changing the system configuration and observing the resulting failure distribution. The paper does not, for example, take a single system and systematically remove or modify individual architectural components (e.g., "MetaGPT without SOPs," "ChatDev without the dedicated verification phase") to measure their marginal contribution — such experiments would be considerably more informative but are outside the paper's scope.

Intervention studies as manipulation checks: The before-and-after failure mode distributions in the intervention studies (Figures 10 and 11) serve as manipulation checks for MAST's diagnostic validity. If MAST-identified failure modes were not "real" — if they were annotation artifacts rather than genuine system behaviors — then interventions targeting those modes should not produce measurable reductions. The fact that the AG2 topology intervention (which specifically addressed coordination failures by introducing a dedicated Verifier agent with termination authority) reduced FC2 failures by 74.2% while the prompt intervention (which primarily improved specification clarity) did not reduce FC2 at all provides convergent evidence that MAST is measuring genuine system properties. Similarly, the ChatDev topology intervention (which added iterative review cycles) reduced FC1 and FC2 failures but barely touched FC3, consistent with the independent diagnostic that verification is the system's primary remaining weakness.

Benchmark difficulty variation: The failure rate comparison across GSM, MMLU, and Olympiad (Table 8) validates that MAST is sensitive to task difficulty in expected ways — harder benchmarks produce higher failure rates per trace, and the distribution shifts toward more FC1 (specification) and FC3 (verification) failures as problems become more complex. The non-monotonic FC2 pattern (GSM has the highest FC2 rate despite being the easiest benchmark) is a genuine puzzle that the paper does not explain — it may reflect domain-specific coordination challenges in math problem-solving (e.g., the Student-Assistant dynamic in MathChat creates more opportunities for FM-2.2 Fail to Ask for Clarification when mathematical terms are ambiguous) that are less relevant in general knowledge (MMLU) or harder math (Olympiad, where the problem structure might be clearer even if the content is harder).

Open-source model comparison: The large gap between Qwen2.5-Coder-32B-Instruct and CodeLlama-7b-Instruct-hf (Table 6) demonstrates that MAST is sensitive to model capability differences and not just architecture differences. CodeLlama shows dramatically higher failures across almost every mode — for example, FM-1.1 (Disobey Task Specification) in ChatDev: CodeLlama 76 vs. Qwen 35; FM-2.3 (Task Derailment): CodeLlama 76 vs. Qwen 9; FM-3.2 (No or Incomplete Verification): CodeLlama 67 vs. Qwen 16. These are not subtle differences — they represent qualitatively different system behavior. However, the paper does not investigate whether these differences are primarily due to model scale (7B vs. 32B parameters), architecture, or training data, making it difficult to attribute the gap to specific model properties.

Human evaluation coverage: Of the 1,642 traces in MAST-Data, only 210 (12.8%) have human-evaluated task completion labels (Table 1: 7 frameworks × 30 traces each). The remaining traces — including all the comparative analyses of LLMs and architectures on ProgramDev-v2 (800 traces) and the math/general agent extensions (732 traces) — rely on automated task completion evaluation or LLM-based judgment. The paper does not report the accuracy of automated task completion evaluation against human judgment for the ProgramDev-v2 benchmark, which means that the task success labels in the extended dataset may have errors that could bias the failure mode correlation analyses in Appendix J.

No false-positive/negative analysis for the LLM annotator per failure mode: Table 2 reports aggregate metrics (accuracy, recall, precision, F1, κ) across all 14 failure modes, but does not break down performance per mode. Given that some modes are very rare (FM-2.4 at 0.85%, FM-1.2 at 1.5%), the aggregate metrics may mask poor performance on low-frequency modes. For instance, if the LLM annotator never predicts FM-2.4 (Information Withholding), its accuracy on that mode would be 99.15% (since FM-2.4 is absent 99.15% of the time) but its recall would be 0 — it would miss every instance. The paper does not provide per-mode recall and precision, making it impossible to assess whether the LLM annotator is equally reliable across all failure modes or only performs well on common ones.

Critical Assessment

Claim: "41% to 86.7% failure rate on 7 SOTA open-source MAS"

This claim is supported by the reported data in Figure 5 (Appendix B), but the paper does not provide enough information to verify whether these failure rates are comparable or meaningful. The seven frameworks are evaluated on different benchmarks with different difficulty levels — Magentic-One on GAIA (62% failure) vs. AppWorld on Test-C (86.7% failure) vs. MetaGPT on ProgramDev (60% failure). There is no common benchmark against which all seven are evaluated, so the range "41% to 86.7%" conflates framework quality with benchmark difficulty. A framework could have an 86.7% failure rate because it is evaluated on a harder benchmark, not because it is worse-designed. The paper acknowledges this implicitly by stating the rates "are measured on different benchmarks, therefore they are not directly comparable" (Figure 5 caption), but the headline claim in Section 1 ("41% to 86.7% failure rate") is presented without this caveat in the main text, potentially misleading readers into thinking these numbers represent a controlled comparison. A stronger experimental design would have included at least one benchmark (e.g., ProgramDev) on which multiple frameworks could be evaluated, enabling direct performance and failure-mode comparisons without the benchmark-difficulty confound.

Claim: "Failures stem from system design issues, not just LLM limitations"

This claim is the paper's central intellectual contribution and is partially supported by the comparative analyses in Appendix F. The evidence that MetaGPT with GPT-4o shows 60–68% fewer FC1 and FC2 failures than ChatDev with GPT-4o on ProgramDev-v2 (Figure 9) demonstrates that architecture matters — two systems using the identical model on the identical benchmark produce substantially different failure profiles. However, the claim that failures are "not just" model limitations requires a stronger counterfactual: would improving the model (e.g., GPT-4o → a hypothetical GPT-5) eliminate the remaining failures, or would they persist because they are inherently architectural? The paper does not and cannot test this — it compares two current models (GPT-4o vs. Claude-3.7-Sonnet) and finds that model substitution changes the failure profile but doesn't eliminate failures. This shows that current model capability differences interact with architecture, but doesn't establish that "improvements in the base model capabilities will be insufficient to address the full MAST" (Section 5.3). That claim requires extrapolation beyond available data — it is a conjecture, and the paper's phrasing ("we conjecture") appropriately hedges it, but the strength of the supporting empirical evidence is more modest than the claim's rhetorical prominence suggests.

The strongest direct evidence for the system-design claim comes from the intervention studies: the ChatDev topology change produces +15.6% improvement on ProgramDev-v0 without changing the underlying model (Table 5). This demonstrates that some failures are addressable through architectural changes alone. However, +15.6% from 25% to 40.6% still leaves 59.4% failure rate — the majority of failures persist even after the intervention. This is consistent with both the system-design hypothesis (the intervention wasn't aggressive enough, more radical redesign is needed) and the model-limitation hypothesis (the remaining failures are due to GPT-4o not being capable enough, regardless of architecture). The paper interprets the persistent failures as evidence for the former, but the data is equally consistent with the latter — the experiments don't distinguish between these interpretations.

Claim: "Well-designed MAS can outperform poorly-designed ones by 60–68% in reducing specification and coordination failures"

This claim, paraphrased from Section 5.1 and Appendix F, is supported by the data in Figure 9 but requires careful interpretation. "60–68%" refers to the relative reduction in failure counts: ChatDev has 118 FC1 failures vs. MetaGPT's 38, a reduction of (118-38)/118 = 67.8%; ChatDev has 116 FC2 failures vs. MetaGPT's 47, a reduction of (116-47)/116 = 59.5%. This is a valid computation from the reported numbers, but three caveats apply:

1. It is a single comparison on a single benchmark (ProgramDev-v2) with a single model (GPT-4o). Whether this advantage generalizes to other benchmarks, other models, or other failure mode distributions is unknown. The Qwen vs. CodeLlama comparison in open-source models (Table 6) shows similar architecture effects but with noisier patterns.

2. It only covers FC1 and FC2 — MetaGPT is worse on FC3. The paper's framing emphasizes the reduction in "specification and coordination failures" but downplays the verification tradeoff. A reader focused only on the headline might conclude MetaGPT is "better designed" overall, but the data shows it makes a different tradeoff (better specification adherence, worse verification) rather than being uniformly superior.

3. "Well-designed" is post-hoc reasoning. The paper attributes MetaGPT's FC1/FC2 advantage to its SOPs and ChatDev's FC3 advantage to its explicit verification phases, but these attributions are qualitative interpretations, not causally tested hypotheses. There could be other differences between the two frameworks (prompt quality, conversation management, error handling) that produce the observed patterns — the paper does not isolate which specific design choices drive the differences.

Claim: "Verification appears even in successful traces"

This claim is strongly supported by Table 7 in Appendix J.1 and is genuinely novel and robust. In ChatDev, FM-3.2 (No or Incomplete Verification) occurs in 10.0% of successful traces and 10.0% of failed traces — identical rates. FM-3.3 (Incorrect Verification) occurs in 20.0% of successful vs. 25.0% of failed — slightly higher in failures but still present in one-fifth of successes. In MetaGPT, FM-3.2 occurs in 16.7% of successful vs. 5.6% of failed — more often in successful traces, and FM-3.3 in 16.7% of successful vs. 27.8% of failed. The sample sizes are small (ChatDev success N = 10, failure N = 20; MetaGPT success N = 12, failure N = 18, computed from the 30-trace human-evaluated subsets), but the pattern is consistent: verification failures are not cleanly separable from task outcomes. This is the paper's most empirically solid finding because it does not require cross-system comparisons or model substitution counterfactuals — it emerges from within-system, within-model analysis of the basic relationship between failure mode labels and task success.

Claim: "Simple interventions yield only modest gains (up to +15.6% success rate increase)"

This claim is supported by Table 5 but "modest" is a judgment call that depends on context. The ChatDev topology intervention improves ProgramDev-v0 from 25.0% to 40.6% — a 62.4% relative improvement over baseline. Whether this is "modest" or "substantial" depends on the cost of the intervention and the requirements of the deployment context. A 15.6 percentage point gain from an architecture change that requires no additional model training or inference cost could be considered excellent in many engineering contexts. The paper's framing that these gains are "modest" serves its argument that structural redesigns are needed, but the data could equally support the opposite conclusion: that architecture matters enormously (a 62% relative improvement!) and that further architecture iteration — rather than fundamental rethinking — is the path forward, especially given that the intervention was a single design iteration, not an exhaustive optimization. The paper does not explore whether additional architecture iterations (e.g., combining MetaGPT's SOP approach with ChatDev's verification phases) would yield further gains — this is a natural follow-up that the current experiments don't address.

Missing experiments that would have strengthened the paper

1. A common benchmark for cross-framework comparison: The most significant missing experiment is evaluating multiple frameworks (at minimum ChatDev and MetaGPT, ideally all seven) on a single shared benchmark so that failure rate differences can be attributed to architecture rather than benchmark difficulty. ProgramDev-v2 is evaluated on both ChatDev and MetaGPT (Figure 9), which partially addresses this, but the other five frameworks are never compared on common ground.

2. Systematic component ablation within a single framework: The paper interprets differences between ChatDev and MetaGPT as evidence for the importance of specific design patterns (SOPs, verification phases), but these are holistic comparisons between complex systems. A within-framework ablation — e.g., "ChatDev with explicit SOP prompts," "ChatDev with the verification phase removed," "ChatDev with CEO termination authority enforced" — would provide much stronger causal evidence about which specific design choices drive which failure modes.

3. Human evaluation of task completion at scale: Only 12.8% of traces have human-verified task outcomes. The remaining traces rely on automated or LLM-based completion evaluation whose accuracy is not reported. If the automated evaluation has, say, 85% accuracy, then ~15% of traces in the extended MAST-Data have incorrect success/failure labels, which could substantially bias correlation analyses like Table 7.

4. Per-mode LLM annotator reliability: Table 2 reports aggregate metrics that may mask poor performance on rare failure modes. Per-mode recall and precision would reveal whether FM-2.4 (0.85% prevalence) or FM-1.2 (1.5%) are reliably detected or essentially invisible to the automated pipeline.

5. Replication across model scales within a family: The paper compares across model families (GPT-4o vs. Claude) and across widely different open-source models (Qwen-32B vs. CodeLlama-7B), but does not compare, for instance, GPT-4o-mini vs. GPT-4o vs. a hypothetical larger GPT variant within the same architecture. Such a scaling comparison would provide evidence about whether failure patterns improve monotonically with model capability or exhibit threshold effects — crucial for assessing the paper's conjecture that model improvements will be insufficient.

6. Statistical power analysis for the intervention studies: The AG2 MathChat experiments use 200 exercises with 6 repetitions, and the standard deviations are reported (±1.18 to ±1.94). The ChatDev experiments use only 32 (ProgramDev-v0) or 164 (HumanEval) tasks with no reported variability. For the ChatDev ProgramDev-v0 results (25.0% → 34.4% → 40.6%), confidence intervals cannot be computed from the provided data, making it impossible to assess whether the differences are statistically significant or within sampling noise. Given that ProgramDev-v0 has only 32 tasks, the standard error on a proportion near 25–40% is approximately 7–8 percentage points, meaning the observed differences (+9.4, +15.6) may not be statistically robust.

7. Cost-effectiveness analysis: The paper reports annotation costs (Table 9) but does not report the compute cost (LLM API calls, tokens generated) of running the MAS frameworks themselves. Without this, it is impossible to assess whether, for example, MetaGPT's superior FC1/FC2 performance comes at higher inference cost than ChatDev — a crucial consideration for practitioners choosing between architectures.

Overall assessment of the experimental support for central claims

The paper's descriptive claims — that MAS fail frequently (Section 1), that failures can be categorized into 14 modes across 3 categories (Section 4), that failure profiles differ by architecture and model (Section 5.1), and that verification failures persist even in successful runs (Appendix J.1) — are well-supported by the MAST-Data corpus and the reported analyses, though with important caveats about benchmark comparability, human evaluation coverage, and per-mode annotation reliability as discussed above.

The paper's causal claims — that failures stem primarily from system design rather than model limitations, that simple fixes are insufficient, and that structural redesigns are necessary — are suggested by the data but not rigorously demonstrated. The comparative analyses show that architecture matters and that within-architecture interventions help, but they do not establish that model improvements cannot solve the same problems (only that current model substitutions don't), nor do they show that the intervention ceiling has been reached (only that one or two rounds of improvement didn't solve everything). These are reasonable conjectures given the evidence, and the paper appropriately hedges them, but readers should distinguish between the well-supported descriptive taxonomy and the more speculative causal interpretations.

The paper's most important contribution — the taxonomy itself and the annotated dataset — is primarily a methodological and infrastructure contribution whose value lies in enabling future work rather than in conclusively answering its motivating question. MAST and MAST-Data provide the diagnostic tools that the field previously lacked; the paper's own experimental analyses using these tools demonstrate their utility without exhausting their potential. The key test of the paper's impact will be whether subsequent work uses MAST to design and evaluate more effective MAS architectures, not whether the paper's own intervention studies achieved high success rates.

6. Limitations and Trade-offs

Difficulty Estimation Cost Is Unaccounted for — the Diagnostic Is More Expensive than the Cure

The assumption or constraint. The entire MAST-based failure diagnosis framework depends on the ability to annotate MAS execution traces with fine-grained failure mode labels. The paper develops an LLM-as-Judge pipeline to make this annotation scalable, reporting costs of $0.37–$4.14 per trace depending on framework (Table 9 in Appendix K). However, the paper explicitly states that the annotation cost is measured only for the API calls to the annotator LLM (OpenAI's o1 model). It does not account for the cost of running the MAS itself to generate those traces — the LLM API calls made by the agents during task execution, which are substantial (ChatDev and MetaGPT traces average over 15,000 lines of text, implying many agent-to-agent conversation turns, each requiring LLM inference).

The paper never provides the inference cost of running the MAS frameworks themselves. A ChatDev execution on ProgramDev involves multiple agents (CEO, CTO, CPO, Programmer, Reviewer, Tester) engaging in multi-turn conversations across design, coding, and testing phases — each turn invokes GPT-4o. The LLM annotator cost for a ChatDev trace is $2.13, but the cost of *generating* that trace may be 10–50× higher. If a developer needs to run their MAS on, say, 100 test problems to diagnose failure patterns, the total cost is (100 × trace generation cost) + (100 ×$2.13 annotation cost), and the former likely dominates.

The consequence. The headline economics of MAST-based diagnosis — "annotation costs $1.80 per trace on average, making failure analysis affordable" — are misleading because they ignore the dominant cost driver. A practitioner evaluating whether to adopt MAST-based failure analysis for their own MAS would face a total cost that is primarily determined by how expensive their system is to run, not by the MAST annotation overhead. For expensive MAS with many agents and long conversation chains (like OpenManus, where annotation alone costs$4.14/trace, implying even higher generation costs), systematic failure diagnosis across hundreds of test cases could be prohibitively expensive. This creates a tension: the systems that most need failure diagnosis (complex MAS with high failure rates) are also the most expensive to diagnose at scale.

Furthermore, the paper's intervention case studies (Appendix H) demonstrate iterative development — run baseline, annotate failures, implement fixes, re-run, re-annotate, compare distributions. Each iteration incurs both generation and annotation costs. The paper does not report the total cost of its own intervention experiments, making it impossible to assess the cost-effectiveness of MAST-guided development relative to other approaches (e.g., simply running more trials and using majority voting, or spending the same budget on better base models).

What evidence exists in the paper. Table 9 reports annotation API costs per framework (ranging from $0.37 to$4.14 per trace), but the paper provides no data on MAS generation costs. Section 3.1 notes that the initial GT analysis of 150 traces required "over 20 hours of annotation per expert" — a human cost figure — but doesn't translate this into LLM API costs for the MAS runs. Appendix K is titled "LLM Annotator Cost" and discusses only the judge pipeline, not total system cost. The paper never provides a total-cost-of-ownership analysis that sums generation and annotation costs for a complete diagnostic cycle.

Mitigation status. The paper does not address this limitation or suggest that generation costs should be reported. It partially mitigates the human-cost side by replacing expert annotators (20+ hours for 150 traces) with the LLM annotator (~$1.80/trace), which is genuinely a massive cost reduction for the annotation component. But this shifts the bottleneck from annotation cost to generation cost without acknowledging the shift. The agentdash Python library (Appendix C) makes annotation cheap and easy, but generating the traces to annotate remains expensive and is left to the user. Future work would need to explore whether failure diagnosis can be performed on shorter or partial traces (reducing generation cost), whether a lightweight "fast diagnosis" mode can approximate MAST labels from fewer agent interactions, or whether the MAST labels themselves can be predicted from system design features without executing the full MAS at all.

The Taxonomy Was Derived and Validated on a Narrow Slice of the MAS Design Space

The assumption or constraint. MAST was derived through Grounded Theory analysis of 150 traces from 5 MAS frameworks (HyperAgent, AppWorld, AG2, ChatDev, MetaGPT) on two task domains (programming and math problem-solving), and validated on two additional frameworks (OpenManus, Magentic-One) with one additional domain (general agent tasks via GAIA). The paper explicitly acknowledges this scope limitation, stating: "While MAST serves as a foundational first step towards unifying the understanding of MAS failures, we do not claim it covers every potential failure pattern" (Section 1).

However, the diversity of the MAS design space is far larger than what is captured here. The paper's frameworks span three architectural paradigms — hierarchical workflow (ChatDev, HyperAgent), assembly line with SOPs (MetaGPT), and star topology (AppWorld, Magentic-One) — but these represent only a subset of possible MAS organizations. Missing from the study are: fully decentralized peer-to-peer architectures with no central coordinator; competitive or mixed-motive systems where agents have partially conflicting goals; systems with dynamic role assignment where agents change responsibilities mid-execution; systems with human-in-the-loop intervention; systems using structured communication protocols (the paper mentions Model Context Protocol and Agent-to-Agent as emerging standards but doesn't evaluate any systems using them); and systems operating in embodied or real-time environments where action timing matters.

The task domains are also constrained: all tasks have well-defined correctness criteria (code compiles and runs, math answers match ground truth, GAIA tasks have verifiable outcomes). This enables the human evaluation of task success that underpins MAST-Data. The paper does not study MAS for open-ended generation, creative tasks, dialogue, or decision-making under uncertainty — domains where "correctness" is ambiguous and failure may be subjective or multi-dimensional.

The consequence. The 14 failure modes in MAST may not cover failure patterns that emerge in architecturally different MAS. For example, a peer-to-peer system with no designated verifier agent might exhibit qualitatively different verification failures than the centralized-verifier pattern captured in FC3. A competitive multi-agent system might exhibit strategic deception or information manipulation that doesn't fit neatly into FC2 (Inter-Agent Misalignment) as currently defined. The paper's IAA validation achieved κ = 0.79 on out-of-domain frameworks (OpenManus, Magentic-One), which is strong but not perfect, and the "out-of-domain" frameworks are still within the same broad paradigm (orchestrated collaboration toward a shared goal).

More practically, a developer working with a MAS architecture substantially different from those studied — say, a swarm of homogeneous agents with emergent role differentiation, or a market-based system where agents bid on subtasks — cannot assume that MAST's 14 modes provide adequate coverage. They may need to extend the taxonomy with new modes, but the paper provides no guidance on how to do this systematically or how to validate new modes for consistency.

What evidence exists in the paper. The generalization validation (Section 3.4) demonstrates that MAST transfers to two new frameworks (OpenManus, Magentic-One) and two new benchmarks (MMLU, GAIA) with κ = 0.79 — a substantial but not perfect agreement, suggesting some failure patterns in these systems are not fully captured by the existing modes. The paper does not report whether annotators needed to add new categories or whether the 14-mode taxonomy was sufficient for these systems. The correlation analysis in Appendix E (Figures 6 and 7) shows low inter-category correlations (0.17–0.32), suggesting the three categories are genuinely distinct, but this analysis is performed on the same systems used to derive the taxonomy — it doesn't test whether new systems would introduce high correlations indicating missing categories.

Table 3 in Appendix B.1 summarizes the seven frameworks studied, and all share the property of being designed for collaborative task completion with predefined agent roles and a shared objective. The paper does not discuss MAS with fundamentally different organizational principles or evaluate whether MAST would require extension for such systems.

Mitigation status. The paper is transparent about this limitation ("we do not claim MAST covers every potential failure pattern") and frames MAST as a "foundational first step" rather than a complete taxonomy. The open-source release of MAST-Data, MAST, and the LLM annotator is partially a mitigation — by making the infrastructure public, the paper enables other researchers to test MAST on new architectures, identify gaps, and propose extensions. However, the paper provides no methodological guidance for how to extend the taxonomy (e.g., how to conduct additional GT analysis, how to validate new failure modes through IAA, how to determine when a new mode is genuinely distinct versus a variant of an existing mode). A developer encountering an unclassified failure pattern in their system would need to either force it into an existing MAST category (potentially distorting the taxonomy's meaning) or develop their own extension methodology from scratch — the paper doesn't provide a protocol for either approach.

Single-Trace Annotation Misses Cross-Trace Failure Patterns and Systemic Weaknesses

The assumption or constraint. The MAST annotation methodology operates at the level of individual execution traces. Each trace is analyzed independently: the LLM annotator (or human expert) reads the complete trace and labels which of the 14 failure modes are present. The resulting MAST-Data maps each trace to its failure mode profile, and the paper's analyses aggregate these per-trace labels into system-level statistics (e.g., "ChatDev exhibits FM-1.3 in X% of traces").

This per-trace approach assumes that failure modes are properties of individual executions and that aggregating across traces captures everything important about a system's failure dynamics. However, some failure patterns may only become visible when examining relationships across traces. For example:

• Systematic failure modes: A system might fail on 100% of traces involving a particular subtask pattern (e.g., "any task requiring authentication across services" in AppWorld), but if those traces are only 5% of the dataset, the aggregated FM-2.4 (Information Withholding) rate will be low (~5%), masking the fact that this is a deterministic failure when the pattern occurs.
• Input-dependent failure modes: FM-1.1 (Disobey Task Specification) might be triggered by specific task phrasings (e.g., ambiguous prompts, prompts with negation, prompts with multiple constraints). Per-trace annotation without analyzing which inputs cause failures cannot distinguish between "this system randomly fails 11.8% of the time on specification compliance" and "this system deterministically fails on specific input types."
• Failure cascades: One trace might exhibit FM-2.6 (Reasoning-Action Mismatch) in early steps, leading to FM-3.3 (Incorrect Verification) in later steps. The per-trace analysis captures both labels, but doesn't capture the causal relationship — that fixing FM-2.6 would also eliminate the downstream FM-3.3. Across the corpus, this means the paper's failure mode counts double-count failures that are causally linked, potentially overestimating the number of independent problems that need to be addressed.
• Interaction effects between failure modes: The paper reports individual failure mode prevalence (Figure 1) and correlation matrices (Appendix E, Figures 6–7), but doesn't analyze whether certain combinations of failure modes are particularly lethal or benign. A trace with both FM-1.5 (Unaware of Termination Conditions) and FM-3.2 (No or Incomplete Verification) might be far more likely to fail than a trace with either mode alone, but the per-trace analysis collapses this into two independent binary labels.

The consequence. The paper's failure mode prevalence statistics and cross-system comparisons may obscure important patterns. If MetaGPT exhibits fewer FC1 failures than ChatDev not because it is better at specification adherence in general, but because it handles one specific common failure pattern much better while being equally bad at others, the aggregate comparison overstates MetaGPT's advantage and provides misleading guidance for intervention — a developer might redesign their system to be more "like MetaGPT" without realizing that they only needed to fix the one specific pattern.

The "fatality analysis" in Appendix J.1 (Table 7) partially addresses this by comparing failure mode occurrence in successful vs. failed traces, but it treats each mode independently and doesn't examine mode combinations. A more informative analysis would identify which sets of failure modes co-occur in failed traces but not in successful ones, revealing the lethal combinations that should be prioritized for intervention.

What evidence exists in the paper. The correlation matrices in Figures 6 and 7 (Appendix E) are the paper's main analysis of failure mode relationships, and they show generally low to moderate correlations (inter-category 0.17–0.32, max within-category 0.63). The paper interprets low correlations as evidence of category distinctiveness, which is valid, but doesn't explore whether low correlations also imply that failure modes rarely co-occur in systematic ways — i.e., whether certain failure combinations are strongly predictive of task failure even if they aren't correlated in the full dataset. The fatality analysis (Table 7) treats each mode's occurrence rate in successful vs. failed traces as an independent property, not examining mode combinations.

The paper does not report any input-conditioned failure analysis — for instance, whether specific task types in ProgramDev (game implementation vs. calculator vs. data structure) trigger different failure profiles, or whether ambiguous vs. specific prompt phrasings produce different outcomes. The ProgramDev dataset is described in Appendix D but its composition (distribution of task types, prompt styles, difficulty) is not characterized, so readers cannot assess whether the reported failure rates generalize across task types or are dominated by a few challenging categories.

Mitigation status. The paper does not address this limitation. The per-trace annotation methodology is presented as the natural unit of analysis, and cross-trace patterns are not discussed as a missing dimension. The open-source release of MAST-Data does enable other researchers to conduct cross-trace analyses (since all traces are available with their failure mode labels and task inputs), but this is an opportunity for future work, not a mitigation by the current paper. A natural extension would be to annotate traces with task-level metadata (task type, prompt features, subtask structure) and analyze failure mode prevalence conditional on these features, enabling fine-grained diagnosis of when and why specific failure modes occur, not just how often.

The LLM Annotator's Reliability on Rare Failure Modes Is Unknown

The assumption or constraint. The paper evaluates the LLM annotator (OpenAI o1 with few-shot examples) against human expert annotations and reports aggregate performance metrics: accuracy 0.94, recall 0.77, precision 0.833, F1 0.80, Cohen's κ = 0.77 (Table 2). These are averages computed across all 14 failure modes and all traces in the held-out calibration set.

However, the 14 failure modes have dramatically different prevalence rates. FM-1.3 (Step Repetition) occurs in 15.7% of traces, while FM-2.4 (Information Withholding) occurs in only 0.85% of traces — a nearly 20:1 ratio. In a held-out set of, say, 30 traces, FM-2.4 would appear in approximately 0.255 traces on average (30 × 0.0085) — meaning in most calibration sets, it appears 0 or 1 times. Aggregate metrics computed by pooling all decisions across all modes will be dominated by the common modes (FM-1.3, FM-2.6 at 13.2%, FM-1.5 at 12.4%), masking potentially poor performance on rare modes. If the LLM annotator has 90% recall on FM-1.3 (common) but 20% recall on FM-2.4 (rare), the aggregate recall of 0.77 would be achieved through a weighted average where the rare mode's poor performance is invisible.

The consequence. Several of the paper's analyses depend on rare failure mode occurrence statistics. The "fatality analysis" (Table 7) reports that FM-2.4 (Information Withholding) appears in 5.0% of ChatDev failed traces and 0.0% of successful traces — but this is based on at most 1–2 traces in a 30-trace human-evaluated subset (5% of 20 failed traces = 1 trace). If the LLM annotator has low recall for FM-2.4, the true occurrence rate could be higher, and the "fatality" conclusion (that FM-2.4 is a "critical bug highly likely to derail the task") might be based on a small number of correctly detected instances while missing others where FM-2.4 was present but not annotated.

Similarly, the comparative analyses in Appendix F (Figures 8 and 9) report per-mode failure counts, including for rare modes. If FM-2.4 is annotated with low recall, the reported counts (e.g., 0–3 instances per system in Figure 9) are lower bounds, and the true difference between systems could be larger or smaller than reported. The paper's claim that FM-2.4 is the rarest failure mode in the taxonomy (0.85% overall prevalence, Figure 1) could partly reflect annotation difficulty rather than genuine rarity — if the LLM annotator systematically misses instances because Information Withholding requires inferring an agent's unstated knowledge (a challenging reasoning task), the true prevalence could be several times higher.

What evidence exists in the paper. The paper provides no per-mode evaluation of the LLM annotator. Table 2 reports only aggregate metrics. The calibration methodology (Section 3.3) describes evaluating the LLM annotator "on a held-out set from our IAA studies" but doesn't specify the size of this set, the distribution of failure modes in it, or whether per-mode metrics were computed. The few-shot examples used in the LLM annotator prompt are drawn from human-annotated data (MAST-Data-human, 21 traces), but the paper doesn't report whether these examples cover all 14 failure modes or whether rare modes are underrepresented in the few-shot set.

The human IAA process achieved κ = 0.88 on the final taxonomy, but this is also an aggregate metric. The paper doesn't report per-mode IAA, so it's unknown whether human experts also disagree more on rare modes — if they do, the LLM annotator's lower agreement on those modes would partially reflect inherent ambiguity in the definitions rather than LLM-specific limitations.

Mitigation status. The paper does not address this limitation or acknowledge that aggregate annotation quality metrics may conceal per-mode variation. The agentdash library (Appendix C) provides per-mode binary labels when annotating a trace, but doesn't provide confidence scores or calibration information that would allow users to assess reliability for specific modes. A mitigation would be to report per-mode recall, precision, and F1 for the LLM annotator against human consensus, or at minimum to report whether the few-shot examples include instances of all 14 failure modes with balanced representation. The absence of this analysis means practitioners using MAST for diagnosis cannot distinguish between "this failure mode genuinely doesn't occur in my system" and "this failure mode occurs but the LLM annotator can't reliably detect it," which undermines the practical utility of the taxonomy for quality assurance.

The Paper Does Not Demonstrate That MAST-Guided Diagnosis Leads to Better MAS Than Alternative Development Approaches

The assumption or constraint. The paper's central value proposition is that MAST enables systematic failure diagnosis, which in turn guides more effective MAS development. The intervention case studies (Appendix H) are designed to demonstrate this: use MAST to identify dominant failure modes in AG2/MathChat and ChatDev, implement targeted fixes, and observe improvements (Table 5) with before-and-after failure mode distributions (Figures 10 and 11).

However, the paper does not compare MAST-guided development against any alternative development methodology. The interventions are designed based on MAST analysis, but there is no control condition showing what would happen if the same developer time and expertise were applied to improving the systems without MAST. Would a developer reading through traces manually and applying their own judgment have achieved the same or better improvements? Would simply using a stronger base model (e.g., GPT-4o instead of GPT-4 for AG2) without any architectural changes have outperformed the MAST-guided architectural intervention? The paper's comparison of GPT-4 vs. GPT-4o in AG2 (Table 5) shows that the base model switch alone (GPT-4 → GPT-4o without any MAST-guided intervention) yields minimal improvement for the baseline system (84.75% → 84.25%, actually a slight decrease), but this isn't framed as a comparison between "change the model" and "apply MAST-guided fixes" as competing development strategies.

The consequence. The paper's claim that MAST-guided development is valuable rests on the assumption that the improvements achieved in the case studies (e.g., +15.6% on ChatDev ProgramDev-v0) would not have been achieved — or would have been achieved less efficiently — without MAST. If an experienced MAS developer could achieve the same improvements by reading a few traces and applying intuition, then MAST's value is primarily in standardizing and communicating failure diagnoses (which is still useful) rather than in enabling better interventions (which is a stronger claim). The paper doesn't provide evidence to distinguish between these interpretations.

More fundamentally, the case studies demonstrate only two rounds of intervention each (prompt improvement, then topology change). Both leave substantial failure rates (AG2 MathChat with GPT-4o at ~89% — meaning 11% of math problems still fail; ChatDev ProgramDev-v0 at 40.6% — meaning 59.4% of programming tasks still fail). The paper interprets this as evidence that "more substantial improvements are needed" and "achieving high reliability may require combinatorial changes" — but an alternative interpretation is that MAST-guided diagnosis, while providing granular failure visibility, doesn't lead to more effective interventions than unguided development. The paper cannot distinguish these interpretations because it doesn't compare against an unguided baseline.

What evidence exists in the paper. The intervention studies (Appendix H, Table 5) report accuracy improvements for MAST-guided changes, but there is no control condition. The failure mode distribution comparisons (Figures 10 and 11) show that interventions changed failure profiles, but don't establish that MAST was necessary to identify which failures to target — a developer without MAST might have noticed that AG2 agents were getting stuck in loops and added a termination condition, or that ChatDev's CPO was overstepping its role and enforced hierarchy, arriving at the same interventions through unaided observation. The paper does not report a "developer time to diagnosis" metric comparing MAST-assisted vs. unassisted debugging, nor does it report whether the identified failure modes were obvious to the developers before consulting MAST.

The paper's strongest implicit argument for MAST's necessity comes from the complexity of the traces — averaging over 15,000 lines — making manual diagnosis infeasible at scale. However, the case studies involve reading and analyzing traces for specific systems, and the paper doesn't quantify how much time MAST saved relative to unassisted trace analysis. The LLM annotator costs $1.80/trace, but a developer reading a 15,000-line trace might take 30–60 minutes, costing far more in labor — so MAST is clearly more scalable for large-scale analysis. But for the specific task of designing two rounds of interventions on one system, a developer might read 10–20 traces manually in a few hours and arrive at similar conclusions, making MAST's marginal value unclear in the small-scale development context that the case studies represent.

Mitigation status. The paper partially mitigates this by framing MAST as infrastructure for the research community rather than claiming it is the uniquely optimal development methodology. The abstract states that MAST-Data is designed "to outline the failure dynamics in MAS for guiding the development of better future systems" and that the contributions enable "widespread research and development in MAS" — the emphasis is on enabling future work, not on proving MAST's superiority over alternatives. However, the intervention case studies are presented as demonstrations of MAST's practical utility, and the absence of a control condition weakens their evidentiary force. A stronger design would compare MAST-guided development against an alternative methodology (e.g., "developer reads 20 traces and proposes fixes" vs. "MAST annotates 100 traces and identifies top failure modes") with metrics for both outcome quality (task success improvement) and process efficiency (developer time, annotation cost, number of iterations to reach a given improvement). The paper doesn't propose or conduct such a comparison.

The Taxonomy Prioritizes Failures That Are Addressable Through System Design, Potentially Obscuring Model-Centric Failure Modes

The assumption or constraint. The paper makes an explicit methodological choice to focus on "identifying failure patterns where improvements in system design, agent coordination, and verification can offer room to improve the reliability of MAS, often independently of or complementary to advancements in the base models themselves" (Section 4, FC1 discussion). The authors acknowledge that "some MAS failures can stem from fundamental limitations of current LLMs, such as hallucination or instruction following" but deliberately exclude or de-emphasize these in the taxonomy. The 14 failure modes in MAST are framed as addressable through engineering changes — better prompts, clearer role specifications, improved verification procedures, restructured workflows — rather than through better base models.

This framing is reflected in the taxonomy's structure. FM-1.1 (Disobey Task Specification) is categorized under "System Design Issues" with the argument that it can stem from "flaws in MAS design regarding agent roles and workflow, poor user prompt specifications, or limitations of the underlying LLM" — but the paper argues that "a well-designed MAS should interpret high-level objectives with minimal but clear user input to mitigate the impact" of prompt quality and model limitations. This implicitly treats model-caused specification failures as a system design problem: if the system were designed better, it would compensate for model weaknesses.

The consequence. The taxonomy may systematically undercount or misclassify failures whose root cause is genuinely model capability limitations rather than system design flaws. A base LLM that fundamentally lacks the reasoning capacity to solve a particular math problem will produce incorrect answers regardless of how well the MAS is designed. MAST might label this as FM-2.6 (Reasoning-Action Mismatch) or FM-3.3 (Incorrect Verification), implying that better agent coordination or verification could have prevented the failure — but no amount of coordination or verification can compensate for the model simply being unable to solve the problem. The taxonomy's design bias toward actionable engineering fixes means it may systematically attribute failures to correctable system flaws when the true bottleneck is model capability.

This has practical consequences for resource allocation. If a development team uses MAST to diagnose their MAS and finds high rates of FC1 and FC2 failures, MAST's framing encourages them to invest in prompt engineering and workflow redesign. But if those failures are actually driven by the base model's instruction-following limitations, the investment in system design may yield diminishing returns while a switch to a stronger base model would have produced larger gains. The paper's own data shows that model substitution (GPT-4o vs. Claude-3.7-Sonnet in MetaGPT, Figure 8) changes failure profiles substantially — GPT-4o reduces FC1 failures by 39.7% — suggesting that model capability is a major driver of the very failure modes that MAST categorizes as "system design issues."

What evidence exists in the paper. The comparative analysis in Appendix F (Figures 8 and 9) provides indirect evidence of this tension. The fact that switching from Claude-3.7-Sonnet to GPT-4o in MetaGPT reduces FC1 failures by 39.7% suggests that a non-trivial fraction of "system design" failures are actually model-dependent — if they were purely architectural, the model switch shouldn't matter. Similarly, the open-source model comparison (Table 6 in Appendix I) shows that CodeLlama-7B exhibits dramatically more FC1 and FC2 failures than Qwen2.5-Coder-32B or the closed-source models, further suggesting that model capability strongly influences failure modes categorized under system design and coordination. The paper does not provide a decomposition analysis estimating what fraction of failures in each category are attributable to model capability vs. system design — such an analysis would require controlled experiments varying model and architecture independently and measuring interaction effects, which are not conducted.

The paper's own Insight 1 (Section 4, FC1) states that "MAS failure is not merely a function of challenges in the underlying model; a well-designed MAS can result in performance gain when using the same underlying model." The emphasis on "not merely" acknowledges that model limitations play a role, but the taxonomy's structure and the paper's rhetorical emphasis on system design primacy may lead readers to underestimate the model's contribution. The +15.6% improvement from ChatDev topology changes (Table 5) is significant but leaves 59.4% failure rate — the majority of failures persist, and the paper does not investigate whether those residual failures are addressable through further design changes or whether they represent a model capability ceiling.

Mitigation status. The paper is transparent about its focus on system-addressable failures and explicitly acknowledges the role of model limitations. Section 4 states: "In developing MAST, we focus on identifying failure patterns where improvements in system design, agent coordination, and verification can offer room to improve the reliability of MAS." This is a scope limitation rather than an oversight. However, the paper does not provide guidance on how to distinguish model-caused failures from design-caused failures in practice, nor does it discuss how MAST should be used in conjunction with model capability evaluation. A practitioner whose MAST analysis reveals high FC1 failures faces an ambiguous signal: invest in prompt engineering and workflow redesign (per MAST's framing), upgrade to a stronger model (per the comparative evidence that model choice matters), or both. The paper provides no framework for making this resource allocation decision, and its emphasis on system design solutions may bias practitioners toward design interventions when model upgrades would be more cost-effective — or vice versa, depending on their prior beliefs.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper fundamentally reframes MAS reliability from a model evaluation problem into a systems engineering discipline. Prior to this work, the dominant diagnostic when a multi-agent system failed was to attribute the failure to the underlying LLM's limitations — insufficient reasoning capability, poor instruction following, hallucination. Improvement strategies followed naturally: wait for better models, fine-tune on domain data, or scale up inference compute. The field's intellectual center of gravity was model-centric.

MAST shifts that center of gravity toward organizational design. The paper's empirical evidence is direct and difficult to dismiss: the same model (GPT-4o) used in two different MAS architectures on the same benchmark (ProgramDev-v2) exhibits a 60–68% difference in failure rates for specification and coordination failures (Figure 9, Appendix F). If failures were primarily driven by model capability, architecture choice shouldn't produce variations of this magnitude — both systems use identical inference engines. The implication is inescapable: MAS architecture is a first-order determinant of reliability, not a secondary optimization on top of model quality. This is not an incremental refinement of existing knowledge; it is a conceptual reorientation that changes what questions the field should be asking. Instead of "how do we make LLMs better at multi-agent coordination?", the question becomes "what organizational structures, communication protocols, and verification mechanisms produce reliable collective behavior from imperfect individual agents?"

The paper also resolves a latent tension in the MAS literature that parallels the Anna Karenina principle quoted in the paper's epigraph. Prior work oscillated between enthusiasm for multi-agent collaboration (diverse reasoning, task decomposition, specialized ensembling) and skepticism grounded in empirical results showing minimal gains over single-agent baselines or best-of-N sampling. These positions were not contradictory — they reflected different systems tested on different benchmarks with different implicit failure dynamics. MAST provides the diagnostic vocabulary that explains why some MAS work and others don't: a well-designed system like MetaGPT suppresses FC1 and FC2 failures through SOP-driven role adherence but leaves FC3 verification as a vulnerability; a differently-designed system like ChatDev invests in explicit verification phases that reduce FC3 failures at the cost of weaker specification enforcement. The taxonomy makes these tradeoffs visible, measurable, and comparable for the first time, converting anecdata into engineering evidence.

The paper's most subversive finding — that verification failures appear even in successful traces (Table 7, Appendix J.1: FM-3.3 Incorrect Verification occurs in 20% of ChatDev successful traces and 16.7% of MetaGPT successful traces) — has implications beyond MAS. It suggests that current approaches to LLM-based verification are fundamentally unreliable in a way that aggregate accuracy metrics conceal. A system that produces correct outputs despite broken verification is a system operating without a safety net: it succeeds by luck in the current distribution but provides no guarantee on the next input. This finding should give pause to anyone deploying LLM-based agents in settings where correctness matters and cannot be externally verified — which is to say, most real-world deployments. The implication is that verification cannot be treated as a post-hoc check; it must be architected into the system's core workflow with the same rigor as the task-execution logic itself.

The paper also provides empirical grounding for what was previously an intuitive argument in the "AI agents that matter" discourse (Kapoor et al., 2024; Anthropic's building-effective-agents guidance). Those works argued that complexity can hinder practical adoption — keep agents simple, avoid over-engineering. MAST provides the mechanism: complexity creates more surface area for the specific failure modes cataloged in FC1 (step repetition, loss of conversation history, unawareness of termination conditions) and FC2 (reasoning-action mismatch, task derailment, information withholding). The argument shifts from "complexity is bad" (a heuristic) to "complexity increases the probability of specific, diagnosable failure patterns" (an engineering claim with empirical support). This makes the simplicity argument testable and refutable rather than a matter of taste: a system designer can now measure whether added complexity increases the target failure modes or whether the complexity buys sufficient coordination benefits to justify the increased failure risk.

Finally, the paper's methodological contribution — the Grounded Theory derivation of the taxonomy followed by LLM-as-Judge scaling — establishes a template for how the field can build shared diagnostic infrastructure. The sequence is: (1) invest substantial expert effort in qualitative analysis of a diverse corpus to derive categories inductively, (2) validate those categories through rigorous inter-annotator agreement studies to ensure they are consistently applicable, (3) calibrate an automated annotation pipeline against the human consensus to enable scalable application, and (4) release the taxonomy, dataset, and tools as open infrastructure. This template is applicable beyond MAS failures — it could be used to build taxonomies for single-agent failure modes, human-AI interaction breakdowns, or safety-relevant agent behaviors. The paper thus contributes not just a specific taxonomy but a methodology for building taxonomies in domains where failure modes are complex, emergent, and lack prior standardized definitions.

Follow-Up Research This Work Enables

Causal isolation of architectural components through within-framework ablation. The paper's comparative analysis (MetaGPT vs. ChatDev in Figure 9) attributes performance differences to high-level design patterns — SOPs vs. explicit verification phases — but these attributions are post-hoc interpretations of holistic system comparisons. A natural follow-up would systematically ablate individual architectural components within a single framework and measure the resulting failure mode distribution shifts using MAST. For example: take ChatDev, remove the CEO approval requirement for conversation termination (hypothesized to increase FM-1.2 Disobey Role Specification), run on ProgramDev-v2 with GPT-4o, annotate with the LLM annotator, and compare failure profiles against the baseline. Alternatively: add explicit SOP prompts to ChatDev agents while keeping the rest of the architecture identical, and measure whether FC1 failures decrease to MetaGPT-like levels. This would transform the paper's correlational evidence ("architectures with feature X have fewer failures of type Y") into causal evidence ("adding feature X to a system reduces failure type Y by Z%"). The MAST annotation pipeline makes this feasible at scale: each ablation variant requires running the MAS on 100 tasks (ProgramDev-v2) and annotating the traces, costing roughly (100 × generation cost) + (100 × $2.13 for ChatDev annotation) — expensive but within research budget range for a systematic study of 5–10 ablations.

Longitudinal study of failure mode evolution with model scaling. The paper demonstrates that model substitution (GPT-4o vs. Claude-3.7-Sonnet in MetaGPT, Figure 8) changes failure profiles, but provides only a single cross-sectional comparison between two models of roughly similar capability. The conjecture that "improvements in base model capabilities will be insufficient to address the full MAST" (Section 5.3) requires testing across a capability gradient. A follow-up study would evaluate a single MAS architecture (e.g., ChatDev or MetaGPT) on a fixed benchmark (ProgramDev-v2) using a sequence of models spanning an order of magnitude in capability: GPT-4o-mini → GPT-4o → a hypothetical GPT-5 or Claude-4 when available, plus open-source models at multiple scales (Llama-3-8B, Llama-3-70B, Qwen2.5-Coder-7B, Qwen2.5-Coder-32B). For each model, compute the full MAST failure mode distribution. The key question: do certain failure modes (e.g., FM-3.3 Incorrect Verification, FM-1.5 Unaware of Termination Conditions) asymptote at non-zero rates even as model capability increases, indicating fundamental architectural bottlenecks? Or do all failure modes trend toward zero, suggesting that the paper's emphasis on system design is overstated for sufficiently capable models? The MAST-Data framework makes this study newly tractable because the annotation cost is fixed per trace regardless of model — the LLM annotator's o1-based pipeline can label traces from any underlying model equally. The paper's existing data (GPT-4o vs. Claude, Qwen vs. CodeLlama) provides the first two points on these curves; extending to 2–3 more model tiers would reveal whether the trends are asymptotic or monotonic.

Input-conditioned failure analysis: when do specific failure modes trigger? The paper's per-trace annotation aggregates failure modes across all inputs without analyzing which task characteristics trigger which failures. The ProgramDev-v2 dataset (100 programming tasks including game implementations, calculators, data structures) could be annotated with task-level metadata: task category, prompt ambiguity (does the specification leave design choices unspecified?), constraint complexity (how many explicit requirements?), subtask dependency structure (does the task require sequential dependencies between components?). With these metadata and the existing per-trace MAST labels, a follow-up analysis could answer questions like: Is FM-1.1 (Disobey Task Specification) uniformly distributed across tasks, or does it concentrate in tasks with ambiguous specifications? Does FM-2.4 (Information Withholding) occur primarily in tasks requiring cross-component coordination? The paper's finding that some failure modes are "more fatal" than others (FM-1.5 and FM-2.4 appear almost exclusively in failed traces, Table 7) could be refined: are these modes fatal conditional on task type, or universally? If FM-2.4 is fatal only when it occurs in coordination-heavy tasks but benign in simple tasks, intervention priorities would shift. The existing MAST-Data provides the failure labels; what's needed is task-level feature annotation, which could be done automatically (LLM-based task classification) at negligible additional cost.

Can MAST predict failure before execution? toward static analysis of MAS designs. The paper's annotation pipeline operates on execution traces — it diagnoses failures post-hoc. A more ambitious follow-up would ask: can we predict which failure modes a given MAS design will exhibit without running it? This would require feature-engineering MAS architectures: number of agents, topology type (hierarchical, star, assembly line), presence/absence of explicit verification agents, conversation termination protocol (any-agent vs. designated-agent), SOP encoding in prompts, maximum context window utilization, etc. With a dataset of ~200 MAS configurations (varying these features systematically within a framework like AG2 that supports flexible topology configuration), each evaluated on a common benchmark and annotated with MAST, one could train a predictor that maps architectural features to expected failure mode distributions. The existing MAST-Data provides a proof-of-concept: the cross-sectional comparisons (Figures 8 and 9) show that architecture features correlate with failure profiles. Extending this to a systematic design space exploration would transform MAST from a diagnostic tool into a design tool — developers could evaluate candidate architectures for expected failure profiles before implementation, analogous to how static analysis tools predict bug-prone code patterns. The key enabler is the LLM annotator's scalability: generating and annotating 200 MAS variants × 100 tasks each = 20,000 traces would cost approximately $36,000 in annotation (at$1.80/trace average) plus generation costs — expensive but feasible for a dedicated research project, and dramatically cheaper than equivalent human annotation.

Stress-testing MAST on adversarial or competitive multi-agent settings. The paper's frameworks all involve collaborative agents working toward a shared goal with predefined roles. MAST was derived from and validated on cooperative systems. A critical stress-test would apply MAST to multi-agent settings with different incentive structures: competitive systems (agents with opposed goals, e.g., negotiation or debate), mixed-motive systems (partial goal alignment, e.g., resource allocation with private preferences), or adversarial settings (one agent attempting to exploit or deceive others). The paper's FC2 (Inter-Agent Misalignment) currently covers failures of coordination in cooperative contexts — information withholding, ignoring input, conversation reset. In competitive settings, these same behaviors might be strategic rather than failure-driven: withholding information is rational if agents have opposed goals. Would MAST's definitions need extension to distinguish between "failure to coordinate" and "strategic non-coordination"? Would new failure modes emerge — strategic deception, collusion, goal hijacking? The LLM annotator could be applied to existing competitive MAS benchmarks (e.g., negotiation datasets, debate environments) with minimal modification; the key question is whether human IAA on such traces would achieve comparable κ to the cooperative setting (κ = 0.88), or whether the taxonomy's assumptions about "correct" agent behavior break down when correctness is relative to agent-specific goals rather than a shared task objective. Negative results (κ dropping substantially in competitive settings) would refine our understanding of MAST's scope; positive results would dramatically expand its applicability.

Cost-aware failure diagnosis: optimizing the tradeoff between diagnostic depth and annotation cost. The paper reports that annotation costs vary substantially by trace length ($0.37/trace for AppWorld vs.$4.14/trace for OpenManus, Table 9) and that generation costs (the LLM API calls made by the MAS itself) are not reported but likely dominate total diagnosis cost. A practical follow-up would investigate whether MAST-level failure diagnosis can be performed on partial traces — e.g., only the first N conversation turns, or only the final outputs, or only a summary generated by a cheaper model — at substantially reduced cost, and what the accuracy tradeoff is relative to full-trace annotation. If partial-trace annotation achieves κ = 0.70 with human experts while reducing cost by 5×, it would make continuous MAST-based monitoring feasible for production MAS where per-trace costs must be minimized. The experiment: for each of 50 traces from MAST-Data-human, create truncated versions (25%, 50%, 75% of conversation turns), run the LLM annotator on each, and compute per-mode agreement with the full-trace human consensus. Identify which failure modes are detectable early in execution (e.g., FM-1.3 Step Repetition may be visible within a few turns) vs. those requiring full traces (e.g., FM-3.1 Premature Termination requires seeing the termination point). This would produce practical guidance for deployment: "to monitor for FC1 failures, annotate only the first 25% of conversation turns (saving 75% of annotation cost); to monitor for FC3 failures, full traces are necessary." The paper provides the human-annotated gold standard (MAST-Data-human, 21 traces) as a calibration set; a larger human-annotated set (50–100 traces) would be needed for robust measurement.

Practical Applications and Downstream Use Cases

MAS development debugging dashboard for framework teams. The paper's case studies (Appendix H) demonstrate MAST's utility as a debugging tool: run the MAS on a benchmark, annotate traces with the LLM annotator, and receive a 14-dimensional failure mode vector per trace plus aggregate statistics per failure category. This can be productized as a development dashboard integrated into MAS frameworks. The agentdash Python library (Appendix C, pip install agentdash) already provides the core annotation API. A framework team (e.g., the ChatDev or MetaGPT maintainers) could integrate this into their CI/CD pipeline: every time a new commit is pushed, run the MAS on a fixed benchmark suite (e.g., ProgramDev-v2, 100 tasks), annotate all traces, and compare failure mode distributions against the previous commit. A regression in FM-1.2 (Disobey Role Specification) from 1% to 5% would immediately flag that a prompt change accidentally weakened role enforcement, even if aggregate task success didn't change. The paper's findings that aggregate accuracy can mask shifting failure profiles (AG2 prompt intervention in Figure 10: accuracy improved but FC2 failures increased from 692 to 796) demonstrate exactly why such monitoring is valuable — without MAST-level visibility, a developer might celebrate a +5% accuracy gain while unknowingly introducing a coordination regression that will manifest later on different input distributions. The cost is manageable: annotating 100 ChatDev traces costs approximately $213 (at$2.13/trace) plus generation costs, which for a well-resourced framework team is a negligible fraction of development costs and dramatically cheaper than the person-weeks of manual debugging that current practice requires.

Model selection for MAS deployments with failure profile matching. Organizations deciding which LLM to use in their MAS currently rely on single-agent benchmarks (MMLU, HumanEval, etc.) or aggregate MAS success rates. The paper's comparative analysis (Figure 8) shows that model choice affects failure profiles in non-uniform ways: GPT-4o reduces FC1 (System Design Issues) by 39.7% relative to Claude-3.7-Sonnet in MetaGPT, but FC3 (Task Verification) failures are nearly identical (77 vs. 83). A deployment team whose application is verification-critical (e.g., code generation where incorrect outputs have high downstream cost) therefore learns that upgrading from Claude to GPT-4o provides minimal benefit for their primary risk category, and should instead invest in verification infrastructure regardless of model choice. Conversely, a team whose application is specification-heavy (e.g., generating reports from structured templates where role adherence matters) would see substantial gains from the GPT-4o switch. The practical workflow: for each candidate model, run the MAS on a representative benchmark, annotate with MAST, compute the failure profile, and select the model whose profile best matches the deployment's risk priorities. The paper provides the template (MetaGPT on ProgramDev-v2 with GPT-4o vs. Claude-3.7-Sonnet, Figure 8) and the tooling (LLM annotator at $2.45/trace for MetaGPT); what's needed is the organization-specific benchmark and risk weighting. For a team spending$50,000/month on LLM API costs for their MAS, a one-time $500–$2,500 investment in MAST-based model selection (200–1,000 annotated traces) could identify that a cheaper model is actually better for their specific failure risk profile — or that the expensive model is worth it but only if paired with a verification architecture upgrade.

Continuous verification monitoring for production MAS. The paper's finding that verification failures (FC3) appear even in successful traces (Table 7: FM-3.3 in 16.7% of MetaGPT successful traces) means that deploying a MAS without monitoring its verification behavior is flying blind — the system can produce correct outputs today while its verification mechanisms are silently broken, guaranteeing failures on future inputs that happen to trigger the unverified edge cases. A production MAS (e.g., an automated code review system or a customer-support routing agent) could integrate MAST-based monitoring as a runtime safety check: sample 5% of production traces, run the LLM annotator, and track FC3 failure mode rates over time. If FM-3.2 (No or Incomplete Verification) spikes from a baseline of 8% to 20%, it signals that something has changed — perhaps a model update weakened the verifier agent's attention to detail, or a new class of user inputs triggers superficial verification checks that pass on simpler inputs. This is analogous to error budget monitoring in site reliability engineering: you don't need to catch every failure, you need to detect when the failure rate changes. The cost is feasible for moderate-scale deployments: at 10,000 production traces per day, a 5% sampling rate = 500 traces/day × $1.80 average annotation cost =$900/day for monitoring. For a system whose failures cost substantially more than $900/day in business impact, this is easily justified. The paper's open-source LLM annotator pipeline makes this deployable today; the missing component is latency optimization (batch annotation, cheaper annotator models for screening) that the paper doesn't explore but that a production engineering team could implement on top of the released infrastructure.

When to Prefer This Method

The paper does not position MAST as a method to be preferred over specific named alternatives for solving MAS failures. MAST is a diagnostic and analytical framework — it tells you why your system failed, not how to fix it. The paper does, however, implicitly distinguish between two development methodologies: (1) MAST-guided development, where systematic failure mode analysis informs targeted interventions (as demonstrated in the Appendix H case studies), and (2) aggregate-metric-driven development, where only task success rates guide improvement efforts. The tradeoff between these is clear from the paper's evidence: aggregate metrics can be actively misleading as development signals. The AG2 prompt intervention (Figure 10, Table 5) improved accuracy from 84.75% to 89.75% while increasing FC2 (Inter-Agent Misalignment) failures from 692 to 796 and FC3 (Task Verification) failures from 305 to 335 — the system got better at math while getting worse at coordination and verification. A developer guided only by accuracy would have shipped this change; a developer guided by MAST would have seen the regression and investigated before deployment.

The decision rule the paper supports is therefore:

• Prefer MAST-guided development when reliability requirements are stringent, failure costs are high, or the system will encounter diverse inputs at deployment that may stress different failure modes than the evaluation benchmark. The cost is the annotation overhead ($1.80/trace average + generation costs) and the development time to interpret failure profiles and design targeted interventions.
• Aggregate-metric-driven development may suffice when the system operates in a narrow, well-characterized input distribution where the benchmark accurately represents deployment conditions, and when failures have low individual cost (e.g., batch processing where occasional wrong outputs can be manually filtered). In this case, MAST's additional diagnostic granularity may not justify its cost — if 89% accuracy is acceptable and the system won't encounter inputs that trigger the regressed failure modes, the developer doesn't need to know that FC2 failures increased.

The paper does not articulate this tradeoff explicitly, but the evidence in Figures 10–11 and Table 5 makes it the natural operational conclusion. The decision rule is not about MAST vs. another method but about the depth of diagnostic signal needed for the deployment context.