The Collaboration Gap — Paper Graph

🎯 Pitch

Models that ace tasks solo can completely collapse when paired with an identical partner, revealing a stark 'collaboration gap' that solo benchmarks miss. Strikingly, just letting the stronger model speak first largely closes this gap, motivating a simple 'relay inference' hand-off strategy.

1. Executive Summary

This paper studies how leading language models perform when required to collaborate on a shared task, using a novel maze-solving benchmark that distributes partial information between two agents and requires them to negotiate moves under minimal output-format constraints across 32 open- and closed-source models. The central empirical finding is a "collaboration gap": models that achieve high solo performance on the same mazes exhibit a sharp performance drop when paired with an identical copy of themselves, with distilled models disproportionately affected (e.g., gpt-5-nano and gemini-2.5-flash-lite showing near-complete collapse in certain pairings). The paper further identifies that interaction order critically shapes outcomes — the stronger agent initiating the dialogue yields significantly better results than the weaker agent leading — and proposes "relay inference", a strategy where a strong model primes the first few turns before handing off to a weaker model, which recovers much of the lost collaborative performance. The authors demonstrate that a single opening message from a strong model can boost weaker models' weighted outcomes substantially (e.g., lifting gpt-4.1-mini homogeneous collaboration from approximately 0.39 to above 0.70), establishing that collaborative capability represents a distinct axis of model competence that current training paradigms fail to capture even when the underlying task is well within a model's solo ability.

2. Context and Motivation

The Core Problem: We Don't Know If AI Agents Can Collaborate

The fundamental question this paper tackles is straightforward but surprisingly unanswered: when two independently developed AI agents need to work together on a shared task without a predefined communication protocol, can they do so effectively? This matters because the trajectory of AI deployment is increasingly pointing toward agent-based systems where multiple specialized agents — built by different organizations, with different capabilities, information, tools, and privileges — must coordinate in real time to accomplish tasks that no single agent can complete alone.

The gap this paper identifies is not a technical oversight but a blind spot in how we evaluate and train language models. Current LM evaluation benchmarks overwhelmingly measure individual capabilities: can the model solve math problems, write code, summarize documents, or answer factual questions? These benchmarks treat the model as a solo performer operating on complete information. But the emerging agentic paradigm — where agents are deployed as part of multi-agent systems — requires a fundamentally different set of skills: grounding shared understanding, resolving conflicting information, coordinating turn-taking, recognizing when to defer to a partner's expertise, and maintaining coherent state across extended multi-turn interactions.

The paper's central concern is that we are building toward a future of multi-agent systems without any systematic evidence that current LMs possess the collaborative capabilities required to make those systems work. Section 1 frames this starkly:

"do our current training strategies create agents capable of dynamic collaboration?"

This is not a hypothetical worry. The paper catalogs massive ongoing investments in multi-agent infrastructure — from communication protocols like Anthropic's MCP, Google's A2A, and Besen's ACP (Section 1) to agent solutions from major providers (Salesforce Agentforce, OpenAI Operator, Google's Gemini agents) to venture capital flowing into agent startups. Governments and corporations are building adoption programs for agentic AI (Capgemini, EY, Gartner). Yet the question of whether the underlying models can actually collaborate effectively remains unexamined at scale.

Why This Problem Matters: Beyond Human-AI Collaboration

The paper draws a crucial distinction between human–AI collaboration and AI–AI collaboration — two problems that the field has largely conflated (Section 1). Human–AI collaboration research has focused heavily on optimizing AI's role as a "helpful assistant" (Bai et al., 2022), where the human provides goals, context, and judgment while the AI executes. This is a fundamentally asymmetric relationship: the human leads, the AI follows. The alignment techniques developed for this paradigm — instruction following, helpfulness, harmlessness — address a scenario where one party has complete authority and the other is subservient.

AI–AI collaboration is qualitatively different. Two agents enter as peers (or near-peers), each with partial information and no predetermined hierarchy. Neither has the authority to unilaterally decide. Success requires:

Mutual grounding: establishing shared reference points when each agent sees different aspects of the problem.
Conflict resolution: reconciling contradictory information without a ground-truth arbiter.
Dynamic role negotiation: determining who proposes moves, who evaluates them, and when to defer — all without preset turn-taking rules.
Theory of mind: reasoning about what the other agent knows, believes, and intends, based on their partial view of the problem.

The paper argues that progress in human–AI collaboration does not automatically transfer to AI–AI settings because these skills are not exercised when the AI always plays the assistant role. Section 5 quotes Barbara Grosz's 1996 presidential address:

"capabilities needed for collaboration cannot be patched on, but must be designed in from the start"

The implication is that collaboration is not an emergent property that will naturally arise as models become more capable at solo tasks — it must be explicitly targeted in training. The paper sets out to test whether this is true.

The Small Model Paradox

The paper's motivating problem is sharpened by a recent argument from Belcak et al. (2025), discussed in Section 5: small language models (SLMs) are the natural future of agentic AI because they are sufficiently capable for many specialized tasks, more operationally practical than large models, and dramatically cheaper. If this argument is correct, the deployment landscape will consist of many specialized SLMs — some handling scheduling, others managing databases, others interfacing with APIs, etc. — that must collaborate when tasks span multiple domains.

But specialization creates a collaboration burden: the more specialized an agent becomes, the more likely it is to encounter situations outside its expertise, and the more it must rely on collaboration with other agents. If smaller, distilled models lose collaborative capability disproportionately — a finding the paper previews and then demonstrates empirically (Section 4.2, Figure 4.1) — then the SLM-centric future hits a wall: the models are individually competent but collectively dysfunctional.

This is what makes the problem practically urgent. The industry is actively building toward a multi-agent architecture (evidenced by the protocol efforts and product launches) while simultaneously pushing toward smaller, cheaper models for deployment. The paper positions itself as measuring whether these two trends are compatible.

Where Prior Work Falls Short

The paper identifies several specific limitations in how the field has approached multi-agent evaluation and collaboration:

1. Prior multi-agent benchmarks conflate collaboration with individual capability. Most existing benchmarks for multi-agent systems (the paper cites Chang et al., 2024 as a survey) evaluate integrated systems where multiple agents are deployed together, but the tasks do not isolate collaboration as a distinct skill. A system might succeed because individual agents are powerful enough to solve the task independently, not because they collaborated effectively. The paper argues that to measure collaboration, you need a task where (a) solo performance can be measured as a baseline, (b) collaboration requires skills beyond what solo performance demands, and (c) the information asymmetry forces genuine interdependence.

2. Tightly orchestrated systems avoid the core challenge. The paper acknowledges that multi-agent systems with predetermined communication protocols (MCP, A2A, ACP) or centrally orchestrated architectures (Guo et al., 2024) have shown promise — including the authors' own prior work on Magentic-One (Fourney et al., 2024). But these systems solve the collaboration problem by eliminating it: they specify exactly how agents should communicate, what formats to use, and who controls what. This works for closed-world deployments but fails in open-world settings where agents encounter novel partners, unexpected task decompositions, or communication partners who don't share the same protocol. The paper quotes Pan et al. (2025) who found that "failures arising from ineffective communication, poor collaboration, [and] conflicting behaviors among agents" represent a critical failure mode even in tightly orchestrated systems, suggesting the problem is deeper than protocol design.

3. Emergent behavior studies lack measurable collaboration outcomes. The paper acknowledges work like Park et al. (2023)'s Generative Agents, which simulates emergent social behaviors in multi-agent settings, as interesting but insufficient. These studies observe what agents do but don't provide controlled, measurable outcomes that isolate collaborative capability. An agent might appear to collaborate because its role-playing instructions happen to produce collaborative-seeming behavior, not because it possesses generalizable collaborative skills.

4. Human–AI collaboration studies measure different skills. Recent work by Wu et al. (2025) and Zhou et al. (2025) has used reinforcement learning to optimize LMs' collaborative capabilities, but focused on human–AI settings where the AI is explicitly in a helpful assistant role. In these setups, the human provides goals and evaluation, and the AI's task is to be useful and responsive — an asymmetric relationship that doesn't test whether the AI can initiate coordination, challenge incorrect information from a peer, or negotiate shared understanding when both parties have partial views.

5. Heterogeneous agent studies are rare and limited. The paper notes that most multi-agent studies explore "homogeneous interactions, where all agents are powered by the same underlying model, or a limited selection of heterogeneous models, likely limiting their generalization" (Section 1, citing Wynn et al., 2025). In the real world, agents will come from different model families trained by different organizations with different capabilities. Understanding cross-family collaboration dynamics — do GPT agents collaborate better with other GPT agents? Can a strong model compensate for a weak partner? — requires systematic heterogeneous evaluation at scale, which prior work hasn't done.

6. Negotiation benchmarks introduce confounding incentives. The paper's authors previously developed a negotiation-based evaluation for agent collaboration (Davidson et al., 2024). While negotiations "have important collaborative elements," they also introduce "incentives to withhold information and/or deceive." This confounds the measurement: poor performance might reflect strategic behavior (rational withholding) rather than collaborative inability. The paper's maze-solving benchmark is designed to strip away these strategic confounds by making the incentives purely cooperative — both agents share the same goal and can only succeed together.

How This Paper Positions Itself

The paper positions itself not as proposing a new collaboration method or training technique, but as providing the first systematic empirical evidence that a collaboration gap exists and characterizing its properties. The contributions are primarily diagnostic rather than prescriptive.

The benchmark as a measurement instrument. The paper designs its collaborative maze-solving benchmark (Section 2) to satisfy four specific requirements that address the limitations above:

Isolate collaborative capabilities: by measuring solo performance on the same task (both with full information and distributed information), the benchmark creates a baseline against which collaborative performance can be compared. Any drop when moving from solo to collaborative settings can be attributed to collaboration-specific demands rather than task difficulty.
Modulate problem complexity: maze parameters (size, wall density, path length) can be adjusted to test whether the collaboration gap scales with task difficulty or remains constant.
Enable scalable automated grading: unstructured outputs (no format constraints) preserve ecological validity — real agents won't conform to a single output schema — but require a grading mechanism that can handle diverse solution representations. The paper uses an LM-based grader with a multi-scheme normalization pipeline (described in Section 2.1 and ablated in Appendix D).
Impose no output-format constraints: this is a deliberate design choice that contrasts with benchmarks specifying JSON outputs or fixed communication protocols. The absence of format constraints forces agents to invent their own coordination mechanisms, which is exactly what open-world deployment will require.

The evaluation as an existence proof. The paper explicitly frames its results as an "existence proof that effective collaboration represents a distinct axis of capability that current training strategies fail to capture" (Section 1). This is a carefully chosen framing: the goal is not to claim that all models fail at all collaborative tasks, nor to propose a fix, but rather to demonstrate that the gap is real, measurable, and large enough to warrant serious attention from the research community.

The relay inference contribution as a diagnostic tool. The relay inference experiments (Section 4.4) serve a dual purpose. Practically, they demonstrate a lightweight intervention (priming with a strong model's initial messages) that partially closes the collaboration gap. Diagnostically, they provide evidence about why the gap exists: if a single opening message from a strong model dramatically improves outcomes, the problem is not that weak models can't collaborate but that they fail to initiate effective collaboration — they don't establish the shared grounding and communication conventions that the strong model does. This points toward specific deficits (grounding failures, poor schema proposals) rather than a general inability to engage in multi-turn interaction.

The difficulty connection. The paper implicitly connects to the broader literature on when and why LLMs exhibit capability failures. Just as the "collaboration gap" paper from the reference example found that test-time compute strategies have difficulty-dependent effectiveness, this paper argues that collaboration failure is not uniform: it disproportionately affects distilled models (Section 4.2), depends critically on interaction order (Section 4.3), and can be mitigated by strategic priming (Section 4.4). This suggests that the gap is not a binary "can collaborate / can't collaborate" property but rather a continuous capability that varies across models and interaction patterns — opening the door to targeted interventions rather than requiring fundamental architectural changes.

3. Technical Approach

3.1 Reader Orientation

This is primarily an evaluation and measurement paper whose core idea is that effective AI–AI collaboration represents a distinct capability that is not automatically acquired through standard language model training, and that this capability can be systematically measured by comparing solo task performance against collaborative performance on the same underlying task under conditions of distributed, partial information. The paper builds a collaborative maze-solving benchmark where two agents each receive an incomplete view of a maze and must communicate in unstructured natural language to negotiate each move, and then uses this benchmark to evaluate 32 models across solo, homogeneous-collaborative, and heterogeneous-collaborative settings to quantify the size and nature of the resulting "collaboration gap."

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected by a sequential pipeline:

Maze Generator — produces a random maze instance and splits it into two partially-obfuscated copies, each missing roughly half the cells. This enforces information asymmetry that requires collaboration.
Two Agent Instances (powered by language models) — each receives one partial maze copy along with collaboration rules. They engage in multi-turn dialogue, taking turns proposing and agreeing on individual moves until reaching the goal or exhausting the turn budget.
Relay Inference Controller (optional) — in relay experiments, monitors the dialogue and swaps one agent for another after a predetermined number of turns $K$ , enabling measurement of priming and recovery effects.
Grader Language Model — processes the raw, unstructured dialogue transcript and extracts the agreed-upon route as a sequence of coordinates or directions, including inferring the coordinate schema (origin, orientation, notation) used by the agents.
Schema-Normalized Solution Checker — transforms the grader-extracted route through a large set of potential coordinate schema transformations (different origins, row/col conventions, indexing schemes) and checks each against the ground-truth maze to determine whether any interpretation yields a valid path from start to goal, producing a binary success/failure outcome and a continuous weighted outcome score.

Information flows as follows: a random maze is generated and split → two agents receive their partial copies → agents alternate sending messages in a conversation history → after termination (goal reached, wall hit, or turn limit), the full transcript is passed to the grader → the grader extracts candidate moves and infers the schema → the schema normalizer enumerates interpretations → the solution checker validates each interpretation against the ground-truth maze → final binary and weighted outcomes are recorded.

3.3 Roadmap for the Deep Dive

First, the maze generation and information distribution mechanism, since the distributed-information design is what creates the need for collaboration and separates collaborative capability from raw maze-solving ability.
Second, the agent interaction protocol — how agents communicate, what rules constrain them, and how turn-taking is structured without prescribing communication format — because this is where models succeed or fail at grounding, conflict resolution, and state tracking.
Third, the outcome metrics (binary success rate and weighted outcome) and the grading pipeline, since these define what "success" means and how unstructured dialogues are converted to quantitative measurements without human annotation.
Fourth, the experimental configurations: solo baselines, homogeneous collaboration, heterogeneous collaboration, and relay inference, including how each setting isolates a different aspect of collaborative capability and what comparisons they enable.
Fifth, the model selection strategy — which 32 models were chosen, why heterogeneous pairings were restricted to a subset, and how model families and distillation status are tracked — because model characteristics (size, distillation, system prompt flexibility) are central to explaining the collaboration gap.
Sixth, the relay inference mechanism, which is the paper's primary constructive contribution: how dialogue priming and recovery swap experiments are structured, what they measure, and why they provide diagnostic evidence about where in the collaboration process weak models fail.

3.4 Detailed, Sentence-Based Technical Breakdown

Maze Generation and Information Distribution

The maze task is designed to satisfy four requirements specified in Section 2.1: it must isolate collaborative capabilities (by having a measurable solo baseline), modulate problem complexity (via adjustable parameters), enable scalable automated grading (since thousands of transcripts must be evaluated), and impose no output-format constraints (preserving ecological validity). The maze generation procedure creates tasks where collaboration is necessary — neither agent can solve the maze alone because neither has the complete map.

Maze generation procedure. A maze is an $N \times N$ grid with designated start and goal positions, populated with path cells (traversable) and wall cells (impassable). The main experiments use $N = 6$ , a wall density of approximately 30%, and an average shortest-path distance between start and goal of 7 to 9 steps. Maze generation is parameterized by $\theta$ , which controls dimensions, wall density, and start-goal separation. The paper specifies that "we start by sampling a maze instance, $m_i \sim M(\theta)$ " (Section 2.1). In addition to the primary 6×6 configuration, the paper reports ablation experiments in Appendix B that vary maze size ( $N \in \{4, 6, 8, 10, 12, 18\}$ ) and wall density ( $p \in \{0, 0.15, 0.30, 0.45, 0.60, 0.75\}$ ), enabling analysis of how the collaboration gap scales with task complexity.

Information splitting — the key design choice. Rather than giving both agents the complete maze (which would let either agent solve it independently), the paper introduces a distributed information setup: each cell of the original maze $m_i$ is randomly assigned to be visible in either one of two partial copies ( $m^1_i$ or $m^2_i$ ) or both, with the constraint that approximately 50% of cells are obfuscated in each copy. The paper states that "we distribute the information by randomly obfuscating half the cells of each copy resulting in $m^1_i$ and $m^2_i$ . Combined, the two copies recover the complete map, i.e., $m_i = m^1_i \cup m^2_i$ " (Section 2.1). Crucially, this is not a strict partition — some cells may be visible to both agents, and some may be visible to only one. This design choice means that agents cannot simply concatenate their maps deterministically; they must communicate to discover which cells each can see, resolve conflicts where both agents can see the same cell but disagree (which should not occur if both are truthful but can occur due to model error), and jointly plan routes through the combined information.

Why random obfuscation rather than a structured split? The random split creates genuine interdependence: neither agent can be confident that their partial view contains a complete path from start to goal. This forces communication about specific cells rather than allowing each agent to solve independently and then compare answers. The alternative — giving each agent a contiguous half of the maze — would make the collaboration task closer to map-merging, which tests a different (and likely easier) capability. Random obfuscation also creates more realistic partial-observability scenarios: in open-world agent deployment, agents will have access to different subsets of information based on their organizational context, privileges, and tool access, and these subsets will not partition cleanly.

Maze representation. Mazes are presented to agents as a visual, text-based grid of $N \times N$ symbols using four characters: @ for the current position (start), * for the goal position, . for a traversable path cell, # for an impassable wall cell, and ? for a hidden (obfuscated) cell. The representation is deliberately visual and spatial rather than structured (e.g., JSON with coordinates), which means agents must interpret the grid layout. This is critical because it forces agents to establish shared coordinate conventions — for example, whether (0, 0) is top-left or bottom-left, whether coordinates are (row, column) or (column, row), and whether indexing starts at 0 or 1. These grounding challenges are exactly what Section 2.2 identifies as core collaborative skills: "they must establish the shared position of the start and goal position... Absent a fixed communication protocol, agents must further agree on how to refer to locations and actions."

Solo baseline variants. To establish individual maze-solving capability independent of collaboration, the paper defines two solo settings. In the "solo: full" setting, a single agent receives the complete maze (all cells visible). In the "solo: distributed" setting, a single agent receives both partial map copies ( $m^1_i$ and $m^2_i$ ) simultaneously. The distributed solo setting is crucial because it controls for the cognitive challenge of integrating distributed information: if a model performs well in distributed solo but poorly in collaborative, the deficit is specifically due to collaboration requirements (communication, grounding, turn-taking) rather than the difficulty of reasoning with partial information. The paper states (Section 3): "to fairly compare solo to collaborative performance on the decomposed task, we can present a single agent with both of the distributed maps to measure its capability to handle distributed information."

Shared rules and incentives. All agents (solo and collaborative) receive the same core rules: (1) only move to adjacent cells (up, down, left, right — no diagonals), (2) cannot move through walls, (3) the game ends when reaching the goal or stepping into a wall. Agents are told "Your game is timed for speed and put on a leader board so hurry up!" — this is a deliberate design choice to incentivize efficient movement rather than exhaustive map exploration. Collaborative agents additionally receive two critical rules: "both agents must agree on a move before it is executed" (preventing unilateral action) and "only one move can be executed at a time" (enforcing turn-by-turn collaboration rather than batch planning). Beyond these constraints and a predefined completion phrase ("ACTI!" to signal goal achievement), agents receive no guidance on communication protocol, output format, or turn-taking procedure. The paper emphasizes: "we do not enforce nor provide any other guidance on the communication protocol the agents must use or output format they must follow" (Section 2.1).

Agent Interaction Protocol and Communication Architecture

The collaborative interaction is implemented as a multi-turn dialogue between two language model instances, mediated by an explicit relay mechanism that addresses API-level constraints across diverse model providers. This section explains how the dialogue is structured, how agent messages are routed, and what implementation choices were necessary to make the benchmark work across 32 models with different API capabilities.

The interaction loop. At each turn, the active agent receives as input: (1) its own partial maze copy (the grid with ? symbols for hidden cells), (2) the collaboration rules (including the requirement for mutual agreement and the completion phrase), and (3) the full dialogue history of all messages exchanged so far (from both itself and the other agent). The agent must produce a message in natural language; the system then routes this message to the other agent, which processes it along with its own information and the dialogue history, and produces a response. This back-and-forth continues until either (a) the predefined completion phrase is uttered (signaling goal achievement), (b) an agent moves into a wall (terminating the game with failure), or (c) the maximum of 50 turns is reached.

Why 50 turns? The paper does not provide an explicit justification for the 50-turn limit, but it emerges from the maze parameters: with 6×6 mazes and solution paths of 7–9 steps, plus backtracking, conflict resolution, and grounding overhead, 50 turns provides sufficient headroom for even inefficient collaborations while preventing runaway loops in degenerate cases. The limit is symmetric — both homogeneous and heterogeneous collaborations use the same cap — enabling fair comparison.

The agent-to-agent communication implementation (Appendix A.2). Since the paper evaluates models from many different providers (OpenAI, Google, Anthropic, xAI, DeepSeek, Meta, Cohere, Microsoft, Moonshot AI, Alibaba), each with different API constraints on message roles and sequencing, a non-trivial engineering challenge arises: how do you make two AI models talk to each other when some APIs only accept a single System message, some require User/Assistant alternation, and some don't allow the first message to have the Assistant role?

The paper describes three candidate approaches and explains why it chose the third:

Approach 1: User as Agent — simply use the User role when relaying messages between agents, so each agent receives the other agent's messages "as if" from a user. The paper rejects this because (a) LMs are aligned to behave helpfully and agreeably toward users, which introduces sycophancy biases (Sharma et al., 2023) that would distort the AI–AI collaboration measurement, and (b) some providers don't allow the first message in a conversation history to be an Assistant message, requiring a "filler" User message whose influence on subsequent behavior is unknown.

Approach 2: Transcript Style — transform the entire conversation history into a single User message summarizing the dialogue as a transcript (following Davidson et al., 2024). The paper rejects this because it pushes models out of their training distribution: "models are optimized for sequential messages being sent back-and-forth — not contributing lines to a script."

Approach 3: Explicit Prefixes (chosen approach) — the system uses a carefully constructed message structure:

A System message informs the agent: "The user will act as an intermediary between you and another agent. The user will directly forward your messages to the other agent and vice versa. The user will not see or modify the messages, but will relay them as is. Messages coming from the other agent will be prefixed with '[other agent]:'. Messages coming from the user will be prefixed with '[user]:'. Do not add any additional prefixes or suffixes to your own messages."
The first User message contains the task instructions, prefixed with "[user]:" to distinguish system instructions from the relay mechanism.
At each turn, the other agent's message is inserted as a User message prefixed with "[other agent]:" (or vice versa — the roles in the message history are swapped appropriately so that each agent always sees alternating User and Assistant messages in their own history). This ensures the first message in every conversation history is of type User (satisfying API constraints), and the prefixes make the distinction between system relay messages and agent messages explicit.

The paper notes (Table 1 in Appendix A.1) that this approach is compatible with the subset of models marked "System Prompt Flexible: Yes" — models that allow a system prompt role and permit it to be inserted at an arbitrary position. Models marked "No" or "Yes*" (where the asterisk indicates the API accepted the system prompt but appeared not to use more than one) received adapted versions of this protocol.

What agents actually see. For the collaborative setting, the first User message to each agent contains (full prompt text in Appendix A.4): the task description ("You are going to play the collaborative maze game 'ACTI' together with another agent"), the maze grid (using @, *, ., #, ? symbols), the legend explaining each symbol, the move rules (adjacent cells only, no diagonals, no walls), the mutual agreement requirement, the one-move-per-turn constraint, the speed incentive, and the completion phrase instruction ("Be sure to say: 'ACTI!', after you reached the goal"). No additional formatting, schema, or protocol guidance is provided — agents must determine for themselves how to describe locations, propose moves, share map information, and negotiate agreement.

The "primer message" as a pivotal moment. The paper's qualitative analysis (Section 4.2, Figure 4.2, and Appendix E) reveals that the opening message in a collaborative dialogue — the "primer" — has disproportionate influence on the entire subsequent collaboration. This is because the first agent to speak sets the communication conventions: coordinate system, schema for describing the maze, granularity of information sharing, and procedural norms (who proposes, who evaluates). A strong model's primer (e.g., o3 in Figure 4.2) immediately provides a fully specified coordinate schema, requests specific missing information, and grounds the starting position explicitly. A weak model's primer (e.g., gpt-4.1-mini in the same figure) offers vague proposals without establishing any shared reference system, making it impossible for the partner to engage efficiently. This asymmetry is the mechanism behind the ordering effects observed in heterogeneous collaboration (Section 4.3) and the effectiveness of relay inference priming (Section 4.4).

Style imitation effects. The paper observes that when a strong model follows a weak model's primer, the strong model partially imitates the weak model's communication style rather than overriding it with a better schema (Figure 4.4, Figure E.1). This is a form of conversational accommodation — the strong model adapts to its partner's conventions rather than imposing its own, even when those conventions are suboptimal. This behavior is consistent with models' alignment to be cooperative and non-disruptive in conversation, but it has the unintended consequence of locking the collaboration into a suboptimal communication protocol that the strong model could have improved. Conversely, when a weak model follows a strong model's primer, the weak model adopts the strong model's conventions (Figure E.1), which is beneficial because the strong model's conventions are typically more precise and complete.

Outcome Metrics

The paper defines two quantitative outcome measures, both derived from the same underlying validation: whether the agents' agreed-upon sequence of moves corresponds to a valid path from start to goal in the ground-truth maze.

Metric 1: Binary success rate. This is the fraction of rollouts where the extracted solution path is correct — that is, it forms a valid, contiguous path from the starting position to the goal position, with each consecutive pair of coordinates being adjacent (sharing an edge), none stepping into a wall cell, and none skipping intermediate cells. The paper reports this for solo and collaborative settings as a straightforward accuracy measure (Tables 3a–3f in Appendix C.3).

Metric 2: Weighted outcome. Because binary success omits information about partial progress, the paper defines a continuous metric that captures how close an agent got to the goal. The weighted outcome is defined in Section 3 (Outcome Metrics) as:

$\text{Weighted Outcome} = \frac{a - b}{a}$

where $a$ is the distance (number of steps) along the optimal shortest path from the start position to the goal, and $b$ is the distance from the last valid position reached by the agents to the goal, also measured along the optimal path.

What it computes: this metric measures the fraction of the optimal path completed. If the agents reach the goal ( $b = 0$ ), the weighted outcome is 1.0 — perfect performance. If the agents never leave the start ( $b = a$ ), the weighted outcome is 0.0 — no progress. If the agents move away from the goal (which is geometrically possible if they backtrack or take an inefficient route) such that their distance from the goal exceeds the optimal start-to-goal distance ( $b > a$ ), the weighted outcome becomes negative — penalizing counterproductive movement. The paper notes explicitly: "Note that this ratio can be negative."

Why this form: the weighted outcome is more informative than binary success for evaluating collaborative behavior because (a) it captures partial progress — an agent that gets within one step of the goal receives a high weighted outcome even if it fails to complete, and (b) it penalizes severe navigation failures (moving away from the goal) that binary success treats identically to "never started." This is particularly important for small models that may make meaningful partial progress through collaboration even when they rarely achieve perfect solves. The paper uses weighted outcomes as the primary metric throughout Section 4 because it provides finer-grained discrimination than binary success, especially for models clustered near 0 or 1 on the binary metric.

Computational note: the distance $b$ is measured along the optimal path from start to goal, not the Euclidean distance. This means that if the agent makes progress but along a suboptimal route, the metric reflects the fraction of the optimal distance covered, not the fraction of the agent's actual path length. This rewards efficient navigation toward the goal rather than arbitrary exploration.

Additional efficiency metrics (Appendix C.4, C.5). The paper also reports two process-level metrics: median tokens per collaboration (Appendix C.4, Figures C.1a–b) and median number of messages per collaboration (Appendix C.5, Figures C.2a–b). These capture collaborative efficiency: how much communication is required to achieve a given outcome level. The paper stratifies these by weighted outcome ranges (Fail: 0–0.25, Partial Fail: 0.25–0.5, Partial Success: 0.5–0.75, Success: 0.75–1.0) to decompose whether longer conversations are associated with better outcomes (suggesting productive communication) or worse outcomes (suggesting confusion and deadlock). The paper notes that token counts "do not account for possible 'thinking' tokens, as these are not always exposed," so they should be interpreted as "efficiency in conveying information to the other agent" rather than computational efficiency.

Grading Pipeline: From Unstructured Transcripts to Validated Routes

Since the benchmark imposes no output-format constraints — a deliberate design choice to preserve ecological validity — the generated dialogues can describe moves in arbitrary formats: cardinal directions ("move up"), coordinate pairs ("(3, 4) to (3, 5)"), row-column notation ("r2c3"), or any combination thereof. This makes deterministic extraction of the solution path infeasible. The paper uses an LM-based grader with a multi-schema normalization pipeline to handle this diversity. The grading pipeline is described in Section 2.1 and detailed in Appendix A.5.

Grader model and task. The paper uses gpt-4.1 as the primary grading model for all main experiments. The grader receives the raw dialogue transcript and a structured extraction prompt (Figure A.6, Appendix A.5 for collaborative settings; Figure A.5 for solo settings). Its task is to produce a YAML object containing:

Route schema inference: the grader must infer four properties of the coordinate convention used by the agents:
- maze_origin: whether the maze coordinates start at 0 or 1.
- maze_orientation: which corner is the origin (top_left, bottom_left, top_right, bottom_right).
- coordinates_orientation: whether coordinates are (row, column) or (column, row).
- coordinates_symbols: whether coordinates are expressed as numbers (e.g., (1, 2)), letters (e.g., (A, B)), alphanumeric (e.g., (A, 1)), or directions (e.g., "up", "down").
Route extraction: for each turn in the dialogue, the grader records the coordinates or direction of any agreed-upon move (with turn_type: "move"), or the coordinates/directions the agents considered without agreeing (with turn_type: "consider"), along with which agent proposed or whether both agreed.

The grader prompt (Figure A.6) provides explicit examples of each schema variant and instructs the grader to "be consistent in the move format you record, e.g., stick with directions or coordinates, do not mix them unless absolutely necessary." The grader outputs only the YAML object, with no additional commentary.

Schema normalization and solution checking. Once the grader extracts a candidate route $z_i$ and an inferred schema, a separate normalization step transforms $z_i$ under a large set of potential schemas to protect against grader errors. The paper states (Section 2.1): "we transform $z_i$ under a large set of potential schemes and evaluate each against the original maze $m_i$ to obtain the final outcome $y$ ." This means the system does not require the grader to perfectly infer the schema; it enumerates plausible alternative schemas, transforms the extracted path for each, and checks whether any transformation yields a valid path. If at least one transformation produces a valid start-to-goal path, the rollout is considered successful. This approach is robust to grader errors where, for example, the grader misidentifies the coordinate orientation but the extracted moves are still semantically correct under some interpretation.

What constitutes a valid path: the solution checker verifies that the extracted route (under some schema transformation) forms a contiguous sequence of adjacent cells from the start position @ to the goal position *, with no intervening wall cells and no illegal jumps. The specific adjacency constraint is that consecutive coordinates must share exactly one edge (up, down, left, or right — no diagonals). The checker requires that the entire path be consistent with the ground-truth maze (the complete $m_i$ before obfuscation).

Error case: unanticipated schemas (Appendix F.1). The paper documents a case where gpt-5 used a coordinate convention ("r1c1" for row 1, column 1) that was not anticipated by the normalization pipeline, causing a valid solution to be incorrectly marked as a failure. The paper states: "The large number of manual author checks do not suggest this to be a widespread problem," but this reveals a fundamental limitation of the approach: the schema normalization can only cover schemas the designers anticipated. As models become more creative in their communication conventions, this limitation will become more significant, suggesting that future work should consider more robust extraction methods (perhaps using the grader to directly produce standardized coordinates rather than inferring a schema).

Grading ablation (Appendix D). The paper conducts extensive ablation studies to validate the grading pipeline's reliability, addressing two concerns:

Intra-model consistency (Appendix D.1): How consistent is gpt-4.1 at grading the same transcripts multiple times? The paper collects three independent grading passes on a stratified sample of solo and homogeneous collaborative transcripts (selected to balance success and failure cases across models from different builders). For solo transcripts, intra-grader consistency is near-perfect (ICC ≈ 0.99, Fleiss' Kappa ≈ 0.99). For collaborative transcripts, consistency is high but slightly lower (ICC ≈ 0.89, Fleiss' Kappa ≈ 0.87), with disagreement concentrated in success cases (14% binary disagreement rate vs. 6% for failure cases). The paper attributes this to the greater ambiguity in collaborative transcripts (multiple agents, conversational language, backtracking).
Inter-model consistency (Appendix D.2): Would a different grader model produce different results? The paper collects additional grading passes using o3 and gemini-2.5-flash on the same transcripts, comparing against the gpt-4.1 reference. The three graders show high agreement (ICC ≈ 0.84, Fleiss' Kappa ≈ 0.77 for collaborative; ICC ≈ 0.88, Fleiss' Kappa ≈ 0.91 for solo). Majority votes are "statistically indistinguishable from the reference" (Table 6a for solo, Table 7a for collaborative). The paper also checks for model-specific biases, finding no evidence that an OpenAI grader unfairly favors OpenAI-generated transcripts over those from Google or xAI. Agent-specific ICC and Kappa values are reported in Tables 6c and 7c; some models show lower consistency (e.g., gpt-5 at ICC ≈ 0.22 for collaborative), but the paper's manual inspection attributes this to gpt-5's unusual success patterns rather than grader bias.

Why use an LM grader rather than human annotation? The paper explicitly states (Appendix D): "Since using human annotators is not feasible given the scale of our experiments (many tens of thousands of transcripts), we instead opt to use LMs as graders." The ablation results validate this choice by showing that the LM grader is consistent, unbiased across model families, and produces results that would not meaningfully change if graded by a stronger model or by majority vote across multiple models. However, the paper acknowledges the possibility that "some grader models might unfairly disadvantage participating models" and designs the ablation to test and rule out this concern.

Experimental Configurations

The paper defines four experimental settings, each designed to isolate a different aspect of collaborative capability. The configurations share the same maze instances (fixed random seeds for reproducibility) and grading pipeline, ensuring that differences in outcomes can be attributed to the collaboration configuration rather than task difficulty or measurement error.

Setting 1: Solo baselines (Section 3, "Solo Baseline"). Each model is evaluated individually in two sub-settings, with at least 100 samples collected per setting:

Solo: Full — the agent receives the complete, fully visible maze. This measures raw maze-solving ability: can the model plan and execute a valid path from start to goal given complete information? The model is given one attempt per maze (the paper states "you get one shot to complete the maze"), receives a "critic" step where it can review its proposed solution ("Carefully review the final solution you have provided above... If needed, you can make adjustments"), and then submits its final solution as "a list of coordinates" (Appendix A.3, Figures A.2–A.3).
Solo: Distributed — the agent receives both partial map copies simultaneously (both $m^1_i$ and $m^2_i$ , each with different cells obfuscated). This measures the ability to integrate distributed information without collaboration: can the model mentally merge two partial views, identify the complete maze structure, and plan a path? The prompt structure is identical to the full setting except that two separate grid views are provided instead of one.

The paper reports both solo variants in Figure 4.1 (gray and yellow markers, respectively), and uses the difference between them to identify models where information integration itself is a bottleneck.

Setting 2: Homogeneous collaboration (Section 3, "Homogeneous Collaboration"). Two independent instances of the same model are paired together, each receiving a different partial map copy. This setting isolates collaborative capability: any performance drop relative to the distributed solo setting cannot be attributed to information integration difficulty (since the solo setting already requires merging partial maps) and must instead be due to the demands of communication — grounding, turn-taking, conflict resolution, and multi-agent state tracking. The paper collects at least 100 rollouts per model, with a maximum of 50 turns per rollout.

The homogeneous collaboration results are the paper's central finding (Figure 4.1, red markers) and define the "collaboration gap": the difference between distributed solo weighted outcome and homogeneous collaborative weighted outcome. The paper emphasizes that this gap is not explained by task difficulty alone — models that achieve near-perfect distributed solo performance (like gpt-5, o3, grok-3-mini) show varying degrees of collaborative degradation, proving that collaboration introduces demands beyond what solo task performance measures.

Setting 3: Heterogeneous collaboration (Section 3, "Heterogeneous Collaboration"). Two agents powered by different models (potentially different families and strengths) are paired together. The paper restricts heterogeneous experiments to a subset of models from "the most popular providers: Anthropic, Cohere, Google, OpenAI, and xAI" to avoid a combinatorial explosion (with 32 models, there are 32 × 31 / 2 = 496 possible unordered pairings). Within each provider, the paper investigates "in-family" pairings (models of different strength from the same builder, e.g., o3 paired with gpt-4.1-mini) and "cross-family" pairings (models from different builders, e.g., grok-3-mini paired with gemini-2.5-flash). For each ordered pairing (where order matters — Agent 1 speaks first), the paper collects at least 50 rollouts with a maximum of 50 turns.

This setting enables analysis of:

Ordering effects: does the collaboration outcome depend on which agent initiates the dialogue? (Results in Figure 4.3 and Tables 2–3 show significant asymmetry.)
Capability complementarity: can a strong model "carry" a weak model, and if so, by how much? (Results show that strong models do not fully replicate their homogeneous performance when paired with weak partners.)
Cross-family compatibility: do models from the same builder collaborate better with each other than with external models? (Results in Table 2a suggest possible within-family affinity, e.g., gemini-2.5-flash-lite improves when paired with gemini-2.5-flash but not with the stronger gpt-4.1.)

Setting 4: Relay inference (Section 3, "Relay Inference"). This is the paper's primary constructive contribution — a lightweight intervention that partially closes the collaboration gap without retraining models. The relay inference experiment simulates a practical deployment scenario: "Given two available agents, A and B, where A is stronger (costlier) than B. Assume we have to deploy an agent to collaborate with another party's agent, and that we cannot control the choice of agent used by the other party." The experiments test two strategies:

Strong primer (Figure 4.6a): A strong model (o3) and a weak model (either gpt-4.1-mini or gemini-2.5-flash-lite) collaborate for the first $K$ turns, where $K \in \{2, 4, 6, 8\}$ . After turn $K$ , the strong model is removed and replaced by another copy of the weak model, which continues the collaboration with the original weak model from that point. This tests whether a strong model can prime the dialogue — establish grounding, propose a schema, and start navigating — such that two weak models can continue productively where the strong model left off. The hypothesis is that the initial grounding is the hardest part, and once established, weak models can execute.
Strong recovery (Figure 4.6b): Two weak models collaborate for the first $K$ turns, after which one is replaced by a strong model (o3). This tests whether a strong model can recover a collaboration that has gone off the rails — whether it can diagnose confusion, re-establish grounding, and redirect toward the goal. The hypothesis is that early errors compound and become increasingly difficult to recover from.

The relay inference budget is 100+ rollouts per model pairing and relay point $K$ , with the relayed rollouts initialized by "freezing" the first $K$ turns of actual AB or BB collaborations (rather than generating fresh rollouts for each $K$ ). This ensures fair comparison across relay points — the same initial trajectories are truncated at different depths.

Why the relay inference setup matters diagnostically. The strong primer results (Section 4.4) reveal that a single opening message from o3 is sufficient to dramatically boost weak models' collaborative performance, demonstrating that the collaboration gap is not a general inability to communicate but rather a specific failure to initiate effective collaboration — to propose a clear coordinate schema, share relevant map information proactively, and establish a coordination procedure. The strong recovery results (diminishing returns as $K$ increases) reveal that early interactions create path dependencies that are hard to escape — once two weak models establish poor conventions or make navigation errors, even a strong model struggles to course-correct. This has direct implications for multi-agent system design: invest compute in the first few messages of a collaboration to establish strong grounding, rather than deploying strong models as backup error-handlers.

Model Selection and Characteristics

The paper evaluates 32 language models from nine builders, representing a deliberate cross-section of the current model landscape along several dimensions relevant to collaborative capability. The complete model list is provided in Table 1 (Appendix A.1), along with annotations for four properties:

Open/Closed: whether model weights are publicly available (Open) or only accessible via API (Closed). This matters because open models can be privately deployed and potentially fine-tuned for collaboration, while closed models' capabilities are fixed by the provider.
Distilled: whether the model is explicitly distilled from a larger model (e.g., gpt-5-mini from gpt-5, gpt-4.1-nano from gpt-4.1). The paper hypothesizes (and confirms in Section 4.2) that distillation disproportionately degrades collaborative capability — a finding with significant practical implications given industry trends toward deploying distilled models.
System Prompt Flexible: whether the model's API (a) allows a system prompt role and (b) permits system prompts at arbitrary positions in the message history. This is a practical constraint on the agent-to-agent communication implementation (see Appendix A.2).

Why these 32 models? The paper states the goal is "to capture generalizable insights relevant to LMs' collaborative capabilities" and to "include as many commercially available open- and closed-source models as possible" (Appendix, "Costs of Multi-Turn Rollouts"). The selection spans:

Frontier closed-source models: gpt-5, o3, grok-4, gemini-2.5-pro, claude-opus-4.1 (the strongest available models at the time of evaluation).
Frontier open-source models: deepseek-R1, llama-4-maverick, qwen-3-235B, kimi-k2 (competitive with closed-source models on many benchmarks).
Distilled/small variants: gpt-5-mini, gpt-5-nano, gpt-4.1-mini, gpt-4.1-nano, gemini-2.5-flash, gemini-2.5-flash-lite, claude-haiku-3.5, llama-4-scout, command-r7b (to test the distillation hypothesis).
Mid-tier general-purpose models: gpt-4.1, claude-sonnet-4, grok-3, gemini-2.5-flash, phi-4 (representative of models deployed in many production settings).
Smaller open models: gemma-3-27b-it, llama-3.3-70B-it, qwen-2.5-72B-it, command-r (to test whether the collaboration gap appears at smaller scales).

Heterogeneous subset selection. For heterogeneous experiments, the paper restricts to models from "the most popular providers: Anthropic, Cohere, Google, OpenAI, and xAI" and "investigates 'in-family' pairings and selected 'cross-family' pairings." The specific pairings are organized in the result tables (Tables 2a–2f in Appendix C.2): Google × OpenAI (Table 2a), OpenAI internal family (Table 2b), xAI × OpenAI (Table 2c), Anthropic × Google × OpenAI × xAI (Table 2d), Anthropic internal family (Table 2e), and Cohere internal family (Table 2f). This structured approach enables systematic analysis of ordering effects and cross-family compatibility without the full combinatorial explosion.

Model quirks affecting results (Appendix C.1). The paper documents model-specific behaviors that affected outcomes:

Cohere command-a and command-r: These models had a "tendency to prematurely 'fantasize' about being able to use the completion phrase" — they would utter "ACTI!" before actually reaching the goal, triggering premature termination. The paper notes that their reported collaborative performance "should be seen as a lower bound on actual performance."
gemini-2.5-pro: This model produced unusually short messages, which worked adequately in homogeneous pairings (it could coordinate with copies of itself despite brevity) but "did complicate collaborating with weaker models in the heterogeneous setting." The weaker but more verbose gemini-2.5-flash often outperformed gemini-2.5-pro in mixed pairings (Table 2a).
claude-sonnet-4: Exhibited two interesting failure modes: (a) getting stuck in "agreement loops" where it kept seeking confirmation on already-agreed moves without proposing the next step, and (b) "splitting up" to explore the maze separately (even though this provides no advantage since all information is shared; Figure F.2). The paper notes that splitting up "is not necessarily incorrect, as this could be interpreted as underspecified in the instructions," but it "breaks the current parser, leading to suppressed performance."
claude-opus-4.1: Despite strong average performance, some solutions were "surprisingly inefficient" — the model would visit nearly every path cell on its way to the goal, consuming the full 50-turn budget.
gpt-5: Achieved near-perfect solo and homogeneous collaborative performance on 6×6 mazes, with failures "due to a parsing shortcoming, rather than a mistake made by the model" (Appendix F.1, Figure F.1).

Relay Inference Mechanism

The relay inference experiments (Section 3, "Relay Inference"; Section 4.4) are the paper's constructive contribution — a lightweight deployment strategy that partially closes the collaboration gap without requiring model retraining or fine-tuning. This section explains the experimental mechanism in detail.

Motivating scenario. The paper frames relay inference around a practical resource allocation problem. Suppose you control two agents: Agent A (strong, expensive) and Agent B (weak, cheap). You must deploy an agent to collaborate with an external party's agent, but you cannot control which agent the external party deploys. If the external party deploys Agent B (the cheap option), you face a choice: should you also deploy Agent B (matching their cost savings) or deploy Agent A (incurring higher cost)? The relay inference experiments test a third option: deploy Agent A for the initial turns to establish grounding, then switch to Agent B — capturing most of the benefit of the strong model at a fraction of the cost.

Experimental design. The paper uses o3 as the strong model and two weak models as test cases: gpt-4.1-mini (OpenAI family) and gemini-2.5-flash-lite (Google family). For each test case, the paper first generates a set of actual collaborative rollouts using two configurations: AB (strong–weak, where strong speaks first in the relay primer experiments, or weak speaks first then strong takes over in the relay recovery experiments) and BB (weak–weak). Each rollout consists of a complete dialogue of up to 50 turns. The paper then "freezes" the first $K$ turns of each rollout, where $K \in \{2, 4, 6, 8\}$ , truncates the dialogue at turn $K$ , and swaps in a different model to complete the remaining turns. At least 100 rollouts are collected per model pairing and relay point $K$ .

Strong primer configuration (Figure 4.6a). For the strong primer experiments:

Baseline 1 (lower bound): BB homogeneous — two copies of the weak model collaborate for the full dialogue. This represents the "no intervention" case.
Baseline 2 (upper bound): AB heterogeneous — the strong model (speaking first) collaborates with the weak model for the full dialogue. This represents the "strong model available for the entire task" case.
Relay treatment: Strong primer at turn $K$ — the strong model and weak model collaborate for the first $K$ turns, after which the strong model is removed and replaced by a second copy of the weak model. The two weak models then continue the collaboration from turn $K+1$ onward, building on the dialogue history established in the first $K$ turns.

The key comparison is whether the relay treatment at turn $K$ achieves weighted outcomes closer to the AB upper bound or the BB lower bound. If even $K = 2$ (a single primer message from the strong model, plus one weak-model response) achieves outcomes substantially above BB, then the initial grounding message is the critical intervention.

Strong recovery configuration (Figure 4.6b). For the strong recovery experiments:

Baseline 1 (lower bound): BB homogeneous — two weak models for the full dialogue.
Baseline 2 (upper bound): AB heterogeneous — the strong model (speaking first) collaborates with the weak model for the full dialogue.
Relay treatment: Strong recovery at turn $K$ — two weak models collaborate for the first $K$ turns, after which one weak model is removed and replaced by the strong model. The strong model continues collaborating with the remaining weak model from turn $K+1$ onward.

The key comparison is whether the strong model can "rescue" collaborations that started poorly. If the strong model achieves outcomes near the AB upper bound even for large $K$ , then collaboration errors are recoverable. If outcomes degrade rapidly as $K$ increases, then early errors create compounding path dependencies that even strong models cannot easily undo.

What the strong model contributes in the primer condition. The strong model in the primer experiments has access only to its own partial map copy — it does not have privileged information about the complete maze. This is a crucial design choice that prevents confounding: if the strong model had access to the full maze and simply solved it, the experiment would measure information superiority rather than collaborative priming. Instead, the strong model's contribution is purely about how it structures the collaboration: establishing a clear coordinate system, proactively sharing its visible map information, requesting specific missing information from the partner, and proposing an initial move with justification. These are all things a weak model could do but typically does not (as shown in the qualitative comparison in Figure 4.2).

What the strong model does in the recovery condition. In the recovery experiments, the strong model enters a collaboration that has already been shaped by the two weak models' initial interactions. It inherits whatever communication conventions, navigation decisions, and misunderstandings have accumulated over $K$ turns. Its task is to diagnose the state of the collaboration, identify errors or inefficiencies, and redirect toward the goal — all while working within (or renegotiating) whatever conventions the weak models have established. The paper's finding that recovery effectiveness degrades with $K$ (Figure 4.6b) implies that established conventions are "sticky" — once two agents have converged on a (possibly flawed) shared understanding, even a more capable agent struggles to unwind and replace it.

Why relay inference matters beyond the specific experiment. The relay inference framework demonstrates a general principle for multi-agent system design: front-loading capability is more cost-effective than back-loading it. The most valuable interventions happen early, when communication conventions are being established and the shared mental model is being constructed. Once these are set — well or poorly — they constrain the subsequent collaboration in ways that are expensive to override. This has implications for protocol design (invest in initial handshake/schema negotiation stages), agent deployment (use stronger models for initial coordination, weaker models for execution), and human–AI collaboration (invest effort in the first few exchanges of a collaboration to establish mutual understanding).

Implementation Details and Practical Considerations

The paper documents several implementation-level details that affect reproducibility and practical deployment.

Non-determinism management (Appendix, "Non-Determinism"). Since most API-accessible models are non-deterministic (He, 2025), repeated collaborations between models can produce different outcomes for the same maze instance. The paper mitigates this by collecting large sample sizes (100+ rollouts for homogeneous, 50+ for heterogeneous) and reporting 95% confidence intervals around all mean metrics. The paper also uses fixed random seeds for maze generation to ensure fair comparison: "all mazes generated during this study used fixed random seeds for a fair comparison between models."

Cost scaling in multi-turn rollouts (Appendix, "Costs of Multi-Turn Rollouts"). The paper acknowledges that replicating the study is expensive. For a collaboration with average message length $K$ tokens and maximum $T$ turns, the input context length at turn $t$ is approximately $t \times K$ , and the average across the rollout is $K \cdot T / 2$ . Since API costs for most providers scale with input token count, later turns become increasingly expensive. The paper does not report the total cost of the experiments, but notes that this cost profile "can partially be attributed to the higher cost of using some frontier models, but also to the increasing cost of generating a subsequent turn."

System prompt compatibility issues (Table 1, Appendix A.1). Several models (marked "System Prompt Flexible: No" in Table 1) do not support the full three-party relay architecture described in Appendix A.2. These include all Google models (gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite, gemma-3-27b) and all Anthropic models (claude-opus-4.1, claude-sonnet-4, claude-haiku-3.5). For these models, the paper used adapted versions of the communication protocol that respected their API constraints, though the specific adaptations are not detailed. This is a practical limitation: the benchmark's communication architecture cannot be perfectly replicated across all models, and the adaptations may introduce subtle differences in how models perceive the collaboration setup.

Unintended model behaviors (Appendix C.1). Beyond the documented quirks, the paper notes that "upon qualitative inspection it was found that the command-a model from Cohere would sometimes 'prematurely' use the predefined completion phrase when fantasizing about a successful outcome. Due to the large scale of our experiments, modifying the instructions was no longer feasible." This reveals a tradeoff in large-scale benchmark design: the fixed prompts must work across all models without per-model tuning, but models can interpret the same instructions differently, leading to measurement artifacts that affect some models more than others. The paper handles this transparently by documenting the affected models and interpreting their results as lower bounds.

Summary of Design Choices and Their Justifications

Random cell obfuscation over structured splits: creates genuine interdependence and forces communication about specific cells rather than enabling map-merging strategies.
Text-based visual grid representation over structured JSON: forces agents to interpret spatial layout and negotiate coordinate conventions, testing grounding capability.
No output-format constraints over prescribed message formats: preserves ecological validity — real open-world agent interactions won't follow a single protocol.
Explicit prefix relay protocol over user-as-agent or transcript approaches: avoids sycophancy biases while maintaining compatibility with diverse API constraints on message roles.
LM-based grading with multi-schema normalization over deterministic parsing or human annotation: handles unstructured output at scale while protecting against grader-specific schema inference errors.
Weighted outcome metric over binary success alone: captures partial progress and penalizes counterproductive movement, providing finer-grained discrimination.
Fixed maze random seeds over independent generation: ensures fair comparison across models and settings.
Restricted heterogeneous subset over full pairwise matrix: enables systematic analysis of ordering and cross-family effects without 496-pair combinatorial explosion.
gpt-4.1 as default grader over other models: validated through extensive inter- and intra-grader ablation showing high consistency and low model-specific bias.
50-turn maximum over unlimited turns: prevents degenerate loops while providing sufficient headroom for inefficient collaborations on 6×6 mazes with 7–9 step solutions.
At least 100 homogeneous / 50 heterogeneous rollouts over smaller samples: ensures confidence intervals are narrow enough to detect meaningful differences despite model non-determinism.
Offline relay inference using frozen initial turns over fresh generation per $K$ : enables controlled comparison by holding the first $K$ turns constant across relay points.
No per-model prompt optimization over tailored instructions: maintains fairness and scalability, acknowledging that "there is likely a set of system prompts and instructions more suitable for a particular model" but that "due to the opaqueness of closed-source models, it is unfortunately impossible to determine if our prompts clashed with existing developer prompts" (Appendix, "Reproducibility Statement").

4. Key Insights and Innovations

Innovation 1: "Collaboration Gap" as a Distinct Capability Axis — Not a Failure of Task Competence

The paper's most fundamental contribution is conceptual rather than methodological: it establishes that effective AI–AI collaboration is a distinct axis of capability that is not predicted by solo task performance, and provides the first systematic empirical evidence that this gap is large, pervasive, and disproportionately severe in distilled models.

Prior to this work, the field's implicit assumption was that if a model could solve a task individually, it could also collaborate effectively on that task — collaboration being treated as "task performance plus communication," with communication being an emergent property of language modeling capability that scales naturally with model size and solo benchmark scores. This assumption was untested. As the paper notes in Section 1, prior benchmarks "reflect the paradigm shift toward agentic use-cases" in some ways but "few reflect... dynamic, long-horizon, multi-agent collaboration in partially observed environments," and none provided the solo-to-collaborative comparison that would isolate collaboration as a distinct skill.

The evidence for this innovation is Figure 4.1, which shows that the collaboration gap is not simply a function of raw maze-solving ability. Several models (o3, grok-3-mini, gpt-5) achieve near-perfect weighted outcomes in the distributed solo setting (yellow markers above 0.8), demonstrating they can handle the cognitive challenge of integrating partial information into a complete path plan. Yet their homogeneous collaborative performance (red markers) varies dramatically: gpt-5 maintains near-perfect performance (~0.99), grok-3-mini drops to ~0.90, and o3 drops to ~0.88 — all statistically significant differences despite comparable solo baselines. If collaboration were simply "solo capability plus communication," these three models would cluster together. They do not, which is the existence proof: collaboration tests something that distributed solo performance does not.

What makes this a fundamental insight rather than an incremental observation: The paper reframes collaboration from an application (something you do once you have capable models) to a capability (something the model must possess independently of task competence). This is the same conceptual move that other fields have made at pivotal moments — for example, when the test-time compute scaling paper (referenced in the example analysis) established that inference-time strategy selection is a distinct optimization problem from pretraining scaling, not merely a consequence of having a good model. The collaboration gap paper makes an analogous move: it argues that solving mazes solo and solving mazes collaboratively are different problems that happen to share a task surface, not different instantiations of the same underlying capability.

The distillation finding sharpens the significance. The paper observes that "distilled models further appear disproportionally affected" (Section 4.2), pointing to gpt-5 → gpt-5-nano and gemini-2.5-pro → gemini-2.5-flash-lite drops. In the distributed solo setting, gpt-5-nano achieves a weighted outcome above 0.6 (yellow marker in Figure 4.1), demonstrating reasonable maze-solving ability. In homogeneous collaboration, it collapses to approximately 0.38 (red marker). This is not a proportional degradation — it is a qualitative shift from "can (partially) solve" to "largely cannot collaborate." The paper's interpretation — that "current distillation strategies lose more than just information from the 'tail end' of a base model's knowledge distribution" (Section 4.2) — challenges the prevailing view of distillation as primarily affecting factual recall and reasoning depth. If collaborative capability is disproportionately lost during distillation, then the industry trend toward deploying distilled models (motivated by cost and operational efficiency, as argued by Belcak et al., 2025) may be producing agents that are individually competent but collectively dysfunctional.

Connection to prior work: The paper deliberately echoes the framing of Grosz (1996): "capabilities needed for collaboration cannot be patched on, but must be designed in from the start" (Section 1, Section 5). By demonstrating empirically what Grosz argued theoretically 30 years ago, the paper provides evidence that the field has not heeded this warning — current training paradigms (pretraining on internet text, instruction tuning, RLHF) do not produce collaborative capability as a byproduct, and distillation actively destroys it. This is a diagnostic contribution: it identifies what is missing rather than proposing a fix.

Innovation 2: Interaction Order as the Critical Lever — Priming Over Recovery

The paper's second conceptual contribution is the demonstration that the opening moves of a collaborative dialogue have disproportionate causal influence on outcomes, and that this asymmetry is exploitable: front-loading capability (having a strong model prime the first few turns) is dramatically more effective than back-loading it (having a strong model attempt recovery after weak models have established faulty conventions).

Prior work on multi-agent systems has largely treated agent contributions as symmetric and cumulative — agents contribute information over time, and outcomes improve as more information is exchanged. Order effects, when studied at all, were treated as secondary concerns potentially reflecting minor asymmetries in information revelation. The dominant implicit model was that collaboration quality depends on the total capability applied to the problem, not on when that capability is deployed.

The paper's heterogeneous collaboration results (Section 4.3, Figure 4.3) establish ordering as a first-order effect. In the OpenAI family panel (Figure 4.3a), o3 paired with gpt-4.1-mini achieves a weighted outcome of 0.77 ± 0.04 when o3 speaks first, versus 0.62 ± 0.06 when gpt-4.1-mini speaks first — a gap of approximately 0.15 that is large relative to the confidence intervals and roughly equal to the difference between gpt-4.1-mini homogeneous (0.39) and gpt-4.1 homogeneous (0.55). Strikingly, when the weak model leads, the strong model is dragged down far below its homogeneous performance (0.88), achieving outcomes comparable to mid-tier models paired together.

The relay inference experiments (Section 4.4, Figure 4.6) provide the causal evidence for why order matters. The strong primer condition (Figure 4.6a) shows that a single opening message from o3 ( $K = 2$ relay point, meaning the strong model contributes just turn 1 before being replaced) lifts gpt-4.1-mini homogeneous collaboration from approximately 0.39 to above 0.70 — a near-doubling of performance — and lifts gemini-2.5-flash-lite homogeneous collaboration from approximately 0.36 to approximately 0.65. These gains come from a model that has access only to its own partial map copy, meaning the intervention is purely about how the collaboration is structured, not about providing privileged information. The strong model's opening message establishes: (1) a clear coordinate convention, (2) a shared understanding of the starting position, (3) an initial information-sharing cadence, and (4) a procedural norm for proposing and evaluating moves. Once established, two weak models can productively continue without the strong model present.

The strong recovery condition (Figure 4.6b) makes the negative case equally powerful. When two weak models are allowed to interact for even 4 turns before a strong model is swapped in, the strong model's recovery effectiveness is significantly reduced compared to the full AB collaboration baseline. By $K = 8$ turns, the strong model achieves weighted outcomes barely above the weak–weak baseline. This is not because the strong model cannot solve the maze — it demonstrably can — but because the conventions and errors established in the initial turns are "sticky." Once two agents converge on a (potentially flawed) shared understanding of the coordinate system, the maze layout, and their current position, an incoming agent must either work within that framework (perpetuating errors) or spend precious turns renegotiating it (consuming the turn budget with meta-communication rather than navigation).

Why this is a reframing rather than an optimization: This finding challenges the "ensemble" or "committee" model of multi-agent systems where adding more capable agents always helps. It instead suggests a sequential-dependency model where collaboration quality is path-dependent: early interactions set attractor states that constrain later interactions. This has direct implications for system architecture — invest compute in the handshake phase (initial grounding, schema negotiation) rather than in error recovery — and for evaluation — benchmarks should measure not just whether agents can collaborate but how they initiate collaboration, since initiation quality predicts downstream success.

Connection to human collaboration research: The paper explicitly ties this finding to the concept of "grounding" from Clark and Brennan (1991) and to studies of linguistic convention formation by Garrod and Anderson (1987) and Garrod and Doherty (1994). In human dyads, the first few exchanges establish shared reference points and procedural norms that persist throughout the interaction. The paper shows that this same dynamic applies to AI–AI collaboration, but with the crucial asymmetry that weaker models fail to initiate effective grounding, and once poor grounding is established, even stronger models cannot easily override it. The concept of the "specialist librarian problem" (Taylor, 1968), cited in Section 5, makes this connection explicit: effective collaboration requires first deciphering the partner's needs and conventions before solving the task, and this deciphering step is precisely where weaker models fail.

Innovation 3: Distributed-Information Benchmark Design That Isolates Collaboration from Task Competence

While the benchmark design is described mechanically in Section 3, its intellectual contribution is the measurement methodology itself — specifically, the use of distributed, partially-obfuscated information as a way to create genuine interdependence that cannot be bypassed by individual capability. This is a measurement innovation that addresses a fundamental confound in multi-agent evaluation.

Prior work on multi-agent systems (the paper cites Chen et al., 2024; Fourney et al., 2024; Tran et al., 2025) typically evaluated integrated systems on tasks that could, in principle, be solved by a single sufficiently capable agent. In these settings, observing that a multi-agent system succeeds does not demonstrate that collaboration was effective — it may simply demonstrate that individual agents were powerful enough to solve the task regardless of the multi-agent scaffolding. The collaboration was decorative rather than necessary. Even when tasks were deliberately decomposed across agents, the decomposition was often logical (each agent handles a subtask they're specialized for) rather than informational (each agent lacks information the other possesses), meaning the collaboration requirement could be satisfied by simple output aggregation.

The paper's distributed-information maze design solves this confound by making collaboration structurally necessary: each agent receives a maze copy with approximately 50% of cells obfuscated, with the obfuscation pattern being random and different across copies, meaning that "combined, the two copies recover the complete map" (Section 2.1). Neither agent can solve the maze independently because neither has a complete view of walls and paths. This creates what game theorists call strategic complementarity: each agent's contribution is essential, and the task cannot be decomposed into independent subtasks that are later merged.

Why this matters beyond mazes: The paper explicitly frames the distributed-information methodology as generalizable: "Our 'distributed' information methodology further offers an exciting approach to designing new collaboration tasks, the exploration of which we leave to future work" (Section 5). The key design principle — take a task that can be solved solo, split the information required to solve it across multiple agents such that no single agent has sufficient information, and compare solo-with-full-information performance to collaborative performance — is applicable to any domain with decomposable information structures: legal document review (each agent sees different clauses), medical diagnosis (each agent has different test results), scientific literature synthesis (each agent reads different papers), code review (each agent sees different modules).

The solo distributed baseline as a measurement innovation: By measuring solo performance on the distributed map (presenting one agent with both partial copies simultaneously), the paper creates a control condition that isolates the cognitive challenge of integrating distributed information from the collaborative challenge of communicating with another agent. A model that performs well in distributed solo but poorly in collaborative is specifically failing at the communication and coordination aspects of collaboration, not at the information integration task itself. This decomposition is what enables the paper to define the "collaboration gap" with precision: it is the difference between distributed solo and homogeneous collaborative performance, not the difference between full-information solo and collaborative.

The role of the gpt-5 parsing failure (Appendix F.1) as a validation of the approach: The paper documents that gpt-5 used an unanticipated coordinate schema ("r1c1" notation) that the normalization pipeline failed to handle, causing valid solutions to be misclassified as failures. While presented as a limitation, this actually validates the benchmark's ecological validity: the model did something creative and effective that the test designers didn't anticipate, which is exactly the kind of flexible, on-the-fly communication that open-world agent deployment will require. The failure was in the measurement apparatus, not in the model, and the paper's transparency about it demonstrates that the benchmark genuinely doesn't constrain output formats — the models are free to innovate, and the measurement challenge is real.

Comparison to negotiation benchmarks: The paper's prior work (Davidson et al., 2024) used negotiations to evaluate agent interactions but acknowledged that negotiations "often contain incentives to withhold information and/or deceive." The maze benchmark eliminates this confound by making the payoff structure purely cooperative: both agents share the same goal, there is no advantage to deception or withholding, and the only path to success is through effective information sharing and coordination. This means that failures can be attributed to collaborative inability rather than strategic behavior, which is essential for a measurement instrument — you want to know that the quantity you're measuring (collaborative capability) is what's driving the outcomes.

Innovation 4: The Grim Implication for Distillation — Scaling Down Disproportionately Destroys Collaboration

While the collaboration gap itself is the paper's headline finding, the paper's analysis of which models exhibit the largest gaps constitutes a distinct insight with significant implications for model deployment strategy. The finding is that distillation disproportionately destroys collaborative capability, and the gap does not scale linearly with model size/compute.

The paper compares models along a distillation chain: gpt-5 (base frontier) → gpt-5-mini (distilled) → gpt-5-nano (further distilled), and similarly gemini-2.5-pro → gemini-2.5-flash → gemini-2.5-flash-lite. In both cases, the drop from base to mini/flash is moderate in the solo settings but becomes severe in the collaborative setting. Figure 4.1 shows gpt-5 at ~0.99 collaborative, gpt-5-mini at ~0.86, and gpt-5-nano at ~0.38 — a collapse of roughly 60% from the base model. For the Gemini family: gemini-2.5-pro at ~0.84, gemini-2.5-flash at ~0.76 (modest drop), gemini-2.5-flash-lite at ~0.36 (collapse of ~57%). These are not marginal degradations — the nano/lite models go from "mostly succeeds" to "mostly fails" in a single distillation step.

Contrast with models that are small but not distilled: The paper notes that grok-3-mini — which xAI reports is "not a distilled model, but purpose-built for its size" — achieves a collaborative weighted outcome of ~0.90, rivaling or exceeding much larger models and dramatically outperforming similarly-sized distilled models like gpt-5-nano (0.38) and gemini-2.5-flash-lite (0.36). This suggests that distillation, specifically, damages collaborative capability in ways that training a smaller model from scratch does not. The paper hypothesizes that this is because distillation captures the "center" of the teacher's output distribution (the most likely tokens, the most common reasoning patterns) but loses the tail behaviors that are essential for collaboration — the ability to propose unconventional communication schemas, to recognize when a partner's coordinate conventions differ from one's own, to recover from misunderstandings by re-grounding, and to adapt one's communication style to a partner's needs.

Why this is a fundamental rather than incremental finding: The distillation result challenges the premise behind the industry's SLM-centric agent deployment vision (Belcak et al., 2025, discussed in Section 5). If smaller models are to lead the agentic age, and those smaller models are predominantly produced through distillation, then the collaboration gap documented here creates a hard ceiling on multi-agent system effectiveness. The paper's finding that grok-3-mini — purpose-built rather than distilled — largely avoids this gap suggests that the problem is not smallness per se but how smallness is achieved. This reframes the model compression research agenda: compression techniques should be evaluated not just on benchmark accuracy but on collaborative capability retention, and distillation-specific losses (perhaps through multi-agent training objectives) may be necessary to preserve the interaction patterns that enable effective teamwork.

The claude-sonnet-4 anomaly as supporting evidence: The paper notes (Section 4.3.2, Figure 4.3d) that claude-sonnet-4 is the "sole exception" to the ordering trend — it actually performs better when collaborating with gemini-2.5-flash or gpt-4.1 than with a copy of itself. Qualitative inspection revealed that claude-sonnet-4 in homogeneous pairs got stuck in "agreement loops" or "split up" to explore separately, while heterogeneous partners broke these patterns. This is further evidence that collaborative capability is not a monotonic function of model quality — the specific interaction patterns a model defaults to can be counterproductive when paired with itself but productive when paired with a model that pushes back or proposes different conventions.

Innovation 5: Collaborative Capability as a Property of Interaction, Not of Individual Models

The paper's final conceptual contribution — distributed across the heterogeneous collaboration and relay inference results rather than localized to a single figure — is the demonstration that collaborative performance is a property of the pairing, not of the individual agents, and that this challenges the reductionist approach of evaluating models in isolation.

In the heterogeneous collaboration results (Figure 4.3, Tables 2–3), the paper shows that collaborative performance does not decompose into individual agent scores. The weighted outcome for a pairing A–B cannot be predicted from A's homogeneous score and B's homogeneous score. For example, gemini-2.5-flash-lite homogeneous achieves ~0.36 and gpt-4.1 homogeneous achieves ~0.55, yet their pairing (with gemini-2.5-flash-lite speaking first) achieves only ~0.36 — the stronger model adds essentially nothing. Conversely, gemini-2.5-flash homogeneous achieves ~0.76 and gemini-2.5-flash-lite paired with it achieves ~0.61 (with gemini-2.5-flash speaking first) — a substantial improvement over the weaker model's homogeneous score, despite gemini-2.5-flash being only moderately strong. The paper suggests this may reflect "a certain affinity for models of their own family" (Section 4.3.2), but the deeper point is that compatibility is not reducible to individual capability scores.

This is a measurement-theoretic insight with practical consequences. If collaborative performance is a property of the interaction rather than of the individual agents, then (a) evaluating models in isolation, as most benchmarks do, systematically misses interaction-quality effects, (b) deployment decisions about which model to use in a multi-agent system should be informed by pair-specific evaluation, not just individual benchmark scores, and (c) the concept of a "strong" model is context-dependent — gpt-4.1 is strong in solo benchmarks but weak as a collaborator when paired with certain models, and grok-3-mini is strong in collaboration despite being mid-sized.

The "imitation" dynamic as a mechanism: The paper's qualitative analysis (Section 4.3.1, Figures 4.4 and E.1) provides a mechanistic explanation: models tend to imitate their partner's communication style rather than imposing their own, even when their own would be more effective. o3, which proposes a fully-grounded coordinate schema when initiating (Figure 4.2), partially abandons this approach and adopts a "greedy local strategy" when following gpt-4.1-mini (Figure 4.4). This accommodation behavior — likely a byproduct of alignment training that makes models agreeable and non-disruptive — means that a strong model's capabilities are latent when paired with a weak initiator; they are present but not expressed because the model prioritizes maintaining conversational coherence over optimizing for task success. This is a form of capability suppression that cannot be detected by evaluating the strong model in isolation.

Implications for evaluation methodology: This insight argues against the current paradigm of leaderboard-driven model evaluation, where models are ranked by individual benchmark scores. If a model's effective capability in deployment depends on which other models it interacts with, and on the interaction order, then individual scores provide at best a partial picture. The paper's framework — evaluating models in homogeneous pairs, then in selected heterogeneous pairings, then under relay inference — provides a template for what collaboration-aware evaluation could look like, though the combinatorial explosion of possible pairings remains a practical challenge for full coverage.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses a custom-generated maze-solving benchmark rather than a pre-existing dataset. Mazes are procedurally generated as N×N grids with parameterized wall density (θ), with the main experiments using 6×6 mazes at 30% wall density and optimal solution paths of 7–9 steps between start and goal (Section 3, "Maze Details"). The benchmark consists of randomly sampled maze instances with fixed random seeds to ensure fair comparison across models and settings. Ablation experiments in Appendix B vary maze size (N ∈ {4, 6, 8, 10, 12, 18}) and wall density (p ∈ {0, 0.15, 0.30, 0.45, 0.60, 0.75}). For collaboration settings, each maze is split into two partially-obfuscated copies with approximately 50% of cells hidden in each (replaced with "?" symbols), where the combined copies recover the complete map. The benchmark does not use train/validation/test splits in the traditional sense — all evaluations are on held-out maze instances not seen during any model's training, and the primary comparison is within-maze across experimental configurations (solo vs. collaborative) rather than generalization to new mazes.

Base models. The paper evaluates 32 language models from nine builders (Table 1, Appendix A.1): OpenAI (gpt-5, gpt-5-mini, gpt-5-nano, gpt-4.1, gpt-4.1-mini, gpt-4.1-nano, o3, o4-mini, gpt-oss-120b, gpt-oss-20b), Google (gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite, gemma-3-27b), Anthropic (claude-opus-4.1, claude-sonnet-4, claude-haiku-3.5), xAI (grok-4, grok-3, grok-3-mini), DeepSeek (deepseek-V3, deepseek-R1), Meta (llama-4-maverick, llama-4-scout, llama-3.3-70B-it), Cohere (command-a, command-r, command-r7b), Microsoft (phi-4), Moonshot AI (kimi-k2), and Alibaba (qwen-2.5-72B, qwen-3-235B). Models were selected to "include as many commercially available open- and closed-source models as possible" (Appendix, "Costs of Multi-Turn Rollouts") and span frontier closed-source, frontier open-source, distilled/small variants, mid-tier general-purpose, and smaller open models. For heterogeneous experiments, the paper restricts to a subset from "the most popular providers: Anthropic, Cohere, Google, OpenAI, and xAI" to avoid combinatorial explosion (Section 3, "Models Used").

Metrics. The paper uses two primary outcome metrics (Section 3, "Outcome Metrics"): (1) Binary success rate — the fraction of rollouts where the agents' agreed-upon solution forms a valid path from start to goal (adjacent, contiguous, no wall cells traversed), and (2) Weighted outcome — defined as (a − b) / a, where a is the distance along the optimal shortest path from start to goal and b is the distance from the last valid position reached by the agents to the goal measured along the optimal path. This yields 1.0 for perfect solves, 0.0 for no progress, and negative values for counterproductive movement (when b > a). The paper primarily reports weighted outcomes for their finer-grained discrimination of partial progress. Additional process-level metrics reported in Appendix C include median tokens per collaboration (Appendix C.4, Figure C.1) and median number of messages per collaboration (Appendix C.5, Figure C.2), stratified by weighted outcome ranges (Fail: 0–0.25, Partial Fail: 0.25–0.50, Partial Success: 0.50–0.75, Success: 0.75–1.0).

Baselines. The paper establishes two solo baselines per model (Section 3, "Solo Baseline"): (1) Solo: Full — the model receives the complete, fully-visible maze and must produce a valid path from start to goal, and (2) Solo: Distributed — the model receives both partial map copies simultaneously (representing the same information available to the collaborative pair) and must integrate them to produce a valid path. Both solo settings include a "critic" step where the model reviews its proposed solution and can make adjustments (Appendix A.3, Figure A.3). These baselines control for raw maze-solving ability and distributed information integration capability respectively, enabling isolation of the collaboration-specific deficit.

Generation budget / compute accounting. The paper does not measure compute in FLOPs or inference-time cost; the relevant budget is measured in number of dialogue turns (capped at 50 for all collaborative settings) and number of rollouts (at least 100 for homogeneous and solo settings, at least 50 for heterogeneous pairings, at least 100 for relay inference conditions). The paper acknowledges that cost scales with context length as dialogues progress: for average message length K tokens and T turns, average input context length is K·(T−1)/2 (Appendix, "Costs of Multi-Turn Rollouts"). Token usage efficiency is reported separately (Appendix C.4) but not used as a primary constraint.

Cross-validation / statistical protocol. All metrics are reported with 95% confidence intervals based on standard errors computed over the rollout sample. Maze generation uses fixed random seeds for fair comparison across models and settings. For grading, the paper uses gpt-4.1 as the primary grader and conducts extensive ablation studies (Appendix D) measuring intra-grader consistency (three independent grading passes, ICC and Fleiss' Kappa) and inter-grader consistency (comparing gpt-4.1 against o3 and gemini-2.5-flash) on stratified samples balanced across success/failure outcomes and model builders. The paper also checks for model-specific grading biases (Tables 4c, 5c, 6c, 7c) and uses McNemar's test and paired t-tests (or Wilcoxon signed-rank if normality is violated) for comparing majority-vote grades against reference grades. There is no train/validation/test split for strategy selection — the paper does not train or optimize any model, eliminating the need for held-out data in the traditional sense.

Main Quantitative Results

Solo Baselines: Most Models Can Solve 6×6 Mazes, Distributed Information Reduces Performance

The solo experiments establish that the underlying task is within most models' capability range when information is complete but becomes challenging when information is distributed, even for a single agent. Figure 4.1 (gray markers) shows that more than half of the evaluated models achieve near-perfect weighted outcomes (above ~0.9) in the Solo: Full condition on 6×6 mazes. This establishes that raw maze-solving — planning a path from start to goal given complete visibility — is not the primary bottleneck for these models. In the Solo: Distributed condition (Figure 4.1, yellow markers), where models must integrate two partial map copies, approximately one-third of models show a significant performance drop. The paper reports that many models maintain high performance (o3, gpt-5, grok-3-mini all above ~0.85 weighted outcome), indicating they can handle the cognitive challenge of merging distributed spatial information without collaboration. This baseline is critical because it means that collaborative failures (discussed below) cannot be attributed to models being unable to integrate partial maps — the deficit is specifically in the communication and coordination required to share information between agents.

The maze size ablation (Appendix B.1, Figure B.1a) shows that solo distributed performance degrades sharply for most models when maze size exceeds N = 6. For N = 8, many models drop substantially from their N = 6 performance, and for N = 12, only gpt-5 and o3 maintain near-perfect weighted outcomes. This confirms that the main 6×6 experiments operate in a regime where the task is challenging enough to create meaningful variance but not so hard that most models fail completely.

Homogeneous Collaboration: The Collaboration Gap Is Pervasive and Severity Varies Dramatically Across Models

The central empirical finding of the paper is Figure 4.1 (red markers): virtually all 32 models experience a statistically significant performance drop when moving from distributed solo to homogeneous collaboration, but the magnitude of this drop varies enormously across models and does not simply track solo capability.

For the strongest models, the collaboration gap is small to moderate: gpt-5 collaborative weighted outcome is approximately 0.99 versus ~0.98 distributed solo (near-zero gap), o3 achieves ~0.88 versus ~0.90 (gap of ~0.02, within overlapping confidence intervals), and grok-3-mini achieves ~0.90 versus ~0.92 distributed solo (gap of ~0.02). For mid-tier models, the gap becomes substantial: gpt-4.1 achieves ~0.55 collaborative versus ~0.78 distributed solo (gap of ~0.23), and gemini-2.5-pro achieves ~0.84 collaborative versus ~0.94 distributed solo (gap of ~0.10). The gap is catastrophic for distilled models: gpt-5-nano drops from ~0.62 distributed solo to ~0.38 collaborative (gap of ~0.24), gemini-2.5-flash-lite drops from ~0.48 to ~0.36 (gap of ~0.12), and gpt-4.1-nano drops from ~0.42 to ~0.24 (gap of ~0.18). The paper notes that "small distilled models that solve mazes well alone may fail almost completely in certain pairings" (abstract). The binary success rates in Appendix C.3, Table 3, reinforce this pattern: gpt-5 achieves 0.97 ± 0.02 binary success, while gpt-4.1-mini achieves 0.01 ± 0.01 — near-total failure on the binary metric despite a distributed solo weighted outcome above 0.5.

Maze size ablation for collaboration (Appendix B.1, Figure B.1b). For the subset of strong performers tested at larger maze sizes (N ∈ {6, 8, 10}), collaborative performance "drops considerably for all models except gpt-5." This suggests that the collaboration gap widens with task complexity — larger mazes require more information sharing, more moves to coordinate, and more opportunities for grounding failures to compound.

Qualitative analysis reveals the mechanism behind the gap: grounding failures in the opening message. The paper's qualitative comparison of o3's and gpt-4.1-mini's first messages (Figure 4.2) demonstrates that strong models immediately provide "a fully determined schema, request for missing information, and immediate next steps" while also grounding the starting position. In contrast, weak models "only attempt to ground the meaning of different symbols, without proposing a communication schema, or clearly defining [their] starting position." This difference in opener quality appears to drive much of the performance differential — collaborations that start with clear coordinate conventions and proactive information sharing succeed; those that start with vague proposals and no shared reference system fail.

Heterogeneous Collaboration: Ordering Effects, Cross-Family Affinity, and the Limits of Strong Models Compensating for Weak Partners

The heterogeneous collaboration results (Section 4.3, Figure 4.3, Tables 2–3 in Appendix C) reveal three key findings that are not predictable from homogeneous scores alone.

Finding 1: Ordering matters substantially. In the OpenAI family (Figure 4.3a), o3 paired with gpt-4.1-mini achieves 0.77 ± 0.04 when o3 speaks first but only 0.62 ± 0.06 when gpt-4.1-mini speaks first — a difference of ~0.15 that is well outside overlapping confidence intervals. This pattern recurs across nearly all pairings: the strong model speaking first yields higher weighted outcomes. The single exception noted is claude-sonnet-4 (Figure 4.3d), which "is the sole exception to this trend: its collaborative performance with gemini-2.5-flash and gpt-4.1 surpasses both of the models' homogeneous performance" — though the paper notes this is within confidence intervals and likely reflects claude-sonnet-4's homogeneous failure modes (agreement loops, splitting up) being disrupted by a partner that doesn't share those patterns.

Finding 2: A strong model cannot fully replicate its homogeneous performance when paired with a weak partner. In Figure 4.3a, o3's homogeneous performance is 0.88 ± 0.03, but when paired with gpt-4.1-mini (even with o3 leading), it achieves only 0.77 ± 0.04. The weak partner drags down the strong model — the strong model cannot simply "solve the maze and tell the weak model what to do" because the rules require mutual agreement on each move. The paper's qualitative analysis reveals that strong models adapt to weak partners' communication styles: when o3 follows a gpt-4.1-mini opener (Figure 4.4), o3 "no longer attempts to ground the complete map, instead opting for a greedy 'local' strategy." This accommodation, while conversationally appropriate, prevents the strong model from deploying its full collaborative toolkit.

Finding 3: Cross-family compatibility varies and suggests possible within-family affinity. In Table 2a (Google × OpenAI), gemini-2.5-flash-lite homogeneous achieves 0.36 ± 0.02. When paired with the stronger gpt-4.1, the outcome depends dramatically on who leads: 0.36 ± 0.07 (gpt-4.1 leading) — essentially no improvement over homogeneous. Yet when paired with gemini-2.5-flash (same family), performance improves to 0.61 ± 0.07 (gemini-2.5-flash leading) or 0.71 ± 0.06 (gemini-2.5-flash-lite leading). The paper interprets this as evidence that "models can display a certain affinity for models of their own family" (Section 4.3.2). The binary success rates in Table 3a mirror this: gemini-2.5-flash-lite homogeneous binary success is 0.04 ± 0.02, improving to 0.39 ± 0.08 when paired with gemini-2.5-flash (flash leading) but only 0.13 ± 0.07 when paired with gpt-4.1 (gpt-4.1 leading).

Finding 4: Collaboration can degrade below the weaker model's homogeneous baseline. Table 2c shows that gpt-4.1 homogeneous achieves 0.55 ± 0.04, but when paired with grok-3 (with gpt-4.1 speaking first), the outcome drops to 0.37 ± 0.11 — significantly below gpt-4.1's solo performance. Similarly, gpt-4.1-mini homogeneous achieves 0.39 ± 0.01, but drops to 0.16 ± 0.07 when paired with grok-3 (grok-3 leading). This demonstrates that heterogeneous pairings can be worse than both models' homogeneous baselines — the interaction itself introduces new failure modes that neither model exhibits alone.

Finding 5: grok-3-mini is an "especially capable collaborator." In Figure 4.3d, grok-3-mini achieves 0.90 ± 0.06 with gemini-2.5-flash, 0.90 ± 0.05 with claude-sonnet-4, and 0.82 ± 0.06 with gpt-4.1 — all values close to its homogeneous 0.90 ± 0.04. The paper attributes this to grok-3-mini being "less agreeable than some other models, actively correcting mistakes introduced by weaker partners" and provides a qualitative example (Figure 4.5) showing grok-3-mini successfully resolving a perceptual disagreement with gemini-2.5-flash through persistent re-grounding rather than accommodation of the partner's error. This contrasts with the accommodation behavior observed in other strong models (o3, Figure 4.4) and suggests that collaborative robustness may require a willingness to challenge a partner's incorrect assertions — a behavior that may be suppressed by alignment training that prioritizes agreeableness.

Collaborative efficiency (Appendix C.4, C.5). The paper reports that models adapt their token efficiency to their partner: "on average, agents tend to adapt their efficiency to their partner" (Figure C.1b). When paired with a more verbose partner, models increase their own verbosity; when paired with a more concise partner, they decrease it. Success-conditioned message counts (Figure C.2) show that "some models require significantly more messages to achieve success: grok-4, deepseek-R1, and qwen-3-235B all use the full 50 turns on average even in successful rollouts, indicating inefficient exploration.

Relay Inference: A Single Strong Primer Message Dramatically Boosts Weak Model Collaboration

The relay inference experiments (Section 4.4, Figure 4.6) demonstrate that the collaboration gap can be substantially closed through a minimal, practical intervention — having a strong model contribute only the opening turns of the dialogue.

Strong primer results (Figure 4.6a). For the gpt-4.1-mini test case (top panel):

BB baseline (weak–weak homogeneous): approximately 0.39 weighted outcome
AB baseline (o3 leading gpt-4.1-mini for full dialogue): approximately 0.77
Relay at K = 2 (o3 contributes only turn 1, then replaced by another gpt-4.1-mini): approximately 0.72 — within ~0.05 of the full AB collaboration
Relay at K = 4: approximately 0.73 — essentially maintains the gain
Relay at K = 6 and K = 8: both approximately 0.70 — slight decline but still far above BB

For the gemini-2.5-flash-lite test case (bottom panel), the pattern is similar: BB baseline ~0.36, AB baseline ~0.65, relay at K = 2 reaches ~0.65, and relay at K = 4–8 maintains approximately 0.58–0.62.

The key quantitative takeaway: a single opening message from o3 (K = 2, meaning the strong model contributes one turn before being removed) lifts gpt-4.1-mini homogeneous collaboration from ~0.39 to ~0.72 — a near-doubling — and lifts gemini-2.5-flash-lite from ~0.36 to ~0.65 — an ~80% relative improvement. The paper notes that "this is noteworthy, as o3 only has access to its own incomplete map copy at this point, avoiding the potential confounding of proposing a solution for the entire map" (Section 4.4). The strong model's contribution is purely structural: it establishes a coordinate convention, grounds the starting position, shares its visible information proactively, and models an effective turn-taking procedure — all without seeing the complete maze.

Strong recovery results (Figure 4.6b). In contrast to the primer results, recovery effectiveness diminishes rapidly with K:

For gpt-4.1-mini (top panel): at K = 2, o3 recovery achieves ~0.62 (close to AB baseline of ~0.77); at K = 4, this drops to ~0.52; at K = 8, it drops to ~0.48 — only marginally above the BB baseline of ~0.39.
For gemini-2.5-flash-lite (bottom panel): at K = 2, recovery achieves ~0.52 (compared to AB baseline ~0.65); at K = 4, drops to ~0.42; at K = 8, drops to ~0.38 — essentially indistinguishable from the BB baseline.

The paper concludes that "the more messages the weak models are allowed to exchange, the harder it becomes for a strong model to recover" and that "using strong models to 'seed' collaborations can be more effective and efficient than using them as backup 'experts', jumping in to course correct" (Section 4.4).

Token efficiency in relay (implicit in the design). While the paper does not report token costs directly for relay conditions, the relay inference design implies a cost model: if the strong model's messages are more expensive (o3 is a reasoning model with higher per-token cost than gpt-4.1-mini), then limiting the strong model to K = 2 turns provides most of the benefit at a fraction of the cost of keeping it engaged for the full dialogue. The paper does not quantify this tradeoff numerically, but the logic is inherent in the relay inference concept.

Ablation Studies and Robustness Checks

Maze size ablation (Appendix B.1, Figure B.1): The paper tests solo full, solo distributed, and homogeneous collaborative performance on maze sizes N ∈ {4, 6, 8, 10, 12, 18} for selected models (Figure B.1a for solo, Figure B.1b for collaborative). For solo tasks, "performance drastically drops in the distributed setting for N > 6" for most models, with only gpt-5 and o3 maintaining near-perfect performance up to N = 12. For homogeneous collaboration, five strong models (o3, grok-3-mini, gpt-5, gpt-5-mini, gemini-2.5-pro) are tested at N ∈ {6, 8, 10}, and "performance drops considerably for all models except gpt-5." This confirms that the main 6×6 experiments operate at a difficulty level that is discriminative (not all models succeed, not all fail) and that the collaboration gap is not an artifact of the specific maze size.

Wall density ablation (Appendix B.2, Figure B.2): The paper varies wall density p ∈ {0, 0.15, 0.30, 0.45, 0.60, 0.75} for three OpenAI models (gpt-4.1, gpt-4.1-mini, gpt-4.1-nano) in solo full and solo distributed settings on 6×6 mazes with approximately 30 rollouts per condition. In the solo full setting, gpt-4.1 and gpt-4.1-mini show "the most variance in the range p ∈ [0.30, 0.45]," while gpt-4.1-nano performance "monotonically degrades as wall density increases." In the solo distributed setting, all three models show performance degradation as wall density increases. This ablation is less comprehensive than the size ablation — it does not extend to collaborative settings — but establishes that the main wall density of 30% is within a non-trivial but not maximally difficult range.

Grader consistency: intra-model (Appendix D.1): Three independent grading passes by gpt-4.1 on stratified solo and homogeneous collaborative samples show near-perfect consistency for solo (ICC ≈ 0.99, Fleiss' Kappa ≈ 0.99; Table 4) and high consistency for collaborative (ICC ≈ 0.89, Fleiss' Kappa ≈ 0.87; Table 5). The lower consistency for collaborative transcripts is attributed to greater ambiguity in multi-agent dialogues. Disagreement is concentrated in success cases (14% binary disagreement rate vs. 6% for failure cases in collaborative; Table 5b), with a "small, symmetric, systemic bias for weighted outcomes conditioned on binary outcome, i.e., regression to the mean" (Appendix D.1.1). Agent-specific analysis shows high ICC (>0.80) and Kappa (>0.6) for most models, with only command-a (ICC 0.72, Kappa 0.57) and gpt-5 (ICC 0.69, Kappa 0.72 for collaborative) showing notably lower consistency.

Grader consistency: inter-model (Appendix D.2): Comparing grades from gpt-4.1, o3, and gemini-2.5-flash shows high agreement: ICC ≈ 0.88, Fleiss' Kappa ≈ 0.91 for solo (Table 6); ICC ≈ 0.84, Fleiss' Kappa ≈ 0.77 for collaborative (Table 7). Majority votes "align almost perfectly with the reference" (Appendix D.2.2), with statistically significant but practically small differences (Cohen's d ≈ 0.06 for weighted outcomes in collaborative). The paper explicitly tests for model-specific grader bias and concludes: "We did not find any evidence for statistically significant noise in the grading process or unfair biases across models" (Appendix, "Auto-Grading"). Agent-specific ICC and Kappa values are reported for all evaluated models (Tables 6c, 7c), with gpt-5 (ICC 0.22 for collaborative) flagged as an outlier but attributed to the model's unusual success patterns (the "failed" runs were due to unanticipated maze schemas, not actual failures) confirmed by manual inspection.

Grader schema robustness (implicit in design): The multi-schema normalization pipeline (enumerating plausible coordinate conventions and checking all transformations) serves as an implicit ablation: if the grader infers the wrong schema, the normalization catches it by testing alternative interpretations. The paper reports only one known failure case (gpt-5's "r1c1" notation, Appendix F.1), which "was not anticipated by our parsers" and resulted in valid solutions being misclassified as failures. The paper notes that "the large number of manual author checks do not suggest this to be a widespread problem."

Message efficiency stratification (Appendix C.5, Figure C.2): By stratifying number of messages by weighted outcome range, the paper implicitly ablates whether longer collaborations are associated with better or worse outcomes. Figure C.2 shows that for most models, successful rollouts (0.75–1.0 weighted outcome) require fewer messages than partial successes (0.5–0.75), which in turn require more messages than failures — suggesting that efficient collaboration is characteristic of success, not the result of simply spending more turns communicating. Some models (grok-4, deepseek-R1, qwen-3-235B) buck this trend, using the full 50-turn budget even in successful rollouts.

Unintended model behaviors as implicit ablations: The paper documents several model-specific failure modes (Appendix C.1) that serve as qualitative ablations: command-a/r's premature completion phrase usage ("fantasizing about a successful outcome"), gemini-2.5-pro's unusually short messages (making heterogeneous collaboration difficult despite strong homogeneous performance), claude-sonnet-4's "agreement loops" and "splitting up" behavior, and claude-opus-4.1's inefficient exhaustive exploration. These are not controlled ablations (the experimental design was fixed across all models), but they identify model-specific collaborative failure modes that contribute to the measured gap.

Critical Assessment

The paper's central claim is that a "collaboration gap" exists — that models capable of solving mazes solo degrade substantially when required to collaborate — and the evidence in Figure 4.1 supports this claim robustly: across 32 models spanning nine builders and a wide range of scales and architectures, virtually all models show a performance drop from distributed solo to homogeneous collaboration, with confidence intervals that generally do not overlap between the two conditions for affected models.

However, the claim about distillation disproportionately destroying collaborative capability — while strongly suggested by the Figure 4.1 data — would benefit from more systematic evidence. The paper identifies gpt-5 → gpt-5-nano and gemini-2.5-pro → gemini-2.5-flash-lite as the key distillation chains, but these represent only two distillation paths. The inference that "purpose-built" small models (grok-3-mini) retain collaboration while distilled models lose it rests on a single counterexample. To strengthen this claim, the paper would need to evaluate more models where training methodology (distilled vs. trained from scratch at a given size) is known, or ideally, compare distilled and non-distilled versions of the same base architecture at the same parameter count — an experiment that is practically difficult given limited model release transparency.

The claim that "a single opening message from a strong model dramatically improves performance" is well-supported by Figure 4.6a, but the experiment has an important limitation: the relay configuration uses the same frozen initial turns for each K value, meaning that the relay at K=2 and relay at K=8 experiments are not independent treatments applied to fresh rollouts — they are the same trajectories truncated at different depths. This is acknowledged as a design choice ("ensuring fair comparison across relay points"), but it means that the strong primer experiment effectively measures how much of the final outcome is determined by the first few turns of the same trajectories, rather than testing whether a strong primer message causes better subsequent weak-model behavior across independent rollouts. A stronger causal test would generate fresh rollouts where the primer message is independently manipulated (strong vs. weak opener) and subsequent behavior is measured.

The grading pipeline reliance on gpt-4.1 is well-ablated for consistency (Appendix D), but the ablation cannot rule out systematic biases that affect all graders equally. If all three grader models (gpt-4.1, o3, gemini-2.5-flash) share a blind spot — for example, all misinterpreting a particular coordinate convention in the same way — the multi-grader consistency analysis would not detect it. The paper's manual inspection of a subset of transcripts partially mitigates this concern, but the scale mismatch (manual checks on "a subset" vs. "many tens of thousands of transcripts") means that systematic grading errors affecting a small fraction of rollouts could go undetected.

The solo distributed baseline is critical to the paper's claim that the collaboration gap is specifically about collaboration rather than information integration. However, the solo distributed setting presents both partial maps to a single agent simultaneously, while the collaborative setting distributes them across agents who must communicate sequentially. This means the solo distributed baseline tests a different cognitive operation — parallel map merging vs. sequential information exchange — and the performance difference may partially reflect the difficulty of sequential information integration rather than collaborative communication per se. An additional baseline where a single agent receives the maps sequentially (simulating the turn-by-turn information revelation of collaboration without requiring multi-agent coordination) would disentangle these effects.

The paper claims in Section 5 that "mazes as a lower bound" — arguing that if a collaboration gap exists in this simple setting, it likely widens in more complex real-world scenarios. This is a conjecture, not an empirical finding. The maze size ablation (Figure B.1b) shows that the gap does widen with maze size for most models, supporting the conjecture directionally, but the paper does not demonstrate generalization to qualitatively different collaboration tasks (e.g., collaborative writing, joint decision-making, multi-agent planning) where the nature of grounding, conflict, and coordination may differ substantially. The claim that mazes "simulate many vital aspects of collaboration" (Section 5) is plausible but unverified — the paper would need to demonstrate that maze collaboration performance correlates with collaboration performance on other, more realistic tasks to establish mazes as a valid proxy.

A notable missing experiment is the combination of relay inference with cross-family pairings. The relay experiments use only o3 as the strong model and only within-family weak models (gpt-4.1-mini and gemini-2.5-flash-lite). Given the paper's finding that cross-family compatibility varies (Table 2a shows gemini-2.5-flash-lite improving with gemini-2.5-flash but not with gpt-4.1), it would be valuable to test whether a strong primer from a different model family (e.g., an xAI model priming a Google model) is as effective as a within-family primer. The paper's "specialist librarian problem" framing (Section 5) suggests this should work, but the heterogeneous results suggest it might not.

The sample sizes — 100+ for homogeneous, 50+ for heterogeneous — are adequate for detecting large effects (the collaboration gap itself) but may be underpowered for some of the finer-grained comparisons. For example, the claim that claude-sonnet-4 heterogeneous performance "surpasses both of the models' homogeneous performance" (Figure 4.3d) has overlapping confidence intervals, making it statistically tenuous. The paper appropriately reports confidence intervals, allowing readers to assess the reliability of specific pairwise comparisons.

Finally, the paper's practical recommendations — invest in initial grounding, use strong models as primers rather than recovery agents — follow logically from the results but are demonstrated only in the specific maze-solving context with a specific strong model (o3). Whether the same relay inference benefits would obtain with different strong–weak model pairs, different tasks, or different information asymmetries is an open question that the paper does not address. The relay inference contribution is best understood as a proof of concept that interaction order can be strategically exploited, not as a general solution validated across tasks and model combinations.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in Headline Comparisons

The constraint. The fundamental experimental design — comparing solo against collaborative performance to measure the "collaboration gap" — requires that the distributed solo baseline establishes what a model could achieve given complete information integration capability. But the distributed solo setting presents both partial maps simultaneously in a single prompt, while the collaborative setting requires agents to share information sequentially through multi-turn dialogue. These are cognitively different operations: the solo setting tests parallel map-merging from a single context window; the collaborative setting tests sequential information exchange with a partner who may communicate imperfectly. The paper does not include a baseline where a single agent receives the two partial maps sequentially (simulating turn-by-turn information revelation without multi-agent coordination), which would disentangle the cost of sequential information integration from the cost of interpersonal coordination.

The consequence. The "collaboration gap" measured in Figure 4.1 (red vs. yellow markers) potentially conflates two distinct deficits: (1) the inability to integrate information that arrives piecemeal over multiple turns, and (2) the inability to coordinate with a partner. A model that performs well in distributed solo but poorly in collaborative might fail because it cannot track partial map information across turns (a memory/context issue), not because it fails at grounding or conflict resolution (a collaboration issue). The paper's interpretation — that the gap measures specifically collaborative capability — is partially confounded by this methodological difference.

What evidence exists in the paper. The paper does not include a "sequential solo" baseline. The solo distributed prompt (Appendix A.3, Figure A.2) presents both map copies in a single message, labeled "Map view 1" and "Map view 2." There is no experimental condition where a single agent receives the maps one at a time across multiple turns. The paper acknowledges the general challenge of attributing performance differences to collaboration specifically (Section 2.2 discusses multiple communication challenges that collaboration introduces), but does not address this specific confound between sequential information integration and multi-agent coordination.

Mitigation status. Not addressed. The paper treats the distributed solo baseline as the appropriate comparison point for collaborative performance without discussing the simultaneous-vs-sequential information presentation difference. A sequential solo baseline would be straightforward to implement (present one map in turn 1, the other in turn 2, ask the model to solve) and would sharpen the attribution of the collaboration gap to interpersonal factors rather than information integration modality.

The $4\times$ and $14\times$ Benchmarks Do Not Generalize Beyond 6×6 Mazes With a Single Task Structure

The constraint. All main experiments in Section 4 use 6×6 mazes with approximately 30% wall density and 7–9 step solution paths. The maze size and wall density ablations in Appendix B are partial: size ablation (Figure B.1) tests solo performance across 4–18 sizes but only includes homogeneous collaboration for five models at N ∈ {6, 8, 10}, and wall density ablation (Figure B.2) tests only solo performance across three OpenAI models. The paper does not test how the collaboration gap scales with maze parameters in a fully crossed design, does not evaluate heterogeneous collaboration at different difficulty levels, and does not test generalization to qualitatively different task structures that require different collaborative skills (e.g., tasks where agents must divide labor rather than share partial views, tasks with asymmetric information, tasks requiring joint planning rather than step-by-step move negotiation).

The consequence. The paper's central claims — that a "collaboration gap" exists, that distillation disproportionately damages collaboration, that ordering effects dominate, that relay inference closes the gap — are demonstrated on a single task configuration. The maze size ablation in Figure B.1b shows that "performance drops considerably for all models except gpt-5" when moving from N = 6 to N = 8 or N = 10, but only five models are tested and the specific pattern of the gap across difficulty levels (does it widen proportionally? do certain models exhibit non-monotonic behavior?) is not established. The claim in Section 5 that the maze setting represents a "lower bound" — that the gap would widen in more complex tasks — is a conjecture without empirical support within the paper. If the gap is partially an artifact of the specific maze representation (textual grid of symbols) rather than a general collaborative deficit, the findings may not transfer to tasks with different information structures (e.g., natural language descriptions rather than grid layouts, continuous rather than discrete action spaces, tasks requiring joint creative generation rather than step-by-step planning).

What evidence exists in the paper. The partial maze size ablation (Appendix B.1) is the only evidence addressing generalizability. It shows that distributed solo performance degrades sharply for most models at N > 6 (Figure B.1a), confirming that the 6×6 configuration sits at a discriminative difficulty level. But the limited collaborative ablation (five models, three sizes) cannot establish whether the collaboration gap exhibits systematic scaling behavior — whether it grows proportionally with task difficulty, whether it affects some model families differently at different scales, or whether ordering and priming effects persist at harder difficulty levels. The paper explicitly identifies the single-task limitation: "Clearly, maze environments do not capture the full complexity of real-world collaboration, nor do they serve as a proxy for all types of collaboration" (Section 5).

Mitigation status. The paper is transparent about this limitation in Section 5 ("Mazes as a Lower Bound") and frames it as a conjecture requiring future validation. However, the strength of the paper's prescriptive claims (collaboration must be "designed in from the start," relay inference is an effective strategy, distillation damages collaboration) implicitly assumes that the maze findings generalize. No experiments with alternative task structures (even simple variants — e.g., mazes described in natural language rather than grid symbols, partially-observable mazes where agents control different avatars, tasks where agents have complementary rather than overlapping partial information) are included.

The Oracle Difficulty Effect Is Not Separable From the Task Design — All Failures Look Like Collaboration Failures

The constraint. The paper's experimental design makes collaboration necessary — neither agent can solve the maze alone because each has only partial information — and defines success as reaching the goal. This design cannot distinguish between two fundamentally different failure modes: (1) the agents collaborated effectively (shared information accurately, resolved conflicts, grounded conventions) but the underlying maze-solving challenge was too hard for either agent, and (2) the agents failed at collaboration specifically (miscommunicated, failed to ground, introduced errors through coordination breakdowns) even though each individually could have solved the maze given the combined information. The distributed solo baseline partially addresses this by showing that models can integrate distributed information when given both maps simultaneously, but a model that achieves 0.62 weighted outcome in distributed solo (like gpt-5-nano) and 0.38 in collaborative (Figure 4.1) might be failing partially because sequential information integration degrades its individual reasoning, not because the partner interaction introduced new failure modes.

The consequence. The paper interprets all collaborative performance degradation relative to distributed solo as evidence of a "collaboration gap" — a deficit specifically in collaborative capability. But some fraction of that gap may reflect that models are worse at reasoning about spatial tasks when information is acquired piecemeal through conversation rather than presented in a structured prompt. This is not the same as failing to collaborate; it is failing to maintain and update a spatial mental model across turns, a capability that matters for collaboration but is not uniquely collaborative. The paper's strongest claims — that the gap represents a "distinct axis of capability" (Section 1) and that "capabilities needed for collaboration cannot be patched on" (Section 5, quoting Grosz) — assume that the measured deficit is attributable to collaboration-specific skills (grounding, coordination, conflict resolution) rather than to domain-general sequential reasoning under partial observability.

What evidence exists in the paper. The paper's qualitative analysis of opener messages (Figure 4.2) provides some direct evidence of collaboration-specific failure: gpt-4.1-mini fails to establish a coordinate schema or ground the starting position, behaviors that are clearly about inter-agent communication rather than individual spatial reasoning. The relay inference experiments (Figure 4.6) provide additional evidence: a strong model's opening message, which can only establish communication conventions (since it has no more information than the weak model would have had), dramatically improves weak-model outcomes. This suggests that at least part of the gap is genuinely about communication initiation rather than individual reasoning. However, neither analysis quantifies how much of the total gap is attributable to communication failures vs. sequential reasoning degradation, and the paper does not include a "sequential solo" condition that would enable this decomposition.

Mitigation status. Not directly addressed. The paper consistently describes the gap as about "collaborative capabilities" (Section 1, Section 5, Section 6) without qualifying that some unknown fraction of the measured deficit may reflect domain-general sequential reasoning limitations rather than collaboration-specific skill deficits. The relay priming results (Figure 4.6a) demonstrate that improving communication structure improves outcomes, which is consistent with the gap being partially collaboration-attributable, but does not bound the attribution. Future work could address this by including a sequential solo baseline and by designing tasks where collaboration-specific failures (e.g., convention mismatch, false agreement) can be detected and quantified separately from individual reasoning errors.

The Strong Primer Strategy Assumes Access to a Sufficiently Strong Model With Compatible Communication Patterns

The constraint. The relay inference experiments (Section 4.4) demonstrate that o3 can effectively prime weak models (gpt-4.1-mini and gemini-2.5-flash-lite), dramatically improving their collaborative outcomes with a single opening message. But this result relies on a specific strong model (o3) with two specific weak models from the same or related model families (both available through major API providers). The heterogeneous collaboration results (Section 4.3.2) suggest that this effectiveness may not generalize: gemini-2.5-flash-lite shows almost no improvement when paired with the stronger gpt-4.1 (0.36, essentially equal to its homogeneous 0.36) but substantial improvement when paired with gemini-2.5-flash (0.61–0.71), suggesting possible within-family compatibility effects that make priming more effective when the strong and weak models share training lineage, architectural patterns, or communication styles.

The consequence. In a realistic multi-agent deployment where organizations deploy models from different builders (the very scenario the paper motivates in Section 1), the strong primer strategy may fail if the strong and weak models come from incompatible families. The paper's own data (Table 2a) shows that cross-family strong–weak pairings do not always yield improvements: a stronger model from one family does not necessarily improve a weaker model from another family, even when the stronger model leads the dialogue. If this pattern extends to relay priming — if a gpt-4.1 opener does not improve a gemini-2.5-flash-lite collaboration the way an o3 opener does — then the relay inference strategy is family-dependent rather than general. An organization deploying a Google model as its weak agent may need a Google strong model (rather than any strong model) to serve as an effective primer, limiting the practical applicability of the approach.

What evidence exists in the paper. The heterogeneous collaboration matrices (Tables 2a–2d) provide indirect evidence. Table 2a shows that gemini-2.5-flash-lite (homogeneous 0.36) paired with gpt-4.1 achieves at most 0.36 (gpt-4.1 leading) — zero improvement from the stronger partner. In contrast, gemini-2.5-flash-lite paired with gemini-2.5-flash (same family) achieves 0.61–0.71. This pattern suggests that the "affinity for models of their own family" noted in Section 4.3.2 may limit cross-family priming effectiveness. However, the relay experiments themselves only test o3 (OpenAI) with gpt-4.1-mini (OpenAI) and gemini-2.5-flash-lite (Google) — one within-family pairing and one cross-family pairing. The cross-family case (o3 → gemini-2.5-flash-lite) does show substantial improvement (Figure 4.6a, bottom panel, from ~0.36 to ~0.65 at K=2), suggesting cross-family priming can work. But this is a single data point with a specific strong model, and the heterogeneous results (Table 2a) caution against assuming this generalizes across all strong–weak cross-family pairs.

Mitigation status. Not addressed. The paper does not systematically test relay inference with different strong–weak model combinations across families, does not measure whether priming effectiveness correlates with the strong and weak models' within-family vs. cross-family heterogeneous performance, and does not provide guidance on how to select an effective primer model for an arbitrary weak agent. The Section 5 discussion of the "specialist librarian problem" (Taylor, 1968) implicitly acknowledges that effective priming requires understanding the partner's "true needs," but the paper does not explore whether current models can serve as effective librarians/primers for arbitrary unfamiliar partners.

The Paper Provides No Path to Closing the Gap Through Training

The constraint. The paper is explicitly diagnostic rather than prescriptive: it measures the collaboration gap and demonstrates a deployment-time mitigation (relay inference), but provides no evidence about how to eliminate the gap through training. No fine-tuning experiments are conducted; no multi-agent training objectives are tested; no data augmentation or prompting strategies are evaluated that might improve a model's inherent collaborative capability. The relay inference approach works around the gap at deployment time but does not reduce it — the weak models are still fundamentally poor collaborators when they must initiate interactions themselves. The paper's central prescriptive claim (Section 5, Section 6) is that "collaborative capabilities cannot be patched on, but must be designed in from the start," and that the research community should "treat collaborative intelligence as a core objective to be designed for, not as an emergent property to be hoped for." But the paper provides no empirical evidence that designing for it is feasible with current training methods.

The consequence. A practitioner reading this paper learns that their distilled models will collaborate poorly and that a strong model can partially compensate by priming interactions, but receives no guidance on how to actually improve the weak models themselves. The relay inference strategy assumes continued availability of a strong (expensive) model for the priming phase; in deployments where all agents must be lightweight (e.g., on-device, edge computing, or cost-constrained settings where even a single o3 message is too expensive), the paper offers no solution. The "designed in from the start" prescription, while rhetorically compelling, is not operationalized — what training data, objective, or architecture would produce collaborative capability? The paper leaves this entirely to future work without even a speculative direction.

What evidence exists in the paper. The paper contains zero training experiments. The models are evaluated as-is, without any attempt to improve collaborative performance through fine-tuning, multi-agent RL, or curriculum learning. The relay inference experiments demonstrate that the gap can be worked around but not closed. The paper's strongest evidence that the gap may be training-related is the contrast between distilled models (gpt-5-nano, gemini-2.5-flash-lite) and purpose-built small models (grok-3-mini), which suggests that training methodology affects collaborative capability retention. But this is observational — the paper does not control for or manipulate training methodology, and the claim that distillation "causes" the gap is correlational. No ablation compares the same architecture trained with and without distillation to isolate the causal effect.

Mitigation status. The paper acknowledges this gap in its framing. Section 6 calls for "a paradigm shift" and "challenge[s] the research community to treat collaborative intelligence as a core objective to be designed for, not as an emergent property to be hoped for." But these are aspirational — the paper provides no concrete research direction, no baseline for what "designing for collaboration" would look like, and no evidence that current training infrastructure can support it. The relay inference contribution, while practical, arguably reinforces the status quo (deploy strong models to compensate for weak ones) rather than solving the underlying problem. Future work on collaborative fine-tuning — e.g., training models on multi-agent interaction trajectories, incorporating collaborative success as a reward signal, or designing multi-agent pretraining objectives — is suggested by implication but not scoped by the paper.

The Grading Pipeline Has Systematic Failure Modes That Cannot Be Detected by Agreement-Based Validation

The constraint. The grading pipeline relies on gpt-4.1 to extract coordinates and infer schemas from unstructured dialogue transcripts, with robustness provided by multi-schema normalization (testing many possible coordinate interpretations) and validated by inter- and intra-grader agreement (Appendix D). However, agreement-based validation can only detect random noise or individual grader bias — it cannot detect systematic failures where all grader models (gpt-4.1, o3, gemini-2.5-flash) share the same misinterpretation. If three models all fail to parse a particular coordinate convention or all misidentify which agent proposed which move, they will agree with each other while being wrong. The paper documents one such case (gpt-5's "r1c1" notation in Appendix F.1) but treats it as an isolated failure. It is unknown whether rarer but still systematic misinterpretations affect other models or interaction patterns — for instance, whether the grader systematically misattributes moves when one agent backtracks or corrects an earlier statement, or whether it fails to handle dialogues where agents switch coordinate conventions mid-conversation.

The consequence. The paper's reported metrics (weighted outcomes, binary success rates) could be systematically biased in ways that correlate with model behavior. If certain models tend to use coordinate conventions that are slightly harder for LM graders to parse (e.g., mixing directional and coordinate descriptions, using relative rather than absolute coordinates, or describing moves at different levels of granularity), those models' performance would be underestimated relative to models that use simpler conventions. The paper's finding that gpt-5 achieves near-perfect collaborative performance (0.99 weighted outcome) despite the documented parsing failure suggests that failures affect a small fraction of rollouts, but without human annotation of a representative sample, the magnitude of systematic grading error cannot be bounded. The agreement-based validation in Appendix D establishes precision (graders agree) but not accuracy (graders are correct).

What evidence exists in the paper. The gpt-5 "r1c1" parsing failure (Appendix F.1) is the only documented case of a systematic schema interpretation failure, and the paper notes that "the large number of manual author checks do not suggest this to be a widespread problem." The grading ablation (Appendix D) focuses on agreement statistics (ICC, Kappa, pairwise correlations) and agent-specific consistency — all measures of reliability rather than validity. The paper does not report human annotation of transcripts to establish a ground-truth accuracy baseline against which the LM grader's performance can be measured. The paper does note (Appendix, "Auto-Grading") that "we did encounter at least one instance of an unanticipated maze schema not captured by our normalization process," suggesting other unanticipated schemas may exist at low frequency.

Mitigation status. Partially addressed through redundancy — the multi-schema normalization pipeline enumerates many coordinate transformations, catching errors where the grader misidentifies the schema but extracts moves correctly enough that some transformation yields a valid path. However, this cannot correct errors where the grader extracts the wrong moves entirely (e.g., missing an agreed-upon move, including an un-agreed move, or missequencing moves). The paper's manual spot-checks of transcripts provide some reassurance but are not systematic. A more robust validation would include human annotation of a stratified random sample of transcripts (stratified by model, outcome, and collaboration type) to establish ground-truth accuracy bounds for the automated pipeline. The paper's scale argument ("many tens of thousands of transcripts") makes full human annotation infeasible, but a sampled validation would still provide important calibration.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new diagnostic instrument and measurement methodology rather than a paradigm shift or a solution. Its core contribution to the landscape is establishing that collaborative capability is measurable, that it is distinct from individual task competence, and that it is largely absent from current models when evaluated at scale — three claims that, taken together, redefine what "model evaluation" means in an agentic deployment context.

The methodological shift: from evaluating models in isolation to evaluating them in interaction. Prior to this work, the dominant evaluation paradigm treated language models as solo performers operating on complete information, with leaderboard rankings determined by individual task scores (Chang et al., 2024). This paper demonstrates that such rankings are insufficient for predicting deployment performance when models must collaborate. A model that ranks highly on individual benchmarks may be a poor collaborator (gpt-5-nano achieves ~0.62 solo distributed weighted outcome but collapses to ~0.38 in collaboration, Figure 4.1), while a mid-sized model can be a strong collaborator (grok-3-mini at ~0.90, rivaling much larger models). If the agentic deployment trajectory the paper documents in Section 1 continues — with multi-agent systems becoming the default architecture for complex tasks — then evaluating models only in isolation will systematically mispredict their real-world effectiveness. The paper's framework (solo baseline → homogeneous collaboration → heterogeneous pairings → relay inference) provides a template for what collaboration-aware evaluation could look like, even though the combinatorial explosion of possible pairings remains a practical challenge for comprehensive adoption.

The reframing of collaboration from an application to a capability. The paper's most important conceptual move is recategorizing collaboration: it is not something you do with capable models (an application layer concern), but something models must possess as a distinct capability (a training and evaluation concern). This echoes the trajectory of other capability discoveries in the field — chain-of-thought reasoning was initially treated as a prompting technique before being recognized as a capability that models possess to varying degrees and that can be explicitly trained. The paper provides evidence that collaboration follows the same pattern: the qualitative analysis of opener messages (Figure 4.2) shows that strong models systematically establish grounding conventions while weak models do not, suggesting that collaborative behavior is not simply "general language capability applied to dialogue" but rather a specific set of behaviors (proactive schema proposal, partner information requests, explicit position grounding) that some models have and others lack. The "grim implication" for distillation (Section 4.2) — that distilling models disproportionately destroys their collaborative capability — further supports this reframing: if collaboration were simply a byproduct of general capability, it would degrade proportionally with other skills during distillation, but it degrades much faster, consistent with it being a distinct skill that distillation's tendency to capture "center of mass" behaviors fails to preserve.

Reconciling prior contradictions in the multi-agent literature. The paper provides a unifying explanation for several tensions in prior work. Studies that reported successful multi-agent collaboration (Chen et al., 2024; Fourney et al., 2024) typically used tightly orchestrated architectures where communication protocols were predetermined — the paper's framework reveals that these successes were not measuring the kind of flexible, on-the-fly collaboration that open-world deployment requires. Studies that reported emergent collaborative behaviors (Park et al., 2023) observed role-playing that looked collaborative but lacked measurable task outcomes — the paper's benchmark provides a controlled environment where collaborative behavior can be causally linked to task success. And studies that reported multi-agent system failures (Pan et al., 2025) identified "ineffective communication [and] poor collaboration" as a failure mode without being able to quantify it — the paper's distributed-information design and solo baseline provide a quantitative measure (the collaboration gap) that precisely captures this failure mode. By providing a measurement instrument, the paper converts a fuzzy set of conflicting observations into a coherent research agenda: measure the collaboration gap, characterize its determinants (model size, distillation, interaction order), and develop interventions (relay inference, future training approaches) to close it.

Research directions that become more attractive. The paper makes several research investments more promising:

Collaboration-aware fine-tuning becomes a high-priority applied problem rather than a speculative curiosity. If the gap can be measured precisely (as the paper demonstrates), then training interventions can be evaluated by their effect on that measurement. The existence of the gap establishes a clear optimization target.
Benchmark design incorporating interaction order becomes necessary rather than optional. The paper shows that which agent speaks first can swing outcomes by ~0.15 in weighted outcome (Figure 4.3a), meaning that any multi-agent benchmark that doesn't counterbalance or report ordering effects is measuring noise as much as signal.
Relay/priming architectures for multi-agent deployment become a practical design pattern with empirical justification. Prior to this work, the idea of using a strong model to prime a weak model's collaboration was unvalidated intuition; the paper provides quantitative evidence that it works (Figure 4.6a) and that it substantially outperforms the alternative recovery strategy (Figure 4.6b).
Distillation research incorporating interaction-quality metrics becomes urgent. The paper's finding that distilled models disproportionately lose collaborative capability suggests that current distillation evaluation (which focuses on individual benchmark scores) systematically overlooks a critical deployment-relevant capability. The paper provides a concrete metric (homogeneous collaborative weighted outcome vs. teacher model baseline) that distillation researchers could adopt.

Research directions that become less attractive. The paper also casts doubt on several approaches:

Purely protocol-driven multi-agent architectures — where collaboration is achieved by specifying fixed communication formats (MCP, A2A, ACP) — appear less promising as a general solution. The paper's whole motivation is that "fixed communication protocols are too rigid to handle the diversity of real-world situations" (Section 1). The collaboration gap results suggest that even if protocols exist, models may lack the underlying collaborative capability to use them effectively — a model that cannot ground coordinates with a partner will struggle with any protocol that requires referential communication.
Emergent-collaboration-by-scaling approaches — the implicit assumption that making models larger and training them on more data will naturally produce collaborative capability — appear inadequate. The paper shows that gpt-5 achieves near-perfect collaboration (~0.99), but o3 (also a frontier model) drops to ~0.88, and models only slightly weaker (gpt-4.1 at ~0.55) show substantial gaps. This suggests that collaboration does not smoothly scale with model capability and may require explicit training attention rather than passive emergence.
Recovery-based multi-agent architectures — where strong models are deployed to fix errors after weak models have failed — lose credibility relative to priming-based architectures. The relay inference results (Figure 4.6) show a steep asymmetry: priming a collaboration with 2 strong-model turns achieves near-full-AB performance, while recovery after even 4 weak-model turns shows substantial degradation.

A new diagnostic tool for the field. Beyond its specific findings, the paper's benchmark design provides a reusable template that other researchers can adapt. The key design principles — (1) take a task solvable solo, (2) split the information needed to solve it across agents, (3) compare solo-with-full-information to collaborative to isolate the collaboration-specific deficit, (4) impose no output-format constraints — can be applied to any domain with decomposable information structures. This template is more valuable than any single finding because it enables the field to move from debating whether models can collaborate to measuring how well they do so under controlled conditions.

Follow-Up Research This Work Enables

Collaboration-specific fine-tuning with the benchmark as a training signal. The paper demonstrates that collaborative capability can be measured with precision (the collaboration gap, weighted outcome), but never attempts to improve it through training. The most natural follow-up is to use the benchmark directly as a training environment: fine-tune a model that exhibits a large collaboration gap (e.g., gpt-4.1-mini, gap of ~0.39 from distributed solo to homogeneous collaborative) on multi-agent interaction trajectories where the outcomes are known. A strong experiment would collect trajectories from successful heterogeneous collaborations (e.g., o3–o3 or o3–gpt-4.1-mini with o3 leading), extract the behavioral patterns that distinguish successful from unsuccessful collaborations (explicit coordinate schemas in opener messages, proactive information sharing in early turns, specific conflict resolution strategies when perceptual disagreements arise), and fine-tune the weak model to adopt these patterns. The dependent variable would be the within-model homogeneous collaborative weighted outcome post-fine-tuning, compared against a control group fine-tuned on high-quality solo trajectories from the same maze distribution. If fine-tuning on collaborative trajectories closes the gap while solo fine-tuning does not, it would establish that the collaborative deficit is trainable and separable from task competence.

Cross-task generalization of the collaboration gap. All results are on a single task structure: partially-obfuscated spatial grid mazes requiring step-by-step move negotiation. A critical open question is whether the collaboration gap reflects a general collaborative deficit or is specific to spatial coordination tasks. A strong follow-up would replicate the benchmark's design principles with qualitatively different task structures: (1) a textual reasoning task where each agent sees different premises of a logical argument and must jointly determine the conclusion, (2) a planning task where each agent has different constraints on available actions and must negotiate a jointly feasible plan, or (3) a creative generation task where agents must co-author a document with each contributing different sections that must maintain consistency. For each, the design would include solo-full, solo-distributed, and homogeneous collaborative conditions. If the collaboration gap magnitude correlates across task types (models that drop heavily in maze collaboration also drop heavily in logical or planning collaboration), that would establish the gap as a general model property. If the correlation is weak (some models collaborate well on mazes but poorly on textual reasoning), that would suggest task-specific collaborative skills, implying that collaboration is not a unitary capability and that evaluation must be multi-dimensional. The paper's finding that orders and priming effects dominate makes this extension particularly important: if task-specific collaborative skills exist, then agent selection and priming strategies must be matched to the task type, not just to the model.

Systematic dissection of distillation's effect on collaborative capability. The paper observes that distilled models show disproportionately large collaboration gaps (gpt-5-nano, gemini-2.5-flash-lite), with grok-3-mini (reported as purpose-built rather than distilled) as a counterexample retaining strong collaborative performance. But this comparison is observational — the paper does not control for or manipulate the distillation process. A strong causal experiment would require access to a model family where the same base architecture is available at the same parameter count in both distilled and non-distilled variants, ideally with the distillation process well-documented. The experiment would measure the collaboration gap for the teacher model, the distilled model, and the same-size non-distilled model (if available), decomposing the total collaboration degradation into a "size effect" (how much collaboration capability is lost simply by reducing parameters) and a "distillation effect" (how much additional capability is lost specifically due to distillation's training procedure, over and above the size effect). The paper's finding that grok-3-mini outperforms similarly-sized distilled models suggests the distillation effect is large, but without the counterfactual this remains suggestive. If the distillation effect is confirmed, follow-up work could test whether modifying the distillation objective — for example, using multi-agent interaction data rather than solo outputs as the teacher's target distribution, or incorporating collaborative success as an auxiliary loss — preserves collaborative capability better than standard distillation.

Dynamic relay inference with adaptive handoff. The paper's relay inference experiments use fixed handoff points ( $K \in \{2, 4, 6, 8\}$ ) and demonstrate that priming with a strong model for even the first turn dramatically improves weak-model outcomes (Figure 4.6a). But the handoff point is predetermined rather than adaptive. A natural extension is dynamic relay inference: the strong model monitors the collaboration and decides when to hand off based on observable signals — for example, when a shared coordinate convention has been explicitly established and the first move has been successfully executed, or when the weak model has demonstrated it can independently propose and evaluate moves within the established convention. This would connect to the "exploration-exploitation" framing the paper briefly acknowledges: the strong model spends expensive tokens to establish grounding (exploration), then hands off to the weak model once sufficient grounding is achieved (exploitation). The experiment would compare fixed-handoff relay (at K = 2, 4, 6, 8) against adaptive-handoff relay (handoff triggered by meeting specific grounding criteria, as judged either by a separate LM evaluator or by the strong model itself), measuring both weighted outcome and total strong-model token cost. If adaptive handoff achieves comparable outcomes to fixed-handoff at lower strong-model cost, it would provide a practical deployment optimization over the paper's current approach. The paper's qualitative analysis of what constitutes effective grounding (explicit coordinate schema, proactive information sharing, established turn-taking) provides candidates for the handoff criteria.

Human–AI collaboration evaluation using the same benchmark. The paper focuses exclusively on AI–AI collaboration and frames its motivation around the distinction between human–AI and AI–AI settings (Section 1, Section 5). But the benchmark's design — distributed information requiring communication, minimal output constraints, automated grading — is immediately applicable to human–AI collaboration as well. A strong follow-up would replicate the homogeneous collaboration experiments with one AI agent replaced by a human, measuring: (1) whether humans collaborating with weak-model partners (e.g., gpt-4.1-mini) achieve better outcomes than the weak–weak AI baseline, which would test whether human collaborative skill can compensate for AI collaborative deficits, (2) whether the human receives different benefits from the strong primer strategy (does o3 priming improve human–gpt-4.1-mini outcomes as much as it improves gpt-4.1-mini–gpt-4.1-mini outcomes?), and (3) whether the collaboration gap for AI models is consistent across AI–AI and human–AI pairings (does a model that collaborates poorly with other AIs also collaborate poorly with humans?). This would connect the paper's findings to the established human–AI collaboration literature (Bai et al., 2022; Wu et al., 2025; Zhou et al., 2025), which has largely used asymmetric assistant-style interactions rather than symmetric peer collaboration. The paper's finding that AI models tend to accommodate their partner's style rather than imposing effective conventions (Figure 4.4) has direct implications for human–AI collaboration: if an AI imitates a human's suboptimal communication patterns, the human may need training in effective AI-collaboration techniques (a point the paper raises in Section 5, citing national AI competence initiatives).

Verifier-based collaborative capability scoring without full rollouts. The paper's evaluation requires running full multi-turn collaborative rollouts (up to 50 turns, at least 100 per model) for each experimental condition, making comprehensive pairwise evaluation infeasible at scale. A practically important follow-up would develop a lightweight collaborative capability predictor: a method for estimating a model's likely collaborative performance from cheaper signals than full rollouts. Candidates include: (1) analyzing the quality of a model's opening message when given a standardized partial maze and instructed to initiate collaboration (does it propose a coordinate schema? ground the starting position? share map information proactively?), which could be scored automatically by an LM evaluator without running the full dialogue, (2) measuring a model's performance on a "collaborative coordination" multiple-choice test — given a partner's message, can the model identify whether the partner is using (row, col) or (col, row) conventions? Can it detect when the partner has made a navigational error? — which would isolate theory-of-mind and grounding capabilities without requiring full interaction, or (3) using the model's behavior in the distributed solo setting (which requires no multi-turn interaction) as a predictor, testing whether the distributed solo weighted outcome predicts the homogeneous collaborative outcome across models. If a cheap predictor correlates strongly with full-rollout collaborative performance, it would enable rapid screening of models for collaborative capability without the combinatorial cost of pairwise evaluation — directly addressing the scalability limitation the paper acknowledges.

Practical Applications and Downstream Use Cases

Cost-efficient multi-agent system deployment with relay priming. The paper's strongest immediately actionable finding is the relay inference result (Figure 4.6a): a strong model's opening message (K = 2, meaning just turn 1) lifts gpt-4.1-mini homogeneous collaboration from ~0.39 to ~0.72 weighted outcome, and gemini-2.5-flash-lite from ~0.36 to ~0.65. In a production multi-agent system where a weak, cheap model is deployed for routine interactions but needs to collaborate with unknown external agents, an organization could deploy a relay architecture where a single strong model instance handles only the first message of each new collaboration (proposing a coordinate schema, grounding the shared context, initiating information sharing) before handing off to the weak model for the remaining turns. The cost model is favorable: if the strong model contributes 1 out of ~20 turns (5% of the interaction) but produces ~80% of the performance improvement (bridging most of the gap between weak–weak and strong–weak performance), the cost-benefit ratio is highly asymmetric. The paper's data suggests this is approximately the case — the K = 2 relay point achieves outcomes close to the full AB collaboration baseline for both weak models tested, while using the strong model for only one turn. For organizations running thousands of collaborative interactions daily, this translates to deploying one expensive model instance for a fraction of each interaction rather than for the full duration, with the bulk of the work handled by cheaper models.

Screening distilled models for deployment fitness in multi-agent contexts. The paper's finding that distilled models exhibit disproportionate collaboration gaps (gpt-5-nano at ~0.38 collaborative vs. ~0.62 distributed solo, gemini-2.5-flash-lite at ~0.36 vs. ~0.48) has immediate relevance for model selection in organizations building multi-agent systems. An organization evaluating whether to deploy a distilled model (cheaper, faster) versus its teacher model (more expensive, slower) in a collaborative agent role can use the paper's benchmark design to measure the collaboration gap for their specific deployment task. If the gap is large (as it is for gpt-5-nano, losing ~0.24 in weighted outcome), the cost savings of distillation may be negated by increased collaboration failure rates requiring human intervention or system resets. The paper's methodology — compare solo-on-task to homogeneous-collaborative-on-task with the same information distribution — provides a direct template: distill the model, evaluate both solo and collaborative performance on a representative task sample, and compute the gap. A gap exceeding some deployment-specific threshold (e.g., >0.15 weighted outcome) would flag the model as unsuitable for collaborative deployment despite acceptable solo benchmark scores. The paper's contrast with grok-3-mini (~0.90 collaborative, purpose-built rather than distilled) suggests that organizations may want to prefer purpose-built small models over distilled models for collaborative roles, even at equivalent solo benchmark performance.

Human training for AI collaboration using the benchmark as a skill-building tool. The paper's Section 5 discusses national initiatives to increase people's competence in interacting with AI (Mason, 2025; GSA, 2024) and connects this to the "specialist librarian problem" (Taylor, 1968): effective collaboration requires learning to decipher the partner's "true needs" before solving them. The maze benchmark — with its distributed information, minimal constraints, and measurable outcomes — could serve as a training environment where humans practice collaborating with AI agents of varying collaborative capability. A human trainee would be paired with different model partners (strong collaborator like o3, weak collaborator like gpt-4.1-mini) and receive feedback on their collaborative effectiveness: did they establish clear coordinate conventions? Did they detect and resolve the partner's misunderstandings? Did they adapt their communication style to the partner's level of grounding? The paper's findings provide the curriculum: start with strong-collaborator models where the human can observe effective grounding behavior, then progress to weak-collaborator models where the human must compensate for the AI's grounding failures (the relay inference "strong primer" role, but with the human serving as the strong model). The quantitative nature of the benchmark (weighted outcomes, binary success) enables objective measurement of improvement over time, and the paper's documentation of specific failure modes (coordinate convention mismatches, agreement loops, premature completion) provides concrete skill targets for training.

Protocol design for agent-to-agent communication informed by failure analysis. The paper identifies specific communication failure modes that cause collaboration breakdowns: coordinate convention ambiguity (Figure 2.2), failure to ground the starting position (contrast between o3 and gpt-4.1-mini in Figure 4.2), style imitation that suppresses strong models' capabilities (Figure 4.4), and perceptual disagreements requiring persistent re-grounding (Figure 4.5). For organizations designing agent-to-agent communication protocols — whether MCP, A2A, or custom solutions — these failure modes provide a concrete checklist for protocol requirements. An effective protocol should: (1) mandate explicit coordinate/identifier schemas in the initial handshake rather than allowing implicit conventions to emerge (preventing Figure 2.2-style mismatches), (2) require grounding of shared reference points (start positions, goal states, task boundaries) before any task-relevant communication, (3) include mechanisms for agents to declare their confidence in assertions and to challenge partners' statements without violating conversational norms (enabling Figure 4.5-style corrections without requiring grok-3-mini's particular assertiveness), and (4) specify turn-taking procedures that allow stronger models to lead without weaker models' openers setting suboptimal conventions. The paper's finding that a single strong-model opening message dramatically improves outcomes suggests that protocols should designate the more capable agent (if known) as the initial speaker, or include a capability-negotiation phase where agents signal their collaborative competence before the first task-relevant message is sent. These protocol design implications are directly grounded in the paper's empirical failure analysis rather than being speculative best practices.