MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents

ArXiv: 2603.09827

🎯 Pitch

Current AI cannot answer questions that require stitching together evidence from multiple body-worn cameras over an entire week, even though this is essential for managing collaborative robot teams. This paper introduces a benchmark where the best proprietary model scores only 41%, and reveals that naive video concatenation fails because models drown in redundant long-horizon data—a problem their structured shared memory and agent-wise retrieval partially solves.

1. Executive Summary

This paper introduces MA-EgoQA, a benchmark and baseline model for question answering over multiple long-horizon egocentric video streams simultaneously collected from embodied agents. Built on the EgoLife dataset (six people wearing camera-equipped glasses for seven consecutive days in a shared house), MA-EgoQA provides 1.7k multiple-choice questions spanning five categories unique to multi-agent settings — social interaction, task coordination, theory of mind, temporal reasoning, and environmental interaction — designed to require evidence distributed across multiple agents and timestamps. The paper also proposes EgoMAS (Egocentric Multi-Agent System), a training-free baseline that combines event-based shared memory (a centralized manager integrating fragmented events across agents into structured 4W1H records — When, Where, Who, What, How) with agent-wise dynamic retrieval (generating agent-specific sub-queries from shared memory to selectively retrieve fine-grained evidence from individual agent streams). EgoMAS with a Gemini-2.5-Flash backbone achieves 41.41% accuracy, outperforming the same backbone operating over concatenated captions by 4.48 percentage points and enabling a Qwen3VL-8B-Thinking backbone to match the 1M-token Gemini baseline, establishing that retrieval-augmented multi-agent architectures substantially improve efficiency and accuracy over naive concatenation only when grounded by structured shared memory that aligns agent perspectives before retrieval.

2. Context and Motivation

The Core Problem: Understanding Multiple Embodied Agents Requires System-Level Video Reasoning We Haven't Built Yet

The fundamental question this paper tackles is: if you have multiple embodied AI agents operating simultaneously in a shared environment, each generating continuous egocentric video streams over days, how do you build a system that can answer questions requiring evidence distributed across those agents and timestamps? The paper frames this through the concrete scenario shown in Figure 1 — a human manager querying the system about task progress ("How many times did you clean up the table?"), where the evidence is scattered across agents and time, and the system must aggregate it to produce an answer.

This is not a hypothetical concern. The authors situate it within a clear trajectory (Section 1): as text-based AI advances increasingly transfer to physical embodied domains like autonomous driving and industrial robotics, we can expect a future where multiple intelligent agents assist and augment human activities in shared environments. A growing body of work already studies how these agents can interact and collaborate as a system — decomposing tasks, executing them in parallel, and leveraging collective reasoning to arrive at better solutions. But there is a critical missing piece: contextual understanding and question answering over the experiences accumulated by these agents.

Why This Matters: Transparency, Control, and Real-World Deployment

The paper identifies several concrete application scenarios that make this problem practically urgent (Section 1):

• Supervisory oversight: A human manager needs to query the system about task progress, anomalies, or historical events across a team of deployed agents.
• Security and accountability: Multiple bodycam feeds from police officers or security personnel generate continuous video streams; querying anomalies across these feeds requires multi-agent temporal alignment.
• Household robotics coordination: Understanding when and which household robots performed specific tasks (e.g., cleaning the bathroom) requires aggregating their individual experience histories into a coherent global timeline.

These are not scenarios where a single agent's memory suffices. The query "how many times did you clean up the table?" — when asked of a system managing multiple agents — requires knowing which agents were involved, when they performed the action, and avoiding double-counting when multiple agents interacted with the same object. This is fundamentally a system-level comprehension problem that goes beyond the capabilities of single-agent video understanding systems.

The paper identifies QA functionality specifically as "fundamental for making multi-agent systems transparent, controllable, and manageable" (Section 1). Without the ability to query the collective experience of embodied agents, the system is a black box — agents operate, but their human supervisors cannot audit, understand, or redirect their behavior based on historical context. This directly connects to the broader themes of AI safety, interpretability, and human-AI collaboration that have become central to the field.

Where Existing Approaches Fall Short — And Why They're the Wrong Foundation

The paper identifies a systematic gap across three distinct research directions that each touch on parts of this problem but fail to address the full challenge.

Gap 1: Multi-Agent Embodied Systems Research Ignores Retrospective Understanding

Prior work on multiple embodied agent systems (Section 2.1) has focused almost exclusively on forward-looking task execution:

• CoELA (Zhang et al., 2024) integrates perception, memory, and execution in a modular framework and coordinates plans across agents using natural language via LLMs — but the memory is used for planning future actions, not for answering retrospective queries about what happened.
• Co-NavGPT (Yu et al., 2023) employs a VLM as a global planner to enable multiple robots to explore complex environments — again, planning-oriented.
• PARTNR (Chang et al., 2024) built a large-scale human-robot collaboration benchmark focusing on planning and coordination, demonstrating that even state-of-the-art LLM-based systems struggle with these tasks — but the evaluation centers on execution success, not retrospective QA.
• Work on communication protocols (Jain et al., 2020; Yu et al., 2023) and structured prompting (Guo et al., 2024) focuses on how agents should coordinate during task performance, not on how the system should integrate experiences after the fact.

The paper's key critique is direct: "while these studies have primarily focused on optimizing action execution and cooperation strategies, the problem of integrating egocentric experiences collected over long periods by multiple agents to perform QA remains insufficiently addressed" (Section 2.1). In other words, the multi-agent systems community has built increasingly sophisticated controllers but has not built the corresponding memory interrogation layer that would make those systems auditable and manageable by human supervisors.

Gap 2: Egocentric Video Benchmarks Are Single-Agent and Short-Duration

Existing egocentric video QA benchmarks (Section 2.2) have two critical limitations that make them unsuitable for the multi-agent scenario:

Duration limitation. Most prior benchmarks operate on video segments far shorter than what embodied agents deployed over days would generate:

• EgoSchema (Mangalam et al., 2023) evaluates minute-level understanding with videos averaging 180 seconds — a $5,000\times$ gap from the 266-hour total in MA-EgoQA.
• EgoThink (Cheng et al., 2024) defines six core reasoning capabilities but uses short clips.
• EgoMemoria (Ye et al., 2024) averages approximately 60 minutes per sample.
• EgoExoLearn (Huang et al., 2024) averages 13 minutes.
• EgoPlan-Bench (Chen et al., 2023) targets embodied planning but with short-duration videos.

The paper notes that "current video LLMs are still limited to processing video at the scale of only a few hours, falling short of the requirements for multi-agent QA" (Section 1). This is a practical hardware constraint — even models with large context windows struggle with the raw volume of long-duration multi-stream video.

Single-agent assumption. Even the benchmarks that push temporal scale maintain a single-agent perspective. Questions can be answered by referencing only one individual's memory — there is no requirement to integrate evidence across agents. This is the case even for EgoLifeQA (Yang et al., 2025), which is built on the same EgoLife dataset as MA-EgoQA:

"its QA benchmark, EgoLifeQA, is designed under a single-agent assumption, and questions can be answered by referencing only one individual's memory" (Section 2.2)

This is the critical distinction. EgoLife provides the raw multi-agent multi-day video data, but EgoLifeQA does not exploit the multi-agent structure — questions are answerable from a single perspective. MA-EgoQA explicitly uses the same underlying dataset but constructs questions that cannot be answered without consulting multiple agents' streams, thereby testing a fundamentally different capability.

Missing capabilities in existing benchmarks. Table 1 provides a systematic comparison. No prior benchmark simultaneously satisfies: long duration (days), cross-video reasoning (evidence from multiple streams), and Theory of Mind evaluation. MA-EgoQA is the first to require all three.

Gap 3: The Scale Mismatch Between Video Understanding Models and Multi-Agent Deployments

The paper identifies a practical systems challenge that compounds the benchmark limitations (Section 1 and Section 3.1). In realistic scenarios, embodied agents operating for days each generate massive video histories. A system managing $N$ agents for $T$ hours produces $N \times T$ hours of video. For the EgoLife setting used in MA-EgoQA: 6 agents × 24 hours × 7 days = approximately 1,008 hours of raw video (reduced to 266 hours after curation, but the illustrative point stands).

This creates two interacting problems:

1. Storage and retrieval at scale: The system must build a representation that is compact enough to be practically stored and searched, yet rich enough to answer detailed queries about specific events, interactions, and mental states. Naively concatenating all raw video or all captions into a model's context window is computationally prohibitive and often degrades accuracy by introducing distracting irrelevant information (as demonstrated in the paper's All Caption Concat baselines, where even Gemini-2.5-Flash achieves only 36.93% accuracy despite a 1M-token context window).

2. The cross-agent alignment problem: Even if individual agent streams are compressed effectively, the system still needs to align events across agents — establishing what happened simultaneously, understanding who observed what, and resolving partially overlapping or complementary perspectives on the same event. This is not just retrieval; it requires constructing a coherent system-level temporal and causal model from fragmented first-person perspectives.

How This Paper Positions Itself

The paper situates itself at the intersection of these gaps, defining a new problem formulation (Section 3.1) that makes explicit what prior work has implicitly avoided: the Multi-Agent Egocentric Video Question Answering task.

The formal definition captures both the scale and the cross-agent reasoning requirements: given $N$ agents each recording $T$ hours of egocentric video, producing $N \times T$ total hours, answer queries that require information from more than two agents. The authors explicitly highlight the two resulting challenges:

"First, the system must build a coherent global understanding across agents. To achieve this, each agent's information should be stored in a way that is temporally and contextually aligned with others. Second, the model must effectively retrieve relevant events from the saved representations to answer each query." (Section 3.1)

The paper's positioning is not to claim that existing models are fundamentally incapable — they are not evaluated on this task because the task didn't exist. Rather, the contribution is defining the task, building the benchmark to enable evaluation, establishing baseline performance, and demonstrating that a structured approach (EgoMAS) meaningfully improves over naive strategies. The paper explicitly frames MA-EgoQA as filling a gap that "calls for systematic investigation and comprehensive evaluation to inform better system design" (Section 1), positioning the benchmark as an enabling tool for future research rather than a terminal contribution.

The five benchmark categories (Section 3.2) are deliberately chosen to capture aspects "unique to the multi-agent setting and essential for human-agent system interactions in the real world" — social interaction (grounding casual conversations across perspectives), task coordination (understanding role assignment and collaborative execution), theory of mind (reasoning about what agents believe, perceive, or misunderstand — a capability shown to be particularly challenging in prior single-agent ToM benchmarks like EgoToM and MuMA-ToM), temporal reasoning (aligning timelines and ordering events across agents), and environmental interaction (aggregating object usage distributed across agents). These categories are not arbitrary; they map directly to the practical supervisory scenarios the paper envisions.

In summary, the paper identifies a systematic blind spot: the multi-agent systems community has focused on action execution, while the video understanding community has focused on single-agent short-duration scenarios. Neither addresses the retrospective system-level understanding that would make deployed multi-agent systems transparent and manageable. MA-EgoQA provides the benchmark infrastructure to close this gap, and EgoMAS provides an initial architectural direction.

3. Technical Approach

3.1 Reader Orientation

This paper contributes two things: a benchmark (MA-EgoQA) and a system (EgoMAS) for answering questions about what happened across multiple embodied agents wearing cameras over many days. The core idea is that answering system-level queries — like "who cleaned the table and when?" — requires first compressing weeks of egocentric video into a structured, searchable memory that aligns events across agents, and then dynamically retrieving only the most relevant evidence from the right agents when a question arrives, rather than dumping all video or captions into a model's context window.

3.2 Big-Picture Architecture (Diagram in Words)

The work has two independent but complementary components:

1. MA-EgoQA Benchmark Construction Pipeline: A multi-stage data generation and filtering system that takes raw multi-agent egocentric video as input, generates candidate question-answer pairs using LLMs with category-specific prompts and templates, and then progressively filters them through zero-shot checks, single-agent solvability tests, cross-model verification, and human validation to produce 1,741 high-quality multiple-choice questions requiring evidence from multiple agents.

2. EgoMAS Inference System: A training-free, retrieval-augmented QA system with three stages — (a) an event-based shared memory constructor that takes 10-minute agent captions and produces structured 4W1H (When, Where, Who, What, How) system-level event records, (b) an agent-wise dynamic retriever that first searches the shared memory for relevant events using BM25, then generates agent-specific sub-queries from those events to retrieve fine-grained evidence from individual agent streams, and (c) a response generator that conditions on both the shared memory context and the per-agent retrieved evidence to produce the final answer.

Information flows as follows: raw multi-agent video → per-agent captioning (every 10 minutes) → shared memory construction (event-based 4W1H aggregation) → query arrives → BM25 retrieval from shared memory → agent-specific sub-query generation → per-agent retrieval from individual agent memories → response generation conditioned on all retrieved context.

3.3 Roadmap for the Deep Dive

• First, the MA-EgoQA benchmark's formal task definition, since it establishes what "success" means and why the problem is harder than single-agent video QA — this frames everything that follows.
• Second, the five benchmark categories and what each evaluates, since the data generation pipelines are category-specific and the categories determine what kinds of reasoning EgoMAS must support.
• Third, the benchmark construction pipeline in detail, from raw video to validated QA pairs, because understanding how the questions are built reveals what makes them genuinely multi-agent and genuinely hard.
• Fourth, the EgoMAS system architecture — shared memory construction, agent-wise dynamic retrieval, and response generation — with precise descriptions of each component's mechanism, hyperparameters, and design rationale.
• Fifth, the hyperparameters, model choices, and ablation decisions that define how EgoMAS is configured.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and systems paper whose core idea is that answering questions about multi-agent embodied experiences requires (1) systematically constructing questions that cannot be solved from a single agent's perspective, and (2) building a memory architecture that aligns events across agents before retrieval rather than retrieving from agents independently.

Task Definition

The paper formally defines the Multi-Agent Egocentric Video Question Answering task in Section 3.1. Let there be $N$ embodied agents, where each agent $A_i$ continuously records an egocentric video stream $V_i$ for $T$ hours. The complete multi-agent video collection is:

$V = \{V_1, V_2, \ldots, V_N\}$

where $V$ is the set of all agent video streams, $V_i$ is the $i$-th agent's stream, $N$ is the number of agents (6 in MA-EgoQA), and $T$ is the recording duration per agent (7 days or approximately 168 hours in the EgoLife dataset).

What it computes: This is not a computational equation but a formal specification of the input data structure. It declares that the system's input is a collection of temporally parallel video streams, one per agent, spanning the same time interval. The total raw video volume is $N \times T$ hours (approximately 1,008 hours for 6 agents × 7 days × 24 hours, reduced to 266 hours after curation).

Why this form: The set notation emphasizes that the streams are independent but temporally aligned — each $V_i$ covers the same $T$-hour window, meaning events across streams can be related temporally. This is the key structural property that makes system-level understanding possible and that distinguishes MA-EgoQA from prior benchmarks where only a single stream exists or where streams cover different non-overlapping periods.

Given a user query $q$, the system must generate a response $\hat{y}$ based on information from $V$. The paper specifies that queries in MA-EgoQA require information from more than two agents, enforcing the multi-agent constraint:

$|\{A_i : \text{evidence for } q \text{ exists in } V_i\}| \geq 2$

where the notation $|\cdot|$ denotes cardinality (number of distinct agents whose video streams contain evidence needed to answer $q$). This is not stated as a formal equation in the paper but is the explicit design criterion stated in Section 3.1.

What this constraint enforces: Every question in MA-EgoQA must require the system to integrate evidence from at least two different agents' perspectives. This rules out questions that could be answered by consulting a single agent's memory, which is the defining limitation of prior benchmarks like EgoLifeQA.

Why this constraint matters: Without this constraint, a benchmark built on multi-agent data could inadvertently contain questions that are trivially single-agent — the model could achieve high accuracy by simply picking one agent's stream and ignoring the rest. This constraint ensures that evaluation measures genuine system-level understanding, not just the ability to find the right single-agent source and answer from it.

The paper identifies two main challenges arising from this formulation (Section 3.1):

1. Global coherence across agents: "each agent's information should be stored in a way that is temporally and contextually aligned with others" — meaning the system cannot treat agent streams as independent silos; it must establish when events across agents happened simultaneously, which agents observed the same event from different perspectives, and how individual actions relate to system-level goals.

2. Efficient multi-agent retrieval: "queries may require referencing different timestamps across multiple agents, retrieval becomes significantly more complex and challenging than in a single-agent setting" — meaning the retrieval system cannot simply search each agent's memory independently and concatenate results; it must reason about which agents are relevant to which parts of the query and align retrieved evidence temporally.

Benchmark Categories

The paper designs five categories prior to generating questions (Section 3.2), with the goal of capturing aspects "unique to the multi-agent setting and essential for human-agent system interactions in the real world." Each category targets a distinct reasoning capability:

Social Interaction (SI): Evaluates the ability to accurately localize and ground casual conversations or affiliative behaviors across video streams. Questions address "how people engage or respond to others, group behaviors without specific goal sequences involving multiple people interacting, and meaningful information during the conversation." The key challenge is that social interactions are distributed — Agent A might initiate a conversation, Agent B might respond, and Agent C might overhear and react — requiring the system to stitch together fragments from multiple perspectives to reconstruct the interaction. Examples include "Who helped each other find scissors, and what were the scissors used for?" (Table S1) and multi-span questions like "How did egg tart impressions shift between the sight and bite?" requiring evidence across four agents at two different timestamps (Table S2).

Task Coordination (TC): Addresses how roles were assigned, responsibilities divided, and actions sequenced toward goal completion in collaborative tasks. This category directly reflects the practical concerns of multi-agent systems research: "Questions in this category address how roles were assigned, responsibilities divided, and actions sequenced toward goal completion, and how decisions were made throughout the execution of tasks." Examples include "How did they resolve concerns about the safety of the power strip?" (Table S3, requiring evidence from Jake and Alice at the same timestamp) and multi-span questions like "In the flower tasks, what roles did participants take in different events?" (Table S4, requiring evidence from Jake across two timestamps and Shure at a third timestamp).

Theory of Mind (ToM): Evaluates reasoning about mental states — "what someone believed, assumed, or misunderstood; whether someone was aware of something or not; what someone could or couldn't see or notice; differences in knowledge or beliefs between people; and intentions behind someone's actions or words." This is explicitly identified as the hardest category across all models. The multi-agent setting makes ToM questions particularly rich because different agents have different partial observability — Agent A might witness something Agent B doesn't, leading to different beliefs about the same event. Examples include "What did Katrina wrongly assume while looking at Nicous and Violet filming?" (Table S5) and "Why didn't Katrina know where to put the big kitchen tools?" (Table S5).

Temporal Reasoning (TR): Requires aligning timelines from different egocentric video streams into a coherent global view. The category has two subcategories:

• Concurrency: What one agent was doing while others performed different activities at the same time. These questions require the system to establish that events across agents' streams occurred simultaneously.
• Comparison: The relative temporal ordering of events across agents. These questions require the system to determine which event happened before another when the events are recorded by different agents and therefore appear in different video streams with possibly imperfect time synchronization.

Examples include "Which pair of events happened at around the same time?" (Table S7, concurrency) and "What happened between when Lucia led the puzzle completion and supply planning and when Alice deep-cleaned the kitchen after hot pot?" (Table S6, comparison).

Environmental Interaction (EI): Explores object usage distributed across agents. Since "interactions are distributed across multiple agents, aggregating them is essential for tracking environmental states and planning appropriate actions to achieve shared goals." Questions explore "object usage, including frequency, first-time use, and which agent engaged with them most." Examples include "Who used the microwave the most on DAY3?" (Table S8), "When was the first time guitar was used on DAY2?" (Table S8), and "How many people used oven on DAY1?" (Table S8). These questions appear simple but require the system to accurately count interactions across all six agents' video streams and correctly attribute object usage, avoiding double-counting when multiple agents interact with the same object simultaneously.

For SI and TC categories, the paper additionally designs multi-span questions that require reasoning across multiple non-contiguous temporal windows, "thereby extending MA-EgoQA into a genuinely long-horizon benchmark." Multi-span questions cannot be answered by retrieving a single temporal window; the system must identify and integrate multiple temporally separated events related to a common theme.

Benchmark Construction Pipeline

The benchmark construction process (Section 4) is a multi-stage pipeline designed to produce high-quality, genuinely multi-agent questions while filtering out trivial, single-agent-solvable, or low-quality candidates. The pipeline has three major stages: (I) QA generation, (II) LLM filtering, and (III) human verification. Figure 4 provides an overview.

Stage I: QA Generation

The generation strategy differs by category because the nature of the questions differs. For SI, TC, and ToM — where question formats are diverse and open-ended — the paper generates a large pool of candidate samples and subsequently filters low-quality pairs. For TR and EI — which involve more structured queries — the paper uses predefined templates and instantiates them with contextual information.

Single-span question generation (SI, TC, ToM):

Each long egocentric video from an agent is divided into fixed-length segments of 5 minutes. For each 5-minute window, the system collects dense captions and transcripts from all available agents during that period and provides them to GPT-4o, together with a system prompt and a category-specific prompt.

The system prompt (Figure S3) instructs the model to:

• Generate 10 multiple-choice questions based on the provided multi-agent captions and subtitles
• Ensure questions can only be answered by referencing memories of multiple people (randomly selected from 2–5 people)
• Generate questions as diverse as possible
• Contain 5 answer options with a single correct answer
• Provide a rationale citing specific timestamps from relevant people
• Include a list of timestamp–name pairs indicating which memories should be reviewed

The category-specific prompts provide additional constraints. For SI (Figure S4), the model is instructed to avoid task- or plan-based questions, factual or time-based questions, and belief or thought-based questions — focusing exclusively on "how people relate to each other in casual ways." Questions must "sound natural and always involve at least three people" and avoid excessive "who" questions in favor of "what," "where," "why," and "how" diversity.

For TC (Figure S5), the model is instructed to focus on "goal-driven collaboration between people working toward a shared, meaningful objective" and to explore roles, responsibility divisions, steps toward goal completion, and decision-making during execution — while avoiding casual interactions, isolated facts, thoughts/beliefs, and temporal ordering questions.

For ToM (Figure S6), the model is instructed to focus on "people's mental states — what someone knows, believes, sees, or intends, especially when these differ from reality or from others' knowledge" — including beliefs, assumptions, misunderstandings, awareness, perception differences, knowledge differences, and intentions — while explicitly avoiding factual, social interaction, task planning, and temporal sequencing questions.

Each generated question must be grounded by more than two agents and be relevant to its assigned category. In total, the paper generates 33.4k, 31.6k, and 34.1k candidate QA samples for the SI, TC, and ToM categories respectively.

Multi-span question generation (SI and TC):

Multi-span questions are constructed by grouping semantically similar single-span questions. The procedure works as follows:

1. For each single-span question $q$ and its paired answer $a$, concatenate them and obtain a sentence embedding $e$ using a text encoder $E(\cdot)$:

$e = E\left([q; a]\right)$

where $[q; a]$ denotes string concatenation of the question and answer, $E$ is the text encoder, and $e$ is the resulting embedding vector.

What it computes: Each QA pair is mapped to a fixed-dimensional vector representation that captures its semantic content. The concatenation of question and answer ensures the embedding represents what the question asks about and what the correct answer is, not just the question text alone.

Why this form: Encoding question+answer together rather than just the question ensures that questions about different topics that happen to share similar wording are not incorrectly grouped. The semantic content of the answer disambiguates the topic: "Who used the scissors?" with answer "Jake (for crafts)" has a different semantic footprint than "Who used the scissors?" with answer "Tasha (for cooking)," and the concatenated embedding captures this distinction.

2. Define pairwise cosine similarity between two samples $i$ and $j$:

$s_{ij} = \frac{\langle e_i, e_j \rangle}{\|e_i\| \|e_j\|}$

where $\langle e_i, e_j \rangle$ is the dot product between the two embedding vectors, $\|e_i\|$ is the Euclidean norm of the first vector, $\|e_j\|$ is the Euclidean norm of the second vector, and $s_{ij}$ is the resulting cosine similarity in $[-1, 1]$.

What it computes: For any two QA pairs, a normalized similarity score between 0 and 1 (for non-negative embeddings) indicating how semantically related they are. Pairs about the same topic (e.g., both about coffee-related discussions) will have high similarity; pairs about unrelated topics will have low similarity.

Why this form: Cosine similarity normalizes by vector magnitude, making it invariant to the overall "intensity" of the embedding — two QA pairs about the same topic will have similar embedding directions regardless of how "strongly" the encoder activates for each one. This matters because question length and complexity can vary, and Euclidean distance would be confounded by magnitude.

3. Construct a similarity graph where each vertex corresponds to a QA pair, and an edge $(i, j)$ exists if $s_{ij} \geq \delta$, where $\delta$ is a predefined threshold.

4. Extract connected components from this graph as groups. Each group contains at least two semantically related QA pairs.

5. For each group, provide all single-span questions and their correct answers to GPT-5 along with a system prompt and category-wise prompt. GPT-5 synthesizes a multi-span question and answer only — false options are created separately to ensure difficulty and quality, rather than being inherited from the single-span samples.

Why separate false option creation: If false options were inherited from single-span questions, they might be inconsistent with the multi-span context (e.g., a distractor that is plausible for a single timestamp might be obviously wrong when the question spans multiple timestamps). Creating false options specifically for the multi-span question ensures they are plausible and challenging in the broader context.

The paper generated 15.9k samples for SI multi-span questions and 16.3k samples for TC multi-span questions.

Template-based question generation (TR and EI):

For TR and EI categories, template-based methods improve efficiency because "questions in these categories share common objectives within each category, such as ordering event timestamps or counting object usage" (Section 4.1).

For TR questions, the process is:

1. Create captions for every agent's videos using temporal windows of three different scales: 30 seconds, 10 minutes, and 1 hour. This multi-scale approach "encourages coverage across multiple temporal scales" — fine-grained actions are captured in 30-second windows, sustained activities in 10-minute windows, and broad context in 1-hour windows.

2. Define two subcategories with different construction procedures:

   • Comparison type (ordering events across agents): Randomly sample different timestamps for each agent, collect their captions at the same temporal scale, and provide them to GPT-5 along with a prompt (Figure S7). GPT-5 generates a question with five answer options.
   • Concurrency type (temporal alignment among agents): Randomly select a single timestamp and gather captions from all agents at that time. GPT-5 generates a question with five answer options.

The TR prompt (Figure S7) includes question format templates such as "When [Person A] was [doing X], and [Person B] was [doing Y], what was [Person C] doing?" and "Which pair of events happened at around the same time?" The full template set is listed in Table S9 and includes: concurrent activity questions, event pair matching, event sequencing (with options A through E describing different orderings), first/last event identification from a set of five candidate events, and between-event temporal reasoning ("What happened between when [Person A] was [doing X] and when [Person B] was [doing Y]?").

For EI questions, the process is:

1. Manually list the objects with which agents interact in the shared environment (examples from Figure 4: computer, microwave, guitar).

2. For each object and for each six-hour or one-day interval, collect all dense captions from all agents that include the object. This step aggregates all mentions of a given object across all agents within the specified time window.

3. Pair the collected captions with predefined QA templates (Table S10) and provide them to GPT-5 (Figure S8). The templates include:

   • "When was the first time ([object] was used/[action] done) on DAY[day]?"
   • "Who (used [object]/did [action]) the most on DAY[day]?"
   • "How many people (used [object]/did [action]) on DAY[day]?"
   • "Who was the last person to (use [object]/do [action]) on DAY[day]?"

4. GPT-5 analyzes interactions between the agents and the object within the interval and generates a question with five answer options, a correct answer index, and a confidence score from 1 to 5.

Why templates for TR and EI: These categories have inherently structured question patterns — temporal ordering, concurrency checking, counting, and first/last/most queries — that are difficult to generate in a purely open-ended manner because the question format must precisely match the logical operation being tested. Templates ensure the questions are well-formed for their intended reasoning type while the LLM fills in the specific content (names, actions, objects, times) from the provided context.

Stage II: LLM Filtering

After generating candidate QA pairs, the paper employs an automatic LLM-based filtering pipeline to remove low-quality or insufficiently challenging samples. All filtering steps use GPT-5 and are applied in the following order.

Zero-shot Filtering: Each question is evaluated without providing any context, testing whether the correct answer can be trivially inferred from the question text alone. The paper queries GPT-5 three times per question: if the model produces the correct answer in more than two out of three trials, the sample is discarded. This eliminates questions where the answer is obvious from the wording alone (e.g., questions where one option is linguistically privileged or where common sense makes the answer unambiguous without video evidence).

Why three trials: Using majority vote across three trials reduces the impact of random guessing. A question with five options has a 20% chance of being answered correctly by random chance on any single trial. Requiring 2 out of 3 correct answers significantly reduces the probability of a sample surviving this filter by random chance alone.

Single Agent Filtering: Since MA-EgoQA is designed to evaluate reasoning in a multi-agent context, the paper removes any sample that can be answered using the memory of a single individual. The procedure is:

1. Extract all human names appearing in the question and the correct answer.
2. Perform inference using each corresponding individual's memory (captions from only that agent across all relevant timestamps).
3. If no human name is present, randomly select one agent and conduct inference with their memory.
4. If GPT-5 produces the correct answer under any single-agent memory condition, the sample is eliminated.

Why this step is critical: This is the filter that enforces the multi-agent requirement. Without it, questions generated from multi-agent contexts might inadvertently be answerable from a single perspective — for example, a question about what Jake and Alice discussed might be fully answerable from Jake's transcript alone if his perspective captures the entire conversation. This filter ensures that MA-EgoQA genuinely tests cross-agent integration.

Cross-model Validation: All preceding filtering stages rely on GPT-5, which may introduce model-specific biases — for instance, GPT-5 might find certain questions challenging while other models find them trivial, or vice versa. To address this, the paper includes additional verification using two external models: Gemini-2.5-Flash and Claude-Sonnet-4. Each sample is re-examined under the same contextual information available at generation time. This step checks not only the correctness of the answer but also validates the question and the false options, ensuring the query is non-trivial. If either external model flags a sample as invalid (e.g., identifying an error in the correct answer, finding a false option that is arguably also correct, or deeming the question unanswerable from the provided evidence), it is removed.

Why two external models: Using two independently developed models reduces the risk that a systematic blind spot shared by GPT-5 and one external model (e.g., both being insensitive to a particular type of temporal ambiguity) allows flawed samples to pass. Requiring agreement from two models with different training distributions and architectures provides stronger validation.

Stage III: Human Verification

Following all automatic filtering steps, a final human verification stage ensures the quality and validity of the remaining candidates. Human verifiers are granted access to the complete set of information — including captions, transcripts, and videos from all agents — rather than just the text captions used during generation. Each verifier evaluates samples according to the same guidelines used during data generation, ensuring consistency with the original design criteria.

In total, four human verifiers reviewed 3,436 candidates (the samples that survived all automatic filtering stages), and 1,741 high-quality samples were selected, representing a 50.7% acceptance rate at the final stage.

Why human verification is necessary after LLM filtering: LLMs can miss subtle errors that humans catch — for instance, a distractor option might be factually incorrect in a way that requires detailed video inspection to verify, or a question might be ambiguous in a way that only becomes apparent when watching the original video rather than reading captions. Human verification provides a ground-truth quality check that complements the scalability of LLM-based filtering.

The final benchmark: After all filtering and verification, MA-EgoQA contains 1,741 questions across five categories. The distribution (Figure 3, left) shows: SI = 376, TC = 484, ToM = 235, TR = 287, EI = 359. The questions are well-distributed across the seven recorded days and the six agents (Figure 3, center), ensuring "the evaluation is not dominated by a single skill or event but provides a diverse set of challenges and requires the model to track and reason about all agents effectively" (Section 3.3). Question types (Figure 3, right) are diverse: What (35.4%), Who (28.2%), Why (20.0%), How (10.1%), When (4.0%), Which (2.2%).

Within SI and TC, single-span vs. multi-span breakdown (derivable from Table 3): SI has 379 single-span and an implied 105 multi-span (based on the single vs. multi accuracy breakdown); TC has 324 single-span and an implied 52 multi-span (from the same source). The multi-span questions are substantially harder, as shown by the accuracy drops in Table 3.

EgoMAS: Egocentric Multi-Agent System

EgoMAS is the training-free baseline system proposed alongside the benchmark. The design directly addresses the two challenges identified in the task definition: building coherent global understanding across agents, and efficiently retrieving relevant events for each query. Figure 5 provides an architectural overview.

EgoMAS operates in three sequential stages: (1) shared memory construction (offline, done once), (2) agent-wise dynamic retrieval (online, per query), and (3) response generation (online, per query).

Event-Based Shared Memory Construction

The shared memory is built offline before any queries arrive. At every 10-minute interval, each embodied agent provides a caption summarizing its own observations during the preceding 10-minute time span. These captions are the same dense captions used throughout the benchmark (generated from the raw video by an external captioning system).

A centralized manager — implemented as an LLM prompted with the construction prompt (Figure S10) — receives the captions from all agents for a given 10-minute window and integrates them into a system-level summary. Rather than producing a flat textual condensation, the manager follows a structured 4W1H format:

• Who: All agent names involved in the event (type: list of strings)
• What: The action that happened (type: string)
• Where: The location of the action (type: string)
• When: Implicit in the 10-minute window timestamp
• How: Important details of the action, including direct quotes and step-by-step descriptions (type: string)

The output for each 10-minute window is a JSON list of event objects, each containing the 4W1H fields. An example from the prompt (Figure S10):

[
    {
        "name": ["Jake", "Katrina"],
        "action": "Prepare for the lunch. Make pancakes and omelets.",
        "location": "Kitchen",
        "detail": "Jake first made batter for the pancakes using powder and water, saying 'I need to make batter for the pancakes.'. And then he turned on the stove ..."
    },
    {
        "name": ["Lucia", "Jake", "Katrina", "Tasha"],
        "action": "Have a lunch together. Discuss about the plan for the party.",
        "location": "Living room",
        "detail": "Lucia brought the plates and placed them on the table, saying 'Let's get started with lunch.'. Jake served the pancakes and omelets to everyone. Katrina poured some juice into glasses. Tasha sat down and thanked Jake for cooking. As they ate, Lucia asked, 'So, what's the plan for the party?'. ..."
    }
]

Why 10-minute windows: The 10-minute granularity balances two competing needs. Shorter windows (e.g., 1-minute) would produce too many memory entries, bloating the shared memory and making retrieval expensive. Longer windows (e.g., 1-hour) would conflate unrelated events within the same summary, reducing retrieval precision. Ten minutes is short enough that most windows contain at most a small number of distinct events, but long enough to keep the total number of shared memory entries manageable for the 7-day dataset.

Why the 4W1H structured format: The explicit field structure enables precise retrieval. A query like "Who cleaned the living room?" can be matched against the "action" and "location" fields while retrieving the "who" field provides the answer. A flat textual summary would require the retriever to understand the semantic relationship between "cleaned" and the various ways the summary might describe cleaning activities, whereas the structured format separates the roles (agent, action, location) making it easier for keyword-based retrieval like BM25 to find relevant entries. Additionally, the structured format makes the shared memory human-readable and audit-friendly.

Design choice over alternatives (Table 5a): The paper compares the 4W1H structure against four alternative memory construction strategies:

• Summary: A flat textual condensation without structured fields (accuracy: 30.67%)
• Triplet: (subject, predicate, object) triple extraction (accuracy: 30.44%)
• Chunk: Raw caption chunks without aggregation (accuracy: 25.96%)
• Graph: A graph-based representation (accuracy: 31.99%)
• 4W1H (Ours): The structured event-based format (accuracy: 35.55%)

The 4W1H format outperforms all alternatives by a substantial margin. The paper's interpretation is that this format "demonstrates its strong ability to abstract and fuse the events across agents." The triplet format likely fails because it captures atomic facts without grouping them into coherent events — two triplets about the same interaction (e.g., "Jake made pancakes" and "Katrina poured juice") become disconnected rather than being recognized as parts of the same breakfast preparation event. The summary format likely loses precision because the retriever must match against natural language rather than structured fields. The 4W1H format provides structured fields for precise matching while using the "detail" field to preserve rich context.

What the shared memory enables: By aggregating individual agent captions into system-level event records, the shared memory provides the "coherent global understanding across agents" that the task definition requires. An event that spans multiple agents — like a group lunch — appears as a single entry in the shared memory with all participants listed, rather than as separate fragments in each agent's individual memory. This alignment happens at construction time (offline) rather than at query time (online), reducing the online retrieval burden.

Agent-Wise Dynamic Retrieval

When a query $q$ arrives, EgoMAS executes a three-step retrieval process:

Step 1: Shared Memory Retrieval. The system retrieves the top-$n$ system-level memories from the shared memory $\mathcal{M}_{\text{shared}}$ using BM25 ranking:

$\mathcal{R}_{\text{sys}}(q) = \text{Top-}n\left\{ (m, s(m, q)) \mid m \in \mathcal{M}_{\text{shared}} \right\}$

where $\mathcal{M}_{\text{shared}}$ is the complete set of shared memory entries (across all 10-minute windows for all 7 days), $m$ is an individual memory entry (a 4W1H event record), $s(m, q)$ is the BM25 relevance score between memory entry $m$ and query $q$, $\text{Top-}n$ selects the $n$ entries with the highest scores, and $\mathcal{R}_{\text{sys}}(q)$ is the set of retrieved system-level memories with their scores.

What it computes: A ranked list of the top-$n$ shared memory entries most relevant to the query, using lexical matching (BM25). For a query like "Who cleaned the living room?", BM25 scores each shared memory entry based on term frequency and inverse document frequency of query terms like "cleaned," "living," and "room" in the entry's text fields.

Why BM25: The paper compares BM25 against dense retrievers in Table 5b:

• BM25: 35.55% accuracy
• DPR (dense passage retrieval): 28.67%
• Qwen3-Embed-0.6B: 33.03%
• NV-Embed-v2 7B: 37.91%

NV-Embed-v2 achieves slightly higher accuracy but "contains 7B parameters and consequently incurs substantial computational overhead." BM25 "employs lightweight keyword-based retrieval and delivers competitive performance, highlighting its practicality." The dense retrievers may suffer from the domain shift — they are typically trained on web text or general QA datasets, not on structured event summaries from egocentric video. BM25's lexical matching is more robust to this shift because it doesn't rely on learned semantic representations that may not transfer.

Hyperparameter $n$: The number of shared memory entries retrieved is set to $n = 20$ in the main experiments. Ablation in Figure S1 shows that performance is stable for $n \in [20, 60]$ and drops when $n$ is too small (insufficient context) or too large (noise dominates).

Step 2: Agent-Specific Sub-Query Generation. From the retrieved system-level context $\mathcal{R}_{\text{sys}}(q)$, EgoMAS generates a set of agent-specific retrieval requests. Using the prompt shown in Figure S9, the LLM is instructed:

"Based on the context, select one person (should be one of the list: Jake, Alice, Katrina, Lucia, Tasha, Shure) and generate query which memory should be referred to from that person. The question could be a sub-question for the original question."

The output is a list of agent–query pairs:

$\mathcal{Q}_{\text{agent}} = \{(a_j, q_j)\}_{j=1}^J$

where $a_j$ is an agent identifier (name), $q_j$ is a sub-query tailored for that agent's memory, and $J$ is the number of agent-specific retrieval requests generated (typically 1–5 agents per query).

What this step accomplishes: The shared memory provides a system-level overview of relevant events, but it is a compressed representation — the 4W1H fields may not contain all the fine-grained details needed to answer a specific question. The agent-specific sub-queries target each relevant agent's full individual memory to retrieve detailed evidence. For example, the shared memory might indicate that "Jake and Katrina prepared lunch in the kitchen at Day 3, 12:00," but to answer a question about "what did Jake say about the pancake batter?", the system needs to retrieve Jake's specific transcript from that time period — which is stored in Jake's individual agent memory but may not be fully captured in the shared memory's "detail" field.

Why generate agent-specific sub-queries rather than using the original query for all agents: The original query is formulated from a system-level perspective (e.g., "Who cleaned the living room?"). Individual agent retrievals benefit from agent-specific reformulation (e.g., for Jake: "cleaning activities in living room," for Katrina: "cleaning living room"). This reformulation can improve retrieval precision by focusing on what each agent specifically did rather than the general event.

Step 3: Per-Agent Memory Retrieval. For each agent–query pair $(a_j, q_j)$, EgoMAS performs agent-level retrieval from that agent's individual memory $\mathcal{M}_{a_j}$:

$\mathcal{R}_{a_j}(q_j) = \text{Top-}k\left\{ (m, s(m, q_j)) \mid m \in \mathcal{M}_{a_j} \right\}$

where $\mathcal{M}_{a_j}$ is agent $a_j$'s individual memory (captions from that agent across all timestamps), $s(m, q_j)$ is the BM25 score between memory entry $m$ and sub-query $q_j$, and $\text{Top-}k$ selects the $k$ most relevant entries. The hyperparameter $k$ is set to $k = 5$ in the main experiments.

To ensure relevance, a score threshold $\tau$ is applied to filter out low-scoring retrievals:

$\tilde{\mathcal{R}}_{a_j}(q_j) = \{(m, s(m, q_j)) \in \mathcal{R}_{a_j}(q_j) \mid s(m, q_j) \geq \tau\}$

where $\tau = 10$ in the main experiments. Entries with BM25 scores below this threshold are considered insufficiently relevant and discarded.

Why a score threshold: Without a threshold, every agent query returns exactly $k$ results regardless of relevance. For agents that are not actually relevant to the query, these $k$ results would be noise — tangentially related entries that could distract the response generator. The threshold ensures that only genuinely relevant entries from each agent are passed to the response generator. The ablation in Figure S1 shows that higher thresholds ($\tau \geq 18$) decrease accuracy by filtering out too many relevant entries, while moderate thresholds ($\tau \in [2, 12]$) maintain stable performance.

Response Generation

The final response is generated by conditioning the backbone LLM on both the retrieved system-level context and the aggregated agent-level results:

$\hat{y} = \mathcal{F}\left(q, \mathcal{R}_{\text{sys}}(q), \tilde{\mathcal{R}}\right)$

where $\mathcal{F}$ is the response generation function (the backbone LLM prompted with the original question and all retrieved evidence), $\mathcal{R}_{\text{sys}}(q)$ is the set of top-$n$ shared memory entries and their scores, and $\tilde{\mathcal{R}} = \bigcup_{j=1}^J \tilde{\mathcal{R}}_{a_j}(q_j)$ is the union of all threshold-filtered per-agent retrieval results.

What the LLM receives as input:

1. The original query $q$
2. The top-$n$ shared memory entries (system-level event summaries with 4W1H fields)
3. For each agent-specific sub-query, the top-$k$ (filtered by $\tau$) agent memory entries

The LLM then generates the final answer, typically as a multiple-choice selection from the five provided options.

Why hierarchical retrieval (shared → per-agent) rather than flat retrieval across all agent memories: A flat retrieval across all agent memories would face two problems. First, the retrieval space is enormous (6 agents × 7 days × 10-minute windows = ~6,048 captions per agent × 6 agents = ~36,288 captions total), making it computationally expensive and prone to retrieving tangentially related entries. Second, flat retrieval loses cross-agent event structure — two captions from different agents that describe the same event from different perspectives would be retrieved independently without any indication that they refer to the same event. The hierarchical approach first identifies system-level events (which inherently align agent perspectives), then drills down into agent-specific details within those events. This preserves the event structure while enabling fine-grained retrieval.

Why centralize the shared memory rather than having agents communicate peer-to-peer: The centralized design (a single manager that aggregates all agent captions) ensures there is a single coherent view of system-level events. In a peer-to-peer design, each agent would maintain its own view of events, and inconsistencies between agents' perspectives would need to be resolved at query time. The centralized shared memory resolves these inconsistencies at construction time (offline), producing a clean event representation that the retriever can search efficiently.

EgoMAS Hyperparameters and Configuration

The paper evaluates EgoMAS with four different backbone models: Gemini-2.5-Flash, Qwen3VL-8B-Thinking, Qwen3VL-8B-Instruct, and Qwen2.5VL-7B-Instruct. The hyperparameters are:

• Shared memory retrieval size $n = 20$: The number of top-ranked shared memory entries retrieved by BM25. Ablation in Figure S1 shows performance is stable for $n \in [20, 60]$; $n = 10$ reduces accuracy (insufficient context), and $n = 100$ also reduces accuracy (noise dominates relevant signal).

• Agent-wise retrieval size $k = 5$: The number of top-ranked agent memory entries retrieved per sub-query. Ablation in Figure S1 shows robust performance across $k \in [1, 20]$, with $k = 5$ providing a good balance of sufficient evidence without excessive context length.

• Score threshold $\tau = 10$: BM25 scores below this threshold cause retrieved entries to be discarded. Ablation in Figure S1 shows accuracy is stable for $\tau \in [2, 12]$ and drops for $\tau \geq 18$ (filtering out too many relevant entries).

• Input context size: EgoMAS uses substantially less input context than concatenation baselines. The paper reports context sizes: EgoMAS (Gemini-2.5-Flash) uses 4.6k tokens on average; EgoMAS (Qwen3VL-8B-Thinking) uses 5.4k tokens; EgoMAS (Qwen3VL-8B-Instruct) uses 7.4k tokens; EgoMAS (Qwen2.5VL-7B-Instruct) uses 5.4k tokens. Compare to the All Caption Concat baselines requiring 128k–1M tokens.

Why the token efficiency matters: The ~100× reduction in context length compared to concatenation baselines has two benefits. First, it reduces computational cost — processing 5k tokens costs roughly 1/20th of processing 100k tokens. Second, it reduces distraction — providing only relevant context (rather than all captions) prevents the model from being confused by irrelevant information that happens to share keywords with the query.

Oracle setting: The paper also evaluates an Oracle setting where the response generator receives "the same context used to generate the questions" — i.e., precisely the ground-truth evidence for each question. Oracle accuracy (Gemini-2.5-Flash: 83.80%, Qwen3VL-8B-Instruct: 73.98%) represents an approximate upper bound on what can be achieved with perfect retrieval, and the gap between EgoMAS and Oracle (roughly 42 percentage points) indicates the headroom available for better retrieval methods.

Ablation Studies on EgoMAS Components

Section 7.2 reports several controlled experiments that isolate the contribution of each EgoMAS component.

Shared memory and dynamic retrieval ablation (Table 4): Using Qwen2.5VL-7B-Instruct as the backbone, the paper tests all four combinations of shared memory (present vs. absent) and dynamic retrieval (present vs. absent):

• No shared memory, no dynamic retrieval: 27.80% accuracy — the model receives concatenated raw captions from all agents as context without any retrieval.
• No shared memory, dynamic retrieval only: 28.20% accuracy — the model generates agent-specific sub-queries and retrieves from individual agent memories, but without a shared memory to identify which events are relevant system-wide.
• Shared memory, no dynamic retrieval: 30.04% accuracy — the model retrieves from shared memory but does not drill down into agent-specific memories; responses are based solely on the compressed 4W1H event summaries.
• Shared memory + dynamic retrieval (full EgoMAS): 35.55% accuracy.

What this ablation reveals: Each component provides a meaningful gain. Dynamic retrieval alone (28.20%) barely improves over no retrieval (27.80%), suggesting that without shared memory to identify which system-level events are relevant, generating effective agent-specific sub-queries is difficult — the system doesn't know which agents to query or what to ask them. Shared memory alone (30.04%) provides a more substantial gain (+2.24 over no retrieval), confirming that the system-level event alignment is valuable even without fine-grained agent retrieval. But the combination (35.55%) substantially outperforms either component alone (+7.75 over no retrieval, +5.51 over shared memory alone), demonstrating that the two components are complementary: shared memory identifies which events are relevant, and dynamic retrieval retrieves detailed evidence about those events from the right agents' memories.

Sensitivity to number of agents (Figure S2): The paper evaluates EgoMAS accuracy when limited to $1, 2, 3, 4, 5,$ or all $6$ available agents. Accuracy increases monotonically with the number of available agents: from 31.99% with one agent (single-agent setting, essentially disabling the multi-agent requirement) to 35.55% with all six agents. This confirms that MA-EgoQA questions genuinely require multi-agent information — performance improves as more agent perspectives are made available, indicating that the additional perspectives provide non-redundant evidence for answering the queries.

Memory retriever comparison (Table 5b): Four retrievers are compared for the shared memory retrieval step:

• DPR (dense passage retrieval): 28.67% accuracy
• Qwen3-Embed-0.6B: 33.03% accuracy
• NV-Embed-v2 7B: 37.91% accuracy
• BM25 (used in EgoMAS): 35.55% accuracy

NV-Embed-v2 achieves the highest accuracy but is a 7B-parameter model, which "consequently incurs substantial computational overhead." BM25 provides competitive accuracy with essentially zero computational cost (it's a statistical retrieval method with no learned parameters), making it the practical choice for deployment.

Memory structure ablation (Table 5a): Five shared memory construction strategies are compared:

• Summary (flat text): 30.67%
• Triplet (subject-predicate-object): 30.44%
• Chunk (raw captions): 25.96%
• Graph (graph-based representation): 31.99%
• 4W1H (EgoMAS): 35.55%

The 4W1H structure outperforms all alternatives by at least 3.56 percentage points. The paper attributes this to the structured format's ability to "abstract and fuse the events across agents" — the explicit Who/What/Where/How fields support precise retrieval while the detail field preserves rich context.

Latency comparison (Figure 6): Measured over 100 randomly selected MA-EgoQA samples, EgoMAS (Qwen2.5VL-7B-Instruct) achieves approximately 35.55% accuracy with approximately 1.3 seconds of inference latency per query. Non-retrieval baselines like Gemini-2.5-Flash (36.93% accuracy) require substantially higher latency (tens of seconds) due to processing 1M-token contexts. Other retrieval baselines like BM25 (36.01% accuracy) also achieve low latency (~1–2 seconds) but lower accuracy than EgoMAS. The key observation is that EgoMAS achieves the highest accuracy among retrieval-based methods while maintaining low latency, "highlighting that EgoMAS is a practical approach for building a multi-agent egocentric video QA system in real-world settings."

Why EgoMAS is training-free: The entire system operates by prompting pre-trained LLMs — no fine-tuning, no training of retrieval components, no learned embeddings. The shared memory construction uses an LLM prompted with the construction template; the agent-wise dynamic retrieval uses an LLM prompted with the sub-query generation template; the response generation uses an LLM prompted with the retrieved context. BM25 requires no training. This makes EgoMAS immediately deployable on any backbone LLM without requiring training data, which is an important practical advantage given the scarcity of multi-agent egocentric video QA training data.

Why EgoMAS can use relatively small backbone models: The key insight is that retrieval offloads the memory burden from the LLM's context window. A model like Qwen2.5VL-7B-Instruct cannot process 266 hours of video captions in its context window — but EgoMAS only provides it with approximately 5k tokens of highly relevant retrieved evidence. The model's reasoning capability (which is strong even in 7B-parameter models) can then be applied to this focused evidence, producing answers that rival much larger models operating over concatenated contexts. This is why EgoMAS (Qwen3VL-8B-Thinking) at 5.4k tokens achieves 40.26% accuracy, surpassing Gemini-2.5-Flash at 1M tokens (36.93%) and GPT-5 at 272k tokens (34.81%).

4. Key Insights and Innovations

Innovation N: Multi-Agent Egocentric Video QA as a First-Class Problem Definition

The paper's most fundamental contribution is not a technical method but the conceptual act of formally defining a problem that the field had systematically overlooked: question answering over multiple temporally-aligned egocentric video streams from embodied agents operating simultaneously in a shared environment. This may sound like a straightforward problem formulation exercise, but it required recognizing an genuine blind spot that existed at the intersection of two well-developed research communities, neither of which had reason to address it on their own.

What the field did before this work. The multi-agent embodied systems community (Zhang et al., 2024; Yu et al., 2023; Chang et al., 2024) built increasingly sophisticated architectures for task execution, planning, and coordination — but their evaluation frameworks measured forward-looking success: did the agents complete the task? The retrospective understanding problem — a human supervisor asking "what happened and why?" — was not on their radar because their systems were evaluated by execution metrics, not by QA accuracy. Meanwhile, the egocentric video understanding community (Mangalam et al., 2023; Cheng et al., 2024; Yang et al., 2025) built benchmarks testing progressively longer videos and more complex reasoning, but always from a single perspective. Even EgoLifeQA, built on the same EgoLife dataset as MA-EgoQA, was "designed under a single-agent assumption" where "questions can be answered by referencing only one individual's memory" (Section 2.2). The paper's Table 1 makes the gap explicit: no prior benchmark simultaneously required long-duration video, cross-video evidence integration, and Theory of Mind reasoning.

Why the formulation is distinctive, not obvious. The insight is that multi-agent retrospective QA is not simply a scaled-up version of single-agent video QA with more input — it requires a qualitatively different capability. When six agents record 266 hours of egocentric video across seven days, the system cannot just search each agent's memory independently and concatenate results. It must establish temporal alignment across streams (did Agent A's "kitchen cleanup" happen at the same time as Agent B's "puzzle completion"?), resolve partial perspectives on the same event (Agent C saw Jake enter; Agent D heard Jake speak but didn't see him), and avoid double-counting interactions that multiple agents participated in (if Jake, Alice, and Katrina all interacted with the oven, how many distinct oven-usage events occurred?). These are system-level integration challenges that single-agent benchmarks, by design, never test.

The paper's formalization in Section 3.1 captures this by specifying that queries must require information from more than two agents. This constraint is not arbitrary — it is what transforms a multi-stream dataset into a genuine system-level reasoning benchmark. Without it, a model could score well by identifying which single agent has the most relevant perspective and answering from that agent's stream alone, which is exactly what happens when the paper evaluates EgoMAS with single-agent memory (Figure 7, substantial performance drop) and what the single-agent filtering stage removes during benchmark construction. This is a fundamental reframing, not an incremental benchmark contribution. It creates an evaluation category that did not exist before, and the difficulty of the resulting benchmark — Gemini-2.5-Flash at 36.93% with 1M-token context, EgoMAS at 41.41% with structured retrieval, Oracle at 83.80% — demonstrates both the challenge and the headroom.

Evidence anchoring. The benchmark construction pipeline's single-agent filtering stage (Section 4.2) is the empirical proof that this is a genuinely new capability being tested: samples solvable from a single agent's memory are explicitly removed. Figure 7 confirms that MA-EgoQA questions cannot be answered well without multi-agent access, with EgoMAS accuracy dropping substantially when restricted to a single agent's perspective. Figure 8 further shows that accuracy monotonically decreases as the number of required agents increases, confirming the benchmark's design intent.

Innovation N: Shared Memory Before Retrieval as an Architectural Principle for Multi-Agent Systems

EgoMAS's architecture embodies a simple but non-obvious principle: when you have multiple agents generating independent observations, align their experiences into system-level events BEFORE retrieval, not after. This inverts the natural tendency to treat each agent's stream as an independent retrieval target and instead constructs a centralized event representation that integrates perspectives at construction time.

What the field did before this work. The default approach to multi-source retrieval — implicitly assumed by the RAG baselines in the paper and by most retrieval-augmented systems — is to index each source independently and retrieve from them in parallel or sequentially, combining results only at the response generation stage. WorldMM-8B (Yeo et al., 2025) exemplifies this: it "retrieves each agent's memory iteratively" (Section 7.2 case study) but without an intermediate aggregation step. The paper's ablation in Table 4 confirms that dynamic retrieval without shared memory achieves only 28.20% accuracy — barely above the no-retrieval baseline of 27.80%. The reason, as the case study in Figure 9 illustrates, is that independent per-agent retrieval can find relevant fragments but cannot reconstruct the system-level event structure that gives those fragments their interpretive context.

Why this is distinctive. The architectural insight is that centralized event alignment and distributed evidence retrieval are complementary operations that must be sequenced correctly. The shared memory (constructed offline) answers the question "what events occurred across the system?" — resolving which agents were involved together, what the system-level action was, and where and when it happened. The agent-wise dynamic retrieval (executed online) then answers "what were the specific details from each relevant agent's perspective?" — drilling into the fine-grained evidence that the compressed shared memory may not preserve.

What makes this non-obvious is that the shared memory is not just an index or a compression — it is a structured transformation of the raw data. The 4W1H format (When, Where, Who, What, How) imposes a specific ontology on the event representation that is designed to support the downstream retrieval and reasoning operations. Table 5a shows that alternative memory structures — flat summaries (30.67%), triplets (30.44%), raw chunks (25.96%), graphs (31.99%) — all underperform the 4W1H format (35.55%) by substantial margins. This is not because the 4W1H format contains more information — it likely contains less detail than full summaries — but because its structured fields make the information searchable by BM25 in ways that map directly to question patterns. A query asking "Who cleaned the living room?" matches the "action" and "location" fields; a query asking "What happened in the kitchen on Day 3?" matches the "location" and temporal window. The structure enables precision.

Significance beyond performance. This is a systems architecture insight rather than a modeling insight — it applies regardless of the specific backbone LLM used (the paper demonstrates it with Gemini, Qwen3VL-Thinking, Qwen3VL-Instruct, and Qwen2.5VL — all benefit). It suggests a general design principle for any multi-agent system that needs to support retrospective QA: invest compute in building a structured shared event log offline, then use lightweight retrieval at query time. This decouples the expensive cross-agent alignment (done once) from the query-specific retrieval (done per query), enabling low-latency responses (~1.3 seconds for EgoMAS, per Figure 6) that rival concatenation baselines requiring orders of magnitude more compute.

Evidence anchoring. Table 4 directly demonstrates complementarity: shared memory alone = 30.04%, dynamic retrieval alone = 28.20%, combined = 35.55%. The ablation on shared memory structure (Table 5a) shows the 4W1H format outperforming alternatives. The case study in Figure 9 visualizes the failure mode of independent per-agent retrieval (WorldMM fails to find the answer) versus the success of the hierarchical shared-then-per-agent approach (EgoMAS succeeds).

Innovation N: Multi-Agent Theory of Mind as a Distinctively Hard Benchmark Category, Not Just an Incremental Addition

The paper identifies Theory of Mind (ToM) as the hardest category in MA-EgoQA — but the innovation is not simply reporting that ToM is hard (which prior work like EgoToM and MuMA-ToM already established in single-agent settings). The distinctive insight is that multi-agent ToM introduces reasoning requirements that are qualitatively different from single-agent ToM, and current models exhibit a specific failure pattern that reveals fundamental limitations in how they integrate perspectives.

What the field did before this work. Prior ToM benchmarks for egocentric video (EgoToM, Li et al., 2025; EgoThink, Cheng et al., 2024) evaluated whether models could infer what a single observed agent believes, perceives, or intends — a third-person ToM task where the model watches one agent's video and answers questions about that agent's mental state. MuMA-ToM (Shi et al., 2025) introduced multi-modal multi-agent ToM but in short-duration clips (~36 seconds average) rather than the days-long setting of MA-EgoQA. These benchmarks established that ToM is challenging, but they did not surface the specifically multi-agent ToM challenges that MA-EgoQA exposes.

Why multi-agent ToM is qualitatively different. In a single-agent ToM setting, the model needs to infer one agent's mental state from that agent's observable behavior — essentially, "what does this person think is happening?" In a multi-agent ToM setting, the model must track divergent mental states across agents who have different partial observability of the same events. Agent A witnessed something Agent B didn't; Agent C misunderstood what Agent D intended. The ToM questions in MA-EgoQA exploit this: "What did Katrina wrongly assume while looking at Nicous and Violet filming?" (Table S5) requires understanding that Katrina's belief was inconsistent with reality; "Why didn't Katrina know where to put the big kitchen tools?" (Table S5) requires understanding that Katrina lacked access to information that other agents had (Shure had placed things randomly).

What makes this genuinely harder is that the model must track not just one agent's mental state but a set of potentially inconsistent mental states, and then reason about which one is relevant to the query. This is not merely "more of the same" reasoning; it requires maintaining multiple belief models simultaneously and understanding how they relate to each other (who knows what, who observed what, whose beliefs are correct vs. incorrect).

The diagnostic value of ToM as a stress test. The paper's results show that ToM consistently has the lowest accuracy across all models (Table 2) — but the more interesting pattern is how models fail. The paper observes that ToM "requires inferring latent mental states, such as goals and beliefs, rather than extracting explicitly observable visual or textual cues" and notes "the inherent ambiguity, the need for recursive mental-state reasoning, and complex goal–belief–action inference" as sources of difficulty (Section 7.1). This is a diagnostic finding: ToM serves as a stress test that reveals whether a system is doing genuine perspective-taking or pattern-matching against surface-level cues. The fact that even EgoMAS — which substantially outperforms baselines on ToM (33.62% vs. 24.26% for Gemini concatenation) — still lags far behind Oracle (70.21% for ToM vs. 83.80% average Oracle) suggests that retrieval alone cannot close the ToM gap; there is a reasoning capability deficit that retrieval does not address.

Evidence anchoring. Table 2 shows ToM as the lowest-scoring category for nearly all models. The Oracle gap for ToM (Gemini Oracle 70.21% vs. EgoMAS Gemini 33.62% = 36.59 percentage points) is larger than the average Oracle gap (~42.4% as stated in Appendix C), suggesting ToM is disproportionately dependent on perfect evidence retrieval because the reasoning itself is fragile when evidence is incomplete or noisy.

Innovation N: Retrieval as a Substitute for Context Length — With Sharp Empirical Boundaries

The paper demonstrates that a 7B-parameter model with structured retrieval (EgoMAS Qwen2.5VL-7B-Instruct, 35.55%, 5.4k tokens) can match or exceed proprietary models operating over concatenated contexts orders of magnitude larger (GPT-5 at 272k tokens, 34.81%; Gemini-2.5-Flash at 1M tokens, 36.93%). But the innovation is not just the efficiency gain — it is the empirical characterization of when retrieval works and when it doesn't for this task, establishing boundaries that are useful for system design.

What the field did before this work. The RAG literature has extensively demonstrated that retrieval can substitute for long-context processing in knowledge-intensive NLP tasks. But video QA — especially multi-agent, days-long egocentric video QA — is a fundamentally different regime. The "knowledge" is not factual (did the Eiffel Tower open in 1889?) but episodic (what did Jake do on Day 3 at 2 PM?). The retrieval targets are not documents but temporal segments of embodied experience. And the cross-agent alignment requirement means that retrieving the right temporal segment from one agent is insufficient if the evidence spans multiple agents. The paper's experiments reveal that vanilla RAG approaches tuned for text QA — BM25 (36.01%), DPR (26.19%), VideoRAG (25.33%) — substantially underperform the structured retrieval of EgoMAS (41.41% with Gemini backbone), confirming that this domain requires domain-specific retrieval design.

The key empirical finding: retrieval quality dominates model scale for this task. EgoMAS with Qwen3VL-8B-Instruct (a relatively small model) at 7.4k tokens achieves 37.68%, surpassing GPT-5 at 272k tokens (34.81%). EgoMAS with Qwen3VL-8B-Thinking at 5.4k tokens achieves 40.26%, surpassing Gemini-2.5-Flash at 1M tokens (36.93%). This is not a small efficiency win — it is a regime change: for multi-agent egocentric video QA, providing a model with the right 5k tokens is more valuable than providing it with all 1M tokens. The paper's latency analysis (Figure 6) reinforces this: EgoMAS achieves ~1.3 seconds per query versus tens of seconds for 1M-token models, making it practically deployable.

But there are hard boundaries — and identifying them is the contribution. The paper does not claim that retrieval universally replaces context. Three boundaries are evident:

1. The Oracle gap (~42 percentage points) shows that retrieval is the bottleneck, not reasoning — even with perfect retrieval, the model (Gemini) achieves only 83.80%, and the gap between EgoMAS (41.41%) and Oracle (83.80%) is dominated by retrieval failures. This means improving the retriever is the highest-leverage research direction.

2. Category-dependent sensitivity to retrieval quality (Tables 2 and 3): ToM and multi-span questions show disproportionately large Oracle gaps, suggesting these categories require more precise retrieval than current methods provide. The paper's own analysis notes that "both aspects [multi-span reasoning and multi-agent knowledge fusion] significantly contribute to increasing the task difficulty" (Section 7.1).

3. Visual information matters but is underutilized (Table S11): Providing video frames in addition to captions improves performance on some categories (SI, TR, EI) but degrades it on others (TC, ToM), suggesting that naive frame inclusion can distract the model. The paper identifies this as an open problem: "future research should focus on adaptively leveraging the necessary modalities and developing effective methods for frame selection" (Appendix H).

Why this is a significant finding beyond this paper. The question of whether retrieval can substitute for context length in long-form video understanding is unresolved in the literature, with conflicting results across domains. This paper provides one of the clearest demonstrations that, for a specific well-defined multi-agent QA task, the answer is a qualified yes — but only when retrieval is structured to match the task's cross-agent alignment requirements. The boundary conditions (Oracle gap, category sensitivity, modality interaction) provide a concrete research agenda for improving retrieval in this domain.

Evidence anchoring. Table 2 shows EgoMAS (Qwen3VL-8B-Thinking) at 40.26% surpassing Gemini-2.5-Flash concatenation at 36.93% and GPT-5 at 34.81%. The Oracle results (Gemini 83.80%, Qwen3VL 73.98%) define the upper bound. Table S11 shows the modality interaction pattern. Figure 6 shows the latency-accuracy tradeoff.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use MA-EgoQA, the benchmark introduced in this paper. It consists of 1,741 multiple-choice questions with five options each, built on the EgoLife dataset (six people wearing camera-equipped glasses for seven consecutive days in a shared house, producing approximately 266 hours of egocentric video). Questions are distributed across five categories: Social Interaction (376), Task Coordination (484), Theory of Mind (235), Temporal Reasoning (287), and Environmental Interaction (359). Each question requires evidence from more than two agents and is answerable only by integrating information across multiple egocentric video streams. The benchmark is designed as a test set — there is no explicit training/validation/test split because EgoMAS is training-free and the LLM-based baselines are evaluated zero-shot. However, the benchmark construction process used two-fold cross-validation in spirit through its cross-model verification stage (Section 4.2), where two external models independently validated each sample.

• Base model(s). EgoMAS is evaluated with four backbone models spanning a range of capabilities and scales: Gemini-2.5-Flash (proprietary, Google), Qwen3VL-8B-Thinking (open-source, ∼8B parameters with chain-of-thought reasoning), Qwen3VL-8B-Instruct (open-source, ∼8B parameters), and Qwen2.5VL-7B-Instruct (open-source, ∼7B parameters). The baselines span an even wider range: proprietary LLMs include Gemini-2.5-Flash and GPT-5; open-source LLMs include Llama-3.1-Nemotron-8B (8B parameters, 1M-token context), Qwen2.5-7B-Instruct-1M (7B parameters, 500k-token context actually used due to limits), Qwen3-30b-a3b-instruct (30B MoE with 3B active, 240k-token context), gpt-oss-120b (120B parameters, 128k-token context), and gpt-oss-20b (20B parameters, 128k-token context). Video LLM baselines include VideoChat-Flash, VideoXL-2, and Qwen2.5-VL-7B. The range of model scales (7B to 120B+ parameters) and architectures (dense, MoE, text-only, video-capable) is chosen to establish that MA-EgoQA is challenging across the capability spectrum and that EgoMAS benefits are not tied to a specific model family.

• Metrics. The primary metric is accuracy (%) — the fraction of the 1,741 questions for which the selected answer matches the ground-truth correct option. Since each question has five options, random chance accuracy is exactly 20%. Results are reported as overall average accuracy across all 1,741 questions and, in some analyses, broken out by category (SI, TC, ToM, TR, EI), by sub-category (single-span vs. multi-span within SI and TC; concurrency vs. comparison within TR), and by the number of agents required to answer the query (Figure 8). Inference latency (seconds per query) is also reported for a subset of methods (Figure 6), measured over 100 randomly selected MA-EgoQA samples. Input context size (number of tokens provided to the response generation model) is reported for each method in Table 2 (ranges from 4.1k to 1M tokens).

• Baselines. The paper evaluates four categories of baselines, each representing a different approach to the multi-agent video QA problem:

  All Caption Concat Baselines (Table 2, top block): These concatenate captions from all agents in chronological order and pass them directly to the LLM along with the question. Models include Gemini-2.5-Flash (1M-token context), Llama-3.1-Nemotron-8B (1M-token context), Qwen2.5-7B-Instruct-1M (500k-token context), GPT-5 (272k-token context), Qwen3-30b-a3b-instruct (240k-token context), gpt-oss-120b (128k-token context), and gpt-oss-20b (128k-token context). The caption granularity depends on the model's maximum context length: 10-minute window captions for the 1M-token models, 1-hour window captions for others. If the resulting context exceeds the limit, captions are truncated until the length fits.

  All Frame Concat Baselines (Table 2, second block): These concatenate all video frames captured by the agents over 7 days and uniformly sample frames until each model can process them without errors. Models include VideoChat-Flash (10k frames sampled, 32k-token context), VideoXL-2 (4k frames, 32k-token context), and Qwen2.5-VL-7B (1.9k frames, 128k-token context). This baseline tests whether visual information alone (without transcripts) can support multi-agent video QA.

  RAG Baselines (Table 2, third block): These use retrieval-augmented generation over the multi-agent video data. VideoRAG (Jeong et al., 2025) employs an InternVideo2 multimodal encoder to extract embeddings for each 30-second clip and its caption, retrieves five clips by cosine similarity to the query, uniformly samples 4 frames per clip, and provides frames + captions to the response generator (Qwen3-VL-8B-Instruct). EgoRAG (Yang et al., 2025) is designed for single-agent egocentric video; since it does not support multi-agent scenarios, the paper uses videos from only a single person. Ego-R1-Agent-3B (Tian et al., 2025) and WorldMM-8B (Yeo et al., 2025) are also designed for single-agent settings; the paper uses an LLM router to select which individual's memory to refer to. BM25 (Robertson et al., 2009) and DPR (Karpukhin et al., 2020) are text-based retrieval methods: both select five 30-second clips using their respective retrieval scoring, and four frames uniformly sampled from each clip, together with their captions, are provided to Qwen3-VL-8B-Instruct as the response generator.

  Oracle (Table 2, bottom): Generates answers using the same context (captions, transcripts, and videos from the specific agents and timestamps) that was used to originally generate the questions. This represents an approximate upper bound on accuracy achievable with perfect retrieval, tested with both Gemini-2.5-Flash (83.80%) and Qwen3VL-8B-Instruct (73.98%).

  Random (Table 2, top row): Always selects a random option from the five choices (theoretically 20% accuracy), serving as the floor.

• Generation budget / compute accounting. EgoMAS is training-free — there is no fine-tuning, no learned retrieval components, no trained embeddings. The "compute" for EgoMAS is measured in two ways. First, input context size (tokens): EgoMAS with Gemini-2.5-Flash uses 4.6k tokens on average, versus 1M tokens for the Gemini-2.5-Flash concatenation baseline — a >200× reduction. Second, inference latency: measured as wall-clock seconds per query over 100 randomly selected samples (Figure 6), EgoMAS (Qwen2.5VL-7B-Instruct) achieves approximately 1.3 seconds versus tens of seconds for non-retrieval models processing large concatenated contexts. For the caption concatenation baselines, compute is implicitly measured by the context length required (128k to 1M tokens). For the frame concatenation baselines, compute is measured by the number of frames sampled (1.9k to 10k). There is no explicit FLOP counting; the paper relies on token counts and latency as practical proxies.

• Cross-validation / statistical protocol. There is no formal cross-validation because EgoMAS is training-free and all models are evaluated zero-shot on the fixed 1,741-question test set. However, the benchmark construction process incorporated several validation stages that serve an analogous quality-control function: cross-model verification using two external models (Gemini-2.5-Flash and Claude-Sonnet-4) to validate each QA pair independently of GPT-5 (the generation model), and final human verification by four annotators reviewing 3,436 candidates and selecting 1,741 (50.7% acceptance). These stages ensure benchmark quality but do not constitute a statistical validation of model performance. No confidence intervals or statistical significance tests are reported for the accuracy numbers. All results in Table 2 are point estimates over the full 1,741-question set.

Main Quantitative Results

The paper evaluates 16 baselines plus EgoMAS (in 4 backbone configurations) plus Oracle (in 2 configurations), for a total of 22 evaluation settings. The results are organized into: overall accuracy comparison (Table 2), sub-category analysis (Table 3), difficulty factor analysis (Figure 7, Figure 8), and efficiency analysis (Figure 6).

Overall Accuracy: MA-EgoQA Is Genuinely Hard, and EgoMAS Outperforms All Baselines

Table 2 presents the main results. The headline finding is that no method exceeds 42% accuracy (compared to 20% random chance and 83.80% Oracle), establishing MA-EgoQA as a challenging benchmark where substantial headroom exists for future improvement.

Random baseline confirms the floor. Random selection achieves exactly 20.00% across all categories, as expected for 5-option multiple choice.

All Caption Concat Baselines show that larger models and longer contexts help, but only modestly. The strongest concatenation result is Gemini-2.5-Flash at 36.93% average accuracy using a 1M-token context window. This is 16.93 percentage points above random chance. GPT-5 — a substantially more capable model — achieves only 34.81% with a 272k-token context, actually underperforming Gemini-2.5-Flash despite presumably stronger reasoning ability, suggesting that context length matters more than raw model capability for this task when using concatenation — Gemini's 1M-token window captures more of the 266-hour video history than GPT-5's 272k-token window, and this additional coverage outweighs GPT-5's reasoning advantage.

Smaller open-source models with moderate context windows perform substantially worse: Qwen2.5-7B-Instruct-1M at 500k tokens achieves 26.08%; gpt-oss-120b at 128k tokens achieves 26.82%; Qwen3-30b-a3b-instruct at 240k tokens achieves 25.56%; and Llama-3.1-Nemotron-8B at 1M tokens achieves only 21.65%, barely above random chance. The Llama-3.1-Nemotron-8B result is particularly notable — despite having a 1M-token context comparable to Gemini, it achieves near-random performance, suggesting that long context alone is insufficient; the model must also be capable of effectively utilizing that context for multi-agent reasoning.

All Frame Concat Baselines perform worst overall. VideoChat-Flash (10k frames, 32k-token context) achieves 23.06%; VideoXL-2 (4k frames, 32k-token context) achieves 20.39% (essentially random); Qwen2.5-VL-7B (1.9k frames, 128k-token context) achieves 25.22%. These results are attributed by the authors to "the absence of transcript information and the presence of many irrelevant frames" (Section 6.2). The frame concatenation approach suffers from two simultaneous problems: (1) without transcripts, the model must extract all semantic information from visual input alone, which is challenging for long-duration egocentric video where important events may be visually subtle (e.g., conversations, object interactions that look similar); and (2) uniformly sampling frames across 7 days of video means most sampled frames are irrelevant to any given query, creating a needle-in-a-haystack problem.

RAG Baselines demonstrate that retrieval helps but naive retrieval is insufficient. The strongest RAG baseline is BM25 at 36.01% with only 8.1k tokens of context — nearly matching Gemini-2.5-Flash's 36.93% at 1M tokens (>100× more context). This is the first concrete evidence that retrieval can substitute for context length in this domain. Other RAG baselines are less successful: WorldMM-8B achieves 27.63% (4.1k tokens), DPR achieves 26.19% (7.8k tokens), VideoRAG achieves 25.33% (8.0k tokens), and EgoRAG achieves 19.57% (since it only uses a single agent's video). The BM25 result suggests that lexical matching over captions is surprisingly effective for this task, likely because the questions are constructed from caption-derived events, creating a direct lexical overlap between query terms and relevant caption text that BM25 exploits. DPR's lower performance (26.19% vs. 36.01% for BM25) indicates that dense retrieval, which relies on learned semantic representations, may suffer from domain shift — the embedding models were not trained on egocentric video caption data and may not capture the specific event structures relevant to MA-EgoQA queries.

EgoMAS substantially outperforms all baselines. The four EgoMAS configurations occupy the top four non-Oracle positions in Table 2:

• EgoMAS (Gemini-2.5-Flash): 41.41% (4.6k tokens) — +4.48 percentage points over Gemini-2.5-Flash concatenation (36.93%)
• EgoMAS (Qwen3VL-8B-Thinking): 40.26% (5.4k tokens) — surpassing GPT-5 (34.81%) and Gemini-2.5-Flash (36.93%)
• EgoMAS (Qwen3VL-8B-Instruct): 37.68% (7.4k tokens)
• EgoMAS (Qwen2.5VL-7B-Instruct): 35.55% (5.4k tokens) — nearly matching Gemini-2.5-Flash concatenation (36.93%) with a ~200× smaller context

The cross-backbone scaling trend is clear: accuracy increases with backbone model capability, from Qwen2.5VL-7B-Instruct (35.55%) to Qwen3VL-8B-Instruct (37.68%, +2.13) to Qwen3VL-8B-Thinking (40.26%, +2.58) to Gemini-2.5-Flash (41.41%, +1.15), suggesting that "stronger reasoning ability enables more effective retrieval strategies and better integration of information retrieved from multiple agents" (Section 6.2). The jump from Qwen3VL-8B-Instruct to Qwen3VL-8B-Thinking is particularly notable (+2.58 percentage points), indicating that chain-of-thought reasoning specifically benefits the multi-agent evidence integration step.

Oracle results define the upper bound and the headroom. Gemini-2.5-Flash Oracle achieves 83.80%; Qwen3VL-8B-Instruct Oracle achieves 73.98%. The gap between EgoMAS (Gemini-2.5-Flash) at 41.41% and Oracle at 83.80% is approximately 42.4 percentage points (as stated in Appendix C). This gap represents the retrieval deficit — the performance that could be recovered by improving retrieval to perfection, holding the reasoning model constant. The fact that even with perfect retrieval, the best model (Gemini-2.5-Flash) achieves only 83.80% (not 100%) indicates that approximately 16% of questions cannot be answered even with perfect evidence, likely due to inherent ambiguity, reasoning errors, or limitations in the caption/transcript representation of the video content.

Category-Wise Analysis: ToM Is Hardest, EgoMAS Improves Most Where Baselines Struggle

Table 2 reports per-category accuracy for all methods, revealing substantial category-dependent variation:

Social Interaction (SI):

• Best baseline: BM25 at 44.68%
• Best EgoMAS: EgoMAS (Qwen3VL-8B-Instruct) at 43.09%
• EgoMAS (Gemini-2.5-Flash): 41.49%
• Gemini-2.5-Flash concatenation: 41.22%
• The gap between EgoMAS and concatenation baselines is relatively small for SI, suggesting that social interaction questions may be answerable from a smaller number of salient event windows that both retrieval and long-context models can identify.

Task Coordination (TC):

• Best baseline: BM25 at 37.60%
• Best EgoMAS: EgoMAS (Gemini-2.5-Flash) at 41.32% — a substantial +4.96 percentage point improvement over Gemini concatenation (36.36%)
• EgoMAS (Qwen3VL-8B-Thinking): 39.88%
• The larger EgoMAS advantage for TC suggests that task coordination questions, which involve understanding role assignment and collaborative execution across agents, benefit particularly from the structured shared memory's event-based alignment — the 4W1H format explicitly captures who did what in a shared task, making retrieval more precise.

Theory of Mind (ToM):

• All models perform worst on ToM. Gemini-2.5-Flash concatenation achieves only 24.26%; GPT-5 achieves 22.55%.
• Best EgoMAS: EgoMAS (Gemini-2.5-Flash) at 33.62% — a +9.36 percentage point improvement over Gemini concatenation, the largest absolute category improvement for EgoMAS.
• Oracle (Gemini-2.5-Flash) achieves 70.21% on ToM, compared to 83.80% average Oracle — the largest Oracle gap of any category (36.59 percentage points), indicating that ToM is disproportionately dependent on perfect evidence retrieval.
• "This is because ToM requires inferring latent mental states, such as goals and beliefs, rather than extracting explicitly observable visual or textual cues" (Section 7.1). The paper further notes that ToM's difficulty stems from "inherent ambiguity, the need for recursive mental-state reasoning, and complex goal–belief–action inference" — factors that retrieval alone cannot fully address.

Temporal Reasoning (TR):

• Best baseline: Gemini-2.5-Flash concatenation at 46.59%
• Best EgoMAS: EgoMAS (Qwen3VL-8B-Thinking) at 47.39%
• EgoMAS (Gemini-2.5-Flash): 39.37% — actually underperforming the Gemini concatenation baseline. This is the only category where EgoMAS with the best backbone does not outperform the corresponding concatenation baseline.
• The TR category's high performance with long-context models (Gemini at 46.59%) suggests that temporal alignment across agents benefits from having all temporal information in context simultaneously, enabling direct comparison of timestamps. EgoMAS's shared memory compresses temporal information into event summaries, which may lose the precise temporal ordering needed for concurrency and comparison questions.

Environmental Interaction (EI):

• Best baseline: GPT-5 at 38.72%
• Best EgoMAS: EgoMAS (Gemini-2.5-Flash) at 48.19% — the highest category accuracy achieved by any method, and a +14.21 percentage point improvement over Gemini concatenation (33.98%), the largest relative improvement for EgoMAS.
• EI questions involve counting object usage (e.g., "How many people used oven on DAY1?") and identifying first/last/most interactions. The structured shared memory's explicit object-action-location-agent fields make these counting and attribution queries unusually well-suited to the EgoMAS approach — the system can retrieve all events involving a specific object and aggregate across agents without the model needing to parse this from concatenated captions.

Aggregating across categories, EgoMAS provides the largest improvements on the categories where baselines struggle most (ToM: +9.36, EI: +14.21) and smaller improvements or slight regressions on categories where baselines are already stronger (SI: +0.27, TR: −7.22). This pattern — retrieval helping most where naive approaches are most confused — is consistent with the hypothesis that structured retrieval is a complement to, not just a more efficient version of, long-context processing.

Multi-Span and Multi-Agent Reasoning Amplify Difficulty

Table 3 breaks out performance by sub-category within SI, TC, and TR, enabling analysis of which question characteristics make the task hardest.

Single-span vs. multi-span (SI and TC):

• For SI single-span questions, EgoMAS (Qwen3VL-8B-Instruct) achieves 46.30%, substantially outperforming BM25 (41.05%) and GPT-5 (37.65%). For SI multi-span questions, EgoMAS achieves only 23.08%, comparable to BM25 (23.08%) and below GPT-5 (26.92% — note: the paper reports 26.92 for GPT-5 on SI Multi in Table 3, though the baseline table reports GPT-5 SI overall at 36.17%, and multi-span is a subset of SI).
• For TC single-span questions, EgoMAS (Qwen3VL-8B-Instruct) achieves 43.27%, outperforming BM25 (39.84%) and GPT-5 (35.62%). For TC multi-span questions, EgoMAS achieves 23.81%, comparable to GPT-5 (27.62%) and above BM25 (18.10%).
• The consistent pattern is that multi-span questions are dramatically harder than single-span questions across all models — single-span accuracy is typically 40–46% while multi-span accuracy is 10–27%. This is expected: multi-span questions require the system to identify and integrate evidence from multiple temporally separated events, which is a more demanding retrieval task. EgoMAS does not show a disproportionate advantage on multi-span questions, suggesting that its retrieval strategy (top-n shared memories → agent-specific sub-queries) may not be well-optimized for queries requiring evidence from widely separated timestamps.

Concurrency vs. comparison (TR):

• For TR concurrency questions, all models perform relatively well: Gemini-2.5-Flash achieves 42.40% (from the GPT-5 row — the table appears to report baselines across rows, not all models; GPT-5 achieves 42.40 on TR Concur). EgoMAS (Qwen3VL-8B-Instruct) achieves 36.00%.
• For TR comparison questions — which require ordering events across agents ("which happened first?") — performance drops substantially: GPT-5 achieves 37.65%, EgoMAS achieves 35.19%, and BM25 achieves only 22.84%.
• The pattern suggests that comparison questions are harder than concurrency questions because they require precise temporal ordering rather than just establishing co-occurrence. The shared memory's 10-minute window granularity may be too coarse for fine-grained temporal ordering, contributing to the EgoMAS regression on TR overall.

Number of required agents (Figure 8): Accuracy consistently decreases as the number of agents required to answer a query increases, measured for EgoMAS, BM25, and VideoRAG. This confirms that "current approaches to multi-agent knowledge fusion remain limited" and that "both aspects [multi-span reasoning and multi-agent knowledge fusion] significantly contribute to increasing the task difficulty" (Section 7.1). The monotonic decrease across all three methods indicates that this is a fundamental challenge, not a quirk of a specific retrieval approach — integrating evidence from more agents is genuinely harder, likely because it requires more retrieval steps, more cross-referencing, and more complex evidence synthesis.

EgoMAS Efficiency: Lower Latency with Competitive or Better Accuracy

Figure 6 plots inference latency (seconds, measured over 100 randomly selected samples) against overall accuracy for a subset of methods. The key observations:

• EgoMAS (Qwen2.5VL-7B-Instruct) achieves approximately 35.55% accuracy at approximately 1.3 seconds per query. This is the highest accuracy among retrieval-based methods at the lowest latency.
• Non-retrieval models show substantially higher latency: Gemini-2.5-Flash (36.93% accuracy) and GPT-5 (34.81%) require tens of seconds per query due to processing 1M-token and 272k-token contexts respectively.
• BM25 achieves 36.01% at low latency (~1–2 seconds) but with lower accuracy than EgoMAS.
• EgoRAG achieves the lowest accuracy (~19.6%) at moderate latency, reflecting its single-agent design limitation.

The latency-accuracy tradeoff strongly favors EgoMAS: it achieves accuracy comparable to the best concatenation baselines (35.55% vs. 36.93% for Gemini-2.5-Flash) while requiring approximately 1/20th to 1/100th the latency and approximately 1/200th the context tokens. For practical deployment where both accuracy and responsiveness matter, this is a decisive advantage.

Does MA-EgoQA Require Multi-Agent Memory? Yes, Decisively

Figure 7 evaluates EgoMAS when restricted to a single agent's memory versus all agents' memory, using two backbones (Qwen2.5VL-7B-Instruct and Qwen3VL-8B-Instruct) across all five categories. The results:

• Qwen2.5VL Single: Across categories, approximately 25–30% accuracy
• Qwen2.5VL All: Across categories, approximately 30–40% accuracy
• Qwen3VL Single: Across categories, approximately 28–33% accuracy (higher baseline due to stronger model)
• Qwen3VL All: Across categories, approximately 35–45% accuracy

The drop from all-agents to single-agent is substantial (approximately 5–12 percentage points depending on backbone and category), confirming that "models should utilize memories from multiple agents to correctly answer MA-EgoQA queries" (Section 7.1). This is the empirical validation of the benchmark's design criterion — the single-agent filtering during benchmark construction successfully removed questions answerable from one perspective, and the performance impact of restricting agent access is clearly measurable.

Ablation Studies and Robustness Checks

Shared memory and dynamic retrieval components (Table 4): On Qwen2.5VL-7B-Instruct backbone, ablating the two core EgoMAS components: without shared memory, without dynamic retrieval = 27.80%; without shared memory, with dynamic retrieval = 28.20% (+0.40); with shared memory, without dynamic retrieval = 30.04% (+2.24); with shared memory, with dynamic retrieval = 35.55% (+7.75 over baseline, +5.51 over shared memory alone). The marginal contribution of dynamic retrieval alone (+0.40 percentage points) is negligible, indicating that agent-specific sub-queries are ineffective without the shared memory to provide system-level event context. The marginal contribution of shared memory alone (+2.24) is meaningful but modest. The combination (+7.75) substantially exceeds the sum of individual contributions (+0.40 + 2.24 = +2.64), demonstrating that the two components are complementary: shared memory identifies which events are relevant at the system level, and dynamic retrieval uses that event context to generate targeted sub-queries that retrieve detailed evidence from the right agents.

Shared memory structure (Table 5a): Five memory construction strategies compared: Summary (flat text, 30.67%), Triplet (subject-predicate-object, 30.44%), Chunk (raw captions without aggregation, 25.96%), Graph (graph-based, 31.99%), 4W1H (structured event-based, 35.55%). The 4W1H structure outperforms all alternatives by at least 3.56 percentage points. The Chunk baseline at 25.96% is barely above the no-retrieval baseline (27.80%), confirming that raw caption aggregation without cross-agent event fusion provides minimal benefit. The Summary and Triplet baselines perform similarly (~30.5%), suggesting that some level of abstraction helps but unstructured or minimally structured formats lose the precision needed for effective retrieval. The Graph baseline at 31.99% is the strongest alternative but still lags behind 4W1H, likely because graph-based retrieval requires different retrieval mechanisms (graph traversal rather than lexical matching) that BM25 cannot exploit.

Memory retriever (Table 5b): Four retrievers compared for the shared memory retrieval step: DPR (28.67%), Qwen3-Embed-0.6B (33.03%), NV-Embed-v2 7B (37.91%), BM25 (35.55%). NV-Embed-v2 achieves the highest accuracy (+2.36 over BM25) but "contains 7B parameters and consequently incurs substantial computational overhead." The paper makes a practical tradeoff: BM25 is chosen for its competitive accuracy with essentially zero computational cost. DPR's poor performance (28.67%) is notable — dense retrieval trained on general-domain QA data does not transfer well to the structured event summaries in EgoMAS, confirming the importance of domain-appropriate retrieval design. The strong performance of NV-Embed-v2 (37.91%) versus Qwen3-Embed-0.6B (33.03%) suggests that larger embedding models capture more of the semantic structure needed for this domain, and the retrieval bottleneck is addressable with better embedding models.

Hyperparameters (Figure S1): Three hyperparameters are ablated:

• Shared memory retrieval size $n$: Optimal at 20–60 (accuracy ~34–35.5%); $n = 10$ drops to ~33%; $n = 100$ drops to ~34%. Too few entries provide insufficient context; too many introduce noise.
• Agent-wise retrieval size $k$: Robust across $k \in [1, 20]$ (accuracy ~34–35.5%). The flat response suggests that if the top-1 result is good, additional results add little; if the top-1 is poor, more results don't help. This may indicate that agent-level retrieval precision is high for relevant agents and low for irrelevant ones.
• Score threshold $\tau$: Stable for $\tau \in [2, 12]$ (accuracy ~35%); $\tau = 18$ drops to ~34%; $\tau = 24$ drops to ~33%. High thresholds filter out too many relevant entries, while low thresholds allow noise through.

The robustness of $k$ is an unexpected finding — it suggests that the bottleneck is not the number of agent-level results but the quality of the shared memory retrieval and the subsequent sub-query generation. If the shared memory correctly identifies relevant events, the sub-queries are well-formed, and the top-1 or top-2 agent results suffice. If the shared memory misses relevant events, no amount of agent-level retrieval can recover.

Sensitivity to number of available agents (Figure S2): EgoMAS accuracy increases monotonically with the number of available agents: 1 agent = 31.99%, 2 agents = ~33%, 3 agents = ~33.8%, 4 agents = ~34.5%, 5 agents = ~35.1%, 6 agents = 35.55%. The gain from adding each additional agent is positive but diminishing — the largest jump is from 1 to 2 agents (+1 percentage point), and subsequent additions provide smaller increments. This confirms two things: (1) MA-EgoQA questions genuinely require multi-agent information (single-agent performance is substantially lower), and (2) having access to all agents provides measurable benefit over partial access.

Input context modality (Table S11): WorldMM-8B improves when video frames are added to text captions (+2.13 to +4.13 percentage points across categories), confirming that "the MA-EgoQA benchmark requires visual knowledge contained in the frames." However, for EgoMAS (Qwen2.5-VL-7B-Inst), adding video frames provides mixed results: SI (+1.33), TR (+1.04), EI (+1.39) improve; TC (−0.62) and ToM (−1.28) degrade. The degradation on TC and ToM suggests that "including video frames is not always beneficial and can even distract the model when unnecessary frames are provided" (Appendix H). The different pattern between WorldMM-8B (always improved by frames) and EgoMAS (mixed) may be because EgoMAS already provides highly relevant text context through retrieval, and adding frames to this already-focused context introduces visual noise without proportional information gain. WorldMM-8B, with less structured retrieval, benefits more from the additional modality. The paper characterizes this as an open problem: "future research should focus on adaptively leveraging the necessary modalities and developing effective methods for frame selection."

Single-agent filtering effectiveness (Figure 7): Already discussed in the main results — restricting EgoMAS to a single agent's memory causes a substantial accuracy drop across all categories and both backbones, confirming that the single-agent filtering stage during benchmark construction was effective at removing questions answerable from one perspective.

Critical Assessment

The experiments establish MA-EgoQA as a genuinely challenging benchmark and demonstrate EgoMAS as an effective baseline — but there are important qualifications to how these results should be interpreted.

Claim: "MA-EgoQA is a challenging benchmark where current approaches struggle." Strongly supported. The evidence is clear and consistent: the best concatenation baseline (Gemini-2.5-Flash, 1M tokens) achieves only 36.93%, the best frame-based model (Qwen2.5-VL-7B) achieves 25.22%, and even EgoMAS — the best overall method — achieves only 41.41%. All of these are far from the Oracle upper bound (83.80%) and substantially above random chance (20%), confirming both that the benchmark is solvable (models do extract signal) and that it is hard (substantial headroom remains). The category-specific results reinforce this: Theory of Mind at 24–34% across models is particularly challenging. The 1,741-question scale is adequate for benchmark purposes, and the distribution across categories, days, and agents (Figure 3) ensures the evaluation is not dominated by a narrow slice of phenomena.

Claim: "EgoMAS substantially outperforms baselines while using less compute." Supported with clear evidence but with an important boundary condition. EgoMAS (Gemini-2.5-Flash) at 41.41% does outperform the same backbone's concatenation baseline at 36.93% (+4.48 percentage points) while using ~200× fewer tokens (4.6k vs. 1M). EgoMAS with a Qwen3VL-8B-Thinking backbone (40.26%, 5.4k tokens) surpasses GPT-5 (34.81%, 272k tokens) and Gemini-2.5-Flash (36.93%, 1M tokens). These are meaningful practical gains. However, the claim needs qualification: EgoMAS does not outperform the concatenation baseline on every category. On Temporal Reasoning (TR), EgoMAS (Gemini-2.5-Flash) achieves 39.37% versus Gemini concatenation at 46.59% — a 7.22 percentage point regression. On Social Interaction, the gap is small (+0.27). The advantage is concentrated in Task Coordination (+4.96), Theory of Mind (+9.36), and Environmental Interaction (+14.21). This means EgoMAS's superiority is category-dependent, and a system deployed in a domain where TR questions dominate might prefer concatenation. The paper does not discuss this category-dependent limitation, but it is clearly visible in Table 2.

Claim: "EgoMAS with small models matches large models with long contexts." Supported but with a specific comparison structure. EgoMAS (Qwen3VL-8B-Thinking) at 40.26% does surpass concatenation Gemini-2.5-Flash at 36.93%, and EgoMAS (Qwen2.5VL-7B-Instruct) at 35.55% nearly matches it. However, these comparisons are between EgoMAS with retrieval and concatenation baselines without retrieval. A fairer comparison would be: does EgoMAS with a small backbone outperform the same small backbone with concatenation? The answer is yes: Qwen2.5VL-7B-Instruct concatenation baseline achieves 25.22% (Table 2, All Frame Concat, but note that the text concatenation for this model is not reported — the All Caption Concat baselines use text-only models, not VLMs). For Qwen3VL-8B-Instruct, no concatenation baseline is reported, so the within-model comparison is unavailable. A stronger test would be to give the concatenation baselines the same retrieval mechanism — i.e., use BM25 to retrieve relevant captions and concatenate only those, then compare to EgoMAS's more structured approach. This ablation is partially addressed by the BM25 baseline (36.01%), which EgoMAS outperforms (35.55% for the Qwen2.5VL backbone vs. 36.01% for the BM25 baseline using Qwen3-VL-8B-Inst backbone — but careful: the response generators differ, so this is not a clean comparison of retrieval architectures). The key missing experiment is EgoMAS's retrieval + shared memory vs. a simple retrieve-then-concatenate baseline using the same backbone and same retriever — this would isolate whether the shared memory structure and agent-wise sub-query generation provide benefit beyond simple retrieval.

Latency advantage is real but incompletely characterized. Figure 6 shows EgoMAS at ~1.3 seconds versus tens of seconds for concatenation baselines. This is a significant practical advantage. However, the paper does not report the offline cost of constructing the shared memory (processing all 10-minute windows through the LLM to produce 4W1H event summaries) or of generating per-agent captions. For a 7-day, 6-agent dataset, there are approximately 6 agents × 7 days × 24 hours × 6 (10-minute windows per hour) = 6,048 ten-minute windows, each requiring an LLM call for shared memory construction. This one-time cost may be substantial, and while it amortizes over many queries, it is not zero. For a deployment answering thousands of queries, this is negligible; for a deployment answering dozens, it may dominate. The paper does not report this cost.

The Oracle setting is not a true upper bound. The Oracle setting provides the "same context used to generate the questions" — but this context is the set of captions and transcripts from the specific timestamps and agents involved in the ground-truth answer. This is a retrieval upper bound but not a reasoning upper bound, because the Oracle model still has to reason over this evidence. The fact that Oracle (Gemini) achieves only 83.80% (not 100%) indicates that some questions contain inherent ambiguity or require reasoning beyond the model's capability even with perfect evidence. But an alternative interpretation is that the "same context" used for generation may not contain all the evidence needed to disambiguate the answer — i.e., the original human question-writers may have used video context beyond what was captured in captions. If so, the Oracle score is an underestimate of what perfect retrieval + perfect reasoning could achieve, and the 42.4 percentage point "retrieval deficit" is partly a "caption fidelity deficit." The paper does not discuss this nuance.

Missing baselines and ablations that would strengthen the paper:

1. A retrieve-then-concatenate baseline using the same backbone and retriever. This would isolate whether EgoMAS's structured shared memory and agent-wise dynamic retrieval provide benefit beyond simply retrieving relevant captions via BM25 and concatenating them. The BM25 baseline in Table 2 uses a different approach (30-second clips + frames + Qwen3-VL-8B-Instruct), making it not directly comparable to EgoMAS configurations. A clean ablation would be: BM25 retrieval from all agent captions → concatenate top-$n$ results → Qwen2.5VL-7B-Instruct. Compare this to EgoMAS (Qwen2.5VL-7B-Instruct) at 35.55%.

2. Ablation on the number of shared memory events per window. The shared memory prompt (Figure S10) instructs the LLM to "try to contain as many key events as possible." What if the LLM misses events? What if it hallucinates events? The paper does not analyze the quality of the constructed shared memory — how often are relevant events present, how often are irrelevant events introduced, and how does this affect downstream QA? Without this analysis, it's unclear whether EgoMAS's performance is limited by retrieval or by shared memory construction quality.

3. Comparison to a multi-agent RAG baseline using agent-specific indices. A natural baseline not included would be: index each agent's captions independently, retrieve top-$k$ from each agent using BM25, concatenate all retrieved results, and answer. This is the natural extension of single-agent RAG to the multi-agent setting. Would this match EgoMAS? The dynamic retrieval ablation in Table 4 (without shared memory, with dynamic retrieval = 28.20%) suggests not — but that configuration uses LLM-generated sub-queries, not direct BM25 retrieval, so it's not equivalent.

4. Statistical significance. The 1,741-question test set is a reasonable size, but the paper reports only point estimates without confidence intervals. The +4.48 percentage point improvement of EgoMAS (Gemini) over Gemini concatenation is based on 1,741 questions — this corresponds to approximately 78 more questions answered correctly, which is likely statistically significant, but without a reported test, the reader cannot be certain.

5. Generalization beyond EgoLife. All experiments use a single dataset (EgoLife), a single environment (a shared house), and a single set of six agents. The paper acknowledges this limitation in Appendix C: "MA-EgoQA is based only on EgoLife and does not include other scenarios in different environments." The results may not transfer to different agent counts, different environments (e.g., workplace, outdoor), different activity types, or different caption quality. This is a fundamental limitation of all new benchmarks, but it means the conclusions about EgoMAS's effectiveness are conditional on the EgoLife domain.

Category-dependent performance deserves more analysis. EgoMAS shows a 14.21 percentage point improvement on Environmental Interaction but a 7.22 percentage point regression on Temporal Reasoning. This is a large and unexplained asymmetry. The paper's brief discussion attributes the TR regression to shared memory compression losing temporal precision — but why does this affect TR more than TC (where EgoMAS improves by ~5 points)? TC questions also involve temporal reasoning about task sequences. A deeper error analysis — what kinds of TR questions does EgoMAS get wrong that concatenation gets right, and vice versa — would substantially strengthen the diagnostic value of the benchmark and provide guidance for improving EgoMAS. This analysis is not present.

Visual information is underutilized, and the paper acknowledges this. Table S11 shows that adding video frames to EgoMAS helps on some categories and hurts on others. The paper identifies this as future work but does not propose a mechanism for selective frame inclusion. The finding that "naive frame inclusion can distract" is valuable but preliminary — it characterizes a problem without solving it. A frame selection mechanism that only includes frames from retrieved temporal windows, or that uses the LLM to decide when visual information is needed, would be a natural extension.

The single-agent filtering during benchmark construction may introduce a bias. Questions that are answerable from a single agent's memory are removed. This means MA-EgoQA systematically excludes the kinds of questions that are naturally easy for single-agent systems. While this is necessary to enforce the multi-agent evaluation criterion, it means the benchmark's difficulty is partially an artifact of this filtering — the questions are hard because easy ones were removed. This is not a flaw (it's by design), but it means that MA-EgoQA scores should not be interpreted as "how well would this system perform on typical queries in a real multi-agent deployment?" In a real deployment, many queries would be answerable from a single agent's memory, and a system's accuracy on those queries matters for user experience. The paper's single-agent filtering removes this lower-difficulty tail of the distribution, making the benchmark a test of multi-agent reasoning capability specifically rather than a test of overall QA performance in a multi-agent setting.

In summary, the experiments convincingly demonstrate that MA-EgoQA is challenging, that EgoMAS outperforms naive concatenation and retrieval baselines, and that structured shared memory with agent-wise dynamic retrieval provides complementary benefits. The headline numbers support the paper's claims. However, the category-dependent nature of EgoMAS's advantage (large gains on TC, ToM, EI; regression on TR), the absence of a clean retrieve-then-concatenate ablation, the uncharacterized offline cost of shared memory construction, and the lack of error analysis limit the strength of conclusions about why EgoMAS works and when it should be preferred. These are natural limitations for a benchmark paper introducing both a new task and a first baseline — they define the research agenda that the benchmark is designed to enable.

6. Limitations and Trade-offs

The Shared Memory Construction and Difficulty Estimation Costs Are Not Accounted For in Headline Efficiency Numbers

The assumption or constraint. EgoMAS's claimed efficiency — 4.6k tokens per query, ~1.3 seconds latency — measures online inference cost only. It does not include the substantial one-time cost of constructing the event-based shared memory. The shared memory is built by processing every 10-minute window of all six agents' video through an LLM to produce structured 4W1H event summaries. For a 7-day, 6-agent dataset, this involves approximately $6 \times 7 \times 24 \times 6 = 6,048$ LLM calls (assuming continuous recording), each requiring caption generation and event extraction. The paper acknowledges this implicitly through the architectural description but never quantifies it as a cost or includes it in any efficiency comparison. The Oracle setting — which provides ground-truth evidence context — similarly represents a theoretical upper bound that would be extraordinarily expensive to approximate in deployment (requiring the equivalent of question-generation-level context identification for each query).

The consequence. For deployments with low query volumes, the amortized cost of shared memory construction may dominate the total compute budget. If a system receives only 100 queries, spending 6,048 LLM calls on shared memory construction plus ~100 calls on retrieval and response generation means the one-time setup cost outweighs query processing cost by ~60×. The headline claim that EgoMAS uses "~200× fewer tokens" than concatenation baselines is therefore misleading for low-query regimes — it measures per-query efficiency while ignoring the substantial upfront investment that enables that efficiency. The decision to deploy EgoMAS versus a simpler retrieve-then-concatenate approach hinges on the query volume: at low volume, shared memory construction is wasteful; at high volume, it amortizes favorably. The paper provides no analysis of this tradeoff.

What evidence exists in the paper. The paper does not report the cost of shared memory construction. Section 5.1 describes the procedure ("At every 10-minute interval, each embodied agent provides a caption summarizing its own observations... A centralized manager then integrates these individual captions into a system-level summary") but gives no token counts, latency measurements, or FLOP estimates for this offline stage. Figure 6 reports only online inference latency. The token counts in Table 2 (4.6k–7.4k for EgoMAS configurations) refer to the context provided to the response generator, not the cost of building the shared memory. The BM25 retrieval baseline (36.01%) does not require shared memory construction, making it the appropriate comparison for low-query deployments — but the paper does not frame it this way.

Mitigation status. Not addressed. Appendix C acknowledges that EgoMAS "performance still falls significantly behind the oracle performance" but frames this as a retrieval quality problem rather than a cost amortization problem. The one-time construction cost is not discussed in the limitations section or anywhere else in the paper. A natural mitigation — using smaller or distilled models for shared memory construction, or constructing shared memory only for windows that are subsequently queried (lazy construction) — is not explored.

The Benchmark Is Built on a Single Environment and Agent Configuration, With No Evidence of Generalization

The assumption or constraint. MA-EgoQA is constructed entirely from the EgoLife dataset, which captures six specific individuals in a single shared house over seven specific days. The paper acknowledges this in Appendix C:

"MA-EgoQA is based only on EgoLife and does not include other scenarios in different environments. This is because EgoLife is currently the only publicly available video dataset providing long-term, egocentric videos captured simultaneously from multiple agents."

This is a legitimate practical constraint — no comparable multi-agent long-duration egocentric dataset exists — but it means every conclusion the paper draws about model capabilities, category difficulty, and EgoMAS effectiveness is conditioned on a single environment, a single set of six individuals with specific interaction patterns, and a single recording setup (camera-equipped glasses in a residential setting).

The consequence. Several aspects of the results may not transfer. The social dynamics (how these six specific people interact, coordinate tasks, and form beliefs about each other) are a single sample from the space of possible multi-agent configurations. Different agent counts (2, 10, 50) would create different retrieval difficulty profiles — the cross-agent alignment problem scales quadratically with agent count, but the paper's 6-agent fixed setting cannot reveal this scaling behavior. Different environments (workplace, outdoor, industrial, healthcare) would shift the distribution of activities and the nature of cross-agent interactions — factory workers coordinating on an assembly line have fundamentally different event structures than housemates having casual conversations. Different recording modalities (stationary cameras, body-mounted sensors, robots with different camera configurations) would change the nature of what is observable and capturable. The paper's finding that BM25 achieves 36.01% while dense retrieval (DPR) achieves only 26.19% (Table 2) may be specific to the lexical patterns of EgoLife captions — in a different domain with different captioning systems, dense retrieval might become relatively stronger or weaker.

What evidence exists in the paper. Only indirect evidence. The paper evaluates 16 baselines + EgoMAS across four backbones, showing consistent relative trends (EgoMAS > BM25 > concatenation, ToM hardest, etc.), but this consistency is within a single data distribution and does not test generalization. The 1,741 questions, while reasonably sized, are drawn from a single dataset without domain shift evaluation. There is no held-out subset from a different environment, different time period, or different agent group. The paper's own suggestion — "We hope more video datasets proposed in this direction and evaluate the model in various types of videos to ensure the generalization" (Appendix C) — is an acknowledgement that generalization remains unverified.

Mitigation status. Not addressed. The paper identifies this as a limitation explicitly in Appendix C and calls for future datasets, but provides no within-paper mitigation. A partial mitigation — such as evaluating on held-out days (e.g., train on days 1–5, test on days 6–7 to test temporal generalization) or held-out agents (train on 4 agents, test on 2) — would have been possible within the EgoLife dataset but was not performed. The fixed 1,741-question set is used for all evaluations without any train/test split that would enable generalization testing.

EgoMAS Does Not Outperform Concatenation on All Categories, and the Failure Mode on Temporal Reasoning Is Unexplained

The assumption or constraint. The paper's primary claim is that EgoMAS "significantly outperforms all baselines" (Section 6.2) and that "EgoMAS with a Gemini-2.5-Flash backbone achieves 41.41% accuracy, outperforming the same backbone operating over concatenated captions by 4.48 percentage points" (Section 7.2, rephrased). This is true in aggregate but masks a significant category-level failure: on Temporal Reasoning (TR), EgoMAS (Gemini-2.5-Flash) achieves 39.37% versus 46.59% for Gemini-2.5-Flash concatenation — a 7.22 percentage point regression. TR is the second-largest category (287 questions, 16.5% of the benchmark), so this is not a negligible edge case.

The consequence. A practitioner deploying EgoMAS in a domain where temporal reasoning questions dominate — for example, a security monitoring system where queries are predominantly "what happened between X and Y?" or "which event occurred first?" — would experience worse performance from EgoMAS than from simply concatenating all captions and using the same backbone model. The paper's aggregate accuracy advantage masks this regime-dependent failure, and without understanding why EgoMAS fails on TR, a practitioner cannot determine whether their deployment domain is at risk. The paper's brief speculation — that shared memory's 10-minute window granularity "may be too coarse for fine-grained temporal ordering" — is plausible but untested, and does not explain why EgoMAS achieves near-baseline performance on TR concurrency questions (36.00% vs. ~40% for baselines, Table 3) while struggling on TR comparison questions (35.19%, where the ordering requirement is strict).

What evidence exists in the paper. Table 2 clearly shows the TR regression: EgoMAS (Gemini-2.5-Flash) at 39.37% vs. Gemini-2.5-Flash concatenation at 46.59%. Table 3 provides sub-category breakdown: EgoMAS (Qwen3VL-8B-Instruct) at 36.00% on concurrency and 35.19% on comparison (vs. GPT-5 at 42.40% and 37.65% respectively — noting the backbone mismatch makes this comparison imperfect, but the trend of EgoMAS struggling on TR is consistent across backbones). The paper notes the Oracle gap for TR (Gemini Oracle 88.15% vs. EgoMAS Gemini 39.37% = 48.78 percentage points) is among the largest of any category, suggesting a retrieval rather than reasoning problem. But crucially, no error analysis is performed — the paper does not examine which TR questions EgoMAS gets wrong, whether they cluster in specific sub-types, or whether the shared memory fails to capture the temporal ordering information that concatenation models successfully extract.

Mitigation status. Minimally addressed. Section 7.1 discusses TR difficulties briefly ("models achieve substantially lower accuracy on queries grounded in multiple spans... and Comparison in TR") but attributes this to multi-span reasoning difficulty rather than EgoMAS-specific retrieval failures. The 10-minute window granularity limitation is not acknowledged as a design tradeoff of the shared memory construction. The paper's call for "future research on constructing a coherent global understanding beyond simple memory sharing among agents" (Section 7.1) is vague and does not propose specific mitigation for the temporal ordering problem. Obvious mitigations — variable-granularity shared memory (finer windows for recent events, coarser for older), timestamp-aware retrieval that preserves temporal ordering metadata, or hybrid approaches that use concatenation for temporal reasoning queries and shared memory for others — are not discussed.

Visual Information Is Underutilized and Naively Including It Degrades Performance on Key Categories

The assumption or constraint. EgoMAS is designed as a text-based retrieval system — it operates over captions (text descriptions of video content) and transcripts (text of spoken conversations), and the shared memory is constructed from text captions. Video frames are not used in the core EgoMAS architecture. When the paper experiments with adding video frames (Table S11), the results are mixed and concerning: for EgoMAS (Qwen2.5-VL-7B-Inst), adding frames improves SI (+1.33), TR (+1.04), and EI (+1.39) but degrades TC (−0.62) and ToM (−1.28). The paper acknowledges this:

"including video frames is not always beneficial and can even distract the model when unnecessary frames are provided" (Appendix H).

This means EgoMAS cannot safely incorporate visual information in its current form — doing so would improve some categories at the cost of degrading others, with no mechanism to decide per-query whether visual information is net beneficial.

The consequence. Some MA-EgoQA questions demonstrably require visual information that is not fully captured in captions. The paper provides examples in Table S12: "Why didn't Katrina know where to put the big kitchen tools?" requires knowing that "Shure had placed things randomly" — a visual state that may not be explicitly described in captions. "Tasha was grabbing some tissues from the table and Alice was switching to a thinner brush to start her eye makeup" involves fine-grained visual actions that captions may summarize coarsely. WorldMM-8B consistently improves when frames are added (Table S11, +2.13 to +4.13 across categories), suggesting the visual signal is real and useful when properly integrated. EgoMAS's inability to benefit from this signal — and the active harm on TC and ToM — means it leaves accuracy on the table for visually-dependent questions while risking degradation on categories where text evidence is already sufficient. In a deployment setting, this creates an impossible choice: include frames and get worse on some queries, or exclude frames and miss visually-cued answers on others, with no principled way to decide.

What evidence exists in the paper. Table S11 provides the direct ablation: EgoMAS (Qwen2.5-VL-7B-Inst) text-only vs. text+video across all five categories. WorldMM-8B serves as a comparison point showing that a different architecture can benefit from frames more consistently. Table S12 provides qualitative examples of questions requiring visual information. The paper reports that "visual information is essential for answering certain questions in MA-EgoQA" but does not quantify what fraction of the 1,741 questions fall into this category.

Mitigation status. Partially addressed as future work. The paper states that "future research should focus on adaptively leveraging the necessary modalities and developing effective methods for frame selection" (Appendix H). This is a clear problem statement but no solution is proposed. A natural approach — using the retriever to identify temporal windows where visual information is likely relevant, and only including frames from those windows — would potentially reduce the distraction problem by limiting frames to query-relevant contexts. But this is not implemented or evaluated. The current EgoMAS design is purely text-based, which is a practical limitation for any deployment where visual details (object states, spatial arrangements, fine-grained actions) are query-relevant.

The Oracle Setting Overestimates the Retrieval Upper Bound Because It Assumes Ground-Truth Evidence Context Is Available

The assumption or constraint. The paper's Oracle setting provides the response generator with "the same context used to generate the questions" (Section 6.1) — specifically, the captions, transcripts, and video from the specific agents and timestamps that contain the ground-truth evidence for each question. Oracle accuracy (Gemini-2.5-Flash: 83.80%, Qwen3VL-8B-Instruct: 73.98%) is interpreted as an approximate upper bound on what could be achieved with perfect retrieval. The gap between EgoMAS (41.41%) and Oracle (83.80%) — approximately 42.4 percentage points — is characterized as the "retrieval deficit" (Appendix C) that improved retrieval could close.

The consequence. This interpretation conflates retrieval quality with evidence fidelity. The "same context used to generate the questions" consists of captions and transcripts that were produced by an automated captioning system from the raw video. This captioning process is itself lossy — it may miss visual details, misidentify objects, fail to capture subtle interactions, or omit contextual information that a human watching the raw video would notice. When the human verifiers reviewed the 3,436 candidates, they had access to the raw video, not just captions — meaning some questions may have been validated based on video evidence that the captions do not fully capture. The Oracle model receives only the captions/transcripts, not the raw video. If a question's correct answer depends on visual evidence that the captions failed to capture, the Oracle model will get it wrong regardless of retrieval quality, but a model that could process raw video might get it right. The 83.80% Oracle accuracy is therefore not a true reasoning upper bound — it is an upper bound on caption-based reasoning given the specific captioning system used. The "retrieval deficit" of 42.4 percentage points is partially a "caption fidelity deficit" — evidence that exists in the video but is missing from the captions cannot be retrieved, no matter how good the retriever is.

What evidence exists in the paper. The paper does not directly measure caption fidelity. Indirect evidence comes from two places. First, the human verification stage (Section 4.2) notes that verifiers had access to "captions, transcripts, and videos from all agents" — implying that some questions required video-level verification beyond what captions alone provide. Second, Table S11 shows that adding video frames to EgoMAS improves performance on some categories (SI, TR, EI), suggesting that captions miss information present in the video. Third, the paper's own qualitative examples (Table S12) highlight questions where visual information is important. The 83.80% Oracle score — substantially below 100% — is itself evidence that some questions are not answerable from captions alone, but the paper does not analyze whether these failures are due to reasoning difficulty or evidence incompleteness.

Mitigation status. Not addressed. The paper treats the Oracle setting as a retrieval upper bound without discussing the caption fidelity confound. A cleaner Oracle would use the raw video (not just captions) as the evidence context, but this is computationally prohibitive for 266 hours of video. Alternatively, the paper could have analyzed the subset of Oracle failures to determine what fraction are due to caption incompleteness versus reasoning difficulty, providing a more precise characterization of the retrieval deficit. The 42.4 percentage point "retrieval deficit" should be understood as an upper bound on what improved retrieval can achieve — the true achievable ceiling may be lower, or may require better captioning in addition to better retrieval.

The Benchmark's Single-Agent Filtering Systematically Removes Easier Questions, Making Results Unrepresentative of Real-World Multi-Agent QA Difficulty

The assumption or constraint. The benchmark construction pipeline (Section 4.2) includes a single-agent filtering stage that removes any question answerable using the memory of a single individual. The procedure extracts agent names from the question and answer, performs inference using each referenced agent's memory alone (or a randomly selected agent if no name is present), and eliminates the sample if the model produces the correct answer under any single-agent memory condition. This is the filter that enforces the benchmark's defining property — questions require multi-agent evidence integration. The paper does not report what fraction of initially generated samples were removed by this filter, but given that only 1,741 of 3,436 human-reviewed candidates survived (50.7%), and earlier filtering stages further reduced the pool, the removal rate is likely substantial.

The consequence. In a real-world multi-agent deployment, the natural distribution of user queries would include many questions answerable from a single agent's perspective. A supervisor might ask "What did Jake do yesterday afternoon?" (single-agent) as often as "Who was involved in the kitchen cleanup across all agents?" (multi-agent). By systematically removing the easier single-agent-solvable questions, MA-EgoQA overstates the difficulty of the multi-agent video QA problem as it would be encountered in practice, and understates the accuracy that a deployed system would achieve — because in deployment, the system would benefit from "free" correct answers on single-agent questions. The benchmark's 36.93% accuracy for Gemini-2.5-Flash concatenation is a measure of multi-agent reasoning capability specifically, not of expected QA performance in a multi-agent setting. This distinction matters for practitioners estimating real-world performance from benchmark scores: if 50% of real-world queries are single-agent-solvable and a system gets those with ~60% accuracy while achieving ~37% on multi-agent queries, overall accuracy would be ~48.5% — substantially higher than the MA-EgoQA score alone would suggest.

What evidence exists in the paper. The paper does not report the fraction of candidate samples removed by single-agent filtering. Figure 7 provides indirect evidence: EgoMAS accuracy drops substantially when restricted to a single agent's memory (e.g., Qwen2.5VL: ~25–30% single-agent vs. ~30–40% all-agents; Qwen3VL: ~28–33% single-agent vs. ~35–45% all-agents), confirming that multi-agent access is necessary for MA-EgoQA questions. But critically, this does not tell us what performance would be on a mixed set of single-agent and multi-agent questions — the natural distribution in deployment. Figure S2 shows EgoMAS accuracy increases with the number of available agents (from 31.99% with 1 agent to 35.55% with 6 agents), but this is measured on the already-filtered multi-agent-only question set, so it reflects how additional agent perspectives help on inherently multi-agent questions, not how single-agent-solvable questions would affect overall accuracy.

Mitigation status. Not addressed. The paper does not discuss the representativeness implications of the single-agent filter, nor does it provide a "mixed difficulty" evaluation that combines multi-agent questions with single-agent-solvable questions to estimate real-world performance. The filtering is a legitimate design choice — it ensures the benchmark tests what it claims to test — but the paper should transparently characterize how this choice affects the interpretation of results. A simple addition would be: retain a held-out set of single-agent-solvable questions (those removed by the filter), evaluate models on both the multi-agent-only set and the mixed set, and report both numbers. This would let practitioners estimate their expected real-world accuracy by combining the two scores with their domain-specific estimate of the single-agent vs. multi-agent query ratio.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper establishes multi-agent egocentric video QA as a first-class research problem with its own evaluation infrastructure and baseline architecture. The shift is conceptual rather than algorithmic: before this work, the two communities that should have been addressing this problem — multi-agent embodied systems and long-form video understanding — were each missing half of it. The multi-agent systems community built controllers and planners that execute tasks but never built the retrospective memory layer that would make deployed systems auditable; the video understanding community built benchmarks for increasingly longer single-perspective videos but never crossed the threshold into system-level comprehension requiring evidence distributed across agents. MA-EgoQA and EgoMAS close this gap by providing both the evaluation framework (the benchmark) and the architectural template (shared memory → agent-wise retrieval → response generation) that define what solving this problem looks like.

The magnitude of this shift is reframing, not paradigm shift. The paper does not propose a new learning algorithm or model architecture that would change how systems are built. Rather, it demonstrates that a specific composition of existing techniques — structured event extraction, BM25 retrieval, and LLM-based sub-query generation — can substantially outperform naive approaches on a carefully constructed evaluation. The reframing is in the problem specification itself: the paper shows that multi-agent retrospective QA is not just single-agent video QA with more input streams, but a qualitatively different task requiring system-level event alignment before retrieval. The experimental evidence for this is not in a single result but in the pattern across ablations: dynamic retrieval without shared memory achieves only 28.20% (barely above no retrieval at 27.80%, Table 4), while shared memory without dynamic retrieval achieves 30.04%, and the combination achieves 35.55% — a 7.75 percentage point gain that substantially exceeds the sum of individual contributions, demonstrating that the two components are genuinely complementary rather than additive.

The paper reconciles a latent tension between two intuitions about long-form video understanding. One intuition says that given enough context length, a sufficiently capable model can attend to relevant information and ignore the rest — this is the implicit assumption behind concatenation baselines that provide 1M-token contexts to Gemini-2.5-Flash. The competing intuition says that retrieval is necessary because long contexts introduce distraction and computational waste — this is the implicit assumption behind RAG baselines. The paper's results show that both intuitions are partially right and partially wrong. Concatenation works surprisingly well on temporal reasoning (Gemini concatenation at 46.59% vs. EgoMAS at 39.37%, Table 2), where having all temporal information simultaneously in context enables direct timestamp comparison. Retrieval-based approaches work dramatically better on environmental interaction (EgoMAS at 48.19% vs. Gemini concatenation at 33.98%, Table 2) and theory of mind (EgoMAS at 33.62% vs. Gemini concatenation at 24.26%), where the relevant evidence is sparse and easily drowned out by irrelevant context. This category-dependent pattern — some questions benefit from global context, others from focused retrieval — is not predicted by either extreme position, and it provides a more nuanced picture than the simple "retrieval beats concatenation" narrative.

The research landscape shifts in several concrete ways. First, it becomes harder to justify building multi-agent embodied systems without including a retrospective QA capability — MA-EgoQA now exists as a standardized evaluation, and a system that can execute collaborative tasks but cannot answer questions about its own operational history is demonstrably incomplete. Second, video LLM development gains a new stress test: the paper's finding that smaller models with structured retrieval can match much larger models with concatenated contexts (EgoMAS Qwen3VL-8B-Thinking at 40.26% vs. Gemini-2.5-Flash at 36.93%, Table 2) suggests that long-context capability is not the only path to strong performance on multi-stream video tasks, and that retrieval architecture design can substitute for context length in this domain. Third, the benchmark's five categories create targeted diagnostic tools for evaluating specific capabilities — in particular, Theory of Mind at 24–34% across models (Table 2) and the large Oracle gap (70.21% Oracle vs. 33.62% EgoMAS, a 36.59 percentage point deficit) identify perspective-taking as the primary bottleneck that neither retrieval nor scale has solved. Fourth, the paper's finding that naively adding video frames degrades performance on TC and ToM (Table S11) while helping on SI, TR, and EI shifts attention from "do we need visual information?" to "how do we adaptively decide when visual information is net beneficial?" — a more precise question that demands frame selection mechanisms rather than blanket multimodal input.

Follow-Up Research This Work Enables

Adaptive frame selection for multi-agent retrieval. The paper demonstrates that adding video frames to EgoMAS improves performance on some categories but degrades it on others (Table S11: SI +1.33, TR +1.04, EI +1.39; TC −0.62, ToM −1.28). A concrete follow-up would design and evaluate a frame selection policy that decides per-query whether to include visual evidence, based on features of the query and the retrieved text context. The simplest version is a classifier trained on the 1,741 MA-EgoQA questions: for each question, evaluate whether EgoMAS (text-only) and EgoMAS (text+frames) produce different answers, use those differences as training labels for a lightweight model that predicts "frames will help" vs. "frames will hurt" from the question text and the BM25 scores of retrieved shared memory entries. A stronger version would select specific frames rather than a binary include/exclude decision, using the shared memory's temporal grounding (the "When" field in 4W1H) to retrieve frames only from the specific 10-minute windows identified as relevant, rather than uniformly sampling across all agents. The key metric is whether adaptive frame selection can recover the text+video gains on SI, TR, and EI without incurring the text+video losses on TC and ToM — achieving accuracy closer to the per-category best of both modalities rather than their average.

Dynamic granularity for shared memory to improve temporal reasoning. The paper's most striking negative result is EgoMAS's regression on temporal reasoning: EgoMAS (Gemini-2.5-Flash) achieves 39.37% vs. Gemini concatenation at 46.59%, a 7.22 percentage point deficit (Table 2). The paper speculates that the fixed 10-minute shared memory granularity "may be too coarse for fine-grained temporal ordering." A concrete follow-up would construct a multi-scale shared memory that stores events at multiple temporal resolutions (30-second, 10-minute, 1-hour) and dynamically selects the appropriate granularity based on the query. For TR comparison questions requiring precise event ordering, the retriever would access the 30-second or even frame-level memory; for TC questions about task coordination that unfold over longer periods, the 10-minute or 1-hour memory would suffice. The experiment is straightforward: construct multi-scale shared memory using the same caption generation pipeline but with variable window sizes, evaluate EgoMAS with this multi-scale memory on TR questions specifically, and measure whether the gap with concatenation narrows. The null result — multi-scale memory doesn't help — would be equally informative, suggesting that the TR deficit is not due to granularity but to the fundamental difficulty of reconstructing temporal order from event summaries rather than raw timestamps.

Verifier training or distillation for shared memory quality estimation. The paper provides no analysis of shared memory construction quality: what fraction of events are correctly identified, what fraction are missed, what fraction are hallucinated? Without this, it's unclear whether EgoMAS's 41.41% accuracy is limited by retrieval from a good shared memory or by the shared memory itself being incomplete. A concrete follow-up would train a lightweight verifier that evaluates shared memory entry quality by comparing the 4W1H event summary against the raw agent captions that were provided as input. Using the EgoLife dataset, one could construct a training set: for each 10-minute window, have human annotators (or a strong LLM like GPT-5 with access to full captions and video) label whether each event in the shared memory is (a) correctly described with all agents and actions accurate, (b) partially correct (missing agents, wrong action details), or (c) hallucinated (event did not occur). Train a small classifier (~0.5B parameters) to predict these labels from the shared memory entry text alone. Then, during EgoMAS inference, filter shared memory entries by predicted quality before retrieval. The evaluation would measure whether filtering low-quality shared memory entries improves downstream QA accuracy, and the analysis would reveal the fraction of EgoMAS errors attributable to shared memory construction failures versus retrieval failures. This would also address the unmeasured construction cost: if a smaller model can produce shared memory of comparable quality to the larger model used in the paper, the amortized cost of construction drops substantially.

Cross-dataset generalization of the multi-agent QA formulation. The paper is built entirely on EgoLife, and generalization is unverified. A concrete follow-up would evaluate the MA-EgoQA pipeline on a different multi-agent egocentric video dataset — either an existing dataset adapted for this purpose (e.g., Ego4D's multi-person scenarios, though these are typically shorter than days-long) or a newly collected one in a different environment (workplace, outdoor, multi-robot). The critical evaluation is not just whether EgoMAS achieves reasonable accuracy on a new domain, but whether the relative performance patterns observed on MA-EgoQA transfer: does ToM remain the hardest category? Does EgoMAS still outperform concatenation on EI and underperform on TR? Does BM25 still outperform dense retrieval? If these patterns are consistent across environments, they represent fundamental properties of multi-agent egocentric video QA; if they vary, then MA-EgoQA's difficulty profile is environment-specific and generalization requires environment-specific calibration. A negative result — EgoMAS performs near-random on a different dataset — would be important for tempering claims about the architecture's generality and would motivate environment-adaptive retrieval strategies.

End-to-end learned retrieval for the multi-agent setting. EgoMAS uses BM25 (lexical matching) and LLM-generated sub-queries (Figure S9 prompt), with no learned components. Table 5b shows that a stronger retriever (NV-Embed-v2 at 37.91% vs. BM25 at 35.55%) improves accuracy, suggesting a learned retriever could close part of the Oracle gap. A concrete follow-up would train a dense retriever on MA-EgoQA-style data: use the Oracle evidence context (the specific captions and timestamps used to generate each question) as positive retrieval targets, and sample other randomly selected windows as negatives. The key question is whether in-domain training overcomes the domain shift that causes DPR (28.67%, Table 2) to underperform BM25 (36.01%). The evaluation would compare: (a) the trained dense retriever vs. BM25 on MA-EgoQA, (b) the trained dense retriever's generalization to a held-out set of days or agents (testing whether it overfits to the specific EgoLife event patterns), and (c) the trained dense retriever's transfer to a different multi-agent dataset (if available). A finding that in-domain dense retrieval substantially closes the BM25 gap but fails to transfer would clarify whether the domain shift problem is fundamental or addressable with more diverse training data.

Closed-loop evaluation with deployed multi-agent systems. MA-EgoQA is a static benchmark constructed from pre-recorded video. A real deployed multi-agent system would generate queries interactively — a human supervisor asking follow-up questions, agents requesting clarification about past events, the system proactively flagging anomalies. A concrete follow-up would deploy EgoMAS in a simulated multi-agent environment where agents continuously record and caption their experiences, a human (or simulated human) issues queries, and the system must answer in real-time while continuing to accumulate new memories. This tests capabilities that MA-EgoQA cannot evaluate: incremental shared memory updates (how does the system handle a new 10-minute window arriving every 10 minutes without full re-indexing?), interactive clarification (can EgoMAS ask the human a follow-up question when the retrieved evidence is ambiguous?), and temporal staleness tradeoffs (does shared memory quality degrade if constructed with lower frequency or smaller models to reduce cost in a streaming setting?). The evaluation would measure not just accuracy but also latency under continuous load, memory growth rate, and query success rate as a function of time since deployment. This would bridge the gap between the static benchmark evaluation and the dynamic multi-agent systems that the paper's introduction motivates.

Practical Applications and Downstream Use Cases

Security and accountability monitoring across multiple body-worn cameras. Police departments, security firms, and event security operations increasingly deploy body-worn cameras that generate continuous egocentric video across multiple personnel. After an incident — a use-of-force event, a perimeter breach, a crowd control operation — supervisors need to reconstruct what happened across multiple perspectives, determine who was where when, and verify whether protocols were followed. MA-EgoQA's task formulation directly models this scenario: multiple agents (officers) recording simultaneously in a shared environment, with queries like "who was the last person to interact with the suspect?" (environmental interaction), "what did Officer A believe about the suspect's location when making the entry decision?" (theory of mind), and "what happened between when Officer B called for backup and when Officer C arrived?" (temporal reasoning, comparison). EgoMAS's architecture maps naturally: shared memory aligns events across officers' camera feeds, and agent-wise retrieval pulls specific evidence from each relevant officer's perspective. The primary practical benefit is latency: EgoMAS achieves ~1.3 seconds per query (Figure 6) vs. tens of seconds for concatenation baselines processing all bodycam footage simultaneously. In a time-sensitive incident review, this speed advantage matters. The 41.41% accuracy with Gemini backbone (Table 2) is insufficient for high-stakes decisions, but the Oracle result (83.80%) demonstrates that with better retrieval — a tractable engineering improvement — the approach could become operationally viable.

Household and workplace robot fleet management. As consumer and industrial robots become more common, homes and workplaces will contain multiple autonomous agents (vacuum robots, lawn mowers, kitchen assistants, inventory robots) operating simultaneously and generating egocentric video of their operations. A human manager — a homeowner or facility supervisor — needs to answer questions like "when was the living room last vacuumed and by which robot?" (environmental interaction), "did the kitchen robot and the inventory robot coordinate on restocking?" (task coordination), or "why didn't the lawn robot know about the garden hose obstruction?" (theory of mind). EgoMAS's shared memory construction can run offline as robots dock and upload their daily footage, amortizing the one-time construction cost over many subsequent queries. The finding that EgoMAS with a 7–8B parameter backbone (Qwen2.5VL-7B at 35.55%, Qwen3VL-8B at 37.68–40.26%, Table 2) approaches or exceeds much larger models with concatenation means that the system can run on modest on-premise hardware rather than requiring cloud-based inference — an important consideration for privacy-sensitive home deployments and for industrial settings with limited connectivity.

Training data generation for multi-agent policy learning. Multi-agent reinforcement learning and imitation learning systems need reward signals and demonstration data that capture successful collaborative behavior. MA-EgoQA's benchmark construction pipeline — particularly the LLM-based generation of questions grounded in specific multi-agent temporal windows — provides a template for automatically generating retrospective QA data from any multi-agent video corpus. In a robot learning setting, after a multi-robot team executes a collaborative task, the system could automatically generate questions like "which robot handed the tool to which robot during the assembly phase?" (task coordination), "when did Robot A realize the gripper was jammed?" (theory of mind), and "how many times did each robot interact with the conveyor belt?" (environmental interaction). These questions and their ground-truth answers — verified against the execution logs and video — can serve as structured reward signals for training better coordination policies, or as evaluation benchmarks for measuring whether learned policies produce interpretable, queryable behavior. The key practical insight from the paper is that structured event extraction (4W1H) enables both QA evaluation and memory compression — the same shared memory that supports retrospective QA can also serve as a structured log for policy debugging and improvement.

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix between EgoMAS and named alternatives — the baselines (concatenation, RAG, frame-based) are evaluated as reference points rather than as competing deployable systems, and EgoMAS is presented as an initial baseline for the new benchmark rather than as a production-ready method. A forced decision matrix would impose a deployment-oriented framing that the paper's experimental design does not support. However, the results contain implicit guidance that can be stated without overclaiming:

• When query volume is high and shared memory construction cost amortizes favorably, EgoMAS's per-query efficiency (4.6k–7.4k tokens, ~1.3 seconds) dominates concatenation baselines (128k–1M tokens, tens of seconds). The break-even query volume depends on the one-time construction cost, which the paper does not report — but for deployments answering thousands of queries, the advantage is clear from Figure 6.
• When queries are predominantly about environmental interaction, task coordination, or theory of mind, EgoMAS substantially outperforms concatenation (EI: +14.21, ToM: +9.36, TC: +4.96 percentage points, Table 2, Gemini backbone). When queries are predominantly temporal reasoning, concatenation may be preferable (TR: −7.22 percentage points), though the paper does not analyze why or propose mitigation.
• When model size or hardware is constrained, EgoMAS enables smaller models (7–8B parameters) to match or exceed much larger models (GPT-5, Gemini-2.5-Flash) using concatenation, as shown by EgoMAS (Qwen3VL-8B-Thinking) at 40.26% vs. GPT-5 at 34.81% (Table 2). This is the strongest practical signal: if you can afford a one-time shared memory construction cost, you can deploy a relatively small model and achieve competitive accuracy with dramatically lower per-query inference cost.
• When the deployment environment is similar to EgoLife (residential, small group, days-long duration, egocentric camera glasses), the results are directly applicable. For significantly different environments (different agent counts, different activity types, different sensing modalities), generalization is unverified, and the relative strengths of different approaches may shift — in particular, the strong performance of BM25 (36.01%, Table 2) may depend on lexical overlap patterns specific to EgoLife captions.