Seeing, Listening, Remembering, and Reasoning: A Multimodal Agent with Long-Term Memory

ArXiv: 2508.09736

🎯 Pitch

A reinforcement-learning-trained multimodal agent that builds an entity-centric memory graph from continuous video and audio streams outperforms a Gemini-1.5-Pro + GPT-4o prompting system by up to 7.7% on hour-scale video QA, yet removing its semantic memory alone causes accuracy to crater by as much as 19.2%, revealing that incremental world-knowledge construction, not raw retrieval, is the real engine of complex long-horizon reasoning.

---

1. Executive Summary

This paper introduces M3-Agent, a multimodal agent framework equipped with long-term memory that continuously processes real-time video and audio streams to build an entity-centric, multimodal memory graph comprising both episodic memory (e.g., "Alice takes the coffee and says, ‘I can’t go without this in the morning’") and semantic memory (e.g., "Alice prefers to drink coffee in the morning"). Trained via reinforcement learning to perform multi-turn reasoning and iterative memory retrieval during task execution, M3-Agent outperforms the strongest baseline—a prompting agent using Gemini-1.5-Pro and GPT-4o—on the newly introduced M3-Bench long-video question answering benchmark by 6.7% (M3-Bench-robot), 7.7% (M3-Bench-web), and 5.3% (VideoMME-long), while ablation reveals that removing semantic memory alone causes accuracy drops of 17.1%, 19.2%, and 13.1% on the respective benchmarks, establishing that the framework's incremental world-knowledge construction via entity-centric semantic memory is the dominant driver of its reasoning capability and that test-time memory retrieval paired with reinforcement learning can surpass prompted commercial models when answering questions that require aggregating distributed evidence across hour-scale videos.

2. Context and Motivation

The Core Problem: Multimodal Agents Cannot Form Coherent Long-Term Memory from Continuous Sensory Streams

The paper addresses a fundamental capability gap in current AI systems: multimodal agents cannot incrementally build, maintain, and reason over long-term memory from continuous real-world sensory input in the way humans do. Section 1 frames this through a relatable scenario: imagine a household robot that, through daily interaction, learns your coffee preference without being explicitly told—it has "gradually formed a memory of you, tracking your preferences and routines through long-term interactions." This requires three interconnected capabilities that existing systems lack: (1) continuously perceiving the world through multimodal sensors (seeing and listening), (2) storing and organizing those experiences into a structured long-term memory while gradually accumulating world knowledge, and (3) reasoning over that accumulated memory to guide future actions.

The problem is not merely about processing long videos—existing long-video understanding systems can handle videos of fixed, finite length. The challenge is building a system that can process infinite streams of multimodal input in an online fashion while maintaining coherent, consistent representations of entities (people, objects, locations) across arbitrarily long timespans, and simultaneously extracting abstract semantic knowledge that enables higher-level reasoning. Section 2.2 emphasizes this distinction:

"Traditional approaches to long video understanding, such as extending the context window in multimodal models or compressing visual tokens to increase temporal coverage, do not scale effectively for infinitely long video streams. In practical settings, such as interactive agent scenarios, reprocessing the entire video history for each new instruction is computationally prohibitive."

This framing positions the problem at the intersection of three active but largely disconnected research areas: long-video understanding, agent memory systems, and world knowledge acquisition. The paper argues that existing work in each area addresses pieces of the puzzle but systematically fails to integrate them into a cohesive system capable of human-like memory formation and reasoning.

Why This Problem Matters: Real-World Agent Deployment and the Limits of Current Approaches

The significance of this problem is both practical and theoretical. On the practical side, the paper is motivated by the vision of deploying general-purpose multimodal agents—particularly household robots and personal assistants—that can operate autonomously over extended periods. Such agents must learn about their environment, their users, and the relationships between entities through observation alone, without requiring explicit reprogramming for each new context. This vision is articulated through the robot scenario in Section 1 and the design of M3-Bench-robot (Section 3.1), which features "seven everyday scenarios where robots are expected to operate: living room, kitchen, bedroom, study, office, meeting room, and gym."

The practical stakes are high because the alternative—reprocessing all historical sensory data each time a query arrives—is computationally infeasible. Section 2.2 notes that in interactive agent scenarios, "reprocessing the entire video history for each new instruction is computationally prohibitive." Without effective memory compression and organization, agents cannot scale to the hour-long, day-long, or lifelong timelines that real-world deployment demands.

On the theoretical side, the problem connects to fundamental questions in AI about how systems can acquire and organize world knowledge from experience. The paper draws explicit inspiration from human cognitive psychology, citing Tulving's [44, 45] foundational distinction between episodic memory (concrete, context-bound experiences) and semantic memory (abstracted, generalized knowledge). By building a system that mirrors this dual-memory architecture, the paper aims to demonstrate that this cognitive division is not merely descriptive but functionally necessary for robust long-term reasoning.

The paper also identifies a specific evaluation gap that motivates its benchmark contribution. As argued in Section 2 and Section 3, existing long-video question answering (LVQA) benchmarks "mainly focus on visual understanding, such as action recognition and spatial/temporal perception, leaving a gap in evaluating higher-level cognitive abilities that rely on long-term memory and are crucial for real-world agents, such as understanding persons, extracting general knowledge, and performing cross-modal reasoning." Table 2 provides a systematic comparison showing that no prior benchmark simultaneously tests for agent-perspective scenarios, cross-modal reasoning, person understanding, and general knowledge extraction—the very capabilities that would distinguish a memory-equipped agent from a passive video understanding system.

Prior Approaches and Where They Fall Short

The paper identifies and critiques three broad families of prior approaches, each capturing a subset of the desired capabilities but failing to deliver an integrated solution.

1. Long-Term Memory in LLM Agents (Text-Only)

The most mature body of prior work addresses long-term memory for text-based LLM agents (Section 2.1). These systems typically store agent trajectories—dialogues or execution logs—by "append[ing] entire agent trajectories, such as dialogues (Wang et al., 2023; Mei et al., 2024; Liu et al., 2024a; Zhong et al., 2024) or execution trajectories (Liu et al., 2024b; Wang et al., 2024b; Shang et al., 2024), directly to memory." More sophisticated variants incorporate summaries, latent embeddings, or structured knowledge representations, and recent systems like Mem0 (Chhikara et al., 2025) and Memory OS (Kang et al., 2025) provide "finer control on memory management."

Where they fall short: These systems are fundamentally text-based. As Section 2.1 argues:

"most existing approaches focus on LLM agents. In contrast, multimodal agents process a broader range of inputs and store richer, multimodal content and concepts in memory. This also introduces new challenges, particularly in maintaining consistency of long-term memory."

The jump from text-only to multimodal memory is not incremental. When an agent must track a person's identity across visual appearances (changing clothing, lighting, angles), audio (different voices, background noise), and textual descriptions, maintaining consistent entity representations becomes a first-order challenge that text-only memory systems never face. Moreover, the paper notes that "just as humans acquire world knowledge through experience, multimodal agents should form internal world knowledge in memory, rather than merely storing description of experience"—a capability that text-only agent memory systems, which typically log raw interactions, do not address.

2. Online Video Understanding Methods

Section 2.2 surveys online video understanding approaches, which are the closest technical relatives to the memorization component of M3-Agent. The paper groups these into two sub-families:

Memory-augmented architectures (MovieChat [42], MA-LMM [14], Flash-VStream [58]) introduce memory modules that "store encoded visual features for future retrieval." These are "suited for online video processing" because they avoid reprocessing entire histories. However:

"they face a fundamental limitation: maintaining long-term consistency. Because they store only visual features, these methods struggle to maintain coherent tracking of entities such as human identities or evolving events over time."

The limitation is structural: compressed visual features may preserve perceptual similarity for retrieval but discard the semantic information needed to recognize that two visually different appearances (e.g., a person in different outfits or lighting) refer to the same entity, or to track abstract properties like relationships and personality traits that have no direct visual signature.

Socratic Models (Zeng et al., 2022) and their derivatives (Lin et al., 2023b; Zhang et al., 2024a) leverage multimodal models to generate video descriptions as language-based memory, improving scalability. Yet these systems "still encounter challenges in maintaining long-term consistency across complex, evolving video content." The issue is that language-based descriptions—e.g., "a man with a beard" or "a woman in a red dress"—are "inherently ambiguous and prone to inconsistencies when accumulated over time" (Section 4.2). When the same person appears across dozens of clips described in slightly different language, the system has no reliable mechanism to link those descriptions to a single identity.

Neither family of online video understanding methods addresses world knowledge construction. They describe what happens in video but do not abstract from those descriptions to form generalized knowledge (e.g., "Alice prefers coffee in the morning" or "the green bin is for recycling") that would enable reasoning beyond direct observation.

3. Existing LVQA Benchmarks

Section 3 (particularly Table 2) provides a systematic comparison of M3-Bench against six prior LVQA benchmarks: EgoSchema, LongVideoBench, HourVideo, MVBench, Video-MME, and MLVU. The critique is precise:

• No agent perspective: Only EgoSchema and MVBench feature videos with an agent present, and neither pairs this with the full suite of memory-reasoning question types.
• No cross-modal reasoning evaluation: None of the prior benchmarks include questions that require integrating visual and auditory information to reach a conclusion.
• No person understanding evaluation: None test the ability to reason about character identity, personality, emotions, or relationships from accumulated observations.
• No general knowledge extraction: None evaluate whether the system can extract abstract, generalizable knowledge from specific observed events.

The paper argues this gap is not incidental but reflects the field's historical focus on perceptual understanding over cognitive reasoning. M3-Bench is explicitly designed to test "higher-level cognitive abilities that rely on long-term memory and are crucial for real-world agents" (Section 2.2)—abilities that existing benchmarks, by construction, cannot measure.

How This Paper Positions Itself Relative to Existing Work

M3-Agent positions itself as an integrative framework that bridges the gaps identified above, rather than as a point improvement on any single prior approach. The positioning operates along four dimensions:

1. From text-only to multimodal agent memory. M3-Agent extends the concept of structured, entity-centric agent memory—developed in text-only LLM agent systems—to the multimodal domain. The key innovation is not the idea of memory graphs per se, but rather the integration of multimodal feature preservation (faces, voices) with language-based semantic memory in a unified graph structure, enabling both perceptual consistency (multimodal features provide stable identity anchors) and semantic richness (textual memory captures abstract knowledge).

2. From passive video description to active world-knowledge construction. Unlike Socratic Models and video captioning approaches that generate descriptions as end products, M3-Agent treats episodic memory generation as an intermediate step toward building semantic memory—generalized knowledge about characters, relationships, and the environment. This distinction is central: the paper's ablation showing that removing semantic memory causes accuracy drops of 17.1%, 19.2%, and 13.1% (Section 5.4, Table 6) empirically demonstrates that world knowledge, not just detailed descriptions, is the critical ingredient for reasoning.

3. From single-turn RAG to multi-turn RL-trained reasoning. The control process (Section 4.3) is explicitly contrasted with "traditional single-turn RAG" (Section 1): rather than retrieving memory once and generating an answer, M3-Agent performs iterative memory retrieval across multiple reasoning steps, with the policy trained via reinforcement learning (DAPO) to optimize this multi-turn process. This connection to RL-trained agent reasoning—rather than prompted reasoning with commercial models—positions the work at the intersection of agent memory and RLHF/GRPO-style policy optimization, fields that have largely developed in parallel.

4. From perceptual-only to memory-reasoning evaluation. M3-Bench redefines what LVQA evaluation should measure by introducing question types (Table 1) that require aggregating evidence across temporally distant segments (multi-evidence reasoning), chaining inferences across modalities (cross-modal reasoning), inferring abstract character properties from behavioral observations (person understanding), and extracting generalizable knowledge from specific episodes (general knowledge extraction). These question types are designed to be impossible to answer from any single video segment, forcing systems to rely on accumulated memory.

The paper's experimental baselines (Section 5.1) are carefully chosen to test each of these positioning claims: Socratic Models test whether passive description suffices (it doesn't—they achieve only 2.0–8.5% on M3-Bench-robot), online video understanding methods test whether perceptual feature storage suffices (MA-LMM achieves 24.4% but is capped by consistency limitations), and prompted commercial agents test whether scale and prompting alone can compensate for trained memory management (Gemini-GPT4o-Hybrid reaches 24.0% on M3-Bench-robot, substantially below M3-Agent's 30.7%). The consistent underperformance of each baseline family on the specific capabilities M3-Agent targets serves as empirical validation of the paper's integrative positioning.

Finally, the paper positions its RL training approach (DAPO, Section 4.4) as a practical contribution to the open-source agent development ecosystem. By initializing from Qwen2.5-Omni (memorization) and Qwen3 (control)—both open-source models—and releasing model weights, code, and data, the paper implicitly argues that effective multimodal agent memory is achievable without proprietary commercial models, though the strongest baselines (Gemini-GPT4o-Hybrid, Gemini-Agent) still rely on closed-source systems. The fact that M3-Agent, trained via RL on open-source foundations, outperforms prompted versions of Gemini-1.5-Pro and GPT-4o—models widely considered more capable in raw terms—strengthens the argument that architectural design (entity-centric multimodal memory, two-stage memory generation, multi-turn RL-trained retrieval) matters more than base model scale for this class of problems.

3. Technical Approach

3.1 Reader Orientation

M3-Agent is a multimodal agent system that processes continuous video and audio streams to build a structured, entity-centric long-term memory, then uses that memory to answer questions through multi-turn reasoning. The system solves the problem of maintaining coherent, queryable world knowledge from infinitely long sensory input by decomposing memory into two complementary forms—episodic memory (what happened) and semantic memory (what can be inferred from what happened)—and training a control policy via reinforcement learning to iteratively retrieve relevant memories rather than relying on single-turn retrieval.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected through two parallel processes:

1. Multimodal LLM (Memorization Model) — processes incoming 30-second video clips with audio, facial recognition results, and speaker identification results to generate episodic memory (detailed scene descriptions) and semantic memory (abstracted knowledge about characters, relationships, and the environment). Implemented as a fine-tuned Qwen2.5-Omni-7b.

2. Long-Term Memory Database — an external, structured store organized as an entity-centric multimodal graph. Nodes represent distinct memory items (text, images of faces, audio of voices), each with a unique ID, modality type, raw content, embedding vector, confidence weight, and metadata. Edges represent logical relationships (e.g., linking a face node to a voice node when they belong to the same person).

3. Multimodal LLM (Control Model) — receives user questions and autonomously performs multi-turn reasoning, invoking search functions to retrieve relevant memories from the database at each turn, until it has sufficient information to answer. Implemented as a Qwen3 model trained via reinforcement learning (DAPO).

4. Search Tools — two retrieval mechanisms: search_node retrieves top-k most relevant individual memory nodes for a query (supports text, image, or audio queries), and search_clip retrieves top-k most relevant 30-second memory clips (each clip's episodic + semantic memory as a unit).

5. External Perception Tools — facial recognition (InsightFace with HDBSCAN clustering) and speaker identification (Gemini-1.5-Pro for ASR + diarization, ERes2NetV2 for voice embedding) that extract identity-anchored features from raw video/audio for consistent entity tracking.

Information flows through two parallel processes:

Memorization (always running): Raw video stream → 30-second clips → perception tools extract faces/voices with global IDs → memorization model generates episodic memory (text describing events, referencing entities by ID) and semantic memory (text describing character attributes, relationships, equivalence mappings, and general knowledge) → new memories inserted into multimodal graph as nodes and edges, with conflict resolution via weight-based voting.

Control (triggered by a question): User question → control model reasons about what information it needs → generates a search query → search tools retrieve top-k relevant nodes or clips from memory → retrieved content appended to context → control model reasons again → repeats until answer is derivable or maximum rounds reached → final answer generated.

3.3 Roadmap for the Deep Dive

• First, the long-term memory data structure (Section 4.1 in the paper) — the node schema, graph organization, edge semantics, weight-based conflict resolution, and the two search functions that operate on it. This is the foundation everything else reads from and writes to.

• Second, the memorization process (Section 4.2) — how 30-second clips are processed: perception tools for consistent entity representation, then the memorization model's two-stage generation of episodic and semantic memory. This is where memories are created.

• Third, the control process (Section 4.3) — the multi-turn search-and-reason loop, the action grammar ([Search]/[Answer]), and the Algorithm 1 pseudocode. This is how the agent uses memory to answer questions.

• Fourth, the imitation learning for memorization (Section 4.4, first part) — the synthetic data generation pipeline for training memory-7b-sft, including episodic memory synthesis (GPT-4o + Gemini-1.5-Pro hybrid), identity equivalence detection via meta-clip voting, and semantic memory synthesis. This explains where the training signal comes from.

• Fifth, the reinforcement learning for control (Section 4.4, second part) — the DAPO training setup, reward function (binary correctness from GPT-4o evaluator), advantage normalization, and the clipped policy objective. This explains how the agent learns to retrieve effectively.

• Sixth, the key design choices and their justifications — why entity-centric multimodal memory over text-only descriptions, why two separate models for memorization vs. control, why RL over single-turn RAG, and why generative episodic+semantic memory over compressed feature storage.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems architecture paper with an RL training methodology. The core idea is that a multimodal agent must maintain an entity-centric, incrementally updated multimodal memory graph—comprising both episodic descriptions and abstracted semantic knowledge—and must be trained via reinforcement learning to perform multi-turn iterative retrieval from this memory, because single-turn RAG and prompted commercial models are insufficient for questions requiring evidence aggregation across hour-scale videos.

---

Long-Term Memory: The Entity-Centric Multimodal Graph

The long-term memory is implemented as an external database storing information in a structured, multimodal format, supporting text, images (faces), and audio (voices). It is organized as an entity-centric multimodal graph, meaning that information related to the same real-world entity (e.g., a specific person's face, voice, name, personality traits, and observed behaviors) is connected through edges, forming a unified representation that persists across time.

Node structure. Each node in the graph represents a distinct memory item and carries the attributes specified in Table 3:

• id: a unique identifier for the node
• type: the modality type—text (natural language memory), image (a face), or audio (a spoken dialogue segment)
• content: the raw content—plain text string, base64-encoded image, or base64-encoded audio
• embedding: a vector representation of the node content used for similarity-based retrieval (text nodes use OpenAI's text-embedding-3-large; face nodes use InsightFace identity embeddings; voice nodes use ERes2NetV2 embeddings)
• weight: a numeric value indicating the confidence or frequency of this memory item; frequently activated or confirmed memories accumulate higher weights
• extra_data: a JSON object containing additional metadata such as timestamps

Edge structure. Nodes are connected by undirected edges representing logical relationships. The primary edge type is co-reference: items sharing the same entity ID are linked, forming the entity-centric structure. For example, a face node (image of a person), a voice node (audio clip of that person speaking), a text node describing that person's personality, and a text node describing an event involving that person would all be connected. Additionally, equivalence edges are explicitly created during memorization when the model infers that a face ID and a voice ID correspond to the same person (e.g., "Equivalence: <face_1>, <voice_2>"). Once linked, the pair is treated as a single character and assigned a unified <character_id> for retrieval purposes.

Weight-based conflict resolution. Because memories are generated incrementally from potentially ambiguous observations, conflicting information may be introduced. For instance, in one clip, <voice_3> might be incorrectly linked to <face_0>, while in several subsequent clips it is correctly linked to <face_1>. M3-Agent resolves this through a voting mechanism: when a memory is generated that already exists (same content or same relationship), the corresponding node or edge is reactivated and its weight incremented. During inference, frequently activated entries accumulate higher weights and override conflicting entries with lower weights. The paper states: "frequently activated entries accumulate higher weights and override conflicting entries with lower weights. This mechanism ensures the robustness and consistency of the memory graph over time" (Section 4.1). This is analogous to a frequency-based truth-discovery mechanism operating continuously as new observations arrive.

What this design achieves. The entity-centric graph structure supports two retrieval modes that the control process can leverage: sequential retrieval (following edges to find all information about a specific entity) and associative retrieval (using embedding similarity to find semantically relevant content regardless of entity). A key architectural property is that entity representations are persistent across time: when a face or voice appears in a new clip, the perception tools match it to an existing node (or create a new one if no match exceeds the threshold), and all subsequent memory generation references that stable ID. This means the textual memories from different temporal segments remain connected through the shared identity nodes, preventing the fragmentation that plagues description-only memory systems.

---

Search Tools: Two-Level Retrieval

The control process accesses long-term memory through two search functions summarized in Table 4, operating at different granularities:

search_node (fine-grained retrieval): Accepts a query (text, image, or audio) and returns the top-k most relevant individual memory nodes. For text queries, the function performs Maximum Inner Product Search (MIPS) between the query embedding (computed via text-embedding-3-large) and all text node embeddings. Multi-modality queries are supported: an image query searches against face nodes, an audio query searches against voice nodes. The function uses a similarity threshold t (set to 0.5 for all experiments) and returns the k most relevant nodes whose similarity exceeds t (with k = 2 throughout the paper to "reduce retrieval randomness and enable consistent evaluation across models," per Appendix 10).

search_clip (coarse-grained retrieval): Returns the top-k most relevant memory clips, where a "memory clip" refers to the agent's episodic and semantic memory generated from a single 30-second segment during clip-by-clip streaming video processing. Each clip is scored by the highest similarity among its constituent memory entries to the query, and the top-ranked clips are returned. In all experiments, k = 2 for this function as well.

Why two levels: The paper argues that different question types require different retrieval granularity. For questions about specific entities (e.g., "What is Tomasz's personality?"), search_node can retrieve precisely the semantic memory nodes about that character. For questions about temporally-situated events (e.g., "What happened after they visited Ding Cha?"), search_clip retrieves the relevant temporal segments. The control model, trained via RL, learns to choose which search function and what query to use at each reasoning step.

Implementation details for face/voice matching (Appendix 10): For face and voice nodes, retrieval uses a different mechanism than text MIPS. Each face node maintains a set of representative feature snapshots (facial embeddings from multiple frames). When a new face feature is extracted, the system computes the average cosine similarity between the extracted feature and all stored snapshots per node. The node with the highest similarity exceeding a pre-defined threshold (0.3 for image, 0.6 for voice) is considered a match; otherwise, a new node is created. Matched nodes are updated with the new features, refining their representations over time. These thresholds were chosen empirically to balance false-positive merges (grouping different people) against false-negative splits (creating duplicate identities for the same person).

---

Memorization: From Raw Video to Structured Memory

During memorization, M3-Agent processes the incoming video stream in a clip-by-clip manner, segmenting it into 30-second clips. For each clip, two types of memory are generated: episodic and semantic. The process has two stages: (1) consistent entity representation extraction using external tools, and (2) text memory generation using the memorization model.

Stage 1: Consistent entity representation via perception tools. The central challenge of building long-term memory from video is maintaining consistent identity representations across arbitrarily long timespans. The paper argues that purely textual descriptions (e.g., "a man with a beard" or "a woman in a red dress") are "inherently ambiguous and prone to inconsistencies when accumulated over time" (Section 4.2). M3-Agent instead preserves the original multimodal features and constructs persistent identity representations.

Two external tools are used:

• Facial recognition: Video frames are uniformly sampled at 5 frames per second. For each frame, InsightFace's buffalo_l model extracts facial attributes: bounding box coordinates, identity embeddings, and detection/quality scores. Low-quality detections (abnormal aspect ratios or very low confidence) are discarded. HDBSCAN clustering is then applied to the embeddings of remaining high-quality faces to group them by character identity, producing a set of face clusters, each assigned a stable face_id that persists across the entire video.

• Speaker identification: Gemini-1.5-Pro extracts audio segments corresponding to distinct speaker voices while simultaneously performing automatic speech recognition (ASR) on each segment. The prompt used is shown in Table 9 (Appendix 10). Segments shorter than 2 seconds are filtered out. The voice embedding model ERes2NetV2 [3] then encodes each segment into a speaker-specific representation. These embeddings are clustered to group segments from the same speaker, producing a set of voice clusters, each assigned a stable voice_id.

Both tools operate globally across the entire video, not per-clip. For each new 30-second clip, the tools extract the faces and voices present and assign them their global IDs (or create new IDs for previously unseen entities). These IDs are then provided as input to the memorization model along with the video clip itself.

Stage 2: Memory generation by the memorization model. The memorization model takes as input: the 30-second video clip (with audio), a list of detected faces with their face_ids, and a list of detected voice segments with their voice_ids, start/end times, and ASR transcripts. It produces two outputs:

• Episodic memory: An ordered list of text sentences describing the concrete events, actions, dialogue, appearances, and environmental details observed in the clip. Each character must be referenced by their face_id or voice_id. For example: "<face_1> wears a red hat and blue top" or "<voice_2> speaks to <face_3>, 'How are you doing today?'" The generation requirements (from the prompt in Table 21, Appendix 15.3) include: describing characters' appearance, actions and movements, spoken dialogue (quoted or summarized), contextual behavior (emotional states, roles, relationships), and environmental/temporal cues. Each sentence must express one event or detail.

• Semantic memory: An ordered list of text sentences expressing high-level, reasoning-based conclusions. This includes five categories:

  1. Equivalence identification: Explicit mappings between face and voice IDs belonging to the same person, in the format "Equivalence: <face_x>, <voice_y>".
  2. Character-level attributes: Inferred abstract properties such as name (if explicitly stated), personality (e.g., confident, nervous), role or profession, interests, and distinctive behaviors.
  3. Interpersonal relationships and dynamics: Descriptions of social roles, emotional tone, power dynamics, and evidence of cooperation or conflict between characters.
  4. Video-level plot understanding: Scene-level narrative summaries including main events, narrative arcs, and overall tone.
  5. Contextual and general knowledge: General knowledge extracted from the clip, such as setting or genre, cultural norms, real-world facts, and attributes/functional roles of objects.

The detailed content requirements for each category are specified in Table 10 (Appendix 11).

Entity identity equivalence and cross-modal linking. A critical function performed within semantic memory generation is cross-modal reasoning to infer equivalences between face and voice IDs. When the model determines that <face_1> and <voice_3> belong to the same person, this equivalence is stored as a semantic memory text entry and simultaneously used to update the memory graph by creating an edge between the two nodes. Once linked, the pair is treated as a single character. During retrieval, connected nodes are unified under a shared <character_id>, enabling the control model to reason about characters consistently across modalities.

Output format and memory integration. Both episodic and semantic memories are generated as Python lists of text strings. Each text entry is stored in the memory graph as a text node, except for entity equivalence relationships, which are stored as edges. The weight-based voting mechanism handles conflicts: if a particular equivalence (e.g., <voice_3> maps to <face_0>) is generated incorrectly in some clips due to challenging conditions, but correctly generated in most clips, the correct mapping accumulates higher weight over time and dominates during retrieval. The paper states: "as correct associations accumulate, the weight of the correct mapping (<voice_3>, <face_0>) increases and dominates. This allows the system to robustly learn and maintain accurate knowledge, even in the presence of occasional local errors" (Section 4.2).

Design rationale for two-memory generation. Why generate both episodic and semantic memory rather than just one? The paper's ablation (Table 6) provides the empirical justification: removing semantic memory causes accuracy drops of 17.1%, 19.2%, and 13.1% on the three benchmarks. The conceptual justification is that episodic and semantic memory serve complementary functions in reasoning. Episodic memory anchors answers in observable evidence—if a question asks "What color was the folder Bob showed the robot?", the answer can be retrieved directly from an episodic memory entry. Semantic memory enables inference beyond direct observation—if a question asks "Is Lucas skilled at cooking?", the answer requires aggregating behavioral observations across multiple scenes and abstracting from them, which is precisely what semantic memory entries (e.g., "<face_x> displays competence and confidence in cooking tasks") provide as pre-computed inferences.

---

Control: Multi-Turn Reasoning with Iterative Memory Retrieval

When a user question is received, the control process is triggered. The control model (a Qwen3-based LLM trained via RL) performs a multi-turn reasoning loop, iteratively deciding whether to search memory for additional information or to answer based on the information already retrieved.

Algorithm 1: The control loop. The process follows the pseudocode in Algorithm 1 (Section 4.3):

1. Initialization: The trajectory τ is initialized with a system prompt and an instruction prompt. The system prompt (Table 22, Appendix 15.3) states: "You are given a question and some relevant knowledge. Your task is to reason about whether the provided knowledge is sufficient to answer the question. If it is sufficient, output [Answer] followed by the answer. If it is not sufficient, output [Search] and generate a query that will be encoded into embeddings for a vector similarity search."

2. Iterative loop (up to H rounds, where H = 5 in all experiments):

   • The policy model π_θ generates a response τ_i conditioned on the current trajectory.
   • The response is parsed into three components: reasoning (a thinking block explaining the model's analysis), action (either [Search] or [Answer]), and content (a search query if [Search], or the final answer if [Answer]).
   • If action is [Search]: The content is used as a query to search the long-term memory. The search can invoke either search_node or search_clip depending on what the model specifies. The retrieved results are appended to the trajectory as a user message containing the search results plus a repeated instruction prompt for the next round.
   • If action is [Answer]: The loop terminates and the content is returned as the final answer.
   • If i = H - 1 (penultimate round): A special last_round_prompt is appended that states "The Action of this round must be [Answer]. If there is insufficient information, you can make reasonable guesses" (Table 22), forcing the model to produce an answer rather than continuing to search indefinitely.

The instruction prompt and inter-turn guidance. At each round, the instruction prompt (Table 22) provides detailed guidance for constructing search queries:

• "You can get the mapping relationship between character ID and name by using search query such as: 'What is the name of <character_i>' or 'What is the character id of {name}'."
• "After obtaining the mapping, it is best to use character ID instead of name for searching."
• "If the answer can be derived from the provided knowledge, the {content} is the specific answer to the question. Only name can appear in the answer, not character ID like <character_i>."
• "The search {content} needs to be different from the previous" — encouraging the model to explore different retrieval angles rather than repeating failed queries.

The ablation in Table 7 shows that removing inter-turn instruction causes accuracy drops of 10.5%, 5.8%, and 5.9% on the three benchmarks, demonstrating that this structured guidance is critical for effective multi-turn retrieval.

The reasoning action. By default, the control model outputs a thinking block (visible in the case study trajectory in Table 17 before each action. The paper also evaluates a variant without reasoning (control-32b-rl w/o reasoning in Table 7), which causes accuracy drops of 11.7%, 8.8%, and 9.5% on the three benchmarks. The case study (Table 17, Appendix 14) illustrates the reasoning's function: the model analyzes what it currently knows, identifies what is missing, and formulates a targeted query. For example, in the first round for the question "Is Tomasz a person with rich imagination or someone who lacks imagination?", the model reasons: "The problem is that there's no provided knowledge about Tomasz in the given context... the first step is to find the character ID for Tomasz," then generates the query "What is the character id of Tomasz."

Contrast with single-turn RAG. Section 1 explicitly distinguishes this multi-turn approach from "traditional single-turn RAG": rather than retrieving memory once and generating an answer, M3-Agent can retrieve information in stages—first finding a character's ID, then querying their attributes, then querying specific behaviors—with each subsequent query informed by what was found in previous rounds. This is analogous to how a human would approach the task: first identifying the relevant entity, then progressively narrowing down to the specific information needed.

---

Imitation Learning for Memorization: Synthetic Data Generation

The memorization model (memory-7b-sft) is trained via supervised fine-tuning (imitation learning) on a synthetically generated dataset. The training data is sourced from an in-house video dataset with permissions for model training, comprising 500 long videos (26,943 30-second clips) and 2,736 question-answer pairs, annotated to the same standards as M3-Bench-web.

Why synthetic data instead of human annotation: The paper does not explicitly justify this choice, but the scale of the needed annotations makes it implicit: generating episodic and semantic memory for 26,943 clips would require enormous human effort, whereas the synthetic pipeline can be automated. The quality validation (Appendix 11.4) shows that a manual review of 353 memory items from 10 randomly sampled clips found 95.5% accuracy, suggesting the synthetic data is sufficiently reliable for training.

Three-stage synthesis process:

Stage 1: Episodic memory synthesis. The paper employs a hybrid annotation strategy combining GPT-4o and Gemini-1.5-Pro, designed to leverage their complementary strengths:

• GPT-4o provides fine-grained visual details but cannot process audio. It receives 16 video frames sampled at approximately 0.5 fps (for a 30-second clip) and is prompted (Table 11) to generate a detailed description of visual elements—colors, objects, movements, environments, people including actions, clothing, expressions—without speculating beyond what is visible. The output serves as a "reference description."
• Gemini-1.5-Pro receives the full video clip (with audio), detected faces with IDs, voice segments with IDs and ASR transcripts, and the GPT-4o reference description. It is prompted (Table 11) to generate the final episodic memory, with instructions to "incorporate correct elements from the [Reference Description], and correct any mistakes you identify" and to "add any additional details visible or inferable from the [Video], [Faces], and [Dialogues] that are missing from the reference."

The resulting episodic memory is a list of sentences, each describing one event or detail, with characters referenced by their face_id or voice_id. The prompts enforce that characters must not be referenced by pronouns or inferred names—only by their provided feature IDs.

Stage 2: Identity equivalence detection via meta-clip algorithm. This is a novel contribution for automatically establishing face-voice mappings from long videos. The key insight is that meta-clips—short segments (≤5 seconds) containing exactly one face identity and one voice identity—provide high-confidence training signal for identity equivalence.

The algorithm (Algorithm 2, Appendix 11.2) proceeds as follows:

1. Global ID extraction: For a given long video, run facial recognition and speaker identification globally to obtain face IDs F = {f_1, ..., f_N} and voice IDs V = {v_1, ..., v_N}.

2. Meta-clip extraction: Divide the video into short clips (≤5 seconds each) using keyframe-based segmentation to ensure visual stability. For each short clip c_t, detect the set of face IDs F_t ⊆ F and voice IDs V_t ⊆ V present. If |F_t| = 1 and |V_t| = 1 (exactly one face and one voice), the clip is added to the meta-clip set C_meta as a pair (c_t, f, v). The logic: in a clip with exactly one visible face and one audible voice, it is highly likely they belong to the same person.

3. Bipartite graph construction: Build a bipartite graph G = (F, V, E) where each edge (f, v) has weight w(f, v) = |{(c_t, f, v) ∈ C_meta}|—the count of meta-clips supporting that face-voice association.

4. Edge filtering: Remove all edges with weight equal to 1 (single-occurrence associations are treated as noise). For each face f, keep only the edge to the voice v* with maximum weight, and only if the proportion of the top edge's weight relative to all edges incident to f exceeds a threshold p: w(f, v*) / Σ_{v_i ∈ N_f} w(f, v_i) ≥ p. Analogous filtering is applied from the voice side. This ensures that only high-confidence, consistently observed mappings are retained.

5. Dictionary construction: The surviving edges form a mapping dictionary M: V → F, mapping each voice ID to its corresponding face ID.

Why this algorithm works: Meta-clips provide natural "labeled data" because the physical constraint of monologue (one person speaking while visible) creates an unambiguous signal. The voting mechanism across all meta-clips from the entire video ensures robustness: even if individual clips have errors (e.g., the speaker is off-screen but another person is visible), the correct mapping dominates in frequency.

The paper reports that for the training dataset, this algorithm produced 10,952 30-second clips with valid identity equivalence annotations, and a manual review of 48 randomly sampled mappings found 95.83% accuracy.

Stage 3: Semantic memory synthesis. Following the same hybrid strategy as episodic memory, GPT-4o first generates preliminary semantic memory based on video frames and the episodic memory (from Stage 1). Then Gemini-1.5-Pro receives the full video clip, episodic memory, and GPT-4o's preliminary semantic memory, and is prompted (Table 12) to generate the final semantic memory across the five categories. The prompt instructs the model to "incorporate correct elements from the reference conclusions, and correct any mistakes you identify" and to produce conclusions that "reflect deeper reasoning and insight, not surface-level observations already evident from the plot description."

Training details. The synthetic dataset comprises 10,752 training samples and 200 validation samples. Fine-tuning is conducted on Qwen2.5-Omni-7b for 3 epochs with:

• Learning rate: 1e-5
• Batch size: 16
• Hardware: 16 GPUs with 80GB memory each
• Optimizer: not explicitly specified, but standard for SFT on this model class

The best checkpoint is selected based on validation set performance (Table 13, Appendix 12) using two metrics: AutoDQ (assesses description quality by comparing generated outputs to reference descriptions) for episodic and non-equivalence semantic memory, and precision/recall/F1 for identity equivalence against ground truth. The 3-epoch checkpoint achieves AutoDQ-F1 of 0.627 and Equivalence-F1 of 0.846, significantly outperforming the prompted baselines (memory-gemini-prompt: AutoDQ-F1 0.606, Equivalence-F1 0.595; memory-7b-prompt: AutoDQ-F1 0.414, Equivalence-F1 0.145).

---

Reinforcement Learning for Control: DAPO Training

The control model is trained via reinforcement learning using the DAPO algorithm (Yu et al., 2025) [54]. The core idea is to train the policy model π_θ to generate effective multi-turn search-and-answer trajectories, where the reward signal is binary correctness of the final answer.

Environment setup. For each video in the training dataset, the long-term memory is pre-generated using memory-7b-sft (the fine-tuned memorization model). For any given question, the agent is restricted to searching within the memory generated from the video associated with that question. This ensures the agent cannot retrieve information from unrelated videos.

Reward function. For each question-answer pair (q, a) sampled from the training dataset D, the policy π_θ rollouts a group of G = 4 trajectories {τ_i}_{i=1}^G using the control loop (Algorithm 1). For each trajectory τ_i, the final submitted answer y_i is extracted. The reward is binary:

$R_i = \begin{cases} 1, & \text{gpt4o\_evaluator}(q, a, y_i) = \text{True} \\ 0, & \text{otherwise} \end{cases}$

where gpt4o_evaluator(q, a, y_i) is the GPT-4o evaluator introduced in Section 3.3 (prompt in Table 18, Appendix 15.1). The evaluator is prompted to assess whether the ground-truth answer can be logically inferred from the agent's answer, not whether they match exactly: "Do not directly compare the surface forms of the agent answer and the ground truth answer. Instead, assess whether the meaning expressed by the agent answer supports or implies the ground truth answer." This is important because the model may produce correct but differently phrased answers.

Why binary reward: The task is framed as success/failure—the agent either answers correctly or it doesn't. A continuous reward (e.g., based on partial correctness or answer similarity) would be harder to define reliably. Binary reward with a validated automatic evaluator (96% agreement with human judges, Section 3.3) provides a clean training signal.

Advantage normalization. The advantage of the i-th response is calculated by normalizing the group-level rewards:

$\hat{A}_{i,t} = \frac{R_i - \text{mean}(\{R_i\}_{i=1}^G)}{\text{std}(\{R_i\}_{i=1}^G)}$

where mean({R_i}_{i=1}^G) is the average reward across the group of G = 4 trajectories for the same question, and std({R_i}_{i=1}^G) is the standard deviation.

What this computes: The advantage measures how much better or worse a particular trajectory is compared to the average trajectory the current policy produces for the same question. Positive advantage means the trajectory led to a correct answer while other trajectories did not; negative advantage means it failed while others succeeded. Normalizing by standard deviation ensures the advantages have consistent scale across questions with different difficulty levels.

Why this form (group-based normalization): This is the key insight of GRPO/DAPO-style algorithms. By computing advantages within a group of rollouts from the same question, the baseline for comparison is the current policy's own performance, not a separately learned value function. This eliminates the need to train a critic network and avoids the bias-variance tradeoffs of value function estimation. The constraint 0 < Σ_{i=1}^G R_i < G in the DAPO objective (Equation 3) ensures that groups with all-correct or all-incorrect trajectories are excluded—they provide no contrastive signal for learning.

Policy optimization objective (DAPO). The DAPO objective is:

$\mathcal{J}_{\text{DAPO}}(\theta) = \mathbb{E}_{(q,a) \sim D, \{\tau_i\}_{i=1}^G \sim \pi_{\theta}^{\text{old}}(\cdot|q)}\left[\frac{1}{\sum_{i=1}^G \sum_{t=1}^{|\tau_i|} \mathbb{I}(\tau_{i,t})} \sum_{i=1}^G \sum_{t=1}^{|\tau_i|} \mathbb{I}(\tau_{i,t}) \cdot \min\left(\frac{\pi_{\theta}(\tau_{i,t}|\tau_{i,<t})}{\pi_{\theta}^{\text{old}}(\tau_{i,t}|\tau_{i,<t})}\hat{A}_{i,t}, \text{clip}\left(\frac{\pi_{\theta}(\tau_{i,t}|\tau_{i,<t})}{\pi_{\theta}^{\text{old}}(\tau_{i,t}|\tau_{i,<t})}, 1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}}\right)\hat{A}_{i,t}\right)\right]$

subject to the constraint 0 < Σ_{i=1}^G R_i < G.

where:

• τ_i is the i-th trajectory in the group
• τ_{i,t} is the t-th token in the i-th trajectory
• τ_{i,<t} represents all tokens before position t in trajectory i
• 𝕀(τ_{i,t}) = 1 if τ_{i,t} is an LLM-generated token, 0 otherwise (non-LLM tokens, such as retrieved memory content, are excluded from the loss)
• π_θ(τ_{i,t} | τ_{i,<t}) is the probability the current policy assigns to token τ_{i,t} given the preceding context
• π_θ^{old}(τ_{i,t} | τ_{i,<t}) is the probability the old policy (from the previous iteration) assigned to the same token
• ε_low = 0.2 and ε_high = 0.28 are the clipping parameters

What this computes: The objective maximizes the probability of tokens that led to correct answers (positive advantage) and minimizes the probability of tokens that led to incorrect answers (negative advantage), but with clipping to prevent the policy from changing too drastically in a single update. The min operator between the unclipped and clipped importance ratios ensures that the objective is a lower bound on the true expected improvement, following the PPO/GRPO trust-region principle.

Key differences from standard GRPO: DAPO introduces two modifications over GRPO:

1. The denominator in the loss normalization: 1 / (Σ_{i=1}^G Σ_{t=1}^{|τ_i|} 𝕀(τ_{i,t})) normalizes by the total number of LLM-generated tokens across all trajectories in the group, rather than averaging per-trajectory then per-group. This gives equal weight to each token regardless of which trajectory it belongs to.
2. Asymmetric clipping: ε_low = 0.2 and ε_high = 0.28. The higher upper clip allows the policy to increase probabilities of good actions more than it decreases probabilities of bad actions. The paper does not justify this asymmetry, but it is a standard practice in DAPO to prevent entropy collapse.

The ablation in Table 7 shows that DAPO-trained control-32b-rl consistently outperforms GRPO-trained control-32b-grpo on all three benchmarks (e.g., 30.7% vs. 30.0% on M3-Bench-robot), validating this design choice.

Training hyperparameters (Table 14):

• Batch size: 32
• GPUs: 16 (8B and 14B) or 32 (32B) with 80GB memory each
• Rollout model parallel size: 1 (8B, 14B) or 2 (32B)
• Learning rate: 1e-6
• Maximum rounds H: 5
• Number of samples per group G: 4
• Total steps: 180
• ε_low: 0.2
• ε_high: 0.28

Why train only on LLM-generated tokens (the indicator 𝕀(τ_{i,t})): The trajectory includes both tokens generated by the policy model and tokens from the environment (retrieved memory content, instruction prompts). Computing loss on environment tokens would be meaningless—they are not under the policy's control. The indicator restricts gradient updates to only the tokens the policy actually chose.

Scaling with model size. Table 7 shows the effect of model scale on RL training performance. After DAPO training:

• control-8b-rl: 24.6%, 40.5%, 50.8% (M3-Bench-robot, M3-Bench-web, VideoMME-long)
• control-14b-rl: 28.2%, 46.9%, 56.0%
• control-32b-rl: 30.7%, 48.9%, 61.8%

The improvements over prompted baselines (control-8b-prompt: 16.4%, 35.7%, 45.3%; control-14b-prompt: 18.3%, 36.9%, 49.1%; control-32b-prompt: 20.7%, 40.9%, 52.5%) demonstrate that (1) RL training provides substantial gains across all model sizes, and (2) the gains from RL scale with model size—larger models benefit more from RL training (10.0 percentage point improvement for 32B vs. 8.2 for 8B on M3-Bench-robot).

---

Summary of Key Design Choices and Their Justifications

• Entity-centric multimodal memory over text-only descriptions: Textual descriptions ("a man in a blue shirt") are ambiguous and inconsistent across time. Multimodal features (face embeddings, voice embeddings) provide stable identity anchors. The entity-centric graph structure ensures that all information about a person remains connected regardless of when it was observed. Ablation evidence: removing identity equivalence from semantic memory causes large accuracy drops (Table 6).

• Two separate models for memorization vs. control over a single unified model: "Memorization relies strong multimodal understanding, while control requires strong reasoning capabilities" (Section 4.4). The memorization model (Qwen2.5-Omni-7b) has native video and audio processing; the control model (Qwen3) is a text-only LLM with advanced reasoning. Training separate models allows each to be optimized for its task.

• Generative episodic + semantic memory over compressed feature storage: Methods like MovieChat and Flash-VStream store visual features for retrieval but lose semantic information needed for abstract reasoning ("maintaining long-term consistency" is identified as their "fundamental limitation" in Section 2.2). Generative text memory captures both perceptual details and abstract knowledge in a unified, queryable format. Ablation evidence: removing semantic memory (while keeping episodic memory) causes 17.1–19.2% accuracy drops (Table 6), proving that abstract knowledge is the critical component.

• Multi-turn RL-trained retrieval over single-turn RAG: Single-turn RAG retrieves once and answers; multi-turn retrieval allows the agent to reason about what it doesn't know, iteratively narrow down, and change retrieval strategies. RL training optimizes this process end-to-end for answer correctness. Ablation evidence: prompted baselines with identical memory but no RL training underperform by 8.0–10.0% (Table 7: control-32b-prompt vs. control-32b-rl).

• DAPO over GRPO for RL training: DAPO's per-token normalization (rather than per-trajectory) and asymmetric clipping (ε_high = 0.28 > ε_low = 0.2) provide more stable optimization for the multi-turn, variable-length trajectories in this task. Empirical evidence: control-32b-rl (DAPO) outperforms control-32b-grpo on all benchmarks (Table 7).

• Synthetic data over human annotation for memorization training: The scale needed (26,943 clips) makes human annotation infeasible. The hybrid GPT-4o + Gemini-1.5-Pro pipeline leverages complementary model strengths to produce high-quality synthetic memories (95.5% accuracy on manual review). The meta-clip algorithm provides automatic, high-confidence identity equivalence labels (95.83% accuracy).

• Weight-based conflict resolution over deterministic overwriting: In an online, incremental system, occasional errors are inevitable. Weight-based voting allows correct information to accumulate dominance over time without requiring perfect accuracy on every clip. This is a form of implicit robustness through redundancy, analogous to how human memory resolves inconsistencies through repeated exposure.

• Five-round maximum with forced answer on the last round: Setting H = 5 balances sufficient retrieval depth against computational cost. The last_round_prompt ensures the system always produces an answer (avoiding infinite loops) while allowing "reasonable guesses" when information is insufficient—mirroring how a human would provide their best estimate rather than refusing to answer.

4. Key Insights and Innovations

Innovation 1: The Dual Episodic-Semantic Memory Architecture as a Necessary Condition for Reasoning, Not an Enhancement

The paper's most fundamental conceptual contribution is demonstrating that the distinction between episodic and semantic memory—borrowed from human cognitive psychology [44, 45]—is not merely a useful organizational metaphor but a functional necessity for multimodal agents. Prior work on agent memory systems (Section 2.1) treats memory primarily as storage: append trajectories, summarize dialogues, embed execution logs, retrieve relevant context. The dominant assumption is that more detailed, better-retrieved storage of what happened is sufficient for answering questions about what was observed. M3-Agent challenges this by showing that episodic memory alone is radically insufficient for the kinds of reasoning that real-world agent deployment demands, and that the gap cannot be closed by better retrieval or larger models—it requires a different kind of memory.

This is not an incremental refinement of existing memory architectures. It is a diagnostic finding about a capability boundary. The ablation in Table 6 is the core evidence: removing semantic memory while keeping episodic memory intact—the system can still describe every event in precise detail—causes accuracy drops of 17.1% (M3-Bench-robot), 19.2% (M3-Bench-web), and 13.1% (VideoMME-long). These are not marginal degradations; they are catastrophic failures, reducing M3-Agent to performance comparable to or below the Socratic Models baselines (which themselves generate only episodic descriptions). The magnitude of these drops tells us something structurally important: a large fraction of the benchmark questions cannot be answered from episodic observation alone, regardless of how perfectly that observation is described or retrieved.

Why? Because the question types in M3-Bench (Table 1) are specifically designed to require inference beyond direct observation. "Is Lucas skilled at cooking?" cannot be answered by retrieving any single event description—no one in the video says "Lucas is skilled at cooking." The answer requires aggregating behavioral evidence across multiple scenes (episodic memory) and abstracting from it (semantic memory: "Lucas displays competence in cooking tasks"). The semantic memory serves as pre-computed inference—conclusions that the memorization model draws at encoding time and stores alongside the raw observations. Without this inference layer, the control model would need to reconstruct these abstractions at query time from raw episodic descriptions, a task that the ablation shows it cannot reliably perform even with multi-turn RL-trained retrieval.

The conceptual significance extends beyond this specific system. The finding implies that any multimodal agent that needs to answer questions about character traits, relationships, object functions, or generalizable knowledge must generate and store inferences at encoding time—deferring all reasoning to query time is not viable. This is a constraint that follows from the structure of the reasoning task, not from implementation details, and it should generalize to other agent architectures and domains. The paper's framing of this as a two-memory system (episodic for "what happened," semantic for "what it means") provides a clean conceptual vocabulary for thinking about this constraint.

The comparison to human memory is not decorative. Tulving's original distinction was motivated by neuropsychological evidence that patients with damage to episodic memory (unable to recall specific events) could still possess intact semantic knowledge, and vice versa. M3-Agent's ablation effectively demonstrates a computational analog: episodic memory and semantic memory are doubly dissociable components of the system, where removing semantic memory breaks reasoning about abstract properties while leaving event description intact. This positions the paper as providing computational evidence for a cognitive architecture that was previously grounded only in human data.

Prior work on Socratic Models (Zeng et al., 2022) and video captioning systems (VideoChat, Video-LLaVA, VILA) generates descriptions but does not abstract from them. Prior work on agent memory (Mem0, A-Mem, MemoryBank) stores structured knowledge but does so for text-only domains where the abstraction problem is simpler—entity attributes are typically stated explicitly in dialogue rather than needing to be inferred from visual and auditory behavior. M3-Agent's contribution is showing that in the multimodal domain, where most knowledge is implicit in behavior rather than explicitly stated, generative semantic memory is not optional. The Socratic Models baselines in Table 5 (achieving only 2.0–8.5% on M3-Bench-robot despite using Gemini-1.5-Pro for description generation) serve as the control condition: detailed episodic description, even from a powerful model, cannot compensate for the absence of inference.

Innovation 2: Entity-Centric Multimodal Memory as a Solution to the Long-Term Consistency Problem

The problem of maintaining consistent entity identity across long timespans is well-known in video understanding. Section 2.2 identifies it as the "fundamental limitation" of memory-based online video understanding methods: "Because they store only visual features, these methods struggle to maintain coherent tracking of entities such as human identities or evolving events over time." The dominant solution paradigm in prior work is to extend the architecture—longer context windows (LongVILA), more sophisticated token compression (VidCompress), or recurrent feature aggregation (MovieChat, Flash-VStream). These approaches attempt to preserve the information needed for identity tracking within the model's internal representations.

M3-Agent takes a fundamentally different approach that can be characterized as externalizing identity representation into a structured multimodal graph. Rather than asking the model to maintain identity consistency internally across time, the system preserves the raw perceptual features (face embeddings, voice embeddings) as persistent, addressable nodes in an external database, and uses these stable anchors to ground all textual memory references. This is a conceptual shift from compression (trying to pack identity information into a lossy representation that preserves enough similarity for retrieval) to indirection (storing a stable pointer to the identity and referencing it).

The innovation is not any individual component—face recognition, speaker identification, and graph databases all exist independently—but rather the integration of these components into a memory formation pipeline that guarantees referential consistency by construction. When the memorization model generates text referencing <face_1>, it is not generating an ambiguous description that might or might not match future descriptions. It is writing to a node that is permanently and uniquely associated with a specific person's facial features, and any future text referencing <face_1> will be connected to the same node regardless of how the person's appearance or the description language changes.

The case study comparison between memory-7b-sft and memory-gemini-prompt (Table 15, Appendix 14) illustrates the practical consequence. In the Gemini-generated memory, a character is described as "a woman, presumably his wife" walking in the background—a guess that could be wrong and provides no mechanism for linking this woman to any future appearance. In the M3-Agent memory, the same person is referenced as <face_1413>, with explicit equivalence mappings ("Equivalence: <face_1413>, <voice_1>") and character-level attributes extracted ("<face_1413> demonstrates a nurturing and caring personality"). When <face_1413> appears in a subsequent clip, the system knows it is the same person regardless of whether the new description matches the old one.

This design has an important theoretical property that the paper does not explicitly name but that is evident from the architecture: identity is maintained by the database structure, not by the model's memory. The model can be stateless with respect to identity tracking—it processes each clip independently, relying on the perception tools to provide stable IDs and the memory graph to maintain connections. This separation of concerns means that errors in the model's text generation (e.g., misidentifying a person's role in one clip) do not corrupt the identity representation—they are stored as fallible text nodes whose weights can be counterbalanced by subsequent correct observations. The weight-based voting mechanism (Section 4.1) provides a principled way to resolve such inconsistencies over time.

This is a fundamental advance over prior approaches because it solves the consistency problem at the architectural level rather than the model-capability level. Extending context windows or improving feature compression can incrementally improve identity tracking, but they do not provide guarantees—if the compressed representation of a person at time t is too different from the representation at time t + 100, the system will fragment the identity. The graph-based approach guarantees that <face_1> at time t and <face_1> at time t + 100 are the same node, regardless of visual similarity. The only failure mode is when the perception tools fail to match a face to the correct node, and the weight-based voting mechanism is specifically designed to make such failures self-correcting over time.

The ablation evidence ("memory-7b-sft w/o equivalence" in Table 6) quantifies the contribution: removing identity equivalence from semantic memory—the system can still generate episodic and semantic text, but loses the cross-modal linking that unifies faces and voices under shared character IDs—causes accuracy drops that are substantial but smaller than removing semantic memory entirely (e.g., 19.5% vs. 13.6% on M3-Bench-robot when removing equivalence vs. removing all semantic memory). This suggests that entity-centric structure provides a meaningful but incremental benefit over having semantic memory without consistent identity grounding, and that the dominant driver of performance is the presence of semantic inference itself.

Innovation 3: Multi-Turn RL-Trained Retrieval as an Alternative to Prompt-Engineered Agent Reasoning

The control architecture—multi-turn iterative memory retrieval trained via reinforcement learning—represents a methodological shift in how multimodal agent reasoning is implemented. The dominant approach in prior work on video-grounded QA and agent systems (Section 2.2, Section 5.1) is prompt engineering: design detailed prompts that instruct a capable LLM (typically GPT-4 or Gemini) to perform retrieval and reasoning, often with carefully specified output formats, in-context examples, and retrieval plans. The Gemini-GPT4o-Hybrid baseline (Table 5) represents the state of this art: Gemini-1.5-Pro for memory generation, GPT-4o for control with an extensively engineered prompt (Table 21, Appendix 15.3) that includes retrieval plans, query format guidelines, and multi-step examples.

M3-Agent replaces this prompt engineering with learned behavior via RL. The control model is not told how to search—it learns through trial and error, with the only training signal being whether the final answer is correct. This is significant because it demonstrates that the retrieval strategy can be optimized end-to-end for a downstream task without manual specification of retrieval heuristics. The trained policy discovers effective search patterns—first find the character ID, then query their attributes, then query specific behaviors—that the case study (Table 17) shows mirror what a human might do, but these patterns emerge from RL optimization rather than being scripted.

The quantitative evidence for this innovation's significance comes from the comparison between control-32b-prompt and control-32b-rl (Table 7): RL training improves accuracy by 10.0%, 8.0%, and 9.3% on the three benchmarks. These are large gains, and they are achieved with the identical memory content—the only difference is how the control model chooses to search and reason. This isolates the contribution of the learned retrieval policy from the contribution of memory quality.

But the deeper significance is not the performance gain itself—prompt engineering could potentially be improved to close some of this gap. Rather, it is the generalizability and scalability argument that RL-trained control enables. Prompt engineering is a manual, task-specific, and model-specific process. The prompts that work for Gemini-1.5-Pro may not work for Qwen3; the prompts optimized for one benchmark may not transfer to another. RL training, in contrast, produces a policy that is optimized for the task distribution it is trained on, and the paper's results show that this policy transfers to unseen benchmarks (VideoMME-long was not in the training data, yet control-32b-rl improves over control-32b-prompt by 9.3%). As the complexity of agent tasks grows—more entities, longer timespans, more diverse question types—the space of possible retrieval strategies expands combinatorially, making manual prompt design increasingly intractable. Learned policies scale with this complexity in a way that hand-crafted prompts cannot.

The ablation on inter-turn instruction (Table 7) provides additional insight. Removing the structured guidance between rounds—the prompts that tell the model it can query character name-to-ID mappings, that search queries should differ from previous ones, that answers should use names not IDs—causes accuracy drops of 10.5%, 5.8%, and 5.9%. This tells us that the RL-trained policy is not discovering these heuristics from scratch—it benefits substantially from structured guidance in the prompt template. The innovation is therefore not that RL eliminates the need for prompt design entirely, but rather that RL can optimize within a structured action space (the [Search]/[Answer] grammar with specific formatting rules) to find effective strategies that prompt engineering would need to manually enumerate.

The comparison between DAPO and GRPO training (Table 7: control-32b-rl vs. control-32b-grpo) shows that the choice of RL algorithm matters for this task. DAPO consistently outperforms GRPO across all benchmarks (30.7% vs. 30.0%, 48.9% vs. 47.7%, 61.8% vs. 58.7%). The paper attributes this to DAPO's per-token normalization and asymmetric clipping, but the conceptual point is broader: multi-turn agent trajectories have structure (variable length, sparse reward, environment tokens interspersed with model tokens) that standard RLHF algorithms are not optimized for, and algorithm design matters for this class of problems. This connects M3-Agent to the growing literature on RL for reasoning models (DeepSeekMath, DAPO) and suggests that agent control is a distinct RL subproblem with its own algorithmic requirements.

Innovation 4: A Targeted Evaluation Framework That Diagnoses Specific Memory-Reasoning Deficits

M3-Bench is not just a new dataset—it is a diagnostic instrument designed to decompose the complex capability of "reasoning with long-term memory" into distinguishable sub-capabilities, each tested by a specific question type. Table 1 defines five question categories (multi-evidence reasoning, multi-hop reasoning, cross-modal reasoning, person understanding, general knowledge extraction) that collectively operationalize what it means for an agent to effectively use its accumulated memory. The innovation is in the taxonomy itself and what it reveals about system failures, not merely in the presence of new videos.

Prior LVQA benchmarks (Table 2) evaluate holistic video understanding—can the system answer questions about what happened in the video? While this is useful for ranking systems, it provides limited diagnostic information about why a system fails. If a model scores 30% on Video-MME, is it because it cannot track characters across time? Because it cannot integrate visual and auditory information? Because it cannot extract abstract conclusions from observations? The aggregate score conflates these distinct capabilities.

M3-Bench's per-question-type breakdown (Table 5, right columns) enables capability-level diagnosis. The results reveal striking patterns that would be invisible in aggregate scores. On M3-Bench-robot, M3-Agent achieves 43.3% on person understanding but only 19.1% on general knowledge extraction—a 24.2 percentage point gap within the same system on the same benchmark. This tells us something specific: the system is substantially better at reasoning about character traits and relationships than at extracting generalizable world knowledge from observations. This is actionable—it directs future work toward improving the semantic memory generation for the "Contextual & General Knowledge" category (Table 10) rather than toward character understanding, which is already relatively strong.

The cross-model comparisons on per-type scores are equally diagnostic. MA-LMM achieves 39.1% on person understanding on M3-Bench-robot—competitive with M3-Agent's 43.3%—but only 22.7% on cross-modal reasoning, where M3-Agent achieves 31.2% (a larger gap). This suggests that MA-LMM's feature-based memory is adequate for tracking character-related information (perhaps because visual features preserve identity well) but insufficient for the cross-modal integration that M3-Agent's explicit equivalence mappings and multimodal graph structure enable. These are precisely the kinds of insights that a diagnostic benchmark is designed to surface.

The benchmark design also reflects a methodological contribution in how video QA datasets should be constructed for agent evaluation. The M3-Bench-robot subset (Section 3.1) uses scripted events with designated reference events for each question, meaning that annotators explicitly identify which events contain the information needed to answer which questions. This provides ground truth not just for answers but for evidence localization—we know exactly which segments of the video should be retrievable to answer each question. This property is not exploited in the current paper (the evaluation only checks final answer correctness, not evidence retrieval quality) but it enables future work on evaluating retrieval precision that current LVQA benchmarks do not support.

The M3-Bench-web subset (Section 3.2) takes a complementary approach: question-driven video collection where annotators select videos that can support the specific question types. This ensures that the benchmark contains questions that genuinely require the target capabilities rather than questions that happen to fit into categories post-hoc. The requirement that questions be "specific, objective, and have a single unambiguous answer that can be reasonably derived from clues in the video" addresses a common criticism of open-ended video QA benchmarks where answer ambiguity makes evaluation unreliable.

The 96% human-agreement rate for the GPT-4o automatic evaluator (Section 3.3) is not just a validation detail—it is an enabling contribution for scalable benchmark use. Manual evaluation of open-ended answers on 4,490 QA pairs (M3-Bench-robot + M3-Bench-web) would be prohibitively expensive for most research groups. The validated automatic evaluator makes M3-Bench practical for regular use, and the methodology (sampling 100 triples, comparing GPT-4o judgments to majority-vote human judgments) provides a template for other benchmark creators.

Innovation 5: A Training Pipeline That Enables Open-Source Models to Surpass Prompted Commercial Models on Memory-Intensive Tasks

The empirical result that M3-Agent—built on open-source foundations (Qwen2.5-Omni-7b for memorization, Qwen3-32b for control) and trained with synthetic data + RL—outperforms a system using prompted Gemini-1.5-Pro and GPT-4o (Table 5) is more than a leaderboard entry. It is a feasibility demonstration that architectural and training methodology choices can compensate for large gaps in base model capability on tasks that require structured memory.

Gemini-1.5-Pro and GPT-4o are widely considered to be among the most capable multimodal and language models available, with Gemini-1.5-Pro specifically noted for its million-token context window and strong video understanding. Yet Gemini-GPT4o-Hybrid—which uses these models within the identical M3-Agent framework (entity-centric memory, episodic + semantic memory generation, multi-turn search)—achieves only 24.0%, 41.2%, and 56.5% on the three benchmarks, compared to M3-Agent's 30.7%, 48.9%, and 61.8%. The framework alone provides benefits over Socratic baselines (Gemini-GPT4o-Hybrid substantially outperforms all Socratic variants), but the trained models provide substantial additional gains.

The significance is not that "open-source beats closed-source"—the specific models and numbers will change. Rather, it is that the components that matter most for this task are trainable behaviors (memory generation quality, retrieval strategy) rather than raw model scale or proprietary capabilities. The memorization model is a 7B parameter open-source model fine-tuned on synthetic data; it outperforms prompted Gemini-1.5-Pro in memory generation quality (Table 6: memory-gemini-prompt vs. memory-7b-sft, with the latter enabling 2.0–9.1% higher accuracy when the same control model is used). The control model is a 32B parameter open-source model fine-tuned via RL; it outperforms prompted GPT-4o in retrieval and reasoning (Table 7: control-32b-rl vs. control-32b-prompt). In both cases, training—not architecture or scale—is the differentiating factor.

This has implications for how the field should allocate resources in building multimodal agents. The default assumption in much current work is that more capable base models (achieved through larger scale, better pretraining data, or proprietary techniques) will translate to better agent performance. M3-Agent's results suggest an alternative: invest in task-specific training (synthetic data generation for memory, RL for retrieval) on moderately-sized open-source models, and you can outperform prompted versions of much more capable models. This is an instance of the broader "boring apes" phenomenon in AI—specialized training on task-relevant data can beat generalist models that are stronger in the absolute sense—but applied specifically to the multimodal agent memory domain.

The training pipeline itself contains a methodological contribution in the meta-clip algorithm (Algorithm 2, Appendix 11.2) for automatic identity equivalence annotation. The challenge of linking faces to voices in multi-person videos is well-known in the video understanding literature (He et al., 2024b [15]). Previous approaches typically require manual annotation or rely on active speaker detection, which can be unreliable. The meta-clip algorithm exploits a simple structural property—short monologue clips contain exactly one face and one voice—to generate high-confidence training labels automatically. The 95.83% accuracy on manually reviewed mappings validates this approach. This is a contribution that is independent of the M3-Agent architecture and could be used by any system that needs to establish face-voice correspondences from long videos.

The scaling analysis within the control model (Table 7, comparing 8B, 14B, and 32B variants both prompted and RL-trained) provides additional insight: RL training benefits increase with model scale. The prompted baselines show modest improvements with scale (16.4% → 18.3% → 20.7% on M3-Bench-robot for 8B → 14B → 32B), while the RL-trained variants show larger gains (24.6% → 28.2% → 30.7%). This suggests that larger models have greater capacity to learn effective retrieval strategies from the RL signal, and that the gap between prompted and RL-trained performance would likely continue to widen with further scaling. This is an empirical scaling trend for agent control that, to the authors' knowledge, has not been previously documented.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation is conducted on M3-Bench, a newly introduced long-video question answering benchmark comprising 1,020 videos and 4,490 QA pairs. It is split into two subsets: M3-Bench-robot (100 real-world, robot-perspective videos with 1,276 QA pairs, average duration 2,039.9 seconds, ~34 minutes) and M3-Bench-web (920 web-sourced YouTube videos with 3,214 QA pairs, average duration 1,630.7 seconds, ~27 minutes). Videos in M3-Bench-robot are recorded across seven everyday scenarios with scripted events annotated with reference-event-to-question mappings; M3-Bench-web videos are collected using a question-driven approach to ensure suitability for memory-based reasoning questions. An additional evaluation is conducted on VideoMME-long, a subset of the Video-MME benchmark (900 videos, 2,700 QA pairs) selected for long-duration videos. All questions are open-ended, with answers evaluated automatically using a GPT-4o-based evaluator (96% agreement with human judges on a 100-sample test set).

• Base model(s). The memorization model is Qwen2.5-Omni-7b, an open-source multimodal model supporting native video and audio input, chosen for its strong multimodal understanding capabilities. The control model is Qwen3-32b (with 8B and 14B variants tested in ablations), an open-source large language model chosen for its strong reasoning capabilities. The trained variants are memory-7b-sft (fine-tuned on synthetic memory data) and control-32b-rl (trained via DAPO reinforcement learning). The paper argues this model selection demonstrates that open-source models can surpass prompted commercial alternatives (Gemini-1.5-Pro and GPT-4o) when augmented with task-specific training.

• Metrics. The primary metric is accuracy—the fraction of questions for which the agent's final answer is judged correct by the GPT-4o automatic evaluator. The evaluator is prompted (Table 18, Appendix 15.1) to assess whether the ground-truth answer can be logically inferred from the agent's answer, not whether they match exactly: "Do not directly compare the surface forms of the agent answer and the ground truth answer. Instead, assess whether the meaning expressed by the agent answer supports or implies the ground truth answer." A stratified breakdown by question type (multi-evidence reasoning, multi-hop reasoning, cross-modal reasoning, person understanding, general knowledge extraction) is reported in Table 5 to enable capability-level diagnosis. For memorization model evaluation during training, AutoDQ [47] is used to assess description quality, and precision/recall/F1 are computed for identity equivalence against ground truth (Table 13).

• Baselines. The paper evaluates against three categories of baselines. Socratic Models adapt the Socratic Models framework [56]: a multimodal model generates descriptions of 30-second video clips, which are stored as long-term memory, and GPT-4o performs RAG-based QA. Four memory generation variants are tested: Gemini-1.5-Pro (full video input), GPT-4o (video frames at 0.5 fps + ASR transcripts), Qwen2.5-Omni-7b (full video input), and Qwen2.5-VL-7b (frames + ASR transcripts). Online video understanding methods include MovieChat [42] (sliding-window feature extraction with hybrid memory), MA-LMM [14] (online processing with 1fps feature extraction, 100-frame temporal modeling), and Flash-VStream [58] (two-stage asynchronous pipeline with 1fps compression). Agent methods prompt closed-source commercial models within the M3-Agent framework: Gemini-Agent (Gemini-1.5-Pro for both memorization and control) and Gemini-GPT4o-Hybrid (Gemini-1.5-Pro for memorization, GPT-4o for control). All agent baselines use H = 5 maximum rounds and k = 2 for search functions, matching M3-Agent's configuration.

• Generation budget / compute accounting. All agent methods and Socratic baselines use the same maximum-round limit (H = 5) and the same retrieval parameters (k = 2 for both search_node and search_clip), making the comparison fair in terms of retrieval budget. The paper does not measure or compare wall-clock inference time, FLOPs, or monetary cost across methods. For online video understanding methods, official pretrained weights and default configurations are used. For Socratic baselines, "extensive prompt engineering" is applied to "optimize performance for each setup." The paper acknowledges that difficulty estimation cost (generating memory for all clips) is not factored into efficiency comparisons.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported for the main benchmark results. The memorization model checkpoint is selected based on a held-out validation set of 200 samples using AutoDQ and equivalence metrics (Table 13). The GPT-4o evaluator's reliability is validated via human agreement on 100 randomly sampled triples (question, reference answer, generated answer), with three authors independently evaluating and GPT-4o compared against majority-vote human judgments, achieving 96% agreement. For training data, videos with corresponding QA pairs are collected from an in-house dataset; no explicit train/validation/test split contamination analysis between the in-house training data and M3-Bench is reported, though M3-Bench is independently collected.

Main Quantitative Results

The central quantitative result is that M3-Agent (implemented with memory-7b-sft for memorization and control-32b-rl trained via DAPO for control) outperforms all baselines across all three benchmarks (Table 5). On M3-Bench-robot, M3-Agent achieves 30.7%, a 6.3% absolute improvement over the strongest baseline MA-LMM (24.4%) and a 6.7% improvement over Gemini-GPT4o-Hybrid (24.0%). On M3-Bench-web, M3-Agent achieves 48.9%, surpassing Gemini-GPT4o-Hybrid (41.2%) by 7.7%. On VideoMME-long, M3-Agent reaches 61.8%, exceeding Gemini-GPT4o-Hybrid (56.5%) by 5.3%.

Comparison Across Baseline Categories

Socratic Models are severely bottlenecked by memory quality. On M3-Bench-robot, the strongest Socratic variant (GPT-4o) achieves only 8.5%, while the weakest M3-Agent configuration (control-32b-prompt with memory-7b-sft in Table 7) reaches 20.7%. The gap is even larger on M3-Bench-web: Socratic GPT-4o achieves 28.7% versus M3-Agent's 48.9%. These results demonstrate that generating unstructured episodic descriptions—even with powerful commercial models and extensive prompt engineering—is fundamentally insufficient for questions requiring memory-based reasoning. The Socratic baselines can describe what happened in each clip but cannot maintain the entity-centric consistency or extract the abstract semantic knowledge needed to aggregate evidence across clips.

Online video understanding methods show domain-specific strengths but are capped by consistency limitations. MA-LMM achieves 24.4% on M3-Bench-robot, the strongest result among all non-M3-Agent methods. However, it drops to 24.3% on M3-Bench-web and 17.3% on VideoMME-long. Flash-VStream follows a similar pattern (19.4%, 23.6%, 25.0%). MovieChat performs substantially worse (11.2%, 12.6%, 19.4%). Notably, MA-LMM's strong robot-benchmark performance is driven by person understanding (39.1%, competitive with M3-Agent's 43.3%), but its cross-modal reasoning is substantially weaker (22.7% vs. 31.2% for M3-Agent). This pattern supports the paper's claim that feature-based memory can track visual identity reasonably well (helping person understanding) but struggles with the multimodal integration needed for cross-modal reasoning—precisely the gap that M3-Agent's explicit equivalence mappings address.

Prompted commercial agents benefit from the M3-Agent framework but fall short of trained models. Gemini-Agent achieves 16.9%, 34.1%, and 55.1% on the three benchmarks. Gemini-GPT4o-Hybrid improves to 24.0%, 41.2%, and 56.5%—substantially better than Socratic baselines, confirming that the entity-centric multimodal graph, dual episodic-semantic memory, and multi-turn retrieval framework provides value even without task-specific training. However, the gap to M3-Agent (30.7%, 48.9%, 61.8%) demonstrates that prompted commercial models, despite their scale and general capability, cannot match trained models in memory generation quality or retrieval strategy optimization for this specific task distribution.

Per-Question-Type Analysis

Table 5 provides accuracy breakdowns by question type, revealing systematic capability patterns:

Person understanding is the strongest capability. M3-Agent achieves 43.3% on M3-Bench-robot and 59.3% on M3-Bench-web for person understanding. This is consistent across systems: MA-LMM, Flash-VStream, and Gemini-GPT4o-Hybrid all show their highest per-type scores in this category. The entity-centric memory design and explicit identity equivalence appear particularly effective for tracking and reasoning about character traits across long timespans.

General knowledge extraction is the weakest capability. M3-Agent achieves only 19.1% on M3-Bench-robot for general knowledge extraction—less than half its person understanding score. On M3-Bench-web, the gap is smaller but still present (53.9% vs. 59.3% for person understanding). This pattern is consistent across systems: all methods struggle most with extracting abstract, generalizable knowledge from specific observations ("Which shelf is suitable for storing vegetables?" from observing refrigerator organization), suggesting this is a fundamental challenge not fully addressed by semantic memory.

Cross-modal reasoning shows the largest relative advantage for M3-Agent over feature-based methods. On M3-Bench-robot, M3-Agent achieves 31.2% cross-modal reasoning versus MA-LMM's 22.7%—an 8.5 percentage point gap. On M3-Bench-web, M3-Agent achieves 44.3% versus Gemini-GPT4o-Hybrid's 37.6% (6.7 point gap). This supports the claim that explicit cross-modal entity linking (face-to-voice equivalence in the memory graph) provides benefits beyond what feature-based or description-based memory can achieve for questions requiring integration of visual and auditory information.

Multi-evidence and multi-hop reasoning show consistent improvement over baselines but remain substantially below person understanding. On M3-Bench-web, M3-Agent achieves 45.9% multi-evidence and 28.4% multi-hop versus Gemini-GPT4o-Hybrid's 35.9% and 26.2%. The gap is particularly large for multi-evidence reasoning (10.0 percentage points), consistent with the claim that multi-turn RL-trained retrieval is more effective than prompted retrieval for aggregating evidence distributed across video segments.

Scaling of Control Model Performance with RL Training

Table 7 presents results for control models of varying sizes and training configurations, with memorization fixed to memory-7b-sft. Several patterns emerge:

RL training provides substantial gains that scale with model size. Prompted baselines: 16.4% (8B), 18.3% (14B), 20.7% (32B) on M3-Bench-robot. DAPO-trained: 24.6% (8B), 28.2% (14B), 30.7% (32B). The absolute improvement from RL training grows from 8.2 points (8B) to 10.0 points (32B), and the relative ranking changes—the best prompted model (32B at 20.7%) underperforms the worst RL-trained model (8B at 24.6%).

DAPO consistently outperforms GRPO. Across all three benchmarks, control-32b-rl (DAPO-trained: 30.7%, 48.9%, 61.8%) exceeds control-32b-grpo (30.0%, 47.7%, 58.7%). The gap is modest but consistent (0.7, 1.2, and 3.1 percentage points respectively), with the largest advantage on the most challenging benchmark (VideoMME-long). This empirically validates DAPO's design choices (per-token normalization, asymmetric clipping) for this multi-turn retrieval task.

Both inter-turn instruction and reasoning are essential. Removing inter-turn instruction from control-32b-prompt causes accuracy drops of 7.9 (M3-Bench-robot), 8.6 (M3-Bench-web), and 4.2 (VideoMME-long) points. Removing it from control-32b-rl causes even larger drops: 10.5, 5.8, and 5.9 points. Removing the reasoning mode from control-32b-rl causes accuracy decreases of 11.7, 8.8, and 9.5 points. These results indicate that (1) structured guidance between retrieval rounds provides information the model cannot discover purely from RL training, and (2) explicit reasoning steps (the thinking blocks visible in the case study, Table 17) are critical for effective multi-turn search, not merely cosmetic.

Impact of Memorization Quality on Final Performance

Table 6 isolates the effect of memorization quality by fixing the control model to control-32b-rl and varying the memorization method:

Fine-tuned memorization outperforms prompted Gemini-1.5-Pro. Using memory-7b-sft (the fine-tuned model) achieves 30.7%, 48.9%, and 61.8% on the three benchmarks, compared to 28.7%, 46.3%, and 52.7% using memory-gemini-prompt. The gap is largest on VideoMME-long (9.1 points), suggesting the fine-tuned model's advantages are most pronounced on diverse web videos rather than the more controlled robot scenarios.

Imitation learning is critical for effective memorization. memory-7b-prompt (the base Qwen2.5-Omni-7b without fine-tuning, using only prompting to generate memories) achieves only 25.3%, 39.9%, and 50.8%—far below the fine-tuned model and even below memory-gemini-prompt. This demonstrates that the base open-source model, despite its multimodal capabilities, cannot match prompted Gemini-1.5-Pro for memory generation without task-specific fine-tuning.

Identity equivalence and semantic memory are both essential but contribute differently. Removing identity equivalence (memory-7b-sft w/o equivalence) drops accuracy to 19.5%, 39.7%, and 52.1%—a large absolute decrease, particularly on M3-Bench-robot (11.2 points). Removing all semantic memory while keeping episodic memory (memory-7b-sft w/o semantic memory) causes even larger drops to 13.6%, 29.7%, and 48.7%. The incremental contribution of identity equivalence beyond having semantic memory without consistent entity linking is approximately 5.9 points on M3-Bench-robot (19.5% vs. 13.6%) and 10.0 points on M3-Bench-web (39.7% vs. 29.7%), suggesting that the entity-centric structure provides greater benefit in more diverse video domains where identity tracking is harder.

Ablation Studies and Robustness Checks

Replacing fine-tuned memory with Gemini-generated memory: Fixing the control model to control-32b-rl and swapping memory-7b-sft for memory-gemini-prompt reduces accuracy by 2.0% (M3-Bench-robot), 2.6% (M3-Bench-web), and 9.1% (VideoMME-long), as shown in Table 6. The much larger drop on VideoMME-long suggests that the fine-tuned memorization model's advantages in detail richness and consistency (visible in the case study comparisons, Tables 15–16, Appendix 14) matter most for diverse, less controlled video content.

Removing character identity equivalence from semantic memory: As shown in Table 6, this ablation (memory-7b-sft w/o equivalence) produces accuracy drops of 11.2% (M3-Bench-robot), 9.2% (M3-Bench-web), and 9.7% (VideoMME-long). This confirms that consistent cross-modal identity tracking—linking faces to voices within a unified character representation—is a major contributor to M3-Agent's performance, particularly for person understanding and cross-modal reasoning questions.

Removing all semantic memory (episodic-only memory): This is the most severe ablation in Table 6, causing decreases of 17.1% (M3-Bench-robot), 19.2% (M3-Bench-web), and 13.1% (VideoMME-long) relative to the full M3-Agent. The magnitude of these drops is the empirical basis for the paper's central claim that semantic memory—abstracted, inferential knowledge—is a necessary component of effective long-term memory for multimodal agents, not merely a helpful addition. Even with high-quality episodic descriptions and multi-turn RL-trained retrieval, the system cannot reliably answer questions requiring inference about character traits, relationships, or general knowledge without the pre-computed abstractions that semantic memory provides.

GRPO vs. DAPO training algorithm: Table 7 compares control-32b-rl (trained with DAPO) against control-32b-grpo (trained with GRPO under otherwise identical conditions, hyperparameters in Appendix 13.2). DAPO achieves 30.7% vs. 30.0% on M3-Bench-robot, 48.9% vs. 47.7% on M3-Bench-web, and 61.8% vs. 58.7% on VideoMME-long. The consistent advantage across all benchmarks, with the largest gap on the most diverse benchmark, validates DAPO's design modifications (per-token loss normalization, asymmetric clipping) for this multi-turn agent task.

Model size scaling for RL-trained control: Table 7 shows that both prompted and RL-trained control models improve with scale (8B → 14B → 32B). For prompted models: 16.4% → 18.3% → 20.7% on M3-Bench-robot. For RL-trained models: 24.6% → 28.2% → 30.7%. The RL gains increase with scale: +8.2 points at 8B, +9.9 at 14B, +10.0 at 32B. This monotonic trend across three model sizes provides evidence that RL training for multi-turn retrieval benefits from larger model capacity, consistent with the hypothesis that larger models can learn more sophisticated retrieval strategies from the binary reward signal.

Removing inter-turn instruction: Table 7 shows the effect of stripping the structured guidance provided between search rounds (the instruction prompt that tells the model it can query character name-to-ID mappings, that search queries should differ from previous ones, that answers should use names not IDs). For the prompted control model, removing this guidance drops accuracy from 20.7% to 12.8% on M3-Bench-robot (a 7.9 point decrease). For the RL-trained model, the drop is even larger: 30.7% to 20.2% (a 10.5 point decrease). The fact that the RL model suffers more from removing this guidance—despite having been trained on data that included it—suggests that the policy has learned to rely on these structural cues and cannot easily recover the same retrieval patterns without them. On M3-Bench-web and VideoMME-long, the drops are smaller for the RL model (5.8 and 5.9 points), suggesting some ability to adapt retrieval strategy without explicit guidance in more diverse video domains.

Removing reasoning mode: This ablation (control-32b-rl w/o reasoning in Table 7) removes the thinking blocks from the control process—the model still generates actions and content, but without the intermediate reasoning step visible in the case study (Table 17). Accuracy drops of 11.7% (M3-Bench-robot), 8.8% (M3-Bench-web), and 9.5% (VideoMME-long) demonstrate that explicit reasoning is not cosmetic: it is functionally essential for effective multi-turn search, likely because it enables the model to analyze what information is currently missing and formulate targeted queries. Without reasoning, the model must generate queries through a more opaque, end-to-end process that apparently degrades retrieval quality.

GPT-4o evaluator reliability: The paper validates its automatic evaluation by constructing a test set of 100 randomly sampled triples (question, reference answer, generated answer) from M3-Agent and various baselines. Three authors independently evaluate correctness, and GPT-4o's judgments are compared with the majority vote of human annotations. The reported 96% agreement provides reasonable confidence in the automatic evaluation, though the sample size (100 triples) is modest relative to the 4,490 total QA pairs. No breakdown by question type, method, or benchmark subset is provided for the agreement analysis, leaving open the possibility that evaluator reliability varies systematically across these dimensions.

Memorization model checkpoint selection (Table 13, Appendix 12): The memorization model was trained for up to 5 epochs, with checkpoints evaluated on a held-out validation set of 200 samples. AutoDQ-F1 peaks at epoch 3 (0.627), while Equivalence-F1 also peaks at epoch 3 (0.846). Both metrics show decline at epochs 4 and 5, suggesting overfitting. The selected checkpoint (epoch 3) substantially outperforms both memory-gemini-prompt (AutoDQ-F1 0.606, Equivalence-F1 0.595) and memory-7b-prompt (AutoDQ-F1 0.414, Equivalence-F1 0.145), confirming the necessity of fine-tuning and the quality ceiling of purely prompted approaches.

Critical Assessment

Claim 1: M3-Agent outperforms the strongest baselines by 6.7%, 7.7%, and 5.3% on the three benchmarks. This claim is directly supported by Table 5, but requires careful interpretation. The "strongest baseline" differs per benchmark: MA-LMM on M3-Bench-robot (24.4%), Gemini-GPT4o-Hybrid on M3-Bench-web (41.2%) and VideoMME-long (56.5%). The headline numbers are accurate, but the comparison is not against a single unified baseline—different method families dominate different benchmarks, which itself is an interesting finding about the complementary strengths of feature-based vs. language-based memory systems.

A limitation is that M3-Agent and the agent baselines use the same underlying framework (entity-centric memory, episodic+semantic memory, multi-turn search). This makes the comparison between M3-Agent and Gemini-GPT4o-Hybrid informative about the value of training vs. prompting, but it does not test whether alternative agent architectures (different memory structures, different retrieval paradigms) could achieve similar or better performance. The comparison against Socratic Models and online video methods provides some cross-architecture evidence, but these baselines operate under different constraints (single-turn RAG, compressed feature storage) and the performance gaps do not isolate the contribution of any single design choice.

Claim 2: Removing semantic memory causes accuracy drops of 17.1%, 19.2%, and 13.1%. This is supported by Table 6 (memory-7b-sft w/o semantic memory vs. full M3-Agent), but the ablation conflates several distinct functions of semantic memory—identity equivalence, character-level attributes, interpersonal relationships, video-level plot understanding, and contextual/general knowledge. The paper does not report separate ablations for each semantic memory category (beyond the equivalence-only removal), making it unclear which semantic memory sub-types drive the performance improvements. Are character attributes sufficient on their own, or is the general knowledge component essential? This granularity would strengthen the diagnostic value of the analysis and guide future work on targeted memory improvements.

Additionally, the semantic memory removal is tested with only one control model (control-32b-rl). It is possible that a different control model architecture—perhaps one with stronger reasoning capabilities—could partially compensate for missing semantic memory by performing inference at query time. The ablation demonstrates that semantic memory is necessary given the current control model, not necessarily that it is architecturally indispensable.

Claim 3: RL training improves accuracy by 10.0%, 8.0%, and 9.3%. Table 7 supports this claim (control-32b-prompt vs. control-32b-rl), but there is a confound: the prompted and RL-trained models differ not only in training but potentially in the effective use of the action space. The prompted model follows a fixed prompt template with specific instructions; the RL-trained model has learned through trial and error which retrieval patterns work. The paper does not test whether more extensive prompt engineering (e.g., few-shot examples of successful multi-turn retrieval trajectories for the specific task distribution) could close some of this gap. The 10.0% improvement should therefore be interpreted as the gain from RL training relative to the specific prompt engineering used, not necessarily relative to the best possible prompted approach.

Lack of statistical significance testing. The paper reports point estimates for all benchmark results without confidence intervals or statistical tests. With 100 videos and 1,276 QA pairs in M3-Bench-robot, a 1–2 percentage point difference between methods could easily fall within sampling error. The larger M3-Bench-web (920 videos, 3,214 QA pairs) provides more statistical power, but the lack of any variance reporting makes it difficult to assess whether the observed differences between adjacent methods (e.g., M3-Agent at 30.7% vs. MA-LMM at 24.4% on M3-Bench-robot) are reliably distinguishable from noise or from the specific train/test split.

Single memorization model architecture. All comparisons in Table 6 vary the memorization quality but keep the control model fixed at control-32b-rl. A missing experiment is testing how control model quality interacts with memorization quality—specifically, does a stronger control model benefit more or less from better memory? The scaling results in Table 7 hint at an interaction (larger control models show larger RL gains), but a factorial experiment crossing memorization quality (gemini-prompt, 7b-prompt, 7b-sft) with control model scale (8B, 14B, 32B, prompted and RL) would directly test whether better memory and better control are complements or substitutes.

M3-Bench-robot difficulty is not fully characterized. The paper notes that M3-Bench-robot presents a "significant challenge" with M3-Agent achieving only 30.7%. However, unlike the main paper, there is no difficulty-bin analysis showing which types of questions or video characteristics drive this difficulty. The hard case analysis (Section 5.5) identifies fine-grained detail reasoning and spatial reasoning as challenge categories, but this is qualitative and based on manual error inspection rather than systematic difficulty decomposition. A breakdown of accuracy by script scenario (living room, kitchen, bedroom, etc.), by video duration, by number of entities, or by the temporal distance between reference events and questions would substantially enrich the diagnostic value of the benchmark.

Missing ablation: the contribution of the graph structure vs. flat memory. The paper advocates for an entity-centric graph structure with edges connecting related nodes. However, no ablation directly tests the graph structure's contribution. One could imagine a flat memory system that stores episodic and semantic text nodes with multimodal embeddings but without explicit entity-centric edges, relying solely on embedding similarity for all retrieval. The memory-7b-sft w/o equivalence ablation partially addresses this by removing equivalence edges, but other edge types (e.g., linking all semantic memories about a character to their face node) could still provide retrieval benefits. A direct flat-memory baseline would clarify whether the graph structure provides benefits beyond what multimodal embeddings alone can achieve.

Training data domain overlap with evaluation. The training dataset comprises 500 long videos and 2,736 QA pairs from the authors' in-house collection, annotated to the "same annotation standards used in the M3-Bench-web dataset" (Section 4.4). The paper does not analyze domain overlap between training and evaluation videos, nor does it report whether any YouTube channels or video categories in M3-Bench-web overlap with the training data. This is a standard concern for learned systems evaluated on web-sourced benchmarks, and the lack of transparency about potential leakage limits confidence in the transfer results (particularly for VideoMME-long, which is independently collected but may share content distribution with the training data).

The "4x efficiency improvement" claim from the introduction is not evaluated. The paper states in the Introduction that M3-Agent "achieves comparable performance with significantly lower computational cost" but does not quantify this claim anywhere in the experimental section. The only efficiency-relevant metric is the H = 5 maximum round limit applied uniformly across all agent methods, which makes them equally expensive in terms of search depth. No experiments sweep H to find the compute-optimal operating point, and no FLOPs or latency comparisons are provided between M3-Agent and the prompted agent baselines (which use more expensive commercial models but may require fewer reasoning steps). This claim appears to be aspirational rather than empirically demonstrated.

The control model sees question timestamps in M3-Bench-robot but not in M3-Bench-web. In M3-Bench-robot, questions are asked at specific timestamps during the video—"the timestamp must precede the robot's corresponding response or action to avoid inadvertently revealing the answer" (Section 3.1). In M3-Bench-web, "all question timestamps are set to the end of the video." This asymmetry means that the robot benchmark tests the agent's ability to answer questions with partial video observation (not having seen the entire video yet), while the web benchmark tests retrospective reasoning over complete videos. The paper does not analyze how this temporal framing difference affects performance or whether the RL training (which presumably used end-of-video questions, since the training data follows M3-Bench-web standards) transfers effectively to mid-video questioning. The potential for distribution shift between training and M3-Bench-robot evaluation is not discussed.

The GPT-4o evaluator's limitations. While the 96% human agreement on 100 samples is reassuring, the evaluator's behavior on specific question types is not analyzed. The prompt instructs GPT-4o to determine "whether the ground truth answer can be logically inferred from the agent's answer" rather than checking exact match. This semantic-equivalence approach is more generous than exact matching but introduces its own biases: GPT-4o may accept vague or partially correct answers that a human would reject, or may penalize correct but unconventionally phrased answers. The agreement analysis aggregates across all question types and methods; if the evaluator is systematically more lenient or strict for certain question categories (e.g., person understanding vs. general knowledge extraction), the per-type scores in Table 5 would reflect evaluator bias rather than genuine capability differences. A per-type agreement breakdown would address this concern.

The VideoMME-long selection process. The paper evaluates on "VideoMME-long" but provides no details about which subset of Video-MME's 900 videos is used or what "long" means in this context. Video-MME contains videos of varying duration; clarifying the selection criteria and the number of videos in this subset is important for reproducibility and for interpreting the 61.8% M3-Agent score relative to the full Video-MME benchmark's difficulty distribution.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted for in the Headline Efficiency Numbers

The assumption or constraint. The memorization process—generating episodic and semantic memory for every 30-second clip of every video—is the prerequisite for the control process to function. For M3-Bench-robot videos averaging 2,040 seconds (~34 minutes), this means processing approximately 68 clips per video through the memorization model, plus running facial recognition, speaker identification, and memory graph insertion for each clip. For M3-Bench-web videos averaging 1,631 seconds (~27 minutes), approximately 54 clips must be processed. The paper acknowledges this cost indirectly but does not account for it in any performance comparison:

"This also introduces new challenges, particularly in maintaining consistency of long-term memory" (Section 2.1), and Section 4.2 notes the system processes video in a "clip-by-clip manner."

Crucially, the memorization cost is incurred once per video, while the control process (the measured task) operates over this pre-built memory. Section 3.3 notes that "our experiments do not account for this cost largely for simplicity."

The consequence. The reported accuracy improvements over Socratic baselines and prompted commercial agents (6.7%, 7.7%, 5.3% on the three benchmarks) are measured assuming memory is already built. In a deployment scenario where an agent encounters a new video and must answer questions about it, the total cost would be memorization cost + control cost. The memorization cost—running the fine-tuned Qwen2.5-Omni-7b on every clip, plus perception tools, plus embedding computation and database insertion—likely dominates the control cost (up to 5 rounds of text-based retrieval and reasoning with Qwen3). The Socratic baselines also incur description generation costs but use the same models per-clip; the key unmeasured difference is whether M3-Agent's richer memory generation saves sufficient control computation to justify its higher memorization overhead.

This matters for a practitioner choosing between approaches. If the task involves processing a large catalog of videos once and answering many questions per video (e.g., video archive search), the upfront memorization cost amortizes well. If the task involves answering one or two questions about a newly encountered video (e.g., a household robot answering an immediate query about the last hour of interaction), the total latency—memorization followed by control—may be substantially higher than a method that processes the raw video directly (like MA-LMM or MovieChat, which perform feature extraction and QA in a single pass without generating separate text memory).

What evidence exists in the paper. The paper provides no measurement of memorization wall-clock time, FLOPs, or monetary cost. The memorization model is a 7B-parameter multimodal model processing 30-second clips; the control model is a 32B-parameter text model generating up to 5 rounds of reasoning and retrieval. No comparison of total inference FLOPs between M3-Agent and any baseline is reported. The paper states in the Introduction that M3-Agent "achieves comparable performance with significantly lower computational cost" but this claim is not quantified in the experimental sections.

Mitigation status. The paper does not address this. Section 8 (Conclusion and Future Work) does not mention efficiency as a concern or future direction. The choice to train the memorization model (Qwen2.5-Omni-7b) at a relatively small scale suggests awareness of efficiency constraints, but no analysis is provided. A practitioner would need to run their own profiling to determine whether M3-Agent's accuracy advantages justify its computational overhead for their specific deployment scenario.

---

Hard Problems Remain Effectively Unsolved

The assumption or constraint. The paper's architecture assumes that questions can be answered by information distributed across the video that the memorization model can capture and the control model can retrieve and reason over. However, Section 5.5 identifies two categories of questions where this assumption fails systematically: fine-grained detail reasoning and spatial reasoning.

"Questions like 'Who wants to eat the ham sausage?' or 'Which coat rack should Emma's hat be placed, taller one or shorter one?' require the agent to extract precise information from its observations. However, retaining all such details in memory is impractical and may cause cognitive overload" (Section 5.5, Hard Case in M3-Bench).

"Another category of hard cases is related to spatial reasoning... Examples include: 'Where can the robot get the snacks?' and 'Is Leo's water cup currently on the second or third shelf from the top of the rack?' Since verbal memory is generally less effective than visual memory for retaining spatial information, the long-term memory should be designed to incorporate richer visual content, e.g., snapshots, to better support spatial reasoning" (Section 5.5, Hard Case in M3-Bench).

The consequence. M3-Agent achieves only 30.7% on M3-Bench-robot and 48.9% on M3-Bench-web (Table 5). On M3-Bench-robot, this means it fails on approximately 69% of questions—a substantial failure rate for a system intended to guide robot behavior in household scenarios. The hard cases identified suggest these failures are not random noise but systematic: the system fundamentally cannot answer questions requiring retention of fine-grained details (which the text-based memory format cannot capture densely enough) or spatial reasoning (which text-based memory inherently struggles to represent). Moreover, the per-question-type breakdown in Table 5 shows general knowledge extraction at only 19.1% on M3-Bench-robot—less than half the person understanding score (43.3%)—indicating a specific capability gap in abstracting generalizable knowledge from specific observations.

The spatial reasoning limitation is architecturally fundamental: M3-Agent converts all visual information into text descriptions. Text is a lossy representation for spatial relationships (relative positions, distances, 3D layouts). The paper's suggestion to add "snapshots" (Section 5.5) acknowledges this but would require a different retrieval mechanism—current search relies on text embeddings and face/voice similarity, not visual-spatial queries. Fixing spatial reasoning would require extending the memory graph to include visual-spatial nodes with different query modalities, which is not a minor modification.

What evidence exists in the paper. Table 5 provides the aggregate accuracy. The hard case analysis in Section 5.5 is qualitative, based on manual error inspection, and does not quantify what fraction of errors fall into the fine-grained detail vs. spatial reasoning categories. No per-question-type breakdown distinguishes these sub-categories—general knowledge extraction (19.1%) is the closest proxy for one class of hard questions, but fine-grained detail and spatial reasoning are not separately evaluated. The human accuracy on M3-Bench-robot is reported as 90.7% (Section 8.3), with counting problems identified as the most common human error, but no breakdown of machine errors is provided.

Mitigation status. The paper identifies these as limitations in Section 5.5 (Hard Case in M3-Bench) and Section 6 (Conclusion and Future Work) suggests "developing richer yet more efficient visual memory" as a future direction, but proposes no concrete solution. The current architecture offers no mechanism for selectively attending to fine-grained details or for representing spatial information beyond textual descriptions.

---

The Revision Model Is Absent: Memory Is Write-Only with No Correction Mechanism

The assumption or constraint. The memorization process generates memory once per clip and inserts it into the memory graph. The weight-based voting mechanism (Section 4.1) resolves conflicts between already-stored memories:

"Conflicting information may be introduced during construction. To resolve this, M3-Agent applies a weight-based voting mechanism during inference: frequently activated entries accumulate higher weights and override conflicting entries with lower weights" (Section 4.1).

However, this mechanism only operates on information that has been explicitly stored. It cannot correct errors of omission (critical details the memorization model failed to notice or describe), errors of interpretation (a character's emotion was misread), or errors that appear uniformly (the same mistake made consistently across all clips, giving it high weight). More fundamentally, there is no mechanism for the control process to feed back corrections: if the control model discovers during reasoning that a memory entry appears to be wrong (e.g., two entries contradict each other and the retrieved evidence supports one over the other), it cannot update the memory graph to reflect this resolution. Memory is write-only from the perspective of control.

The consequence. Memory errors accumulate over time. The paper validates the memorization model's synthetic data at 95.5% accuracy (Appendix 11.4), but this is measured on the synthetic training data distribution. On real, uncurated video streams—especially long-duration, multi-person, noisy environments—the memorization model (a 7B-parameter fine-tuned model) will make errors. The weight-based voting mechanism provides robustness against occasional random errors (e.g., a single incorrect face-voice mapping) when correct mappings dominate in frequency. But it provides no robustness against errors that are systematic (e.g., the memorization model consistently misidentifies sarcasm as sincerity) or omission-based (critical evidence for a future question was never stored because the memorization model deemed it uninteresting). These error types compound over time: as memory grows, the probability that some critical fact is missing or misrepresented increases, and the control model has no way to recover from this—it can only search what was stored.

This is particularly concerning for deployment scenarios where memory accumulates over days or weeks. The paper's experiments use single-video memory (each video's memory is built once and queried). In a lifelong agent scenario (the stated motivation in Section 1—"a household robot can autonomously carry out household tasks without your explicit instructions; it must have learned the operational rules of your home through daily experiences"), memory errors from early interactions would persist and potentially corrupt later reasoning, with no corrective feedback loop.

What evidence exists in the paper. The paper does not measure memory error accumulation over time or the robustness of downstream QA to memory errors. The ablation replacing memory-7b-sft with memory-gemini-prompt (Table 6) provides an upper bound on the impact of memory quality differences (2.0–9.1% accuracy drops depending on benchmark), but this compares two different memory generation processes, not the effect of errors within a single process. The weight-based voting mechanism is described architecturally (Section 4.1) but never empirically validated—no experiment demonstrates that accumulated weights actually resolve conflicts correctly, or measures how often conflicts occur or how voting accuracy improves over time.

Mitigation status. Not addressed. The paper's future work (Section 6) mentions "enhancing attention mechanisms for semantic memory formation" but does not discuss memory correction, revision, or feedback from control to memorization. The absence of a correction mechanism is an architectural gap between the stated goal (lifelong learning from daily experiences) and the current implementation (single-pass memory generation with simple frequency-based conflict resolution).

---

Generalization Beyond the Training Distribution Is Unverified

The assumption or constraint. The paper evaluates M3-Agent on three benchmarks: M3-Bench-robot (100 robot-perspective household videos), M3-Bench-web (920 diverse YouTube videos), and VideoMME-long (a subset of Video-MME). The training data consists of 500 in-house long videos with QA pairs, annotated "to the same annotation standards used in the M3-Bench-web dataset" (Section 4.4). The memorization model is a fine-tuned Qwen2.5-Omni-7b; the control model is an RL-trained Qwen3 variant.

The generalization claim is implicit in the evaluation on three distinct benchmarks and in the comparison against prompted commercial models (Gemini-1.5-Pro and GPT-4o) that are not fine-tuned on the specific task distribution. However, the paper does not specify whether the 500 training videos overlap in domain, source, or content type with M3-Bench-web (both YouTube-derived) or VideoMME-long (also web-sourced). There is no analysis of domain transfer, no hold-out of specific video categories for out-of-distribution testing, and no evaluation on non-video tasks that would test whether the memory architecture generalizes beyond LVQA.

The consequence. A practitioner cannot determine from the reported results whether M3-Agent's advantages are specific to the (implicit) training distribution or transfer to genuinely novel domains. Several aspects of the setup raise concerns about distributional overfitting:

• Training data leakage risk. M3-Bench-web is constructed from YouTube videos. If the 500 in-house training videos are also YouTube-sourced and share content categories, channels, or even individual videos with M3-Bench-web, the control model may have been trained on questions and memory from very similar content. The paper does not describe the training data source beyond "our in-house video dataset, which we have permissions for model training" (Section 4.4), making leakage assessment impossible.

• Single modality family (video QA). All evaluation is on long-video question answering. The paper's stated motivation includes robot interaction and lifelong agent operation, which might involve non-QA tasks (e.g., action execution, dialogue generation, proactive assistance). The architecture's applicability to these tasks is unverified.

• Single base model family (Qwen). While the comparison against Gemini-1.5-Pro and GPT-4o provides some cross-model evidence, the trained components (memory-7b-sft, control-32b-rl) use a specific model family. It is unknown whether the same architecture and training recipe would yield similar gains with a different base multimodal model (e.g., LLaVA, InternVL) or a different LLM (e.g., Llama, DeepSeek).

• No evaluation on short videos or single-image tasks. The architecture is designed for long videos, but a practitioner might wonder whether the memory framework provides any benefit for shorter content or whether the overhead of memory graph construction is strictly unnecessary below some duration threshold. No duration-ablation is provided.

What evidence exists in the paper. The paper evaluates on three benchmarks with different characteristics (robot-perspective controlled vs. diverse web), which is a reasonable breadth. The VideoMME-long results (61.8% M3-Agent vs. 56.5% Gemini-GPT4o-Hybrid) provide the strongest generalization evidence since VideoMME is independently collected and not used for training. However, the VideoMME-long selection criteria are unspecified (Section 5.3: "Video-MME-Long" with no details on which subset of the 900 Video-MME videos is used or what duration threshold defines "long"). The human agreement validation for the automatic evaluator (96% on 100 triples, Section 3.3) uses answers from M3-Agent and various baselines but does not analyze whether evaluator reliability varies across benchmarks or question types.

The ablation on training data distribution is absent entirely: no experiment tests M3-Agent trained on M3-Bench-web-style data and evaluated only on M3-Bench-robot (the strongest domain-shift test available), or vice versa. The control model's RL training (Section 4.4) uses questions and memory from the 500 in-house videos; the relationship between these videos and the evaluation benchmarks is not characterized.

Mitigation status. The paper does not acknowledge this as a limitation. The release of model weights, code, and data is noted but does not substitute for generalization analysis—it enables others to perform such analysis but does not provide it. Future work could include cross-domain evaluation (training on one video category, testing on held-out categories) and testing on non-LVQA agent tasks to establish the architecture's generality.

---

Single-Turn Memory Generation Cannot Recover from Information Unavailable in a Single Clip

The assumption or constraint. The memorization model processes video in 30-second clips and generates episodic and semantic memory for each clip independently—or more precisely, conditioned only on the current clip's video, audio, face IDs, and voice IDs:

"During memorization, M3-Agent processes the incoming video stream in clip-by-clip manner, generating two types of memory: episodic memory... and semantic memory..." (Section 4.2)

The semantic memory generation is particularly affected by this clip-level granularity. Character-level attributes, interpersonal relationships, and general knowledge often require evidence distributed across multiple clips. For example, inferring "Lucas is skilled at cooking" from observing him across several cooking scenes is a cross-clip inference. However, the memorization model sees only one 30-second window at a time. It can make inferences within that window (e.g., "in this clip, Lucas appears competent at chopping vegetables"), but it cannot aggregate across windows.

The control model, during multi-turn reasoning, can retrieve multiple clips and combine their evidence. But this means inference across clips is deferred entirely to query time. The semantic memory entries stored at encoding time are only single-clip inferences; the cross-clip synthesis must be performed by the control model on demand.

The consequence. This architectural choice limits the quality of semantic memory—specifically, the "Character-Level Attributes," "Interpersonal Relationships," and "Contextual & General Knowledge" categories (Table 10). The memorization model might generate inconsistent or fragmented semantic memories across clips: in clip 10, "<face_1> appears confident and assertive"; in clip 25, "<face_1> appears nervous and hesitant." Without cross-clip context, the model cannot determine whether this represents a genuine character trait (the person is confident in some situations, nervous in others), a change over time, or an error in one of the inferences. The weight-based voting mechanism partially addresses this for equivalence mappings (where accumulated correct mappings dominate) but not for graded or context-dependent attributes where both observations may be valid.

The consequence is visible in the per-question-type performance: person understanding (43.3% on M3-Bench-robot, Table 5) is M3-Agent's strongest category, while general knowledge extraction (19.1%) is its weakest. Person understanding benefits from entity-centric tracking (consistent IDs) and per-clip behavioral observations, even if those observations are individually generated. General knowledge extraction—e.g., "Which shelf is suitable for storing vegetables?" from observing multiple refrigerator interactions—requires synthesizing observations across clips into an abstract rule, which single-clip memorization cannot do and which the control model, based on the ablation evidence, struggles to perform at query time.

What evidence exists in the paper. The semantic memory ablation (Table 6, memory-7b-sft w/o semantic memory) demonstrates the value of semantic memory in aggregate, but does not distinguish between single-clip inferences (which the current architecture generates) and cross-clip inferences (which it cannot). The case studies (Tables 15–16, Appendix 14) show examples of generated semantic memories that are rich but clip-local. No experiment compares clip-level vs. video-level semantic memory generation (e.g., by providing the memorization model with access to previously generated semantic memories as context, enabling it to build on prior inferences).

Mitigation status. The paper does not discuss this limitation. Section 4.2 describes the memorization process as clip-by-clip without addressing the inference aggregation problem. The control process's multi-turn retrieval (Section 4.3) provides a partial mitigation—the control model can retrieve multiple clips and synthesize evidence at query time—but this is less efficient than encoding cross-clip inferences in memory (each query must re-derive the same synthesis) and is limited by the control model's reasoning capability and the retrieval quality (relevant clips may not be retrieved). Future work could explore hierarchical memory generation where clip-level semantic memories are periodically aggregated into higher-level memories, but this is not proposed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a reframing contribution to the multimodal agent literature: it establishes that long-term memory for embodied agents is not primarily a storage problem but an inference-at-encoding-time problem. Prior work on agent memory (Section 2.1, Section 2.2) converged on the assumption that storing detailed observations—whether as raw trajectories, compressed features, or language descriptions—and retrieving them effectively at query time is sufficient for memory-based reasoning. M3-Agent demonstrates this assumption is false, at least for the class of reasoning required by real-world agent tasks.

The decisive evidence is the semantic memory ablation (Table 6), which shows that removing abstracted, inferential memory while keeping detailed episodic descriptions—the system can still describe every event in precise language—causes accuracy drops of 17.1% (M3-Bench-robot), 19.2% (M3-Bench-web), and 13.1% (VideoMME-long). These are not marginal degradations from losing a helpful auxiliary signal; they are catastrophic failures that reduce a trained system to performance at or below Socratic model baselines that use only prompted commercial models with simple description-based memory. The magnitude of these drops encodes a structural claim: a large fraction of the questions that matter for agent deployment cannot be answered from direct observation alone, regardless of how perfectly that observation is described or retrieved. The inference must happen when the memory is stored, not deferred entirely to query time.

This reframing has several downstream consequences for the field's research priorities:

It resolves a latent tension between two parallel literatures. The Socratic Models tradition (Zeng et al., 2022; Lin et al., 2023b; Zhang et al., 2024a) treats video-to-text generation as sufficient for downstream reasoning—generate detailed captions, store them, retrieve relevant ones, and ask an LLM to answer. The online video understanding tradition (MovieChat, MA-LMM, Flash-VStream) treats compressed feature storage with temporal indexing as sufficient—the features preserve perceptual similarity for retrieval, and the downstream model reasons over retrieved features and text. Both traditions implicitly assume that the bottleneck is retrieval quality: if you can find the right clips, the information in those clips is sufficient to answer the question.

M3-Agent shows the bottleneck is more fundamental: the information in the clips, in its raw or lightly-described form, is not sufficient for many questions. "Is Lucas skilled at cooking?" requires aggregating behavioral evidence across multiple scenes and abstracting a character trait. No single clip contains the answer; no retrieved set of episodic descriptions directly states it. Semantic memory solves this by performing the abstraction at encoding time and storing the inference alongside the observation. The Socratic and online video understanding approaches are therefore not just suboptimal—they are architecturally incapable of answering a meaningful fraction of questions, regardless of how much retrieval quality improves. This reframing makes memory generation quality (how well the system abstracts and infers) at least as important a research target as retrieval quality (how well the system finds relevant stored content), a rebalancing that is not reflected in current research emphasis.

It establishes a new performance floor for agent memory benchmarks. Prior to this work, LVQA benchmarks (Table 2) evaluated systems on whether they could answer questions about video content, but did not decompose performance into perceptual (did you see it?) versus inferential (did you understand what it means?) components. M3-Bench's question type taxonomy (Table 1) provides this decomposition, and the per-type results in Table 5 reveal that all tested systems—including M3-Agent—have dramatically uneven capability profiles. Person understanding is consistently the strongest category across systems; general knowledge extraction is consistently the weakest. This pattern is not an artifact of one model's design but appears to be a fundamental difficulty gradient that any memory-equipped agent must grapple with. The benchmark thus provides a diagnostic toolkit for the field: a new memory architecture can be evaluated not just on aggregate accuracy but on whether it shifts the relative strengths and weaknesses across capability dimensions.

It redirects attention from model scale to task-specific training for agent memory. The comparison between M3-Agent (open-source Qwen2.5-Omni-7b for memorization, Qwen3-32b for control, both task-specifically trained) and Gemini-GPT4o-Hybrid (prompted Gemini-1.5-Pro for memorization, prompted GPT-4o for control) shows that training—synthetic data fine-tuning for memory generation, RL for retrieval strategy—can overcome substantial base model capability gaps. The prompted system uses models widely considered more capable in raw terms, yet underperforms by 6.7%, 7.7%, and 5.3% on the three benchmarks. This inverts the default assumption in much current agent work that better base models will naturally yield better agent performance. For memory-intensive tasks specifically, the evidence suggests that specialized training on the memory generation and retrieval tasks matters more than base model scale, a finding that should influence how research groups and practitioners allocate compute between pretraining larger models versus fine-tuning and RL-training moderate-scale models on targeted data.

It identifies verifier-less memory consistency as an open problem. The paper's weight-based voting mechanism (Section 4.1) provides a simple form of consistency maintenance—frequently observed facts dominate infrequently observed ones—but the system has no mechanism for detecting or resolving errors that are systematic (the memorization model consistently misunderstands a relationship), omission-based (a critical fact was never stored), or inferential (two individually correct observations support a false conclusion when combined). This is not a limitation unique to M3-Agent; it is a gap in the entire paradigm of write-only memory generation. The field has developed sophisticated verifier mechanisms for model outputs in RLHF and reasoning contexts (process reward models, outcome reward models, self-consistency checking), but no analogous mechanisms exist for verifying or correcting memories stored in an agent's long-term memory. M3-Agent's architecture makes this gap visible and quantifiable: every error in memory generation propagates forward into all future queries that depend on that memory, with no corrective feedback loop. This positions memory verification and self-correction as a newly salient research direction.

Follow-Up Research This Work Enables

Cross-clip semantic memory aggregation via hierarchical memory generation. The current memorization model processes 30-second clips independently and generates semantic inferences from single-clip observations. This means cross-clip inferences—e.g., synthesizing a character's personality from behavior across multiple scenes, or extracting a general rule from repeated observations—are deferred entirely to the control model at query time. The per-type performance gap (43.3% person understanding vs. 19.1% general knowledge extraction on M3-Bench-robot, Table 5) suggests the control model struggles with this synthesis.

A direct follow-up would introduce a hierarchical memory generation pipeline: clip-level semantic memories (as currently generated) are periodically aggregated by a second-stage model that takes multiple clips' semantic memories as input and produces higher-level, cross-clip inferences. The experiment would compare three conditions on M3-Bench: (1) the current single-level semantic memory, (2) two-level hierarchical memory with aggregation every N clips, and (3) an oracle upper bound where a human annotator writes cross-clip semantic memories. The key measurement is whether the general knowledge extraction score narrows toward person understanding levels, which would indicate that the inference bottleneck is in memory generation architecture, not in the fundamental difficulty of the questions.

Memory verification through retrospective consistency checking. M3-Agent's memory is write-only: once a memory is stored, no mechanism downstream checks whether it remains consistent with subsequently observed facts. A natural extension is a retrospective verification module that periodically scans the memory graph for contradictions and either flags them or resolves them automatically. Concretely: after processing a new batch of clips, run a consistency-checking prompt over all memory nodes connected to the same entity, asking the model to identify contradictions (e.g., "Memory A says Alice prefers coffee; Memory B says Alice prefers tea"). When contradictions are found, use the weight-based voting mechanism or a more sophisticated resolution strategy (e.g., temporal recency weighting, explicit re-querying of relevant video segments) to update or flag entries.

The experiment would measure: (1) memory accuracy over time with vs. without verification, using a ground-truth memory annotation on a held-out video, (2) downstream QA accuracy as memory accumulates (does verification prevent the degradation that likely occurs as error-prone memories compound?), and (3) whether the verification module itself introduces errors (false-positive contradiction detection that corrupts correct memories). The meta-clip algorithm (Algorithm 2) provides a starting point for automatic ground-truth identity mappings against which verification accuracy could be measured.

Selective attention and saliency-weighted memory generation. The paper identifies fine-grained detail retention as a hard case: "retaining all such details in memory is impractical and may cause cognitive overload" (Section 5.5). Currently, the memorization model generates memory with uniform detail across all aspects of a clip—every character's clothing, every object in the scene, every line of dialogue. This is computationally wasteful (most details will never be queried) and potentially harmful (important details are buried in noise, reducing retrieval precision).

A follow-up would train the memorization model to produce saliency-weighted memory: alongside each memory entry, predict a relevance score indicating how likely this detail is to be needed for future questions. The training signal could come from the training dataset's QA pairs: for each question, trace back which memory entries (in which clips) contain the evidence needed to answer it, and label those entries as high-saliency; all others are low-saliency. At inference time, memory entries below a saliency threshold are either stored at lower weight or omitted entirely, focusing the memory graph's capacity on information that is likely to matter.

The experiment would sweep saliency thresholds and measure the accuracy-efficiency tradeoff curve: how much memory storage can be saved at each level of accuracy retention? This directly addresses the unexamined efficiency claims in the paper and would produce actionable guidance for practitioners on how aggressively they can prune memory without degrading QA performance.

Explicit spatial memory nodes with visual snapshots. The paper identifies spatial reasoning as a systematic failure mode and suggests "incorporating richer visual content, e.g., snapshots" (Section 5.5). A concrete implementation would add a new node type to the memory graph: spatial snapshot nodes that store key frames (or short video segments) with associated spatial metadata. When the memorization model detects a scene that is spatially informative (e.g., a room layout, object placements, relative positions), it stores a snapshot node linked to the relevant location entity, with a text description of the spatial relationships and the raw visual data for direct inspection.

The control model would need a new retrieval mode: when a question has spatial content (identifiable via keyword or embedding-based classification), the system retrieves both text and snapshot nodes. The snapshots are provided to the control model as images (the current control model is text-only; this would require upgrading to a multimodal control model or using a separate visual-spatial reasoning module). The experiment would measure improvement specifically on spatial reasoning questions, ideally on a targeted spatial reasoning subset of M3-Bench or a newly constructed diagnostic benchmark, against the current text-only baseline.

RL training with process rewards for retrieval quality. The current RL training uses only outcome reward: binary correctness of the final answer (Equation 1). This provides no signal about whether individual retrieval steps were effective, which means the policy receives sparse, delayed feedback—it may take many training episodes to discover that certain query formulations work better than others.

A process-reward variant would provide intermediate rewards for retrieval quality: when the model issues a [Search] action, compute the similarity (or downstream usefulness) of the retrieved results to the evidence needed for the correct answer. If the training data includes reference-event-to-question mappings (as M3-Bench-robot scripts do, Section 3.1), automatic process rewards can be generated: a retrieval is "good" if it returns memory from a clip that contains a reference event for the current question. The RL objective would combine outcome reward (final answer correctness) with process reward (retrieval precision), potentially using the DAPO framework extended to handle per-token rewards that are non-zero for search-action tokens.

The experiment would compare: (1) outcome-only RL (current method), (2) process-reward-augmented RL, and (3) an oracle upper bound where the model receives perfect retrieval (the ground-truth reference clips are always returned). The key measurements are: training sample efficiency (how many episodes to reach a given accuracy), final accuracy, and—crucially—whether the learned retrieval strategy transfers better to unseen question types or video domains, since process rewards provide a richer training signal that may encourage learning generalizable retrieval heuristics rather than question-specific patterns.

Cross-architecture memory benchmarking. The paper compares M3-Agent against Socratic Models, online video understanding methods, and prompted agents, but these baselines differ along multiple dimensions simultaneously (memory format, retrieval mechanism, model type, training). A targeted ablation study would isolate the memory architecture contribution by implementing multiple memory backends behind a common control interface and comparing them on M3-Bench:

• M3-Agent memory (entity-centric multimodal graph with episodic + semantic text nodes)
• Flat text memory (all episodic and semantic text entries stored in a flat embedding index with no graph structure or entity linking—this tests the graph's contribution)
• Feature-only memory (MA-LMM-style compressed visual features with temporal indexing, queried by the same RL-trained control model—this tests whether language-based memory is necessary or whether feature-based memory can support the same reasoning when paired with strong retrieval)
• Oracle memory (human-written ground-truth episodic and semantic memory for each clip—this provides an upper bound on what perfect memory could achieve)

All backends would use the same RL-trained control model (control-32b-rl), controlling for retrieval strategy. The experiment would produce a decomposition of M3-Agent's accuracy into components attributable to: (a) memory format (graph vs. flat), (b) memory modality (language vs. features), and (c) memory quality (model-generated vs. human-written). This decomposition is currently unavailable and would provide the field with clear guidance on which architectural investment yields the highest return.

Practical Applications and Downstream Use Cases

Video archive search and question-answering for enterprise knowledge management. Organizations accumulate large repositories of meeting recordings, training videos, and procedural documentation in video format. Current search tools index these by metadata (title, date, auto-generated transcripts) and support keyword search over transcripts, but cannot answer questions requiring cross-video evidence aggregation or inference beyond explicit statements. M3-Agent's architecture—specifically, the upfront memorization phase that processes videos once and builds a queryable multimodal graph—directly addresses this use case. The amortization argument is favorable: memorization cost is paid once per video, and the built memory supports arbitrarily many queries with low per-query control cost (up to 5 rounds of text-based retrieval).

A concrete deployment scenario: a company with 10,000 hours of recorded meetings wants to enable queries like "Which team first raised concerns about the Q3 timeline, and what was the eventual resolution?" Answering this requires: (1) identifying the relevant individuals across multiple meetings (entity-centric memory), (2) extracting timeline information from episodic memory, (3) inferring resolution from semantic memory about decisions made. The per-question-type results in Table 5—particularly the 43.3% person understanding and 28.4% multi-hop reasoning scores on M3-Bench-web—provide a lower-bound accuracy estimate for such queries, with the caveat that enterprise meeting content may be more structured than general YouTube videos (potentially increasing accuracy) but also more domain-specific (potentially decreasing it if the memorization model's training distribution does not cover enterprise terminology).

Household robot memory for personalized assistance over extended deployment. The paper's motivating scenario (Section 1) is a household robot that learns user preferences and household rules through daily observation. The current system architecture—clip-by-clip memorization, entity-centric multimodal graph, multi-turn retrieval—is directly deployable in this setting with modest extensions. The robot would process its egocentric video stream through the memorization pipeline continuously, building a graph that connects people (faces, voices, names) to their preferences (semantic memory: "Alice prefers coffee in the morning"), locations (where objects are stored), and routines (when certain events typically occur).

The key practical consideration is the incremental update cost. Each 30-second clip triggers: perception tool extraction (face recognition at 5 fps, speaker identification with Gemini-1.5-Pro), memorization model inference (Qwen2.5-Omni-7b, fine-tuned), embedding computation (text-embedding-3-large), and graph insertion. A real-time deployment would need to complete this pipeline within 30 seconds to keep pace with the incoming stream. The paper provides no latency measurements, but the memorization model's size (7B parameters) and the perception tools' per-frame costs suggest this may be feasible on a single GPU-equipped edge device for the perception and generation steps, with the embedding API call being the potential bottleneck (network latency to OpenAI's API). A practical implementation would likely replace text-embedding-3-large with a local embedding model to eliminate this dependency.

The benefit is measured by the accuracy on M3-Bench-robot (30.7%), which uses videos recorded from a robot's perspective with household scenarios. However, the 30.7% figure should be interpreted as a lower bound for a deployment where memory accumulates over days or weeks (not just ~34 minutes as in the benchmark), since more accumulated memory provides richer evidence for person understanding and general knowledge extraction, but also as an upper bound if memory errors compound over time (a degradation the paper does not measure).

Automated annotation and quality assurance for long-form video content. Media companies, educational platforms, and content moderators need to answer substantive questions about long-form video content: "Does this documentary accurately represent the historical timeline?" or "Are there instances of unsafe behavior in this instructional video?" These questions require evidence aggregation across the full video duration and inference beyond surface-level content—precisely the capabilities M3-Bench tests.

M3-Agent's memorization-then-query architecture is well-suited to this batch-processing use case. Videos are processed once through the memorization pipeline, and the resulting memory graph supports multiple queries with different intents (content accuracy, safety compliance, educational quality) without re-processing the video. The cost structure is favorable: memorization cost scales linearly with video duration, while the value scales with the number of queries per video. For a content platform processing millions of hours of video, the upfront memorization cost could be amortized across multiple downstream QA tasks.

The primary practical barrier is accuracy. M3-Agent's 30.7% on M3-Bench-robot and 48.9% on M3-Bench-web are below the reliability threshold for automated decision-making in high-stakes contexts (content moderation, educational assessment). However, the system could serve as a triage filter: flag videos with low-confidence answers for human review, while automatically handling high-confidence cases. The confidence signal could come from the weight-based voting mechanism (memories with high weight and low contradiction are more reliable), from the control model's reasoning trace (does it express uncertainty?), or from the retrieval quality (were relevant clips successfully retrieved?). The paper does not evaluate confidence calibration, so this application would require additional calibration work.

When to Prefer This Method

The paper does not explicitly position M3-Agent against named alternatives with a clear decision rule. The experimental baselines (Socratic Models, online video understanding methods, prompted agent variants) are evaluated to demonstrate M3-Agent's superiority, not to characterize the conditions under which each approach is preferable. The paper's claims are architectural ("our framework outperforms") rather than conditional ("in situation X, use approach A; in situation Y, use approach B"). Forcing a tradeoff matrix here would require speculating about performance regimes the paper does not analyze—such as the breakeven video duration below which memory graph construction is not worth its overhead, or the accuracy threshold above which prompted commercial models become competitive with trained open-source models. The paper provides no evidence on either point.