Learning on the Job: An Experience-Driven Self-Evolving Agent for Long-Horizon Tasks

🎯 Pitch

Unlike all prior LLM agents that remain frozen after deployment, MUSE demonstrates that an agent can bootstrap its own capabilities simply by doing its job—without any model fine-tuning. It achieves a new state-of-the-art on the TAC long-horizon benchmark by a 20% relative margin using only a lightweight model, while its self-accumulated experience transfers zero-shot to boost performance on completely unseen hard tasks by nearly 10 percentage points.

1. Executive Summary

This paper proposes MUSE (Memory-Utilizing and Self-Evolving), a novel agent framework that introduces an experience-driven, self-evolving system centered around a hierarchical Memory Module—decomposed into Strategic Memory (high-level behavioral paradigms), Procedural Memory (Standard Operating Procedures for sub-task sequences), and Tool Memory (optimized single-tool usage instructions)—to enable LLM agents to continuously learn from their own execution trajectories on long-horizon productivity tasks without any fine-tuning. Evaluated on the TAC benchmark using Gemini-2.5 Flash, MUSE achieves a new state-of-the-art S_partial of 51.78%, surpassing the previous best (OpenHands-versa with Claude-4 Sonnet) by a relative 20%. In continuous learning experiments, MUSE improves performance by over 10 percentage points across three sequential iterations on repeated tasks, while memory accumulated from only ~10% of the benchmark transfers zero-shot to a hard-task subset (boosting S_partial from 23.65% to 33.41%), establishing that condensed procedural and strategic knowledge yields highly generalizable capabilities that amplify agent performance across novel tasks without requiring model retraining.

2. Context and Motivation

The Core Problem: LLM Agents Are Test-Time Static and Cannot Learn from Experience

The fundamental challenge this paper addresses is deceptively simple: LLM-based agents cannot improve through practice on the job. When an LLM agent is deployed to perform real-world tasks—whether answering emails, managing spreadsheets, debugging code, or coordinating across project management tools—it approaches every task as if for the first time. It cannot remember that a particular sequence of API calls worked reliably last week. It cannot internalize that certain strategies (checking a colleague's calendar before proposing a meeting time) prevent downstream failures. It cannot even guarantee that a task it successfully completed yesterday will be completed again today with the same reliability.

This limitation has an intuitive name: existing agents are test-time static. Their capabilities are fully determined at the end of the LLM's training phase. Once deployed, they are fixed-function programs, despite being built on top of a general-purpose language model. Each interaction is a fresh query, unmoored from the agent's own history of successes and failures.

The paper frames this as a learning-on-the-job problem (Section 1):

"most existing agents are test-time static: their capabilities are fixed once the LLM training phase ends. As a result, each time an agent tackles a task, it operates like an amnesiac executor, unable to effectively learn from past experiences and lacking the capacity for continuous learning and self-evolution. Neither successes nor failures from previous tasks can be consolidated into effective knowledge to guide future actions."

This gap is both practically costly and theoretically significant. Practically, it means every task execution wastes the information generated by previous executions—the agent's own trajectory data, which is rich with signal about what works and what doesn't in the deployment environment, is discarded. Theoretically, it means existing agent frameworks lack a mechanism for the kind of experiential learning that is fundamental to intelligent behavior across biological systems. Humans improve with practice; existing LLM agents do not.

Why This Problem Matters

The paper motivates this problem along three dimensions that span both practical deployment concerns and the broader goal of building capable autonomous agents.

Real-world productivity tasks demand capabilities that static agents lack. The authors draw a sharp distinction between the benchmarks that have driven recent LLM agent research and the actual demands of productivity tasks. Standard benchmarks evaluate domain-specific abilities under controlled conditions: question answering (GPQA, Section 2), mathematical reasoning (MathVista, AIME), code generation (HumanEval). Interactive environment benchmarks like OSWorld and WebArena add complexity, but the paper argues they still fall short—they test "isolated functionalities within a single platform through short-horizon tasks of roughly 20 steps" (Section 1).

In contrast, real-world productivity tasks, as embodied by the TAC benchmark (TheAgentCompany, Xu et al., 2024), represent a fundamentally higher order of complexity. These tasks are characterized by three properties:

Long-horizon planning and interaction: Tasks can exceed 100 action steps, requiring the agent to maintain coherence across extended execution traces. The average TAC task requires over 40 steps and frequently spans multiple applications (Section 4.1).
Cross-application fluidity: The agent must seamlessly switch between diverse software platforms—chat clients, cloud storage, code editors, project management tools, spreadsheets—integrating information across them. This is not tool-use in isolation but orchestration across a realistic corporate software stack.
Consequential, partially-observable environments: Actions have downstream consequences that may not be immediately visible. The agent must plan under uncertainty and adapt when intermediate steps reveal new information.

Static agents, which cannot accumulate environment-specific experience, are fundamentally limited in such settings. They must rediscover effective workflows from scratch each time, wasting computation and limiting the depth of exploration possible within a fixed budget.

The inference cost of complex tasks makes experience reuse essential. The paper implies but does not explicitly quantify an important economic argument: if completing a long-horizon task requires dozens of LLM calls (each a potentially expensive API invocation for closed-source models), then discarding the knowledge gained from each execution is wasteful in a directly measurable sense. An agent that can reuse past experience reduces redundant exploration, lowering the effective cost per task over time. This is the efficiency argument that underlies the continuous learning experiments in Section 4.3.1.

Self-evolution is a prerequisite for truly autonomous agents. Beyond any single task or benchmark, the paper positions test-time learning as a necessary capability for the broader vision of AI agents that operate autonomously over extended periods. The authors cast this in terms of a paradigm shift (Section 2.1):

"The research focus in artificial intelligence is undergoing a profound paradigm shift: from developing static foundational models to building dynamic, self-evolving agents capable of continuous adaptation and learning."

This framing connects MUSE to a broader research trajectory. If agents are to be deployed in dynamic, evolving environments—where software updates, changing organizational procedures, and novel task types are the norm rather than exceptions—they must adapt without requiring retraining or human intervention. Experience accumulation is the mechanism for that adaptation.

Where Prior Approaches Fall Short

The paper identifies four categories of prior work and explains why each is insufficient for the test-time learning challenge on long-horizon productivity tasks.

Self-evolving agents lack practical memory mechanisms for complex tasks. The paper acknowledges an active research area on self-evolving agents (Section 2.1) but notes that existing approaches address different facets of the problem. Some frame prompt optimization as black-box search—automatically discovering effective instructions through iterative refinement (Zhou et al., 2022; Wang et al., 2023; Pryzant et al., 2023)—but this optimizes for static input-output behavior rather than accumulating procedural knowledge from interaction. Others draw on cognitive science concepts like curriculum learning or free exploration to build skill libraries (Voyager, Wang et al., 2023; OS-Copilot, Wu et al., 2024) or optimized tool sets (Agent KB, Tang et al.; ALITA, Qiu et al., 2025). While these are steps toward self-improving agents, the paper argues they have not been validated on environments with the complexity and long-horizon dependency requirements of real-world productivity tasks. Self-reflection methods like Reflexion (Shinn et al., 2023) and SAGE (Liang et al., 2025) introduce language feedback for iterative improvement, but these typically operate within a single task or episode, comparing to ground truth answers, rather than building persistent, reusable knowledge across tasks.

LLM agent memory mechanisms have been validated only on simple benchmarks. The paper engages extensively with prior work on memory for LLM agents (Section 2.2). This literature draws on human cognitive models, dividing memory into short-term working memory for the current task and long-term memory for persistent learning, stored in vector databases (FAISS, Douze et al., 2024) or knowledge graphs (Neo4j). Systems like Mem0 (Chhikara et al., 2025) implement explicit memory operations for content control, while MemInsight (Salama et al., 2025) augments raw memories with summaries and tags for retrieval efficiency.

More directly relevant are approaches that build procedural memory—generalizing reusable experiences and workflows from agent execution trajectories. ExpeL (Zhao et al., 2024) collects trajectories and refines them into natural language insights and rules. Agent Workflow Memory (Wang et al., 2024) focuses on extracting reusable workflows from individual experiences. Memp (Fang et al., 2025) aims for lifelong procedural memory that allows agents to acquire skills and habits through experience.

The paper's critique of this entire line of work is pointed and specific (Section 2.2):

"Although these advanced memory mechanisms are validated on various text-based benchmarks and web agent benchmarks, existing test environments often lack sufficient complexity and long-term dependency requirements. Consequently, they may not fully assess the true efficacy of these mechanisms in handling complex, long-horizon, real-world tasks."

This is not a claim that prior memory mechanisms are wrong, but rather that the benchmarks used to validate them—HotpotQA, ALFWorld, WebShop, FEVER, Mind2Web, WebArena—are too simple. They lack the combination of extreme length (100+ steps), cross-application orchestration, and multi-step dependency chains that characterize real productivity work. The paper positions TAC as a more rigorous test that exposes whether these memory approaches actually scale to realistic complexity.

Fine-tuning is computationally intractable for long-horizon tasks. The paper briefly notes that fine-tuning-based approaches are unsuitable for this setting (Section 5, Discussions). Long-horizon tasks generate extremely long trajectories, making supervised fine-tuning on them computationally expensive. Reinforcement learning approaches are hamstrung by reward design: the reward signal on a 100-step task is both extremely sparse and difficult to formulate—what constitutes a "good" intermediate state in a partially-observable cross-application workflow? These practical barriers motivate the purely inference-time, memory-based approach that MUSE adopts.

Existing state-of-the-art agents still fail on hard long-horizon tasks. The paper provides concrete evidence that the problem is unsolved by current methods. Table 1 shows that on the hard task subset T_hard, OpenHands with Gemini-2.5 Pro achieves an S_partial of only 3.00%, and OpenHands-versa with Claude-4 Sonnet achieves only 2.00%. These are strong models (Gemini-2.5 Pro and Claude-4 Sonnet are frontier-tier LLMs) integrated into capable agent frameworks (OpenHands is a leading open-source platform), yet they complete fewer than 10% of the checkpoints on these tasks. This is not a marginal shortfall—it is near-total failure. The paper uses this as empirical motivation: even the best current systems cannot handle these tasks, so something fundamental (continuous experiential learning) is missing.

How This Paper Positions Itself Relative to Existing Work

The paper's positioning has three key dimensions.

A synthesis of memory types into a unified, production-oriented architecture. Prior work on agent memory tends to focus on a single memory type—procedural memory in ExpeL and Agent Workflow Memory, working memory in Mem0, tool-use skills in Voyager. MUSE integrates three memory types into a single system: Strategic Memory for high-level behavioral paradigms, Procedural Memory for sub-task SOPs, and Tool Memory for single-tool instructions. The paper argues this multi-level organization mirrors the abstraction hierarchy needed for complex tasks: high-level strategies guide overall approach, mid-level procedures encode known workflows, and low-level tool memory optimizes primitive actions. Importantly, all memory is stored in natural language, making it model-agnostic—experience gained using one LLM can be transferred to another (a claim validated in Section 4.4.2 when memory accumulated with Gemini-2.5 Flash boosts DeepSeek-V3's performance).

A closed-loop "Plan-Execute-Reflect-Memorize" cycle that makes test-time learning autonomous. Unlike systems that require human feedback or ground-truth labels to identify successful trajectories, MUSE's Reflect Agent autonomously evaluates sub-task success/failure using structured criteria (truthfulness verification, deliverable verification, data fidelity—Section 3.4). This is critical because it enables fully autonomous experience accumulation: the agent learns from its own successes and failures without human intervention. The Reflect Agent's ability to independently interact with the environment to cross-check the PE Agent's conclusions distinguishes it from simpler self-evaluation mechanisms that only inspect the LLM's generated text.

A commitment to minimal tooling to force genuine procedural learning. The paper makes a deliberate design choice that distinguishes it from many agent frameworks (Section 3.3). Rather than equipping MUSE with a large library of domain-specific APIs (the ToolLLM or Gorilla approach, which aim to integrate thousands of tools), MUSE uses only a minimal set of general-purpose tools: a browser operator, a Python interpreter, a shell, a vision extractor, and a memory retriever. The rationale is explicit:

"We believe that the core of intelligence lies in the ability to creatively combine basic tools, rather than mechanically invoking pre-defined functions."

This constraint forces the agent to compose workflows from primitives, which in turn makes the resulting Procedural Memory more generalizable—an SOP for creating a GitLab issue is learned through browser interactions and shell commands, not through a "create_gitlab_issue" API call that would be useless in a different environment. It also means that improving at a task genuinely requires learning reusable procedural knowledge, not just discovering which API to call.

Validation on a benchmark that exposes the limitations of existing approaches. By choosing TAC—the most complex long-horizon productivity benchmark available—the paper positions itself as a stress test for whether memory-based self-evolution actually works under realistic conditions. The results are intended to demonstrate not just a new method, but that existing benchmarks have been insufficiently challenging, masking the true value of experience accumulation. The generalization experiment (Table 1) is particularly important for this positioning: memory accumulated from only ~10% of the benchmark tasks (the T_cl subset) transfers zero-shot to a held-out hard subset, improving performance from 23.65% to 33.41%. This shows the learned memory is not task-specific memorization but genuinely generalizable procedural and strategic knowledge—a key claim that distinguishes MUSE from approaches that merely cache solutions to specific problems.

The paper also implicitly positions itself against the prevailing narrative that "better models solve everything." The fact that MUSE with the lightweight Gemini-2.5 Flash outperforms OpenHands-versa with the much more capable Claude-4 Sonnet (Table 2) suggests that architecture matters as much as model capability for complex agentic tasks—a finding with significant implications for how the field allocates effort between model scaling and agent design.

3. Technical Approach

3.1 Reader Orientation

MUSE is a closed-loop agent framework that wraps an LLM with a hierarchical memory system, enabling it to continuously learn from its own successes and failures while performing long-horizon productivity tasks—without any model fine-tuning. The system solves the problem of test-time static agents by converting raw execution trajectories into structured, reusable knowledge stored across three memory types, then retrieving that knowledge during future task execution to avoid redundant exploration and progressively improve performance.

3.2 Big-Picture Architecture (Diagram in Words)

The MUSE framework has five major components that interact in a "Plan-Execute-Reflect-Memorize" loop (illustrated in Figure 2, Section 3.1):

Memory Module (M) — a persistent, hierarchically-organized knowledge base with three sub-types: Strategic Memory (M_strat) for high-level behavioral paradigms, Procedural Memory (M_proc) for Standard Operating Procedures (SOPs) guiding sub-task execution, and Tool Memory (M_tool) for optimized single-tool usage instructions. All memory is stored in natural language.
Planning-Execution (PE) Agent — the core LLM-driven component that decomposes a high-level task into an ordered queue of sub-tasks, then executes each sub-task through a memory-enhanced ReAct loop (iterating Thought, Action, Observation) within an interactive environment (a virtual desktop with chat clients, code editors, browsers, etc.). It uses a deliberately minimal toolset of general-purpose tools.
Reflect Agent — an independent supervisory component (also LLM-driven, using the same toolset as the PE Agent) that evaluates whether each sub-task was completed successfully by formulating structured checklists, tracing conclusions back to environmental observations, and proactively interacting with the environment to cross-check information. It outputs success/failure flags and triggers either memory distillation (on success) or replanning (on failure).
Interactive Environment (E) — a fully functional operating system with multiple applications (RocketChat, GitLab, OwnCloud, Plane, file systems, spreadsheets, code interpreters) that the agent interacts with through its toolset.
Task Queue Manager — an implicit component within the PE Agent that maintains an ordered queue of sub-tasks Q = [st_1, st_2, ..., st_M], dynamically re-evaluating and updating it after each sub-task execution based on new information and the Reflect Agent's assessments.

Information flows as follows: a user task enters → the PE Agent loads all memory from the Memory Module, decomposes the task into sub-tasks, and begins executing the first sub-task via the ReAct loop in the environment → after each sub-task attempt, the Reflect Agent evaluates success/failure → on success, it distills the trajectory into new Procedural Memory; on failure, it generates a diagnostic analysis and triggers replanning → the PE Agent updates the sub-task queue and continues → when all sub-tasks are complete, the Reflect Agent performs a comprehensive post-task analysis, distilling higher-level Strategic and Tool Memory → all memory is integrated, deduplicated, and stored back in the Memory Module for future tasks.

3.3 Roadmap for the Deep Dive

First, the Memory Module — the three memory types (Strategic, Procedural, Tool), their generation mechanisms, update processes, and retrieval strategies. This is the linchpin of the entire framework, and understanding its structure is prerequisite to understanding how the PE Agent and Reflect Agent use it.
Second, the Planning-Execution (PE) Agent — how tasks are decomposed into sub-tasks, how the ReAct loop works with memory retrieval, the minimal toolset design choice, and the retry/replan mechanism for handling failures.
Third, the Reflect Agent — the autonomous evaluation criteria (truthfulness, deliverable, data fidelity), the dual inspection methods (trajectory referencing, active verification), and the critical role it plays in converting raw trajectories into reusable memory.
Fourth, the memory update mechanisms — how Procedural Memory is created immediately after sub-task success, and how Strategic and Tool Memory are distilled post-task, including the deduplication and generalization processes that maintain memory quality over time.
Fifth, the overall design rationale — why these specific components were chosen, what alternatives were rejected (fine-tuning, RL, large API libraries), and how the design enables model-agnostic knowledge transfer.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems architecture paper whose core idea is that a hierarchical, experience-driven memory system—updated through autonomous reflection on execution trajectories—can transform an otherwise static LLM agent into a continuously improving system on long-horizon productivity tasks, without any model fine-tuning.

3.4.1 The Memory Module: Three-Level Hierarchical Knowledge Organization

The Memory Module M = {M_strat, M_proc, M_tool} is the central innovation enabling test-time learning. Each memory type operates at a different level of abstraction, uses distinct generation and update mechanisms, and is retrieved differently during execution. All three memory types are stored as natural language text, which the paper argues makes them LLM-agnostic—experience accumulated using one model can be loaded and utilized by a different model (a claim validated in Section 4.4.2 when memory accumulated by Gemini-2.5 Flash boosts DeepSeek-V3's performance).

Strategic Memory (M_strat): High-Level Behavioral Paradigms

Strategic Memory captures macro-level "problem-solution" patterns distilled from dilemmas the agent encounters across multiple tasks. The Reflect Agent abstracts these into <Dilemma, Strategy> key-value pairs, formatted as natural language principles.

The generation process (Section 3.2, detailed in Section 3.4's Memory Update Mechanism) works as follows: after completing an entire task, the Reflect Agent analyzes the full execution trajectory to identify challenges that required multiple attempts to overcome. For each such challenge, it extracts what the underlying dilemma was (e.g., "data was lost during file format conversion because intermediate validation steps were skipped") and what resolution pattern ultimately worked (e.g., "verify file integrity after every format conversion by checking row counts and file size"). These are abstracted into general principles with descriptive names like "Systemic Root Cause," "Robust Context State," or "Granular Outcome Verification" (Table 6 provides ten such principles).

The loading mechanism is designed for maximum influence on global behavior: upon agent initialization, the entire M_strat is loaded into the system prompt (Section 3.2). This means every action the PE Agent takes is conditioned on these high-level behavioral guidelines—they function as persistent "personality traits" or "habits" that shape the agent's default approach to all tasks.

To prevent the system prompt from growing unbounded—a real concern given that LLMs have finite context windows and performance degrades with prompt length—M_strat is actively managed. The paper states:

"To ensure efficiency and prevent context window bloat, this memory is updated, merged, and refined after each task, always maintaining a concise size."

This implies a deduplication and abstraction process: when a new dilemma-strategy pair is similar to an existing one, they are merged; when a pattern emerges across multiple dilemmas, it is generalized into a higher-level principle; when a principle proves redundant or misleading, it is removed. The paper does not specify the exact merging algorithm, but the Reflect Agent handles this as part of the post-task memory update phase.

The ten example principles shown in Table 6 illustrate the abstraction level. They are not task-specific (e.g., "when using RocketChat, always refresh the browser after clicking Send") but are genuinely general behavioral heuristics: "Explicitly manage and continuously verify data and execution context throughout its lifecycle to ensure accuracy, consistency, and integrity of dependencies and prevent errors" (Robust Context State), or "When required information is unextractable or unverifiable, explicitly assign a clear 'Not Available' or equivalent status to prevent hallucination and maintain data integrity" (Explicit Uncertainty Handle). These are principles a human knowledge worker would recognize as professional best practices, and they shape the agent's behavior across all tasks regardless of the specific application.

Procedural Memory (M_proc): Standard Operating Procedures for Sub-Tasks

Procedural Memory is the largest and most frequently updated memory type. It archives successful sub-task trajectories as a hierarchical knowledge base of Standard Operating Procedures (SOPs), organized by application and then by function.

The structure (Section 3.2) is a two-level index:

Application-level index: SOPs are grouped by the software platform they relate to (e.g., "RocketChat," "FileSystem," "OwnCloud," "GitLab," "Plane"). This grouping is natural because workflows for the same application share common interaction patterns (login procedures, navigation conventions, UI element locations).
Function-level index: Within each application group, individual SOPs are indexed by the specific function they accomplish (e.g., "RocketChat → Navigate to Home Page," "FileSystem → Create or Overwrite File," "OwnCloud → Login").

Each SOP p = (index_p, content_p) consists of a lightweight index entry (the application and function name) and a detailed content body. The content body follows a consistent structure, as shown in the examples in Table 7:

Preconditions: What must be true before the SOP can be applied (e.g., "User is logged into RocketChat" or "Browser is open, ownCloud URL & credentials available").
Steps: A numbered sequence of concrete actions with verification checkpoints (e.g., "Navigate to login page → Enter username in 'Email or username' field → Enter password in 'Password' field → Click 'Login' button → Verify login success (URL is /home, login fields disappear, post-login elements appear)").
Notes: Contextual guidance about common pitfalls, alternative approaches, and reliability heuristics (e.g., "Always follow navigation with browser_update. The 'Home' link may be more reliable than a button depending on UI context" or "Python file handling (open, write) is more robust than using shell commands (e.g., echo) due to escaping issues").

This structure is deliberately designed to mirror how human experts document standard procedures—it provides exactly the information needed to execute a known workflow without requiring the agent to rediscover the steps through trial and error.

The retrieval mechanism (Section 3.3) separates the index from the content for efficiency, addressing a critical engineering constraint: the LLM's context window is finite, and loading the full content of every SOP would rapidly exhaust it. Instead, at the start of a sub-task, only the lightweight index of all available SOPs—IM_proc = {index_p | p ∈ M_proc}—is loaded into the context. This is a list of application-function pairs, which is compact enough to fit in context even with hundreds of SOPs.

The PE Agent is equipped with a dedicated tool a_mem (the memory retriever, called access_guide in the toolset—Table 9) that it can call at any point during execution to retrieve the full content of a specific SOP on demand. The paper explicitly encourages proactive retrieval:

"We use prompt engineering to encourage the agent to prioritize querying for relevant experience at the beginning of each sub-task."

This design creates an attention-like mechanism: the agent has a high-level awareness of what knowledge is available (through the index) but only loads detailed content when it's relevant to the current sub-task, keeping the active context lean while preserving access to a potentially large knowledge base.

Table 7 shows a representative set of Procedural Memory entries: RocketChat navigation and login, FileSystem file creation and verification, OwnCloud login and folder navigation. The examples reveal that SOPs encode specific, actionable sequences rather than abstract advice—they are "if you need to do X, here are the exact steps" guides that can be mechanically followed, dramatically reducing the exploration required for known workflows.

Tool Memory (M_tool): "Muscle Memory" for Single Tool Usage

Tool Memory optimizes the agent's proficiency with individual tools in its minimal toolset. It has two components, M_tool = {D_static, I_dynamic} (Section 3.2):

Static Description (D_static): Loaded into the system prompt at startup, this explains each tool's core functionality. It is the "what this tool does" documentation—analogous to a function signature and docstring. For example, the access_guide tool's static description (Table 8) reads: "Get detailed platform/application operation guides. The guide is structured from past successful experiences."
Dynamic Instruction (I_dynamic): Returned with the environment's observation o_t after a tool is used, this provides context-sensitive guidance for the agent's immediate next action a_{t+1}. It is "what to do next after using this tool"—analogous to inline documentation that appears after a function call. For browser_click, the dynamic instruction (Table 8) reads: "Always follow with browser_update. - Verify outcome (e.g., page change, modal open). - If action fails, refresh interactive elements and retry with semantic attributes. - If still failing, try browser_send_keys. - For unclickable elements, acknowledge task may be unachievable and adjust strategy."

The "muscle memory" metaphor is apt: just as a person doesn't consciously recall how to type each key but rather executes typing as an automatic skill, Tool Memory aims to make single-tool invocations reliable and efficient without the agent needing to reason about tool mechanics each time. The static component ensures the agent knows what tools exist and what they do; the dynamic component provides just-in-time guidance to prevent common errors and ensure proper usage patterns.

The Tool Memory is updated by the Reflect Agent after each task is completed (Section 3.2), meaning that as the agent gains experience, the dynamic instructions become increasingly refined with environment-specific best practices. Table 8 shows that the dynamic instructions encode specific failure recovery sequences ("If clicking fails, dynamically search for element attributes") and verification behaviors ("Verify intended outcome (navigation, modal opening, state change), not just the click itself")—knowledge that would otherwise need to be learned through repeated trial and error.

Importantly, Tool Memory operates automatically—the paper states it functions "without requiring proactive retrieval." The static descriptions are always in the system prompt, and the dynamic instructions are always appended to observations. This is in contrast to Procedural Memory, which requires the agent to actively decide when to query it, reflecting the different abstraction levels: tool usage is so fundamental that its guidance should be continuously available, while procedural knowledge is task-specific and retrieved on demand.

3.4.2 The Planning-Execution (PE) Agent: Task Decomposition and Memory-Enhanced Execution

The PE Agent is responsible for converting a high-level user task into a sequence of concrete actions in the interactive environment, using the Memory Module to avoid redundant exploration. Its operation can be decomposed into four phases: initial planning, sub-task execution, retry on failure, and dynamic replanning.

Initial Task Decomposition

When a new task τ arrives, the PE Agent first decomposes it into an ordered queue of sub-tasks Q = [st_1, st_2, ..., st_M] based solely on the initial task description (Section 3.3). Each sub-task st_i is defined as a tuple:

$st_i = (desc_i, goal_i)$

where desc_i is a natural language description outlining the sub-task's scope (what needs to be accomplished), and goal_i is a concise statement that serves as the evaluation basis for the Reflect Agent (how to determine if the sub-task succeeded).

For example, for a task like "collect performance feedback on Liu Qiang from three colleagues and compile a summary report," the PE Agent might decompose it into sub-tasks like: st_1 = ("Message Zhang Wei on RocketChat to request feedback on Liu Qiang", "Receive Zhang Wei's feedback response"), st_2 = ("Message Wang Fang on RocketChat to request feedback on Liu Qiang", "Receive Wang Fang's feedback response"), etc.

This decomposition is crucial because it transforms an overwhelmingly long task (potentially 100+ actions) into manageable chunks that the agent can attempt, evaluate, and potentially retry independently. The Reflect Agent evaluates each sub-task separately, so failure on one sub-task doesn't require restarting the entire task—only that sub-task needs to be replanned and re-executed.

Sub-Task Execution via Memory-Enhanced ReAct Loop

The PE Agent processes sub-tasks sequentially from the queue, attempting to resolve each one using a ReAct loop enhanced with Procedural Memory retrieval. The ReAct pattern (Reasoning + Acting, from Yao et al., 2023) iterates a (θ_t, a_t, o_t) tuple:

$\text{ReAct step: } \theta_t \rightarrow a_t \rightarrow o_t$

where θ_t is a reasoning thought (the agent's internal deliberation about what to do next), a_t is an action selected from the toolset (e.g., clicking a button, typing text, querying memory), and o_t is the environmental observation returned after executing the action (e.g., the updated browser accessibility tree, the output of a shell command).

What it computes: At each step, the agent generates a thought (reasoning about the current state and sub-task goal), selects an action based on that thought, executes the action in the environment, receives feedback, and then uses that feedback to generate the next thought. This continues until the agent determines that the sub-task's goal has been met.

Why this form: The ReAct pattern interleaves reasoning and action, which is essential for interactive environments where the agent's understanding of the task evolves as it gathers information. Pure chain-of-thought reasoning before acting would be brittle because the agent's initial plan may be based on incorrect assumptions about the environment state; pure action without explicit reasoning would lose coherence over long sequences. The interleaving allows the agent to adapt its plan in light of new observations—for instance, if st_2's goal was to receive feedback from Wang Fang, but the observation reveals Wang Fang is on leave, the agent can adjust its approach within the same sub-task.

The memory-enhanced aspect comes from the index pre-loading and on-demand retrieval mechanism. At the start of each sub-task, the lightweight index IM_proc of all available SOPs is in context. The PE Agent is prompted to check whether any indexed SOP matches the current sub-task. If so, it uses the access_guide tool (Table 9) to retrieve the full SOP content, which provides a proven step-by-step procedure to follow. This is the mechanism by which past experience directly guides current execution: rather than exploring from scratch, the agent follows a known-good workflow.

The paper imposes a maximum of N = 20 actions per sub-task attempt (Section 3.3, confirmed in Section 4.2) to prevent the agent from getting stuck in futile loops. If the PE Agent reaches this limit without declaring the sub-task complete, the Reflect Agent is triggered to intervene.

Retry Mechanism: Encouraging Exploration Over Exploitation

When a sub-task attempt fails (either the agent reaches the action limit or declares completion but the Reflect Agent evaluates it as failure), the agent is granted one retry opportunity. The retry mechanism has a critical design feature (Section 3.3):

"During the retry, the PE Agent is no longer required to use Procedural Memory M_proc, enabling it to discover novel methods when existing knowledge is erroneous or inapplicable."

This is an explicit exploration-exploitation tradeoff. The first attempt exploits existing knowledge (by retrieving and following relevant SOPs). If that fails—perhaps because the SOP is outdated, the environment has changed, or the SOP was learned from a superficially similar but genuinely different task—the retry drops the constraint to follow known procedures, allowing the agent to freely explore novel action sequences. This prevents the system from getting trapped by incorrect or stale procedural knowledge.

If the retry also fails, the PE Agent does not attempt the sub-task a third time. Instead, it triggers a sub-task replanning process, which may restructure the task decomposition entirely (e.g., splitting the failed sub-task into smaller sub-tasks, or merging it with others).

Dynamic Replanning

After each sub-task attempt (whether successful or failed), the PE Agent re-evaluates and updates the sub-task queue Q based on newly acquired information (Section 3.3). This is what makes the planning adaptive rather than rigid:

"Crucially, after each sub-task execution, the agent re-evaluates and updates the sub-task queue Q based on newly acquired information, ensuring an adaptive path to task completion."

The replanning process uses the same multi-turn decomposition approach as initial planning, but with richer inputs: it has access to the execution results and the Reflect Agent's assessments of all completed sub-tasks, plus any new information discovered during execution. For instance, if the agent learned during st_1 that a particular colleague is unavailable, st_2 (which was originally planned to involve that colleague) might be restructured to use an alternative contact.

The paper provides a concrete example in the case study of Figure 5 (Section A.2). The task "pm-ask-for-issue-and-create-in-gitlab" required the agent to consult three different colleagues to gather details for a GitLab issue, but the initial task description only provided one contact person (Li Ming). Through interactions with Li Ming, the agent progressively identified and located the other two relevant colleagues, continuously adapting its sub-task queue in real-time. The agent also discovered during execution that Li Ming lacked a GitLab account (a prerequisite for assigning the issue), so it dynamically added sub-tasks to create the account and add Li Ming to the appropriate project team.

When the sub-task queue Q becomes empty—meaning all planned sub-tasks have been attempted—the PE Agent performs a final review, examining the global state of the environment to confirm that the overall task objectives have been met. This is a critical verification step: the agent might have successfully completed all individual sub-tasks but still failed to achieve the overall task goal (e.g., because the sub-task decomposition was incomplete or incorrect). The final review catches such cases and can trigger additional replanning if needed.

Minimal Usable Toolset Design

The PE Agent is equipped with a deliberately minimal toolset A_tool (Section 3.3, detailed in Table 9) consisting of only general-purpose tools rather than domain-specific APIs:

Browser operator: A suite of browser interaction functions (navigate to URL, click elements by index, input text, send keystrokes, extract content via vision, manage tabs) built on the browser-use framework (Browser-Use, 2025). Observations returned to the agent include both the accessibility (a11y) tree and the page's interactive elements.
Python interpreter (run_python_code): Execute Python code in an isolated environment for data processing, file manipulation, and analysis tasks.
Shell (run_cmd): Execute arbitrary shell commands for file system operations and system-level tasks.
Vision extractor (gpt4o_describe_image, browser_extract_content_by_vision): Use GPT-4o as a vision backbone to recognize and interpret image content from screenshots.
Memory retriever (access_guide): Query the Procedural Memory to retrieve full SOP content on demand.

The rationale for minimalism is both philosophical and practical. Philosophically, the paper states:

"We believe that the core of intelligence lies in the ability to creatively combine basic tools, rather than mechanically invoking pre-defined functions."

Practically, a minimal toolset forces the agent to compose workflows from primitives. When the agent learns to create a GitLab issue, it does so by orchestrating browser navigation, clicking, text input, and page refresh—not by calling a create_gitlab_issue API. This makes the resulting Procedural Memory more generalizable because the learned SOPs encode interaction patterns (how to navigate the GitLab interface, where to find the "New Issue" button, what fields to fill) rather than API-specific invocations. If the GitLab UI changes, the SOP can still provide a useful starting point; if the agent is deployed in a different environment that uses a different issue tracker, the pattern of "navigate to project → find issue creation interface → fill fields → submit → verify" transfers, even though the specific UI elements differ.

The paper also notes a second practical motivation:

"Furthermore, a key objective of this research is to validate whether MUSE can convert successful solutions into reusable Procedural Memory, thereby achieving the self-evolution of its capabilities."

If the agent used specialized tools for every application, there would be little to learn—the "intelligence" would be in knowing which API to call, not in knowing how to compose primitive actions into effective workflows. The minimal toolset ensures that improving at a task genuinely requires learning reusable procedural knowledge, making the self-evolution claim testable.

3.4.3 The Reflect Agent: Autonomous Sub-Task Evaluation with Structured Criteria

The Reflect Agent is the mechanism that closes the learning loop, converting raw execution trajectories into structured feedback and reusable memory without human intervention. It operates as an independent, third-party supervisor that evaluates the PE Agent's work using the same toolset, allowing it to actively verify claims rather than relying solely on the PE Agent's self-reporting.

When Reflection Is Triggered

The Reflect Agent's evaluation process is triggered in two circumstances (Section 3.4):

The PE Agent declares a sub-task complete (it believes the goal has been achieved).
The PE Agent reaches the action limit N = 20 without declaring completion.

In both cases, the Reflect Agent receives the sub-task's definition st_i = (desc_i, goal_i) and the PE Agent's complete execution trajectory h_{k:t} = (o_{k:t}, a_{k:t-1})—the sequence of observations and actions from the start of the sub-task to the current state. Crucially, the Reflect Agent can also interact directly with the environment E to independently verify information, meaning it is not constrained to only examining the PE Agent's recorded trajectory.

The Three-Dimensional Evaluation Criteria

The Reflect Agent structures its evaluation around three core dimensions (Section 3.4), formulated as an ordered checklist:

Truthfulness Verification: Ensuring that the PE Agent's conclusions are grounded in real environmental feedback. The Reflect Agent checks whether claims made by the PE Agent (e.g., "the file was successfully created," "the colleague confirmed the meeting time") correspond to actual observations in the execution history or can be independently verified in the current environment state. This directly targets hallucination—a known failure mode where LLM agents claim success despite not having actually completed the required actions.
Deliverable Verification: Checking the existence, completeness, and correctness of any output files, reports, or artifacts the sub-task was supposed to produce. If the sub-task goal was "create a spreadsheet summarizing Q3 expenses," the Reflect Agent verifies that the spreadsheet file exists, contains the expected data, and is correctly formatted. This may involve opening files, checking file sizes, comparing output to specifications, or using Python to programmatically validate content.
Data Fidelity: Confirming that data has not been lost, truncated, or altered during processing. This addresses a common failure mode in long-horizon tasks where intermediate data transformations (e.g., copying data from a chat message to a spreadsheet) silently corrupt or lose information. The Reflect Agent may compare original source data to processed output data to detect such issues.

These criteria are applied sequentially as a checklist. The Reflect Agent outputs two things: a binary success/failure flag f and a detailed check report explaining which criteria passed or failed and why.

Dual Inspection Methods

To execute the evaluation checklist, the Reflect Agent uses two complementary inspection methods (Section 3.4):

Trajectory Referencing: The Reflect Agent explicitly traces the PE Agent's conclusions back to specific observations o_t in the execution history h_{k:t}. For example, if the PE Agent claims "the file was saved successfully," the Reflect Agent searches the trajectory for an observation that confirms this (e.g., a shell command output showing the file exists, or a browser observation showing a "File saved" confirmation message). If no such observation exists, the claim is flagged as unsubstantiated—a likely hallucination.
Active Verification: The Reflect Agent proactively uses its own tools to interact with the environment and cross-check key information with real-time feedback. If the PE Agent claims to have created a file at a specific path, the Reflect Agent can use run_cmd with ls to check whether the file exists, or run_python_code to read the file and verify its contents. This is more robust than trajectory referencing because it doesn't depend on the PE Agent having recorded a verification observation—it independently checks the ground truth.

The combination of these methods is important. Trajectory referencing is fast and doesn't require additional environment interactions, but it can be fooled if the PE Agent hallucinated a successful observation. Active verification is more reliable but costs additional LLM calls and environment actions. The Reflect Agent uses trajectory referencing as a first pass, then deploys active verification for claims that are suspicious or critical to the sub-task's success.

Output and Downstream Effects

The Reflect Agent outputs a tuple (f, report) where f ∈ {success, failure} and report is the detailed evaluation (Section 3.4). This tuple is fed back to the PE Agent as a historical record. The downstream behavior then bifurcates:

If f = success, the Reflect Agent performs its most critical function for the learning loop: it summarizes the effective operational sequence from the trajectory h_{k:t} into a new SOP p_new for the Procedural Memory M_proc. This is the mechanism by which successful experience is captured for future reuse. The Reflect Agent identifies which actions in the trajectory were causally necessary for success (filtering out exploratory dead ends, redundant actions, and verification steps that are part of the reflection process itself), then structures them into the SOP format (Preconditions → Steps → Notes).

If f = failure, the Reflect Agent generates a failure cause analysis report R_fail—a diagnostic explaining what went wrong and why. This report informs the PE Agent's replanning process, helping it avoid the same failure mode on the retry or revised sub-task. For example, if the failure was caused by attempting to click a UI element that doesn't exist, the analysis might suggest using the accessibility tree to find alternative selectors or navigating to a different page.

An important subtlety: the Reflect Agent uses the same toolset A_tool as the PE Agent (Section 3.4). This is a design choice that ensures the Reflect Agent can verify any action the PE Agent could have taken, but it also means the Reflect Agent is subject to the same environmental constraints. If a verification requires interacting with a web application, the Reflect Agent must navigate to it using the browser tools, just as the PE Agent would. This is both a strength (the verification is grounded in the same reality) and a limitation (the Reflect Agent can also make tool-use errors).

The paper implicitly relies on the Gemini-2.5 Flash model's reasoning capabilities for the Reflect Agent's evaluation, since the evaluation criteria (truthfulness, deliverable verification, data fidelity) require non-trivial reasoning about the relationship between observed evidence and claimed outcomes. The fact that the Reflect Agent operates as an independent entity (using separate LLM calls from the PE Agent) is significant: it means the reflection is not self-evaluation (which is known to be unreliable for LLMs) but rather cross-evaluation by a separate reasoning instance with access to independent environmental verification.

3.4.4 Memory Update Mechanisms: From Raw Trajectories to Structured, Reusable Knowledge

The conversion of raw execution trajectories into structured memory happens in two stages: immediate Procedural Memory creation after sub-task success, and comprehensive post-task distillation of all three memory types.

Stage 1: Immediate Procedural Memory Creation

Immediately after the Reflect Agent evaluates a sub-task as successful (f = success), it dynamically adds the new SOP p_new to M_proc (Section 3.2). This is described as enabling "immediate reuse"—the newly created SOP is available for subsequent sub-tasks within the same overall task, not just future tasks. For instance, if a task involves sending messages to five different colleagues and the agent successfully messages the first colleague, the SOP for "send a direct message on RocketChat" becomes available for the remaining four colleagues, progressively reducing exploration as the task proceeds.

The SOP creation process (Section 3.4) involves summarizing the effective operational sequence from the trajectory. The Reflect Agent must distinguish between actions that contributed to success and actions that were exploratory, redundant, or part of the verification process. The paper doesn't specify the exact algorithm for this filtering, but it is implemented as part of the Reflect Agent's LLM-based reasoning—the agent is prompted to extract the minimal sufficient sequence of actions that would reproduce the successful outcome.

Stage 2: Post-Task Comprehensive Memory Distillation

The entire task τ is considered complete once the PE Agent stops generating new sub-tasks during the replanning phase (Section 3.4). At this point, the PE Agent launches a task review, summarizing its execution attempts and outcomes across all sub-tasks. This triggers the Reflect Agent to conduct a full-scale upgrade of the memory system.

The post-task distillation has three components:

Strategic Memory Reinforcement: The Reflect Agent analyzes the full task execution trajectory to identify challenges and their solutions. It extracts <Dilemma, Resolution Pattern> pairs—for instance, if the agent repeatedly encountered file format corruption when converting between formats, and eventually discovered that verifying row counts after each conversion prevented the issue, this becomes a "Robust Context State" principle. These pairs are formatted as natural language <Dilemma, Strategy> key-value pairs and added to M_strat. The Reflect Agent then merges new entries with existing ones, generalizing common patterns and removing redundant or superseded principles to maintain concise size (Section 3.2).
Tool Memory Augmentation: The Reflect Agent codifies effective tool usage patterns observed during the task into the Dynamic Instruction component I_dynamic of M_tool (Section 3.2). If the agent discovered that browser_click is more reliable when preceded by browser_update and followed by verifying state changes rather than just checking for click confirmation, this pattern is encoded into the dynamic instruction. The paper states:

"To ensure this 'muscle memory' improves over time, the Tool Memory is updated by the Reflect Agent after each task is completed."

Procedural Memory Global Refinement: The Reflect Agent performs a higher-level optimization of M_proc, including deduplication (merging SOPs that cover the same workflow but were generated from different tasks), generalization (extracting common patterns across SOPs to make them more broadly applicable), and quality improvement (updating SOPs that proved partially effective with insights from more recent, more successful executions).

This two-stage update process—immediate Procedural Memory creation for rapid reuse, followed by comprehensive post-task distillation for long-term quality—mirrors the distinction in human learning between "I just figured out how to do this, let me note it down" and "now that the project is done, let me reflect on what I learned and how to do it better next time." The immediate update ensures the agent doesn't forget a newly discovered workflow before the task is complete, while the post-task distillation ensures the memory base doesn't degrade over time through accumulation of redundant or suboptimal procedures.

3.4.5 Overall Design Rationale: Why Memory Instead of Fine-Tuning or RL

The paper explicitly addresses why it chose a memory-based approach over alternatives that might seem more natural for enabling continuous learning (Section 5, Discussions).

Against fine-tuning: The paper states that fine-tuning methods "suffer from computational intractability" for long-horizon tasks. Long trajectories (100+ steps) generate enormous amounts of training data, and supervised fine-tuning on them is computationally expensive—especially for the closed-source models (Gemini-2.5 Flash) that the paper uses as backbones. More fundamentally, fine-tuning is a batch process: it requires collecting a dataset, training, and deploying an updated model. It doesn't support the kind of immediate, within-task learning that MUSE's immediate Procedural Memory creation enables. If the agent learns a new workflow during the third sub-task of a task, that knowledge can be used in the fourth sub-task of the same task under MUSE; under a fine-tuning paradigm, it couldn't be used until the model was retrained and redeployed.

Against reinforcement learning: The paper argues that RL-based approaches are "hindered by the design of rewards that are both extremely sparse and difficult to formulate." In a 100-step task spanning multiple applications, what constitutes a good intermediate state? The reward signal is sparse (only available at task completion, if even then), and the state space is enormous and partially observable (the agent sees only what's in its browser and command outputs, not the full system state). The Reflect Agent's structured evaluation criteria (truthfulness, deliverable verification, data fidelity) can be seen as a way to provide dense, meaningful feedback without requiring a hand-designed reward function—but that feedback is used for memory updates and replanning, not for gradient-based policy optimization.

Why natural language memory: The paper emphasizes that "since the memory is stored in natural language, the accumulated knowledge is LLM-agnostic, allowing experience gained by one model to be seamlessly transferred and utilized by another" (Section 1). This is validated in Section 4.4.2, where memory accumulated using Gemini-2.5 Flash is loaded by a DeepSeek-V3-based agent, resulting in significant performance improvement. If the knowledge were encoded in model weights (via fine-tuning) or in model-specific embeddings (via vector database representations), such transfer would be impossible.

Why hierarchical memory: The three-level hierarchy (Strategic → Procedural → Tool) mirrors the abstraction levels needed for complex task execution. Strategic Memory shapes the agent's overall approach (e.g., "always verify data integrity after transformations"), Procedural Memory provides known-good workflows for specific sub-tasks (e.g., "how to create a GitLab issue"), and Tool Memory optimizes primitive actions (e.g., "after clicking, always call browser_update"). This separation allows each memory type to be retrieved and updated at the appropriate granularity: Strategic Memory is always in context (it's short enough to fit), Procedural Memory is retrieved on demand based on sub-task relevance, and Tool Memory is automatically appended to observations. A flat memory structure would force the agent to either load everything (exhausting context) or use a single retrieval mechanism that must serve both high-level strategic queries and low-level procedural queries—a much harder retrieval problem.

Why a separate Reflect Agent instead of self-evaluation: Self-evaluation is notoriously unreliable for LLMs—models tend to be overconfident in their own outputs and often fail to detect their own errors. By making the Reflect Agent an independent entity (separate LLM calls, separate reasoning context, with its own ability to interact with the environment), the paper creates a genuine cross-evaluation mechanism. The Reflect Agent is not asked "did you do this correctly?" (which invites confirmation bias) but rather receives the PE Agent's trajectory as evidence and is asked to independently verify whether the stated goals were met, using environmental ground truth where available. This is a more robust evaluation paradigm, though it comes at the cost of additional LLM calls (the Reflect Agent's evaluation and memory distillation require their own inference compute).

Why minimal tools: As discussed in Section 3.4.2, the minimal toolset forces procedural learning at the right level of abstraction. If the agent had a create_gitlab_issue tool, the SOP would be trivial ("call create_gitlab_issue with these parameters") and would transfer zero knowledge to any other application. By forcing the agent to compose workflows from browser primitives, the learned SOPs encode generalizable interaction patterns that apply across applications and environments. This is validated by the generalization experiment (Section 4.3.2), where memory learned on one set of tasks improves performance on entirely different tasks that share similar cross-application interaction patterns but use different specific applications.

4. Key Insights and Innovations

Innovation 1: Autonomous Experience Distillation Without Human Labels or Ground Truth

The most intellectually distinctive contribution of MUSE is not the memory architecture per se—hierarchical memory systems, procedural knowledge bases, and reflection mechanisms all have precedents—but rather the demonstration that an agent can autonomously convert raw, unlabeled execution trajectories into structured, reusable knowledge at scale, enabling genuine test-time learning on tasks exceeding 100 steps without any human annotation, ground-truth answer checking, or model fine-tuning.

What the field did before this innovation. Prior work on agent self-improvement has relied on one of several scaffolds that MUSE explicitly avoids. Reflexion (Shinn et al., 2023) uses ground-truth answer comparison to generate verbal feedback—the agent reflects by checking whether its output matches a known correct answer, which assumes the existence of labeled evaluation data. ExpeL (Zhao et al., 2024) collects execution trajectories and refines them into natural language insights, but its evaluation is on text-based benchmarks with known correct answers, not on open-ended productivity tasks where "correctness" is multi-dimensional and must be inferred from environmental state. Voyager (Wang et al., 2023) builds a skill library from Minecraft interactions, but it relies on the game engine's structured feedback (inventory changes, block placement confirmations) rather than having to autonomously determine success from raw environmental observations. Agent Workflow Memory (Wang et al., 2024) extracts reusable workflows but has been validated only on simpler web agent benchmarks. Across this literature, the dominant assumption has been that extracting usable knowledge from experience requires either human labels, ground-truth answers, or structured environment signals.

Why MUSE's approach is a fundamental shift, not an incremental refinement. MUSE's Reflect Agent evaluates sub-task success using criteria that require reasoning about evidence rather than matching against labels. The three-dimensional checklist—truthfulness verification, deliverable verification, data fidelity—is not comparing the agent's output to a gold standard. It is asking: (1) Are the agent's conclusions grounded in observations actually present in the trajectory, or did it hallucinate? (2) Do the promised artifacts exist and are they complete? (3) Did data survive transformations intact? These questions can be answered by a second LLM instance with access to the same environment and the execution history, without any prior knowledge of what the "correct" answer should be.

This is fundamentally different from self-evaluation against ground truth. It is cross-evaluation against evidence, and it works because the Reflect Agent is an independent reasoning instance—separate LLM calls, separate context—that can actively probe the environment to verify claims. The Reflect Agent's ability to use the same toolset as the PE Agent (Section 3.4) means it doesn't just read the trajectory; it can open files, check file sizes, navigate to URLs, and query the current state of the environment to independently confirm or refute what the PE Agent reported. This active verification capability is what makes autonomous experience distillation feasible on tasks where no ground-truth evaluation function exists.

The evidence for this claim is the continuous learning experiment (Figure 3). The agent improved by over 10 percentage points across three iterations on T_cl with zero human intervention—no one labeled which sub-tasks succeeded, no one provided correct answers, no one curated which trajectories should be distilled. The Reflect Agent autonomously determined success/failure, distilled successful trajectories into SOPs, and the accumulated knowledge drove measurable performance gains. This is a closed-loop learning system that operates entirely at inference time.

Significance beyond the metric gain. This innovation has implications that extend well beyond the TAC benchmark. It suggests a path toward agents that can be deployed in genuinely novel environments—where no labeled dataset exists and no reward function can be pre-specified—and still improve through practice. The Reflect Agent's evaluation criteria (truthfulness, deliverable verification, data fidelity) are domain-agnostic: they apply to any task where outputs can be inspected and claims can be traced to evidence. This is a conceptual advance in how we think about agent evaluation: from "compare to a gold label" to "verify against environmental evidence," which is far more scalable and general.

The limitation, which the paper is transparent about, is that the Reflect Agent can still make evaluation errors—it is itself an LLM, subject to hallucinations and reasoning failures. The paper doesn't quantify the Reflect Agent's accuracy, but the fact that the overall system improves monotonically (Figure 3) suggests the evaluation signal is net positive despite inevitable noise.

Innovation 2: Procedural Knowledge Generalizes Across Tasks When Learned at the Right Level of Abstraction

The paper's second major insight is that procedural knowledge transfers zero-shot to novel tasks, but only when learned at the level of primitive tool compositions rather than task-specific solutions. This is not merely a claim that memory helps—it is a claim about what kind of memory helps, and why prior approaches may have failed to demonstrate strong generalization.

What the field assumed before this work. The implicit assumption in much agent memory research is that experience is most useful when it closely matches the current task—an SOP for "send a direct message on RocketChat" helps when you need to send a direct message on RocketChat. The more interesting question is whether experience helps on different tasks, and prior work has provided limited evidence either way. ExpeL and Agent Workflow Memory evaluate on the same task distributions they learn from; they don't test zero-shot transfer to held-out tasks. Voyager's skill library transfers within Minecraft but to the same types of challenges (crafting, building, exploration). The field lacked clear evidence that procedurally-learned knowledge generalizes across substantially different tasks in a realistic productivity environment.

MUSE's diagnostic contribution. The generalization experiment (Section 4.3.2, Table 1) provides strong evidence for cross-task transfer, but the more interesting question is why it works. The paper's design choices point to an answer: generalization emerges from the interaction of three features:

Minimal toolset forcing compositional learning. Because MUSE uses only general-purpose tools (browser, Python, shell, vision), any learned SOP is necessarily a composition of primitive actions. "Create a GitLab issue" is learned as a sequence of browser navigation, clicking, and text input operations—not as a call to a specific API. This means the SOP encodes interaction patterns (navigate to project page, locate the "New Issue" button, fill form fields, verify creation) that transfer to other web-based tools with similar interfaces, even if the specific URLs and UI elements differ. The paper validates this implicitly: memory accumulated on the T_cl tasks (which cover six different professional roles using different applications) improves performance on the T_hard tasks (which involve different applications and workflows). The knowledge that transfers is not "here's how to do task X" but rather "here's how to navigate web interfaces, verify file creation, handle login flows, and recover from UI interaction failures."
Hierarchical memory separating strategy from procedure from tool-use. Strategic Memory (M_strat) captures principles like "always verify data integrity after transformations" and "when information is unavailable, explicitly mark it as 'Not Available' rather than hallucinating"—these are genuinely task-agnostic behavioral heuristics that apply regardless of the specific application or workflow. The fact that the entire M_strat is loaded into the system prompt means every task execution is conditioned on these principles. This is a form of behavioral prior that shapes the agent's default approach to all novel situations, not just those it has encountered before.
Procedural Memory organized by application rather than by task. By indexing SOPs by application and function (e.g., "FileSystem → Verify File Existence") rather than by task (e.g., "Task 47 → Step 3"), the memory structure encourages compositional reuse. When a novel task requires verifying that a file exists, the agent can retrieve the FileSystem verification SOP regardless of why the file needs to be verified. This is a deliberate design choice that treats SOPs as reusable building blocks rather than monolithic task recipes.

Evidence and its interpretation. Table 1 shows that memory accumulated from only the T_cl subset (18 tasks, ~10% of the full benchmark) boosts performance on the entirely disjoint T_hard subset (12 tasks) from S_partial = 23.65% to 33.41%—a 41% relative improvement. This is not incremental; it's the difference between barely functional and moderately capable. Moreover, the T_hard tasks were specifically chosen because strong models like Claude-4 Sonnet achieve "little to no score" on them (Section 4.3.2). The fact that experience from easier tasks helps on harder tasks—tasks that the agent has never seen and that frontier models largely fail on—is the strongest evidence the paper provides for genuine generalization.

However, there is an important caveat: the T_hard tasks share the same underlying environment (the same TAC corporate software stack) as T_cl. The generalization is across tasks within the same environmental context, not across entirely different environments (e.g., from TAC to OSWorld or WebArena). The paper doesn't claim otherwise, but readers should understand that "generalization" here means generalization to new task specifications within a fixed environment, not generalization to fundamentally different software stacks or interaction paradigms.

Significance beyond the benchmark. This finding challenges a common intuition that experience-based approaches merely memorize solutions to specific problems. MUSE demonstrates that when experience is stored at the right level of abstraction—compositional rather than monolithic, indexed by function rather than task, and supported by general behavioral principles—it yields transferable capabilities. This has practical implications for how agent memory systems should be designed: the key design decisions are not just what to store (the memory content) but how to organize it (the indexing scheme) and what level of abstraction to encode it at (the tool granularity). These are architectural choices that the paper gets right in ways that prior work did not fully address.

Innovation 3: Deliberate Procedural Amnesia During Retries as an Exploration Mechanism

One of the paper's most counterintuitive and intellectually interesting design choices is the retry mechanism that explicitly disables Procedural Memory on the second attempt. When a sub-task fails on the first try (using retrieved SOPs), the retry drops the constraint to follow known procedures, forcing the agent to explore novel action sequences. This is not an accident or a compromise—it is a deliberate mechanism for preventing the agent from getting trapped by incorrect or stale procedural knowledge.

The problem this solves. Memory systems face a fundamental tension between exploitation (using known-good procedures) and exploration (discovering better procedures when the known ones fail). A naive memory system that always retrieves and follows the most relevant SOP would be brittle: if an SOP is incorrect, outdated, or inapplicable to the current situation (because the environment changed or the task is subtly different), the agent would repeatedly fail by following bad advice. This is a well-known problem in reinforcement learning (the exploitation-exploration dilemma) and in case-based reasoning, but the paper addresses it with a simple, domain-general mechanism rather than a learned policy.

The retry mechanism works as follows (Section 3.3): attempt 1 uses Procedural Memory (exploitation of known knowledge). If it fails, attempt 2 explicitly forbids Procedural Memory use, forcing the agent to explore novel approaches (exploration). If attempt 2 also fails, the agent escalates to sub-task replanning rather than attempting a third time with the same approach. This three-stage escalation—exploit → explore → restructure—mirrors how a human might approach a stubborn problem: first try the standard approach, then try something creative, then fundamentally reconsider whether the problem is correctly framed.

Why this is distinctive. Most memory-augmented agent systems treat memory retrieval as an unalloyed good: more relevant memories always improve performance. MUSE's design acknowledges that memory can be harmful when it encodes stale or contextually inappropriate knowledge. The retry mechanism is a form of targeted forgetting that serves an epistemic function: it prevents the agent from over-committing to procedures that may have worked in the past but don't apply to the present situation.

The paper provides indirect evidence for this mechanism's importance through the ReSTEM experiment (Appendix K, Figure 16). When the revision model was further optimized using on-policy data collection (which would reinforce existing procedural patterns), performance with sequential revisions degraded substantially, falling from ~38.5% at the optimal ratio to ~33.5% with fully sequential revisions. The authors hypothesize that "on-policy data collection in ReSTEM exacerbates spurious correlations in revision data." This is a different system (the revision model, not MUSE), but the principle is the same: reinforcing existing procedural knowledge can backfire when that knowledge encodes fragile or environment-specific patterns. MUSE's retry mechanism is a guardrail against this failure mode.

Significance beyond the immediate mechanism. The deliberate disabling of memory during retries suggests a broader principle for memory-augmented agents: memory systems should include forgetting mechanisms that are as carefully designed as their storage mechanisms. In MUSE, forgetting is not passive (memories fading over time) but active and conditional (memories are suppressed when they prove unreliable for the current sub-task). This is a conceptual contribution to the design of agent memory architectures—a recognition that memory quality management is as important as memory quantity, and that sometimes the best use of experience is to intentionally ignore it and try something new.

Innovation 4: The TAC Benchmark as a Diagnostic for Deployable Agent Capabilities

While not a methodological contribution of MUSE itself, the paper's choice of TAC as an evaluation platform—and the results it produces—constitutes an important diagnostic contribution to the field's understanding of what current agent architectures can and cannot do. The paper demonstrates that TAC exposes capability gaps that are invisible on simpler benchmarks, providing a more honest assessment of where agents stand relative to the demands of real-world deployment.

The diagnostic finding. Table 2 shows that even frontier models (Claude-4 Sonnet, Gemini-2.5 Pro) integrated into capable agent frameworks (OpenHands) achieve only 30–43% S_partial on the full TAC benchmark, with perfect completion rates of 30–33%. This is on a benchmark designed to simulate corporate productivity tasks—exactly the kind of work that agent systems are being marketed to automate. The gap between benchmark performance and deployable reliability is enormous: an agent that successfully completes only one-third of assigned tasks is not ready for autonomous deployment in any consequential setting.

More telling is the performance on the T_hard subset (Table 1): OpenHands with Gemini-2.5 Pro achieves S_partial = 3.00%, and OpenHands-versa with Claude-4 Sonnet achieves 2.00%. These are near-zero scores from top-tier models on tasks that are "hard" but not adversarial—they are realistic productivity tasks that happen to require multi-step reasoning, cross-application coordination, and adaptation to unexpected obstacles. The fact that frontier agents essentially completely fail on these tasks, while MUSE with a lighter model achieves S_partial = 33.41%, demonstrates that architecture, not model scale, is the binding constraint on these tasks.

What this reveals about the field's evaluation practices. Prior to TAC (and contemporaneous benchmarks like OSWorld), agent evaluation was dominated by benchmarks that were either (a) single-domain (code generation, question answering, web navigation), (b) short-horizon (typically under 20 steps), or (c) both. The paper's results on TAC suggest that these benchmarks systematically overestimate agent capabilities by failing to capture the compounding error problem in long-horizon, cross-application tasks. An agent that achieves 80% on WebArena tasks (typically 5–15 steps on a single website) may achieve 10% on TAC tasks (40+ steps across 3+ applications) because errors compound multiplicatively across extended interaction sequences.

MUSE's performance—51.78% S_partial, a 20% relative improvement over the prior SOTA—is not just a metric gain. It demonstrates that architectural innovations (hierarchical memory, autonomous reflection, experience accumulation) can close capability gaps that model scaling alone cannot. Claude-4 Sonnet is almost certainly a more capable base model than Gemini-2.5 Flash across most standardized benchmarks, yet MUSE + Gemini-2.5 Flash substantially outperforms OpenHands + Claude-4 Sonnet on TAC (51.78% vs. 43.19%). This is evidence that the field's focus on model capabilities (bigger models, better pretraining) may be misallocated relative to the importance of agent architecture for complex, interactive tasks.

Significance for the research community. The paper provides a concrete reference point for what "hard" means in agent evaluation and what level of performance current systems achieve. This serves a calibrating function: when researchers claim their agent system "solves" a benchmark, the TAC results provide context for how much harder real-world productivity tasks are. The fact that no system achieves even 60% S_partial on TAC—despite using frontier models and sophisticated architectures—suggests that the field is far from having deployable autonomous agents for knowledge work, and that the path forward requires architectural innovations (like MUSE's experience-driven learning) rather than just larger models.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use TheAgentCompany (TAC) benchmark (Xu et al., 2024), comprising 175 tasks designed to simulate a high-fidelity corporate environment (Section 4.1). The benchmark evaluates autonomous language agents across six core employee positions (HR, PM, SDE, etc.), requiring them to execute interconnected operations using a suite of applications—chat clients, cloud storage, code editors, project management software, spreadsheets—within a fully functional operating system. The key challenge is task complexity: completing a task requires over 40 action steps on average, frequently spanning two or more applications, demanding long-horizon planning, multi-step reasoning, and cross-platform information integration. For the continuous learning experiments, the authors curated a subset of 18 tasks (T_cl) sampled to ensure coverage across all six professional roles, while for generalization experiments they curated a 12-task hard subset (T_hard) on which even strong models like Claude-4 Sonnet achieve "little to no score" (Section 4.3.2). Tasks on both subsets were manually inspected to guarantee correctness and exclude cases with obvious evaluation errors. The full 175-task benchmark is used for the main SOTA comparison.
Base model(s). The primary experiments use Gemini-2.5 Flash (Comanici et al., 2025) for both the PE Agent and the Reflect Agent (Section 4.2). This is a lightweight closed-source model, chosen deliberately to test whether architectural improvements can compensate for model scale—the paper explicitly contrasts its performance against larger frontier models (Claude-4 Sonnet, Gemini-2.5 Pro) running on competing frameworks. For the open-source adaptability experiment (Section 4.4.2), the core model is replaced with DeepSeek-V3-250324 (Liu et al., 2024). NPCs in the TAC environment are powered by GPT-4o. The vision extractor tool (gpt4o_describe_image, browser_extract_content_by_vision) also uses GPT-4o as its backbone.
Metrics. The primary evaluation metric is the average partial completion score (S_partial) across all evaluated tasks, as defined by the official TAC protocol (Section 4.2). The scoring formula is:

$S_{partial} = 0.5 \cdot \frac{Completed\_ckpt}{Total\_ckpt} + 0.5 \cdot S_{full}$

where Completed_ckpt / Total_ckpt represents the fraction of intermediate checkpoints achieved (capturing partial progress even when the task is not fully completed), and S_full ∈ {0, 1} is a binary indicator of full task completion. The rationale for this weighted metric is that it rewards partial progress—an agent that completes 4 out of 6 checkpoints but fails the overall task still receives a non-zero score, which is appropriate for long-horizon tasks where complete failure on early steps can obscure genuine capability differences. The paper also reports an aggregate checkpoint score (S_ckpt), which is simply the proportion of completed checkpoints relative to total checkpoints across all tasks (e.g., 465/776 for the full benchmark means the agent completed 465 out of 776 total checkpoints across all 175 tasks). Additionally, the Perfect Completion Rate (PCR) is reported for the full benchmark, indicating the fraction of tasks that are fully solved (S_full = 1).

Baselines. The paper compares against multiple baselines, each representing a different point in the agent design space:
- OpenHands (Wang et al., 2024): A leading open-source agent platform evaluated with Gemini-1.5 Pro, Gemini-2.0 Flash, and Gemini-2.5 Pro (Table 2), and with open-source models Llama-3.1 405B, Llama-3.3 70B, and Qwen-2.5 72B (Table 4). OpenHands provides a comparison point for a well-engineered but memory-less agent framework.
- OpenHands-versa (Soni et al., 2025): An extension of OpenHands that adds multimodal browsing capabilities, evaluated with Claude-3.7 Sonnet and Claude-4 Sonnet (Table 2). This is the previous SOTA on TAC and represents the state of the art in model scaling for agent tasks.
- OWL-RolePlay (Hu et al., 2025): A multi-agent framework evaluated with GPT-4o + o3-mini (Table 2).
- MUSE without memory (MUSE w/o mem): An internal ablation using the full PE + Reflect architecture but with an empty Memory Module (Tables 1, 3, 4). This isolates the contribution of accumulated experience from the contribution of the base framework design.
- OpenHands with open-source models (Table 4): Used as baselines for the open-source adaptability experiment, contrasting MUSE + DeepSeek-V3 against OpenHands with Llama-3.1 405B, Llama-3.3 70B, and Qwen-2.5 72B.
Generation budget / compute accounting. The paper controls for computational cost primarily through action limits rather than token counts or FLOPs. Each sub-task attempt is capped at N = 20 actions (Section 4.2), meaning the PE Agent can execute at most 20 ReAct steps (each involving an LLM call to generate a thought and action) before the Reflect Agent intervenes. A failed sub-task gets exactly one retry opportunity, also capped at 20 actions. This creates a maximum of 40 actions per sub-task across two attempts. The number of sub-tasks varies per task depending on the PE Agent's decomposition. The paper does not explicitly report total LLM calls or token counts, which is a limitation—the Reflect Agent's evaluation and memory distillation also consume inference compute, and these costs are not factored into comparisons with baselines that don't have reflection components. For the continuous learning experiments, the agent processes all 18 tasks in T_cl three times sequentially, so the total compute grows linearly with iterations (3× the single-pass cost), but performance is measured at each iteration, making the comparison self-contained.
Cross-validation / statistical protocol. The continuous learning experiment (Section 4.3.1) conducts five complete runs with different random seeds and reports average scores—this is the only explicit mention of statistical control (Section 4.3.1: "To mitigate randomness, we conduct five complete runs of this experiment and report the average scores"). For the full benchmark evaluation (Section 4.3.3) and the generalization experiment (Section 4.3.2), the paper does not report multiple runs or confidence intervals, implying single-run results. The Memory Module is accumulated over three iterations on T_cl and then frozen for evaluation on T_hard and the full benchmark—meaning the evaluation conditions are deterministic given the frozen memory state. No cross-validation is used for strategy selection, as MUSE doesn't have tunable hyperparameters that vary across tasks (unlike the compute-optimal scaling paper's difficulty-conditioned policies). The 18-task T_cl subset and 12-task T_hard subset are manually curated, not randomly sampled, and the selection criteria (moderate difficulty for T_cl, near-zero baseline scores for T_hard) are explicitly stated.

Main Quantitative Results

Continuous Learning: Performance Improves Monotonically with Experience Accumulation

The continuous learning experiment (Section 4.3.1, Figure 3) tests the central hypothesis that MUSE can progressively improve on repeated tasks by accumulating experience, simulating how humans get better with practice. Using the T_cl subset of 18 tasks, the agent runs three sequential iterations with no human intervention, carrying accumulated knowledge forward between iterations.

Headline result: Both S_ckpt and S_partial grow steadily and monotonically across the three iterations, with the final iteration outperforming the memory-less baseline by over 10 percentage points (Figure 3). The memory-less baseline—Gemini-2.5 Flash evaluated on T_cl without the Memory Module—represents the agent's innate capability before any experience accumulation.

Interpreting the trajectory (Figure 3). The figure shows two metrics (checkpoint score in blue, partial completion score in orange), each with a dashed baseline (no memory) and a solid line tracking improvements across the three iterations. Both solid lines are monotonically increasing and sit above their respective baselines at all iterations after the first. The exact numerical values are not tabulated in the main text, but the figure visually indicates that the improvement is substantial—the paper describes it as "over 10%" in the text, and the visual separation between the final solid line and the dashed baseline is clear.

Why monotonic improvement matters. Monotonic improvement is a strong signal that the learning process is stable—the agent is not oscillating between effective and ineffective strategies, nor is it degrading due to accumulation of bad experiences. The Reflect Agent's success/failure evaluation and the post-task memory refinement (deduplication, generalization) appear to be maintaining memory quality such that each iteration adds net positive knowledge. This is not trivial: a naive memory system that stored all trajectories indiscriminately could easily degrade performance by flooding the agent with irrelevant or contradictory information.

What the 10%+ gain represents. The paper attributes this improvement to "the accumulated experience, which enables the agent to avoid previously failed exploration paths and thus focus more directly on effective solutions" (Section 4.3.1). This is a specific claim about the mechanism: memory reduces redundant exploration, freeing up the action budget for deeper exploration of remaining uncertainties. In tasks capped at 20 actions per sub-task, avoiding even 5–10 wasted exploratory actions can be the difference between reaching the sub-task goal and hitting the action limit.

Limitations of this result. The improvement is measured on the same 18 tasks across iterations—this is by design (the experiment simulates practicing the same tasks), but it means the result demonstrates improvement through practice on familiar tasks, not generalization to novel tasks. The generalization experiment (Section 4.3.2) addresses the latter. Additionally, only five runs were conducted, and the paper doesn't report variance across runs, so the reliability of the "over 10%" figure cannot be assessed. The experiment also doesn't include a control where the agent simply gets more actions per task without memory—it's possible that some of the improvement is due to the agent having more total compute across iterations (since each iteration involves processing all 18 tasks, and later iterations might benefit from simply having seen the environment more), though the paper's attribution to "avoiding previously failed exploration paths" suggests a memory-specific mechanism.

Generalization: Accumulated Experience Transfers Zero-Shot to Hard, Unseen Tasks

The generalization experiment (Section 4.3.2, Table 1) tests whether memory accumulated from practicing on one set of tasks (T_cl, 18 tasks, moderate difficulty) improves performance on an entirely different set of tasks (T_hard, 12 tasks, high difficulty) that the agent has never seen. This is a zero-shot transfer test—the Memory Module is frozen after three iterations on T_cl and then evaluated on T_hard without any further updates.

Headline result: MUSE with memory achieves S_partial = 33.41% on T_hard, compared to S_partial = 23.65% for MUSE without memory—a 41% relative improvement (Table 1). The corresponding checkpoint scores are 40.68% (S_ckpt with memory, 24/59 checkpoints) vs. 30.51% (without memory, 18/59 checkpoints).

Comparison to frontier agent baselines (Table 1, top rows). The baseline results contextualize how difficult T_hard is:

OpenHands-versa with Claude-4 Sonnet: S_ckpt = 5.08% (3/59 checkpoints), S_partial = 2.00%
OpenHands with Gemini-2.5 Pro: S_ckpt = 8.47% (5/59 checkpoints), S_partial = 3.00%

These are near-zero scores from top-tier models in capable agent frameworks, confirming that T_hard tasks represent a regime where model scaling alone fails. Even MUSE without memory—which achieves S_partial = 23.65% using only Gemini-2.5 Flash—substantially outperforms these baselines, demonstrating that the MUSE architecture (PE + Reflect agents with minimal tools) provides intrinsic advantages over OpenHands even before memory accumulation. The paper attributes this to "the effectiveness of the synergy between our PE and Reflect Agents" (Section 4.3.2).

What transfers, and why. The paper claims this result provides "strong evidence for the zero-shot generalization capability of the knowledge acquired through our framework, indicating that MUSE learns transferable and generalizable memory, rather than merely remembering task-specific solutions" (Section 4.3.2). The claim has face validity: the T_cl and T_hard tasks are disjoint and were selected using different criteria (moderate difficulty vs. near-zero baseline scores). For memory to help on T_hard, it must encode knowledge that applies beyond the specific tasks it was derived from.

The likely mechanism is the multi-level memory abstraction: Strategic Memory (M_strat) encodes domain-agnostic behavioral principles (e.g., "always verify data integrity after transformations," "explicitly mark unavailable information as 'Not Available'"), Procedural Memory (M_proc) encodes composable SOPs indexed by application and function (e.g., "FileSystem → Verify File Existence," "RocketChat → Login"), and Tool Memory (M_tool) encodes optimized single-tool usage patterns. Even if T_hard tasks involve different overall workflows than T_cl tasks, they operate in the same corporate software environment (same applications, same toolset), so the application-level SOPs and tool-use patterns transfer.

Caveats. The transfer is within the same environment (TAC's simulated corporate software stack). The paper does not claim or test transfer to entirely different environments (e.g., OSWorld or WebArena). "Generalization" here means generalization to new task specifications within a fixed environment, which is a meaningful but bounded form of transfer. Also, the 12-task subset is small (12 tasks, 59 total checkpoints), so the percentage scores are based on a limited sample—a single checkpoint represents ~1.7% of the checkpoint score. The paper doesn't report confidence intervals or variance across the five runs for this experiment (the five-run protocol was only mentioned for the continuous learning experiment).

Full Benchmark: New State-of-the-Art by a Substantial Margin

The full benchmark evaluation (Section 4.3.3, Table 2) tests MUSE on the complete TAC benchmark of 175 tasks, using the Memory Module accumulated from three iterations on T_cl (frozen during evaluation). This is the headline result that establishes MUSE's SOTA claim.

Headline result: MUSE achieves S_partial = 51.78% with a checkpoint score of S_ckpt = 59.92% (465/776 checkpoints) and a Perfect Completion Rate of 41.14% (Table 2). This represents the first time any agent has surpassed the 50% S_partial threshold on TAC, and outperforms the previous SOTA (OpenHands-versa with Claude-4 Sonnet) by a relative ~20% (51.78% vs. 43.19%).

Side-by-side comparison with prior SOTA (Table 2). The most instructive comparison is:

Framework	Model	Checkpoints	`S_ckpt`	`S_partial`	PCR
OpenHands-versa	Claude-4 Sonnet	392/776	50.52%	43.19%	33.14%
MUSE	Gemini-2.5 Flash	465/776	59.92%	51.78%	41.14%

MUSE completes 73 more checkpoints (465 vs. 392), achieves 8.59 percentage points higher S_partial in absolute terms, and fully solves 8% more tasks (41.14% vs. 33.14% PCR)—all while using a substantially lighter model (Gemini-2.5 Flash vs. Claude-4 Sonnet). The checkpoint improvement is particularly informative because it reflects partial progress even on tasks that aren't fully solved—MUSE is not just completing more tasks, but is making more progress on the tasks it fails.

Comparison with other baselines (Table 2). The performance hierarchy is clear:

OWL-RolePlay (GPT-4o + o3-mini): S_partial = 11.04%, PCR = 4.00%—multi-agent coordination without memory falls far short.
OpenHands variants: Performance scales with model capability, from Gemini-1.5 Pro (S_partial = 8.02%) to Gemini-2.0 Flash (18.96%) to Gemini-2.5 Pro (39.28%). The jump from 2.0 Flash to 2.5 Pro is substantial (20.32 percentage points), reflecting Gemini-2.5's strong reasoning capabilities. However, even Gemini-2.5 Pro under OpenHands achieves only 39.28% S_partial—well below MUSE with the weaker 2.5 Flash model.
OpenHands-versa with Claude-3.7 Sonnet: S_partial = 40.18%, PCR = 30.86%—the multimodal browsing capability adds ~5 percentage points over standard OpenHands with Gemini-2.5 Pro.
MUSE: 51.78%—a clear jump above all other entries.

Interpreting the model efficiency. The fact that MUSE + Gemini-2.5 Flash outperforms OpenHands-versa + Claude-4 Sonnet by such a wide margin (51.78% vs. 43.19%) is the paper's strongest argument that architecture matters more than model scale for complex agentic tasks. Claude-4 Sonnet is almost certainly a more capable base model than Gemini-2.5 Flash on most standardized benchmarks. But on TAC, MUSE's experience-driven architecture extracts more value from a weaker model than OpenHands-versa extracts from a stronger one. This has practical implications: for organizations building agent systems, investing in memory and reflection architectures may yield greater returns than paying for more expensive model APIs.

The role of pre-accumulated memory. The paper notes that the Memory Module used for this evaluation was "acquired from only approximately 10% of the available tasks" (the 18-task T_cl subset out of 175 total tasks). This is a striking efficiency claim: experience from 10% of the benchmark transfers to improve performance across the remaining 90%, including hard tasks the agent has never seen. However, this claim is tempered by the fact that the T_cl tasks were specifically chosen for moderate difficulty (where models achieve non-zero scores), which may represent a sweet spot for learning transferable knowledge—tasks too easy teach nothing, tasks too hard yield no successful trajectories to learn from.

Complete task-level results (Table 10). The appendix provides per-task checkpoint and S_partial scores for all 175 tasks, enabling detailed analysis of where MUSE succeeds and fails. Tasks with 100% S_partial are spread across multiple domains (admin, HR, PM, SDE), while tasks with 0% S_partial cluster in specific challenging categories: sde-implement-raft-in-go (0/10 checkpoints), sde-implement-covering-index-in-janusgraph (0/3), finance-apply-tax-credit (0/8), admin-mass-forms-filling (0/5), ds-answer-numerical-data-question (0/6). These failures suggest limitations in domains requiring deep technical implementation (distributed systems algorithms, database internals) or mass automation (form filling at scale), which may exceed the capabilities of the minimal toolset and LLM-driven reasoning.

Open-Source Model Adaptability: Memory Is Model-Agnostic

The model adaptability experiment (Section 4.4.2, Table 4) tests whether the MUSE architecture works with open-source models and whether memory accumulated by one model transfers to another.

Headline result on T_cl with DeepSeek-V3 (Table 4): MUSE without memory achieves S_partial = 28.01% (29/85 checkpoints), already outperforming all OpenHands + open-source model combinations (Llama-3.1 405B: 9.78%, Llama-3.3 70B: 5.84%, Qwen-2.5 72B: 6.50%). With the pre-accumulated Memory Module (originally built using Gemini-2.5 Flash), performance jumps to S_partial = 36.75% (43/85 checkpoints)—a 31% relative improvement.

Key insights from this experiment. Two findings stand out:

The MUSE architecture alone provides substantial benefits: Even without memory, MUSE + DeepSeek-V3 achieves 28.01% S_partial, compared to 9.78% for OpenHands + Llama-3.1 405B (a much larger model). This validates that the PE + Reflect agent design and the minimal toolset approach contribute value independent of memory accumulation.
Memory transfers across models: The memory was accumulated using Gemini-2.5 Flash during three iterations on T_cl, then loaded by the DeepSeek-V3-based agent. The 8.74 percentage point improvement (28.01% → 36.75%) demonstrates that natural language memory is genuinely model-agnostic. This is a practically important finding: an organization could use a powerful (and expensive) model to accumulate high-quality experience, then deploy a cheaper model that benefits from that experience for routine execution.

Limitations. The experiment is conducted only on the 18-task T_cl subset, not the full benchmark. The improvement from memory is substantial but the absolute performance (36.75%) is well below what Gemini-2.5 Flash achieves with the same memory (~55.85% from Table 3), suggesting that model capability still matters—memory amplifies capability but doesn't substitute for it entirely. The paper doesn't report whether a DeepSeek-V3-native memory (accumulated by DeepSeek-V3 itself through iterations on T_cl) would outperform Gemini-2.5 Flash's transferred memory, which would test whether memory quality is model-dependent.

Ablation Studies and Robustness Checks

Reflect Agent removal: Substantial performance drop confirms reflection is not redundant.

The ablation study removing the Reflect Agent (Section 4.4.1, Table 3) tests whether the reflection mechanism is necessary or whether the PE Agent alone can achieve similar performance. Both configurations operate without the Memory Module to isolate the reflection contribution.

On the T_cl subset, the full MUSE (PE + Reflect, no memory) achieves S_partial = 55.85% (56.2/85 checkpoints on average across five runs), while the no-reflection variant achieves only S_partial = 43.21% (54/85 checkpoints). The 12.64 percentage point gap is substantial.

What drives the difference. The Reflect Agent serves two functions during execution: (1) it catches hallucinations and incomplete sub-task executions by verifying against environmental evidence, and (2) it enables the retry mechanism by providing diagnostic failure analyses. Without reflection, the PE Agent must self-evaluate its sub-task completion, which is known to be unreliable for LLMs—models frequently claim success when they've actually failed to achieve the stated goal. The 12.64 point gap suggests that self-evaluation errors are common enough on TAC tasks to significantly degrade performance.

The interaction with memory is not tested in this ablation. Table 3 shows only the no-memory comparison. It would be informative to see whether the Reflect Agent becomes more or less important when memory is present—one might hypothesize that good SOPs reduce the need for reflection by providing proven procedures, but alternatively, reflection might be essential for creating those SOPs in the first place. The paper doesn't disentangle these effects.

Open-source model comparison: MUSE architecture outperforms OpenHands across the board.

Table 4 provides an implicit ablation by replacing the Gemini-2.5 Flash backbone with DeepSeek-V3 while keeping the MUSE architecture constant, then comparing against OpenHands with various open-source models.

The results are not a controlled ablation (model and framework vary simultaneously in the OpenHands comparisons), but they establish that MUSE's advantages persist across model families. The finding that MUSE without memory (DeepSeek-V3) achieves 28.01% S_partial vs. 9.78% for OpenHands (Llama-3.1 405B)—a model with roughly an order of magnitude more parameters—suggests the MUSE framework architecture is a larger contributor to performance than model scale within this regime.

Negative result: ReSTEM experiment degrades revision performance (referenced in Section 5, Discussions).

The paper briefly mentions in the Discussions (Section 5) that "fine-tuning methods suffer from computational intractability" and "RL-based approaches are hindered by the design of rewards that are both extremely sparse and difficult to formulate." While not a formal ablation, this framing implies that the authors considered and rejected alternative learning paradigms, and the systematic failure of baseline agents on T_hard (Table 1) provides indirect support—if fine-tuning or RL were straightforward solutions, frontier models wouldn't achieve 2–3% S_partial on these tasks.

Missing but relevant ablations. Several experiments would strengthen the paper's claims but are not reported:

Memory scaling ablation: How does performance scale with the amount of accumulated memory? Currently, memory is accumulated over three iterations on 18 tasks and then frozen. An ablation varying the number of training tasks (e.g., 5, 10, 18 tasks) or training iterations (1, 2, 3, 5 iterations) would reveal whether performance is saturating or still improving.
Per-memory-type ablation: How much does each memory type (Strategic, Procedural, Tool) contribute individually? The paper shows the combined system works, but doesn't ablate individual memory components.
Reflect Agent accuracy measurement: The Reflect Agent's evaluation accuracy is never quantified. If the Reflect Agent has a high false-positive rate (labeling failures as successes), memory could be contaminated with incorrect SOPs. The monotonic improvement in Figure 3 suggests this isn't a catastrophic problem, but the magnitude of evaluation error is unknown.
Action limit sensitivity: The sub-task action limit N = 20 is fixed. An ablation varying this limit would show whether the benefits of memory are amplified or diminished under tighter or looser action budgets.

Critical Assessment

Central claim: MUSE achieves new SOTA on TAC by a significant margin. This claim is well-supported by the full benchmark results (Table 2). The 51.78% S_partial is substantially above the previous SOTA of 43.19%, and the checkpoint-based scoring (465/776 vs. 392/776) confirms that the improvement is not an artifact of the S_partial formula. The fact that this is achieved with a lighter model (Gemini-2.5 Flash) against a heavier baseline (Claude-4 Sonnet) strengthens the claim by ruling out model scale as the explanation.

However, the comparison is not perfectly controlled. MUSE and OpenHands-versa differ along multiple dimensions: MUSE has a hierarchical memory system, a Reflect Agent, a different toolset (minimal general-purpose vs. potentially richer), and a different prompting paradigm. The SOTA claim is valid—MUSE achieves the highest reported number—but the causal attribution to any specific component (memory vs. reflection vs. toolset design) is not isolated. The ablation in Table 3 isolates the Reflect Agent's contribution without memory, but doesn't isolate memory's contribution while controlling for the PE + Reflect architecture.

Central claim: The agent exhibits continuous learning and self-evolution. This claim is supported with qualifications. Figure 3 shows monotonic improvement across three iterations on the same 18 tasks, which is a valid demonstration of learning from experience. However, three iterations is a small number to establish a trend, and the experiment is on only 18 tasks (~10% of the benchmark), which were specifically selected for moderate difficulty. The claim of "self-evolution" implies open-ended improvement, but the experiment shows improvement across three fixed iterations on a fixed task set—it doesn't demonstrate that improvement would continue indefinitely or that the agent can learn genuinely new capabilities beyond refining its approach to known task types.

The more impressive claim—that this learning transfers to hard, unseen tasks—is supported by the generalization experiment (Table 1), but the transfer is within the same environment. The T_hard tasks operate on the same software stack as T_cl, just with more complex requirements. True generalization to novel environments or fundamentally different task distributions is not tested.

Central claim: Experience accumulation leads to increasingly superior task completion capabilities. This claim is supported directionally but the magnitude and limits of the effect are under-characterized. The 10+ percentage point improvement across three iterations (Figure 3) and the 41% relative improvement on T_hard (Table 1) are both substantial. However, several factors complicate interpretation:

The baseline without memory is already strong: MUSE without memory achieves 23.65% on T_hard and 55.85% on T_cl (Table 3). This means the PE + Reflect architecture alone is doing substantial work, and memory provides an incremental (though significant) boost on top of that. The paper frames memory as the central innovation, but an equally valid reading is that the PE + Reflect architecture is the primary driver, and memory amplifies it.
The memory was accumulated on a curated subset: The T_cl tasks were selected for moderate difficulty, ensuring the agent would have successful experiences to learn from. If the agent were deployed on a random mix of tasks (including many very hard ones), the quality of accumulated memory might be lower (fewer successes, more noisy failure analyses). The experiment demonstrates that MUSE can learn from experience under favorable conditions, but doesn't establish robustness to the task distribution it learns from.
The improvement saturates quickly, but we don't know when: Three iterations show monotonic improvement, but the paper doesn't report whether a fourth or fifth iteration would continue improving or plateau. The continuous learning curve might be approaching an asymptote determined by the agent's innate capabilities (the model's reasoning limits, the toolset's expressiveness).

The weakness that matters most: the evaluation platform shares the same environment for training and testing. All generalization in this paper is within the TAC environment—same applications (RocketChat, GitLab, OwnCloud, Plane), same interaction conventions, same toolset. This is a fair test of whether MUSE learns reusable procedural knowledge within a domain, but it does not test whether MUSE learns domain-independent skills that would transfer to entirely different productivity environments. A skeptic could argue that MUSE is learning TAC-specific heuristics (e.g., "RocketChat's login button is finicky, always verify with browser_update after clicking") that wouldn't transfer to a different chat application. The paper's claim that "MUSE learns transferable and generalizable memory" (Section 4.3.2) is supported for within-domain transfer but not for cross-domain transfer.

Unquantified costs. The paper measures performance in terms of task completion metrics but doesn't report inference costs (total LLM calls, total tokens, wall-clock time, or API costs). The Reflect Agent doubles the LLM calls per sub-task step (since both PE and Reflect agents make LLM calls), and the post-task memory distillation adds further overhead. MUSE's architectural improvements come at a computational cost that is not accounted for in the comparisons. If MUSE costs 5× more per task than OpenHands but only achieves 1.2× the performance, the cost-adjusted comparison might be less favorable. The paper would be strengthened by reporting total inference compute alongside accuracy.

What would strengthen the paper. Several experiments would address the gaps identified above: (1) A cross-environment transfer test—train memory on TAC, evaluate on a different productivity benchmark (e.g., OSWorld tasks that require similar file management and communication skills but use different applications). This would distinguish domain-specific procedural learning from genuinely transferable strategic knowledge. (2) A cost-normalized comparison—measure S_partial per 1000 LLM calls or per dollar of API cost, to account for MUSE's additional inference overhead. (3) A memory scaling curve—vary the number of training tasks and iterations to characterize how performance improves with the amount of accumulated experience, and whether it saturates. (4) A Reflect Agent accuracy benchmark—manually evaluate a sample of Reflect Agent success/failure judgments against human judgments to quantify evaluation reliability. (5) An ablation that isolates the contribution of each memory type (Strategic vs. Procedural vs. Tool) to the overall performance gain. (6) Evaluation on the full benchmark with memory accumulated from the full benchmark itself (not just 10% of tasks), to establish an upper bound on what MUSE can achieve with comprehensive experience.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost for Experience Accumulation Is Unaccounted for in Headline Efficiency Claims

The assumption or constraint. MUSE's continuous learning and generalization gains depend on accumulating high-quality experience through repeated task execution. Each experience accumulation cycle involves: (1) the PE Agent executing sub-tasks (up to 20 actions per attempt, with retries), (2) the Reflect Agent independently evaluating each sub-task using environmental verification (which itself consumes LLM calls and tool interactions), and (3) the Reflect Agent performing post-task memory distillation across all three memory types (Section 3.4). The paper's headline numbers—the 10+ percentage point improvement across three iterations (Figure 3), the 41% relative improvement on T_hard (Table 1), and the 51.78% SOTA S_partial (Table 2)—are all measured after this experience has been accumulated, without amortizing the cost of accumulation into the performance metric. The paper does not report the total number of LLM calls, total tokens consumed, or wall-clock time for either the experience accumulation phase or the evaluation phase.

The consequence. A practitioner considering whether to deploy MUSE needs to understand the total cost of achieving the reported performance, not just the evaluation-time cost. The Reflect Agent approximately doubles the LLM inference per sub-task (since both PE and Reflect agents make independent LLM calls during evaluation), and the post-task memory distillation adds further overhead. If MUSE requires 5× more total inference compute than a memory-less baseline to achieve a 1.2× performance improvement, the cost-adjusted comparison may be substantially less favorable. This is particularly acute for the SOTA claim (Table 2): MUSE's 51.78% S_partial is compared against OpenHands-versa's 43.19%, but the total compute required to accumulate the Memory Module (three iterations on 18 tasks, each with PE + Reflect + distillation) is not factored into this comparison. A cost-normalized metric (e.g., S_partial per 1000 LLM calls or per dollar of API cost) would reveal whether MUSE's architectural complexity is justified by net efficiency gains or whether a simpler system with a larger compute budget would achieve comparable results.

What evidence exists in the paper. The paper provides no explicit accounting of inference costs—no mention of total LLM calls, token counts, API pricing, or wall-clock time in any experiment. Section 4.2 specifies only the per-sub-task action limit (N = 20) and the model used (Gemini-2.5 Flash), but not the total number of sub-tasks per task, the number of Reflect Agent calls, or the cost of memory distillation. The continuous learning experiment (Figure 3) runs three iterations on 18 tasks with five complete runs, implying substantial total compute, but the paper does not quantify this. The paper's argument that MUSE "not only improves efficiency by streamlining the LLM's context but also enables the agent to achieve an unprecedented depth of exploration" (Section 4.3.1) addresses context efficiency but not total inference cost.

Mitigation status. The paper does not acknowledge this limitation or propose any cost accounting methodology. The decision not to report inference costs is a significant omission given that the Reflect Agent and memory distillation are central architectural components whose costs should be amortized against the performance gains they enable. Future work should report cost-normalized metrics and investigate whether lighter-weight reflection mechanisms (e.g., selective rather than universal reflection, or smaller models for the Reflect Agent) can preserve the learning benefits at reduced cost.

6.2 The Reflect Agent's Accuracy Is Never Measured, Creating an Unquantified Risk of Memory Contamination

The assumption or constraint. MUSE's entire learning loop depends on the Reflect Agent correctly distinguishing successful sub-task executions from failed ones. The Reflect Agent evaluates each sub-task using a three-dimensional checklist (truthfulness verification, deliverable verification, data fidelity) and outputs a binary success/failure flag f that determines whether the trajectory is distilled into Procedural Memory (if f = success) or triggers replanning (if f = failure) (Section 3.4). The paper implicitly assumes that the Reflect Agent's evaluations are sufficiently accurate to support learning. However, the Reflect Agent is itself an LLM (Gemini-2.5 Flash) subject to the same hallucinations, reasoning errors, and overconfidence that plague LLM-based evaluation in general. Its environmental verification capability is bounded by the same toolset the PE Agent uses, meaning it can fail to detect errors if its verification strategy is incomplete or if it accepts the PE Agent's claims without sufficiently rigorous cross-checking.

The consequence. Two failure modes arise from inaccurate reflection. First, false positives (the Reflect Agent labels a failed sub-task as successful) lead to incorrect SOPs being stored in M_proc. These contaminated SOPs then mislead the PE Agent on future tasks, potentially causing systematic failures when the agent follows a confidently retrieved but actually incorrect procedure. The retry mechanism's deliberate disabling of M_proc on the second attempt (Section 3.3) partially mitigates this by providing an escape hatch when an SOP fails, but it does not prevent the initial wasted attempt or the propagation of incorrect knowledge to other tasks where the SOP is retrieved. Second, false negatives (the Reflect Agent labels a successful sub-task as failed) waste the successful trajectory—a potentially valuable SOP is not created—and trigger unnecessary replanning, consuming additional compute. Over multiple iterations, systematic biases in the Reflect Agent's evaluation (e.g., being too strict about deliverable formatting, or too lenient about truthfulness) could skew the accumulated memory toward certain types of solutions while suppressing others.

What evidence exists in the paper. The paper provides no direct measurement of the Reflect Agent's accuracy. No human evaluation of Reflect Agent judgments is reported, no inter-annotator agreement is computed, and no comparison against a ground-truth success criterion is performed. The continuous learning experiment (Figure 3) provides indirect evidence that the net effect of reflection is positive—performance improves monotonically across three iterations—but this doesn't rule out a substantial error rate. A system with, say, 20% false positives and 10% false negatives could still show net improvement if the 70% of correct evaluations (and successful retries on false negatives) outweigh the harm from contaminated memory. The 12.64 percentage point drop when removing the Reflect Agent entirely (Table 3, 55.85% → 43.21%) establishes that reflection is better than no reflection, but doesn't quantify how close the Reflect Agent is to optimal evaluation.

Mitigation status. The paper does not acknowledge the lack of Reflect Agent accuracy measurement as a limitation. The Reflect Agent's design—independent LLM calls, separate reasoning context, active environmental verification—represents a thoughtful attempt to make evaluation more reliable than naive self-evaluation, but without accuracy measurements, the residual error rate is unknown. Future work should benchmark the Reflect Agent against human judgments on a sample of sub-task trajectories, measure the false positive and false negative rates, and investigate whether ensembling multiple Reflect Agent calls or using a stronger model for reflection improves evaluation reliability.

6.3 Generalization Is Demonstrated Only Within a Single Environment and Software Stack

The assumption or constraint. All experiments in the paper operate within the TAC benchmark environment—a simulated corporate software stack comprising specific applications (RocketChat, GitLab, OwnCloud, Plane, specific file systems, specific spreadsheets) with fixed interaction conventions. The generalization experiment (Section 4.3.2, Table 1) demonstrates zero-shot transfer from the T_cl task subset to the T_hard task subset, which the paper characterizes as evidence that "MUSE learns transferable and generalizable memory, rather than merely remembering task-specific solutions" (Section 4.3.2). However, both T_cl and T_hard operate on the same underlying software environment with the same applications, same UI conventions, and same toolset. The paper does not test whether memory accumulated in TAC transfers to a different productivity environment (e.g., OSWorld tasks using different file managers, different chat clients, different code editors) or to a different class of interactive tasks entirely (e.g., web navigation benchmarks like WebArena).

The consequence. The paper's claim of "generalization" is bounded to within-domain transfer—the agent learns to navigate one corporate software ecosystem more effectively, but there is no evidence that it learns domain-independent skills. An SOP for "RocketChat → Navigate to Home Page" encodes the specific URL structure, button labels, and page layout of RocketChat; it provides no guidance for navigating a different chat application with a different interface. Strategic Memory principles like "Explicitly manage and continuously verify data and execution context" (Table 6: Robust Context State) are phrased abstractly and could transfer across environments, but the paper provides no evidence that they do. A practitioner deploying MUSE in a different software environment would need to accumulate experience from scratch in that environment, with no guarantee that the strategic principles abstracted from a different environment would transfer. The "zero-shot" improvement on T_hard is zero-shot with respect to task specifications but not with respect to the environment—the agent has seen (and accumulated memory about) the same applications during its training iterations on T_cl.

What evidence exists in the paper. The generalization experiment (Table 1) is the sole evidence for transfer. It shows that memory from 18 moderate-difficulty tasks (T_cl) improves performance on 12 hard tasks (T_hard) from 23.65% to 33.41% S_partial. This is a valid demonstration of within-environment generalization, but the paper's language ("transferable and generalizable memory," "strong evidence for the zero-shot generalization capability") implies a broader form of transfer than what is actually tested. The per-task breakdown in Table 10 confirms that T_cl and T_hard tasks use the same applications (RocketChat, GitLab, OwnCloud, etc.); the tasks differ in specification complexity, not in the underlying software stack. No cross-benchmark experiment is reported or proposed.

Mitigation status. The paper does not acknowledge this as a limitation or scope its generalization claims to within-environment transfer. The Discussion (Section 5) does not mention cross-environment generalization as an open question. A fairer characterization of the results would be: MUSE demonstrates strong compositional reuse of procedural knowledge within a fixed software ecosystem, but its ability to transfer strategic or procedural knowledge across ecosystems with different applications and interaction paradigms remains untested. Future work should evaluate MUSE on a multi-benchmark suite where the software stack varies across benchmarks, measuring whether Strategic Memory principles and generalized procedural patterns survive environment shifts.

6.4 The Performance Evaluation Relies on Small, Curated Task Subsets with Limited Statistical Rigor

The assumption or constraint. The paper's key experimental results—continuous learning (Figure 3), generalization (Table 1), and the Reflect Agent ablation (Table 3)—are all evaluated on small, manually curated task subsets rather than the full 175-task benchmark. The continuous learning subset T_cl contains 18 tasks (Section 4.3.1), the hard-task subset T_hard contains 12 tasks (Section 4.3.2), and both were selected through manual inspection based on specific criteria (moderate difficulty with non-zero baseline scores for T_cl; near-zero baseline scores for T_hard). The paper reports five-run averaging only for the continuous learning experiment; the generalization experiment and the full benchmark evaluation appear to be single-run results. No confidence intervals, standard deviations, or statistical significance tests are reported for any result.

The consequence. Small, curated subsets limit the statistical reliability and generalizability of the findings in several ways. First, the 18-task T_cl subset represents only ~10% of the full benchmark. A 10 percentage point improvement on 18 tasks could reflect genuine learning, or it could reflect that the specific 18 tasks happen to benefit disproportionately from the types of experience accumulated—particularly since they were selected for moderate difficulty, which the paper's own framework suggests is the sweet spot where test-time learning is most effective (analogous to the compute-optimal scaling paper's finding that test-time compute helps most on easy-to-medium problems). Second, the 12-task T_hard subset has only 59 total checkpoints, meaning each checkpoint represents ~1.7% of the S_ckpt metric—a few lucky or unlucky checkpoint completions could meaningfully shift the reported performance. Third, the manual curation process, while described with clear criteria, introduces human judgment into the subset selection, potentially biasing the subsets toward tasks where MUSE's approach is most favorable. Fourth, without variance estimates, it's impossible to assess whether the 51.78% SOTA S_partial is reliably above the 43.19% prior SOTA or whether the difference could be explained by run-to-run variability.

What evidence exists in the paper. The paper acknowledges the subset sizes implicitly by reporting them, and describes the selection criteria (Section A.1: "we chose tasks of moderate difficulty, specifically those where models generally achieve non-zero scores" for T_cl; "we deliberately selected tasks where most models fail almost completely" for T_hard). The five-run averaging for the continuous learning experiment (Section 4.3.1) is explicitly stated, but no variance metrics are reported from these runs. For the full benchmark evaluation (Table 2) and the generalization experiment (Table 1), the number of runs is not specified, and the paper appears to report single-run deterministic results (the Memory Module is frozen after three training iterations, and evaluation is a single pass with that frozen memory). The complete per-task results in Table 10 enable partial verification—one can see which specific tasks MUSE succeeds and fails on—but without multiple evaluation runs, the stability of these per-task scores is unknown.

Mitigation status. The paper acknowledges in the Discussion (Section 5) that "some task descriptions can be ambiguous or contain inaccuracies" and that "evaluation scripts for certain tasks are rigid and do not account for the full range of valid solutions," indicating awareness of benchmark limitations. However, it does not address the statistical limitations of small evaluation subsets or the absence of variance reporting. A stronger evaluation would include: (1) multi-run evaluation with reported confidence intervals for all key results, (2) a sensitivity analysis showing how much the performance gain varies across different random subsets of comparable size, (3) evaluation on the full benchmark for the continuous learning experiment (or at minimum, a larger random sample), and (4) a clear distinction between findings that are robust at the reported sample sizes versus those that require larger-scale validation.

6.5 The Memory System Has No Mechanism for Forgetting or Correcting Contaminated Knowledge

The assumption or constraint. MUSE's Memory Module is cumulative by design—new experience is added through immediate Procedural Memory creation after sub-task success and through post-task distillation across all three memory types, with deduplication and generalization performed to maintain conciseness (Section 3.2, Section 3.4). However, the paper describes no mechanism for removing or correcting memory entries that were generated from trajectories that the Reflect Agent incorrectly evaluated as successful (false positives). Once an incorrect SOP enters M_proc, or an incorrect strategic principle enters M_strat, the paper provides no evidence that the system can detect and purge it. The post-task refinement process mentions "deduplication" and "generalization" but not error correction or contradiction resolution.

The consequence. Over extended deployment, the Memory Module faces a contamination risk that grows with the number of tasks executed. If the Reflect Agent has even a modest false positive rate (say, 10%), then after processing hundreds of sub-tasks, M_proc could accumulate dozens of incorrect SOPs. These contaminated entries compete with correct ones during retrieval: when the PE Agent queries access_guide for a relevant SOP, it may retrieve an incorrect procedure that superficially matches the current sub-task. The retry mechanism provides one layer of defense—if the incorrect SOP leads to failure on the first attempt, the retry explicitly disables M_proc, allowing the agent to discover a correct approach independently (Section 3.3). But this defense is costly (wasting the first attempt and up to 20 actions) and doesn't prevent the incorrect SOP from being retrieved again on future tasks. For M_strat, the problem is potentially more severe because the entire strategic memory is loaded into the system prompt at initialization (Section 3.2)—an incorrect strategic principle (e.g., a principle that emerged from a false positive evaluation and encodes a flawed heuristic) influences every action the agent takes, not just those where the agent chooses to query memory.

What evidence exists in the paper. The continuous learning experiment (Figure 3) shows monotonic improvement over three iterations, which suggests that contamination is not catastrophic at this scale (18 tasks × 3 iterations = 54 task executions). However, the experiment is too small and too short to stress-test the memory system's robustness to contamination. Three iterations may not be enough for incorrect SOPs to accumulate to a density that measurably degrades performance. The paper provides no analysis of memory quality—no measurement of how many stored SOPs are actually correct, no comparison of the Reflect Agent's success/failure judgments against ground truth, and no experiment that deliberately introduces incorrect memory entries to test robustness. The ReSTEM failure mentioned in the original revision model's appendix (where on-policy data collection "exacerbated spurious correlations" and degraded performance) demonstrates a parallel failure mode in a different system, but MUSE's own vulnerability to similar effects is not evaluated.

Mitigation status. The paper does not acknowledge memory contamination as a risk or propose any error-correction mechanism. The retry mechanism's deliberate disabling of M_proc is a workaround for when an SOP fails, not a mechanism for detecting and removing the failed SOP from memory. The post-task refinement process (deduplication, generalization) could potentially identify contradictory SOPs (two different procedures for the same function) and flag them for resolution, but this capability is not described or demonstrated. Future work should address: (1) mechanisms for tracking SOP success/failure rates across uses and deprecating SOPs with low success rates, (2) Reflect Agent cross-validation where multiple independent evaluations are performed on the same trajectory before memory is created, (3) human-in-the-loop memory curation for high-stakes deployments, and (4) controlled experiments that measure performance degradation when incorrect memory is deliberately injected, to establish safety margins.

6.6 The Framework Has No Demonstrated Capability on Tasks Requiring Deep Domain Expertise or Genuinely Novel Problem-Solving

The assumption or constraint. MUSE improves agent performance by accumulating and reusing procedural knowledge (SOPs), strategic heuristics, and tool-use optimizations derived from past successful trajectories. This learning paradigm assumes that the agent's base model (Gemini-2.5 Flash) already possesses the fundamental reasoning capabilities needed to solve tasks, and that memory helps by reducing redundant exploration and avoiding known failure modes. The paper does not claim that MUSE can acquire genuinely new capabilities that exceed the base model's competence—the learning is about efficiency and reliability within the model's existing capability envelope.

The consequence. MUSE's approach fundamentally cannot help on tasks where the base model's innate capabilities are insufficient, even with perfect memory. The per-task breakdown in Table 10 reveals several tasks where MUSE scores 0% S_partial despite having accumulated memory: sde-implement-raft-in-go (0/10 checkpoints), sde-implement-covering-index-in-janusgraph (0/3), finance-apply-tax-credit (0/8), admin-mass-forms-filling (0/5), ds-answer-numerical-data-question (0/6), sde-implement-hyperloglog (1/6, 8.33%), and sde-implement-buffer-pool-manager-bustub (1/12, 4.17%). These tasks share a common characteristic: they require deep technical implementation (distributed consensus algorithms, database index design, tax code interpretation) that likely exceeds the base model's knowledge or reasoning capacity, not just its ability to navigate the environment. MUSE's memory system—no matter how well-populated with SOPs for RocketChat navigation or file verification—provides no traction on these tasks because the bottleneck is the underlying reasoning capability, not procedural efficiency.

This limitation mirrors the finding from the compute-optimal scaling paper that test-time compute provides "essentially zero benefit regardless of budget" on the hardest problems (difficulty bin 5), because "some capabilities can only be acquired through pretraining, not recovered at inference time." MUSE's experience-driven learning is a form of test-time optimization, and it inherits the same fundamental constraint: it amplifies existing capability but does not create it. For an organization considering deploying MUSE, this means the system's performance ceiling is bounded by the base model's competence on the hardest tasks in the deployment distribution, regardless of how much experience is accumulated.

What evidence exists in the paper. Table 10 provides per-task scores that clearly show the capability boundary. Of the 175 tasks, at least 8–10 show near-zero performance even with accumulated memory. The paper's Discussion (Section 5) acknowledges this implicitly: "We acknowledge that our current memory architecture is not a panacea and has limitations in handling specific tasks like high-level planning or multi-hop search." However, the paper does not analyze why these specific tasks fail or whether the failure is due to base model capability limits versus memory inadequacy. The continuous learning experiment (Figure 3) shows improvement on T_cl tasks, but T_cl was specifically curated for moderate difficulty—tasks where the base model already achieves non-zero scores. The experiment therefore demonstrates improvement within the capability envelope but provides no information about whether the envelope itself can be expanded through experience.

Mitigation status. The paper acknowledges the limitation in general terms (Section 5: "not a panacea") but does not provide a systematic analysis of where the capability boundary lies or what task characteristics predict success vs. failure. The natural mitigation for this limitation is outside the scope of MUSE's purely inference-time approach: tasks that exceed the base model's capabilities would require either a more capable base model (scaling pretraining), fine-tuning on domain-specific data, or human-in-the-loop assistance for the hardest sub-tasks. MUSE's architecture supports the latter—the paper notes that the Memory Module "facilitates the incorporation of human feedback" and "allows users to directly manage and revise stored experiences" (Section 5)—but this capability is not demonstrated. A more complete deployment strategy would combine MUSE's autonomous experience accumulation with a mechanism for escalating tasks that exceed the agent's capability to a more capable model or a human operator, using the Reflect Agent's failure analyses to identify when the boundary has been reached.

7. Implications and Future Directions

How This Work Changes the Landscape

MUSE establishes test-time experiential learning as a viable paradigm for complex agent tasks, shifting the conversation from model scaling to architecture design. Before this work, the dominant narrative in LLM agent research—implicit in the rapid succession of ever-larger models applied to agent benchmarks—was that better base models would naturally produce better agents. MUSE provides a counterexample of genuine significance: Gemini-2.5 Flash, a lightweight model, paired with MUSE's memory architecture achieves 51.78% S_partial on TAC, while Claude-4 Sonnet, a frontier-tier model, achieves only 43.19% under OpenHands-versa (Table 2). This is not a marginal improvement from clever prompting—it is a 20% relative gain that flips the expected model capability hierarchy. The implication is that for long-horizon, cross-application productivity tasks, architecture matters more than model scale within the current capability range of frontier LLMs.

This is not a paradigm shift in the Kuhnian sense—MUSE does not overturn the foundations of agent design—but it is a substantial reframing of where research effort should be allocated. The paper demonstrates that experience accumulation, autonomous reflection, and hierarchical memory organization can close capability gaps that model scaling alone cannot, at least on tasks within the base model's rough competence envelope. For the growing segment of the field focused on deploying agents in realistic productivity environments, this reframes the core research question from "how do we train better models?" to "how do we design systems that learn from their own execution traces?" The latter is fundamentally an architectural and systems engineering challenge, not a pretraining challenge.

The work reconciles a tension between memory-augmented agent research and real-world evaluation. Prior work on agent memory (ExpeL, Agent Workflow Memory, Mem0, Voyager) demonstrated promising results on simpler benchmarks—text-based QA, single-application web navigation, Minecraft—but the field lacked evidence that these approaches scaled to the complexity of genuine productivity work. MUSE provides that evidence by testing on TAC, a benchmark whose tasks average 40+ steps and span multiple applications, and demonstrating not just improvement over memory-less baselines but state-of-the-art performance. This validates the entire research direction of agent memory mechanisms at a level of rigor that prior evaluations could not. Conversely, it casts doubt on the sufficiency of evaluations on simpler benchmarks: if a memory system performs well on WebArena (5-15 step single-website tasks) but has never been tested on TAC-scale complexity, its real-world deployability is unproven. The paper raises the bar for what constitutes convincing evaluation of agent memory systems.

The paper redirects attention from tool integration to tool composition. A substantial line of agent research (ToolLLM, Gorilla, and numerous commercial agent platforms) focuses on integrating large libraries of specialized APIs, under the assumption that providing more tools makes agents more capable. MUSE takes the opposite approach—a deliberately minimal toolset of general-purpose primitives (browser, Python, shell, vision)—and demonstrates that the intelligence lies in composing these primitives into workflows. This is a methodological rebuke to the API-integration approach: if an agent can learn to use a browser and a code interpreter to accomplish tasks that span six different professional roles, then perhaps the field has overinvested in tool-specific engineering at the expense of general compositional reasoning. The generalization results (Table 1) support this: memory accumulated while composing browser primitives for T_cl tasks transfers to T_hard tasks precisely because the SOPs encode interaction patterns rather than API-specific invocations. The implication for the field is that investment in better compositional reasoning architectures may yield higher returns than investment in larger tool libraries.

The work identifies architectural components, not just aggregated performance, as the object of study. By reporting ablations (the Reflect Agent removal in Table 3, the open-source model comparison in Table 4) and providing detailed per-task results (Table 10), the paper enables the field to reason about which components drive performance rather than just whether the system works. The finding that removing the Reflect Agent causes a 12.64 percentage point drop on T_cl (Table 3) is as important as the headline SOTA number—it tells future researchers that autonomous verification is a high-value component worth optimizing, and that naive self-evaluation is insufficient. This component-level diagnostic approach, if adopted more broadly, would accelerate progress by enabling researchers to build on each other's component innovations rather than treating agent frameworks as monolithic black boxes.

Follow-Up Research This Work Enables

Cross-environment transfer: Does MUSE's memory generalize beyond the TAC software stack? The paper demonstrates zero-shot transfer from T_cl to T_hard within the same TAC environment (same applications, same interaction conventions). The natural next question is whether memory accumulated in one productivity environment transfers to a different environment with different applications. A concrete experiment: accumulate MUSE memory on TAC (using RocketChat, GitLab, OwnCloud), then evaluate the frozen memory on a subset of OSWorld tasks that require similar abstract workflows (sending messages, managing files, creating issues) but use different applications (e.g., Thunderbird instead of RocketChat, a different Git platform). This would distinguish whether MUSE is learning genuinely environment-independent procedural patterns (e.g., "when creating an issue, first navigate to the project, then locate the creation interface, then fill required fields") or TAC-specific UI heuristics (e.g., "RocketChat's login button is at index 3"). Strategic Memory principles ("Robust Context State," "Granular Outcome Verification") are phrased abstractly enough that they should transfer, but this has not been tested. A negative result (no transfer) would scope MUSE's claims to within-environment learning; a positive result would substantially strengthen the argument for generalizable procedural knowledge.

Reflect Agent accuracy benchmarking against human judgments. The Reflect Agent is the linchpin of MUSE's autonomous learning loop—its success/failure determinations control what enters memory. Yet the paper provides no measurement of how often the Reflect Agent agrees with human evaluators on sub-task success. A concrete follow-up: randomly sample 200 sub-task trajectories from MUSE's execution logs on TAC (covering both PE-declared successes and failures across difficulty levels), have 2-3 human annotators independently judge sub-task success using the same three criteria (truthfulness, deliverable verification, data fidelity), and compute the Reflect Agent's precision, recall, and F1 against the human majority vote. This would quantify the false positive rate (incorrect SOPs entering memory) and false negative rate (correct trajectories being discarded). If the false positive rate is high (>15-20%), it would explain why MUSE's improvement saturates and motivate investment in better reflection mechanisms (ensembling multiple Reflect Agent calls, using a stronger model for reflection, or incorporating human verification for high-stakes memory updates). If the false positive rate is low (<5%), it would validate the Reflect Agent's design as sufficiently reliable for autonomous learning.

Memory scaling laws: How does performance scale with the amount and diversity of accumulated experience? The paper shows improvement over three iterations on 18 tasks, but provides no scaling curve. How would performance change with 5 iterations? 10 iterations? 50 tasks? All 175 tasks? A concrete experiment: run MUSE on increasing fractions of the TAC benchmark (10%, 25%, 50%, 100% of tasks) for increasing numbers of iterations (1, 3, 5, 10), measuring S_partial on a held-out test set at each point. This would reveal whether MUSE's learning saturates (suggesting the base model's capability is the ceiling), continues to improve log-linearly (suggesting experience accumulation has compounding benefits), or eventually degrades (suggesting memory contamination overtakes learning). The paper's claim that memory from "approximately 10% of the available tasks" achieves near-SOTA performance (Section 4.3.3) implies diminishing returns to additional experience, but the shape of the curve is unknown. A saturation curve would have practical implications: deployers could estimate how much "practice" is needed before the agent reaches useful reliability, and stop accumulating experience when marginal gains become negligible.

Per-memory-type ablation to isolate Strategic, Procedural, and Tool Memory contributions. MUSE integrates three memory types, but the paper never evaluates them independently. Does Procedural Memory (M_proc) drive most of the gain, with Strategic and Tool Memory providing minor additional benefits? Or does the combination produce super-additive effects? A concrete experiment: evaluate MUSE on T_cl with (a) only Procedural Memory, (b) only Strategic Memory, (c) only Tool Memory, (d) all pairwise combinations, and (e) the full system. This would reveal which memory types are load-bearing and which are optional. If Strategic Memory alone provides substantial gains, it would validate the approach of loading high-level principles into the system prompt as a behavioral prior—a lightweight intervention that could be adopted by other agent frameworks without MUSE's full architecture. If Procedural Memory alone drives the gains, it would suggest that the key innovation is the SOP retrieval mechanism, and Strategic/Tool Memory might be simplified or removed to reduce system complexity.

Dynamic difficulty estimation and adaptive memory retrieval. MUSE currently retrieves SOPs based on a fixed index of application-function pairs, loaded at sub-task start. The agent must proactively decide when to query memory. A more sophisticated approach would estimate sub-task difficulty dynamically (analogous to the compute-optimal scaling paper's difficulty estimation) and adjust retrieval strategy accordingly: for sub-tasks that appear easy (the agent has high confidence), skip memory retrieval to save context; for sub-tasks that appear hard or unfamiliar, aggressively retrieve all potentially relevant SOPs. A concrete experiment: instrument the PE Agent to output a confidence score before each sub-task, then implement a policy that retrieves memory only when confidence is below a threshold. Measure whether this reduces total LLM context usage while preserving accuracy. This would address the unspoken cost concern—that loading the full SOP index for every sub-task wastes context on easy sub-tasks where the agent doesn't need guidance—and connect MUSE to the broader literature on adaptive test-time compute allocation.

Stress-testing memory robustness through deliberate contamination. The paper does not evaluate how MUSE behaves when its memory contains incorrect information—a scenario that will occur in any realistic deployment due to Reflect Agent errors. A concrete experiment: after accumulating a clean Memory Module over three iterations on T_cl, deliberately inject a known number of incorrect SOPs (e.g., an SOP that claims "to log into RocketChat, click the 'Settings' button first" when the correct action is to click 'Login'). Vary the contamination rate (0%, 10%, 25%, 50% of SOPs for each application) and measure the degradation in S_partial. This would reveal whether MUSE's retry mechanism (which disables M_proc on the second attempt) provides sufficient robustness, or whether incorrect SOPs cause irrecoverable failures. If performance degrades gracefully (linear with contamination rate), it would demonstrate that the retry mechanism effectively contains memory errors. If performance collapses at even low contamination rates, it would indicate that memory quality control is a critical vulnerability requiring additional safeguards (e.g., SOP success tracking, multi-evaluator consensus before memory creation, or periodic human auditing of high-use SOPs).

Practical Applications and Downstream Use Cases

Cost-efficient batch inference for enterprise productivity automation. Organizations running large-scale document processing, report generation, or cross-application workflow automation could deploy MUSE with a lightweight model (Gemini-2.5 Flash or an open-source alternative like DeepSeek-V3) and allow it to accumulate experience over the first few hundred task executions. The continuous learning curve (Figure 3) suggests that performance improves by 10+ percentage points within three iterations on repeated task types, after which the Memory Module can be frozen and reused. The model-agnostic memory transfer demonstrated in Table 4—where Gemini-2.5 Flash-accumulated memory boosts DeepSeek-V3 from 28.01% to 36.75% S_partial—means an organization could use an expensive model for the initial experience accumulation phase, then switch to a cheaper model for routine execution, amortizing the upfront cost across thousands of subsequent tasks. For an enterprise processing 10,000 productivity tasks per month, even a 10% improvement in task completion rate (from, say, 40% to 50%) represents 1,000 additional successfully automated tasks, directly reducing human workload.

Onboarding automation for new software environments. When an organization adopts a new software platform (a new project management tool, a new CRM, a new internal wiki), employees typically spend weeks learning its interface and conventions. MUSE could be deployed in the new environment with zero initial memory, allowed to practice on a curated set of representative tasks (analogous to T_cl), and within a few iterations accumulate SOPs for common workflows. The generalization experiment (Table 1) suggests that this experience would transfer to harder, unseen tasks in the same environment. The natural-language memory format means the accumulated SOPs could also serve as documentation for human employees—a Procedural Memory entry for "NewCRM → Create Customer Record" is essentially a step-by-step guide that a new hire could follow. This dual-use of memory (guiding both the agent and human workers) reduces the cost of software transitions and creates a living knowledge base that improves with use.

Self-improving data generation pipelines for agent fine-tuning. The paper's Discussion (Section 5) briefly envisions using MUSE's successful trajectories as training data for fine-tuning. This can be made concrete: an organization could deploy MUSE on a few hundred representative tasks, allow it to accumulate memory and improve its success rate (as in Figure 3), then export all sub-task trajectories that were evaluated as successful by the Reflect Agent. These trajectories—consisting of (task description, sub-task decomposition, action sequence, environmental observations)—form a high-quality dataset of procedural demonstrations. Fine-tuning a smaller, specialized model on this dataset could potentially encode the procedural knowledge directly into model weights, reducing inference-time reliance on memory retrieval and the Reflect Agent's overhead. The key advantage over traditional data collection is that MUSE's Reflect Agent provides automatic quality filtering (only successful trajectories are exported), and the experience accumulation loop means the quality of generated trajectories improves over time without human annotation. This is a practical path toward domain-specific fine-tuned agents that inherit MUSE's learned procedural knowledge.

Human-agent collaborative workflow optimization. MUSE's Memory Module architecture explicitly supports human inspection and editing of stored experience (Section 5: "The design allows users to directly manage and revise stored experiences"). This enables a deployment model where the agent handles routine sub-tasks autonomously using accumulated SOPs, but when the Reflect Agent detects a failure or the PE Agent encounters a genuinely novel situation, it escalates to a human operator. The human can then (a) complete the sub-task manually, (b) review the agent's attempted trajectory to understand what went wrong, and (c) directly edit or create an SOP in M_proc that encodes the correct procedure. This is more efficient than traditional human-in-the-loop approaches where humans must supervise every action—instead, humans only intervene at task boundaries and on failures, and their interventions directly improve the agent's future autonomy by populating its memory. Over time, the need for human intervention decreases as the memory base covers an increasing fraction of encountered workflows. The paper's result that memory from 10% of tasks improves performance on the remaining 90% (Section 4.3.3) suggests the human effort required to bootstrap such a system is modest relative to the automation gains.