Memory Transfer Learning: How Memories are Transferred Across Domains in Coding Agents

🎯 Pitch

Memories of test-driven verification routines from a machine learning task can help fix software bugs, boosting average performance by 3.7%, but copying raw code fragments from an unfamiliar domain actively hurts accuracy—abstraction is what makes cross-domain memory transferable.

1. Executive Summary

This paper presents the first systematic investigation of Memory Transfer Learning (MTL) for coding agents, challenging the prevailing assumption that memory utilization must be restricted to homogeneous task domains. Through experiments spanning 6 coding benchmarks—including LiveCodeBenchv6, SWE-Bench Verified, and MLGym-Bench—with GPT-5-mini as the base agent, the authors analyze how memories constructed in four representation formats (Trajectory, Workflow, Summary, and Insight) transfer across distinct task types when retrieved from a unified cross-domain memory pool. The study yields a headline improvement of 3.7% average performance over zero-shot baselines and demonstrates that abstraction dictates transferability: high-level Insight memories—which encode procedural meta-knowledge such as test-driven verification routines and environmental adaptation strategies—generalize effectively across domains, whereas low-abstraction Trajectory memories often induce negative transfer due to brittle implementation anchoring, establishing that cross-domain memory benefits coding agents primarily when memories are abstract enough to supply transferable behavioral guidance rather than domain-specific code.

2. Context and Motivation

The Core Problem: Coding Agents Are Walled Off From Valuable Cross-Domain Experience

The fundamental question this paper addresses is simple but largely unexplored: when a coding agent successfully solves a problem in one domain—say, debugging a machine learning pipeline—can the experience it gains help it solve a completely different kind of coding task, like fixing a software engineering bug? The existing paradigm of memory-based self-evolving agents answers this question with a de facto "no." As the authors state in Section 1:

"existing approaches typically restrict memory utilization to homogeneous task domains, failing to leverage the shared infrastructural foundations, such as runtime environments and programming languages, that exist across diverse real-world coding problems"

This is a significant blind spot. Consider what happens in practice: a coding agent deployed in a real-world software engineering environment encounters a wide spectrum of programming problems—repository-level bug fixes (SWE-Bench style), function-level competitive coding problems, machine learning model development, scientific paper code replication, and more. Despite their surface diversity, these tasks share a deep, underlying substrate: they all operate within Linux shell environments, interact with common programming languages (Python, C++, R), navigate cross-file dependency structures, and require similar meta-skills like test-driven verification, iterative debugging workflows, and environment adaptation. Yet current self-evolving agents treat each benchmark or task type as an isolated memory silo, preventing the agent from exploiting this shared foundation.

The practical consequence is that agents are forced to re-learn the same procedural knowledge from scratch in each new domain. An agent that has learned—through costly trial and error—that it should write inline test harnesses before submitting code changes in an ML competition gains no benefit from that hard-won insight when it later confronts a repository-level software engineering task, even though the same principle applies. The memory sits unused because it was generated in a "different domain." This is the gap that Memory Transfer Learning aims to close.

Why This Problem Matters: Efficiency, Generalization, and Deployment Practicality

The importance of this problem extends beyond academic curiosity into three concrete areas:

1. Sample efficiency and computational cost. Self-evolving agents improve by generating experiences (trajectories), extracting reusable knowledge from them, and applying that knowledge to future tasks. Each successful or failed trajectory represents a real computational investment—the agent spent time and FLOPs interacting with an environment. When memories are siloed by domain, the return on that investment is artificially capped: a trajectory generated on one benchmark provides zero benefit on another, even when the underlying task structures share common ground. MTL promises to amortize the cost of experience generation across a much wider range of downstream tasks, making self-evolution more computationally efficient.

2. Generalization as a proxy for real-world deployment. Real-world coding environments do not come with neat benchmark labels. A software engineering agent handling GitHub issues will encounter problems that blend elements of debugging (finding subtle logic errors), repository exploration (navigating unfamiliar codebases), environment configuration (fixing dependency issues), and algorithmic implementation—all in a single task. An agent that can only draw on memories from tasks that look superficially similar will miss relevant guidance from tasks that differ in surface form but share deep procedural structure. MTL tests whether agents can make these non-obvious connections, which is a more realistic test of their ability to generalize than evaluations within a single benchmark.

3. Democratizing agent capabilities. If cross-domain memory transfer works, a community of researchers and practitioners could potentially share abstract memory pools (e.g., collections of Insight memories encoding general coding best practices) that benefit agents regardless of the specific benchmark or deployment context. This would lower the barrier to building capable coding agents by reducing the need for extensive domain-specific experience generation. The authors hint at this in Section 4.4.4 when they show that memories generated by one model (e.g., GPT-5-mini) can improve the performance of a different, weaker model (e.g., Qwen3-Coder)—suggesting that meta-knowledge is model-agnostic and potentially shareable.

Where Prior Approaches Fall Short

The paper identifies three distinct categories of prior work, each with specific limitations that MTL addresses:

Single-domain self-evolving agents (the dominant paradigm). Most memory-augmented coding agents—including ReasoningBank (Ouyang et al., 2025), AWM (Wang et al., 2024c), ReMe (Cao et al., 2025), and MemEvolve (Zhang et al., 2025)—generate and retrieve memories exclusively within the same benchmark or task domain:

"existing memory-based self-evolving agents are primarily evaluated within the same benchmark or task domain, overlooking the potential value of memories generated from other task domains that may be highly beneficial to agent performance." (Section 2.2)

This is not a minor oversight. The authors show in Table 2 that ReasoningBank—which uses in-domain Insight memories—achieves only a 1.7% improvement over zero-shot (58.4% → 60.1% on the subset of three benchmarks where it's compared), while MTL with cross-domain Insight memories achieves a 4.6% improvement on the same subset (58.4% → 63.0%). The single-domain approach leaves substantial performance on the table because it cannot access the broader pool of relevant meta-knowledge. Moreover, single-domain memory pools are inherently limited in size: you can only generate so many trajectories on one benchmark before the returns diminish. Cross-domain pools aggregate experiences from many domains, naturally growing larger and more diverse (a property the authors explicitly validate in Section 4.4.3, showing that memory pool size and domain count both positively correlate with transfer effectiveness).

Unified memory pools without mechanistic analysis (AgentKB). AgentKB (Tang et al., 2025) represents the closest prior work to MTL. It constructs a large unified memory pool spanning multiple task types—including general reasoning, web interaction, and coding—and retrieves from this pool to support software engineering tasks. However, the authors identify two critical limitations:

"it does not provide a deeper analysis of the underlying mechanisms of memory transfer, including which forms of knowledge are transferable and how transfer-oriented memories should be generated in contrast to in-domain knowledge." (Section 2.3)

In other words, AgentKB demonstrates that cross-domain memory can help, but it doesn't explain why it helps or what kind of memory transfers best. This is analogous to the difference between showing that a drug works and understanding its mechanism of action—the former enables application, but the latter enables principled improvement. AgentKB leaves open the key design questions: Should I store raw trajectories or abstract insights? Should I prioritize memories from domains that are superficially similar to my target task, or is diversity more important? What kinds of failures should I expect when transferring memory across domains?

Furthermore, AgentKB's memory pool is extremely heterogeneous—mixing coding, web, and general reasoning tasks. The authors argue that this "miss[es] the opportunity to exploit coding-specific shared principles that are unique to programming tasks" (Section 2.3). By restricting the memory pool to coding-only domains (competitive coding, repository-level engineering, ML development, scientific code, DevOps), MTL analyzes transfer within a shared infrastructure (Linux shells, common languages, cross-file dependencies) where the potential for useful meta-knowledge transfer is highest. This domain focus allows the paper to isolate the abstraction-transfer relationship more cleanly than would be possible in a pool mixing fundamentally different task modalities.

Transfer learning without memory mechanisms. Traditional transfer learning in NLP relies on parametric adaptation—fine-tuning model weights on source domain data before evaluating on target domains (Howard & Ruder, 2018; Houlsby et al., 2019). In-context learning (Dong et al., 2024; Min et al., 2022) represents a non-parametric alternative where knowledge is provided directly in the prompt. However, these approaches differ from MTL in a crucial way: the knowledge being transferred is externally provided (human-curated examples or labeled data), not self-generated from the agent's own experience. In MTL, the agent is the source of its own transferable knowledge—it extracts memories from its own prior inferences, and these memories encode procedural and strategic insights that would be difficult to specify a priori. This connects MTL to the broader self-evolution paradigm while extending it beyond single-domain constraints.

How This Paper Positions Itself

The paper positions MTL as both a phenomenon to be understood and a principle to be exploited. This dual framing is important because it dictates the paper's structure and contributions.

As a phenomenon, MTL is the observation that coding agents can benefit from memories generated in different task domains—something prior work either ignored (single-domain approaches) or demonstrated without explaining (AgentKB). The paper systematically characterizes this phenomenon by measuring its magnitude (3.7% average improvement), its dependence on memory format (Insight > Summary > Workflow > Trajectory), its scaling properties (larger pools and more domains → better performance), and its failure modes (domain-mismatched anchoring, false validation confidence, misapplied best practices). This characterization fills the gap left by AgentKB's black-box demonstration.

As a principle, MTL is a design philosophy for memory-based agents: memories should be generated with cross-domain transfer in mind, prioritizing abstraction over specificity. The paper argues that existing memory generation pipelines, which focus on faithfully capturing what happened in a trajectory, produce memories that are often too brittle for transfer (Section 4.3.4's case study shows Trajectory leading the agent to execute incompatible R-language commands in a C++ project). Instead, memory generation should be oriented toward extracting generalizable insights from experience—a shift in emphasis that has direct implications for how self-evolving agents should be built.

The paper also positions itself relative to three open research questions it explicitly states in Section 1:

RQ1 (Does cross-domain memory help?): Answered affirmatively with quantitative evidence (Table 1, Table 2) showing consistent improvements across benchmarks and models.
RQ2 (Why does it help?): Answered through the meta-knowledge analysis (Figure 3) showing that transferred memories primarily supply procedural and behavioral guidance (test-driven verification, iterative workflow discipline, environmental adaptation) rather than task-specific code or algorithms.
RQ3 (What factors influence transfer effectiveness?): Answered through the abstraction-transfer correlation (Section 4.3), showing that higher-abstraction formats outperform lower-abstraction ones, that even within a single format task-agnostic memories outperform task-specific ones (Table 4), and that memory pool size and domain diversity scale with performance (Figure 6).

This tripartite structure—existence, mechanism, modulating factors—distinguishes the paper from prior work that either ignores cross-domain transfer entirely or demonstrates it without dissecting it. The authors explicitly frame this as a "first holistic investigation" (Section 5), establishing MTL as a new sub-area within memory-based self-evolving agents rather than merely a technique.

A Subtle but Important Distinction: Transfer vs. Retrieval

One aspect of the paper's positioning that deserves emphasis is the implicit distinction between memory transfer and memory retrieval. The paper does not claim that the memories being retrieved are specifically designed for the target task—indeed, by construction, the memory pool excludes any memories from the target benchmark. The transfer benefit comes from the fact that some fraction of memories from other domains encode knowledge that is applicable even though it wasn't intended for the current task. This is fundamentally different from fine-tuning or domain adaptation approaches where the transferred knowledge is curated for the target domain.

This distinction matters because it means the effectiveness of MTL is inherently probabilistic: you are retrieving from a pool of memories that were generated for other purposes and hoping that some of them are relevant. This explains why memory pool size matters (Figure 6)—a larger pool increases the probability that at least one of the retrieved top-N memories contains applicable meta-knowledge. It also explains why retrieval quality is a bottleneck (Section 4.4.5): even if useful memories exist in the pool, the retrieval mechanism must surface them. The authors show that simple embedding similarity actually outperforms more sophisticated methods like LLM reranking and task-adaptive memory rewriting (Table 7), suggesting that retrieval for cross-domain transfer is a non-trivial research problem in its own right—one that current static methods do not adequately solve.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

This paper builds a memory-augmented coding agent pipeline that stores experiences from solving coding tasks and later retrieves them to assist with new, different kinds of coding tasks. The system solves the problem of siloed experience: current agents only learn from tasks within the same benchmark or domain, wasting the shared procedural knowledge (how to test, how to debug, how to navigate unfamiliar codebases) that transfers across all programming tasks regardless of their surface differences.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a two-phase pipeline:

Coding Agent (mini-swe-agent + harbor) — the agent that solves coding tasks by interacting with a Linux shell environment through a sequence of reasoning steps, bash commands, and observations of command outputs. It produces full execution trajectories.
LLM Judge — evaluates each completed trajectory as a success or failure by comparing the final state against the benchmark's ground-truth criteria. This binary label determines which memory generation prompt template to use.
Memory Generator (offline) — takes the raw trajectory and its success/failure label, then prompts an LLM (GPT-5-mini) to extract a structured memory in one of four formats with varying levels of abstraction: Trajectory, Workflow, Summary, or Insight. Each format strips away different amounts of task-specific detail.
Memory Pool (cross-domain, indexed by embeddings) — aggregates all memories from all benchmarks except the target benchmark being evaluated. Each memory is embedded using OpenAI's text-embedding-3-small model and stored with its feature vector for similarity-based retrieval.
Memory Retriever (online, per-query) — at inference time, embeds the current task description (or a generated coding plan for some formats), computes cosine similarity against all stored memory embeddings, selects the top-N = 3 most similar memories, and injects them into the agent's system prompt before the agent begins solving the task.

Information flows in two phases. Phase 1 (offline memory construction): For each benchmark, the coding agent solves tasks → trajectories are labeled by the LLM judge → the memory generator produces structured memories → memories are embedded and added to the cross-domain pool. Phase 2 (online inference with memory): A new task arrives → the retriever embeds the task and fetches top-3 similar memories from the pool (excluding memories from the same benchmark) → the retrieved memories are prepended to the agent's system prompt → the agent solves the task using the memories as guidance.

3.3 Roadmap for the Deep Dive

First, the four memory formats (Trajectory, Workflow, Summary, Insight) and their generation procedures, because the format determines what information is preserved and at what level of abstraction — this is the core variable the paper manipulates.
Second, the memory pool construction and indexing strategy, including how embedding spaces differ across formats (Figure 4) and what this implies for retrieval behavior.
Third, the retrieval mechanism itself, including the crucial distinction between task-based embedding for Trajectory memories versus plan-based embedding for Workflow, Summary, and Insight — a design choice that reflects whether the memory contains an explicit task reference.
Fourth, the full experimental configuration: models, benchmarks, evaluation metrics, and the cross-validation logic that ensures fair comparison between zero-shot and MTL conditions.
Fifth, the mathematical formalization of the abstraction-transfer relationship (Appendix C), which provides a theoretical grounding for the empirical finding that abstraction drives transfer effectiveness.
Sixth, the controlled experiments and analysis framework — including how memory benefit categories were derived (Figure 3), how abstraction was isolated within a single format (Table 4), and how memory pool scaling was measured (Figure 6) — since these analytical methods constitute part of the technical approach.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that memories generated from heterogeneous coding task domains can improve agent performance on a target domain, and that the effectiveness of such transfer depends critically on the memory's level of abstraction — with more abstract formats (encoding procedural meta-knowledge) outperforming more concrete formats (encoding raw execution traces), because meta-knowledge avoids the brittle implementation anchoring that causes negative transfer.

The Coding Agent Infrastructure (mini-swe-agent + harbor)

The paper does not design a novel agent architecture; instead, it adopts an existing, well-tested coding agent stack to ensure that the results are not confounded by agent-specific artifacts. The agent used is mini-swe-agent (Yang et al., 2024), a streamlined version of SWE-agent designed for repository-level code editing tasks. The evaluation platform is harbor (Team, 2026), a container-based framework that launches isolated Linux environments for each task, executes the agent's bash commands, captures standard output and return codes, and enforces per-task time and resource limits. The base language model driving all components — the agent's reasoning, memory generation, and the LLM judge — is GPT-5-mini, used consistently throughout the main experiments (with DeepSeek V3.2 and Qwen3-Coder-480B-A35B-Instruct used only for cross-model transfer validation in Section 4.4.4).

The agent operates in a standard reasoning-action-observation loop. At each step, the agent produces a reasoning segment r_i (explaining what it intends to do and why), issues a bash command a_i (e.g., grep -rn "def aggregate" django/db/models/, cat <<'EOF' > fix.patch, or python -m pytest tests/test_aggregates.py), and receives an observation o_i from the harbor environment (the command's stdout, stderr, and return code). The full inference history for a task t is denoted as:

$H = (t, [(r_1, a_1, o_1), (r_2, a_2, o_2), \ldots, (r_n, a_n, o_n)])$

where t is the task description (a natural language prompt from the benchmark, such as "Fix the FieldError when an Aggregate contains a window expression in Django"), each r_i is a natural language reasoning step, each a_i is a bash command, each o_i is the captured shell output, and n is the total number of interaction steps in the trajectory.

What it captures: the complete, step-by-step record of how the agent approached and (successfully or unsuccessfully) resolved the task, including both its internal deliberation (r_i) and its external interactions with the environment (a_i, o_i).

Why this representation: the (r_i, a_i, o_i) triple structure is the standard format used across the self-evolving agent literature (Zheng et al., 2024; Wang et al., 2024c; Ouyang et al., 2025), making the paper's memory formats directly comparable to prior work. The inclusion of observations o_i is critical because it captures the environment's feedback — the agent can see not just what commands were issued but what happened when they were executed, which is essential for extracting procedural knowledge about how to safely interact with the shell.

LLM Judge: Binary Success/Failure Labeling

Before memory generation, each trajectory must be labeled as a success or failure because the memory generation prompts differ fundamentally between the two cases: success trajectories yield memories that encode "what to do" (positive patterns), while failure trajectories yield memories that encode "what to avoid" (anti-patterns). The paper uses an LLM-based judge — GPT-5-mini prompted to assess whether the final state of the trajectory satisfies the benchmark's task requirements. The exact prompts for this judge are not reproduced in the paper (they are referenced as following prior work Ouyang et al., 2025 and Cao et al., 2025), but the principle is standard: the judge receives the original task description and the final output of the trajectory and determines whether the task was completed correctly according to the benchmark's evaluation criteria.

Design choice: LLM judge vs. programmatic evaluation. Each benchmark already has a programmatic evaluation protocol (e.g., running unit tests for SWE-Bench Verified, checking output correctness for LiveCodeBench). The paper could have used these directly to label trajectories. However, the LLM judge approach has two advantages in this context: (1) it provides a unified labeling interface across all six benchmarks, simplifying the memory generation pipeline, and (2) it produces labels that can capture partial correctness or near-miss failures that binary test-pass/fail metrics might miss (though the paper does not explicitly analyze this granularity). The trade-off is that the LLM judge introduces its own errors — misclassifying some trajectories — but the paper does not report judge accuracy or analyze its impact.

Downstream use: the binary label determines which of two prompt templates is used for memory generation. For Workflow, Summary, and Insight formats, there are separate prompts for success and failure trajectories (e.g., "Workflow Generation Prompt for a Success Trajectory" vs. "Workflow Generation Prompt for a Failed Trajectory" in Appendix E). The failure prompts instruct the LLM to extract anti-patterns — "a bad or misleading strategy that led to failure" — rather than positive workflows, ensuring that memories encode both positive guidance and cautionary signals.

Memory Format 1: Trajectory

Trajectory memory is the least abstract format — it preserves the raw execution trace with minimal transformation. Formally, given the full inference history H, the trajectory memory M_T is constructed as:

$M_T = (t, [(a_1, o_1), (a_2, o_2), \ldots, (a_n, o_n)])$

where t is the original task description and each (a_i, o_i) pair is a command and its shell output, extracted from H by discarding the reasoning segments r_i.

What is preserved: every concrete bash command the agent executed and every observation the environment returned, in chronological order. This includes both successful commands (e.g., finding the right file with grep, applying a correct patch with cat <<'EOF' > file.py) and failed commands (e.g., a TypeError from a Python script, a missing dependency error). The reasoning r_i is stripped out because the goal is to provide low-level action traces that the agent can reference, not high-level strategic explanations.

What is discarded: the agent's internal deliberation about why it chose each command. This means the retriever must infer relevance purely from the surface similarity between the stored task t and the current task.

Retrieval mechanism for Trajectory: because Trajectory memories include the original task description t, the retrieval embedding is computed directly from t — the task description text is passed through text-embedding-3-small to produce a dense vector. The current task description is embedded identically, and cosine similarity is computed between the current task embedding and stored task embeddings. This assumes that tasks with similar natural language descriptions will benefit from similar command sequences — an assumption that makes sense for task-specific encoding but breaks down when the surface form of tasks differs across domains (e.g., a Django bug fix and an ML pipeline optimization will have very different task descriptions even if they share procedural best practices).

Memory generation procedure: Trajectory is the only format that does not involve an LLM for generation. It is a purely extractive transformation: concatenate commands and observations, strip reasoning, and store. The paper describes this as "we concatenate all commands and codes called by the agent a_i and their execution results o_i from H without reasoning sentences r_i" (Section 3.1.1). This makes Trajectory the cheapest format to generate (no additional LLM calls) but also the most brittle for transfer, as demonstrated in the results.

Memory Format 2: Workflow

Workflow memory extracts only the meaningful action subsequence from the full trajectory, discarding both reasoning and failed or irrelevant commands. The generation process uses an LLM (GPT-5-mini) prompted with the full history H and the task description t, along with format-specific instructions (the full prompts are in Appendix E, Figures 7–8).

Formally, Workflow memory M_W is:

$M_W = (g, [a_i, a_j, \ldots, a_k])$

where g is a natural language goal describing when this workflow should be applied (e.g., "Create a single final R source file containing implementations...") and [a_i, a_j, ..., a_k] is an ordered subsequence of the original commands a_1, ..., a_n, selected by the LLM as the "core strategy" or "key command steps that made the trajectory succeed" (for success cases) or the "incorrect or harmful pattern" (for failure cases).

What the LLM is instructed to do (success prompt):

Identify the goal of the workflow: "what kind of subproblem it solves and when it should be used."
Extract the sequence of bash commands that capture "the core strategy behind the success," focusing on "reusable patterns rather than one-off details."
Long commands may be shortened as long as the core action remains identifiable.
The output must be a valid JSON object with keys "goal" and "workflow" (an array of command strings).

What the LLM is instructed to do (failure prompt):

Identify the goal describing "what kind of mistake or failure pattern this workflow represents."
Extract commands illustrating "the incorrect or harmful pattern, rather than how it was fixed."
The output must similarly be a JSON object with "goal" and "workflow".

Key design difference from Trajectory: Workflow is explicitly curated — the LLM selects which commands are worth preserving and which are noise. This makes Workflow memory much shorter than Trajectory (typically 3–8 commands vs. 20–40), which the paper argues "leads to less danger of distractions from unrelated information." However, the curation is performed by the same model that will later use the memories, creating a potential self-reinforcement bias: the LLM may select commands that look important to it but are not actually the causal drivers of success.

Retrieval mechanism for Workflow: unlike Trajectory, Workflow memories do not include the original task description t — they only have the goal g and the command list. Therefore, the retrieval embedding cannot be computed from t. Instead, the paper uses a plan-based retrieval strategy: before retrieving memories, the agent is prompted to "write 4-5 sentences of coding plan to solve the given task" (Section 3.2.2). This plan is embedded using text-embedding-3-small, and cosine similarity is computed against the embeddings of the Workflow memories. The intuition is that the plan captures the strategic intent of the agent, which should align with the strategic intent encoded in the workflow goal g, even if the surface task descriptions differ. This is a crucial departure from Trajectory's task-based retrieval and is shared by Summary and Insight formats.

Memory Format 3: Summary

Summary memory adds an explicit layer of analysis and explanation on top of the trajectory. Rather than just extracting commands (like Workflow) or preserving the raw trace (like Trajectory), Summary prompts the LLM to produce a structured natural language account of what happened and why it worked (or failed). The generation prompt (Appendix E, Figures 9–10) instructs the LLM to produce two components:

$M_S = (s_t, s_e)$

where s_t is a task summary ("a short description of the task that was being solved, written in two or three sentences") and s_e is an experience summary ("a one-paragraph summary of the entire trajectory... [including] the code environment, the key actions or commands, the overall approach, the final outcome, and why this trajectory succeeded/failed").

What the experience summary must contain (success prompt):

The code environment (programming language, framework, repository structure).
The key actions or commands.
The overall approach.
The final outcome.
"Why this trajectory succeeded, highlighting useful strategies, checks, or decisions that contributed to the correct result."

What the experience summary must contain (failure prompt):

The same structural elements.
"Why this trajectory failed, highlighting incorrect assumptions, missing checks, or flawed strategies so that others can avoid repeating them."

Abstraction mechanism: Summary is more abstract than Workflow because it translates concrete commands (e.g., grep -rn "def aggregate" django/db/models/) into natural language descriptions of intent (e.g., "the agent first searched for the Aggregate class definition in the Django source tree"). This linguistic translation strips away language-specific and framework-specific surface forms while preserving the strategic logic. The paper's analysis in Figure 4 and Figure 5 shows that Summary embeddings are more intermingled across benchmarks than Workflow embeddings, confirming that the format produces less domain-specific representations.

Retrieval mechanism: identical to Workflow — the agent generates a coding plan, embeds the plan, and retrieves the most similar Summary memories based on cosine similarity between the plan embedding and the Summary memory embedding. The plan is embedded rather than the summary text s_t + s_e being embedded directly; the paper embeds the entire Summary memory as a single text chunk (the concatenation of s_t and s_e) and compares against the plan embedding.

Memory Format 4: Insight

Insight is the most abstract memory format, explicitly designed for cross-task generalization. Following the memory design of ReasoningBank (Ouyang et al., 2025), each Insight memory M_I consists of three components:

$M_I = (i_t, i_d, i_c)$

where i_t is the title (a short, descriptive label), i_d is the description (a one-sentence summary), and i_c is the content (1–3 sentences of generalized insights). The generation prompt (Appendix E, Figures 11–12) contains a critical instruction that distinguishes Insight from all other formats:

"Do not mention specific files or details, but rather focus on the generalizable insights."

This is an explicit constraint to remove domain-specific references. For a success trajectory, the LLM is asked to "extract and summarize useful insights" that are "helpful and generalizable for future similar tasks." For a failure trajectory, the LLM is asked to "reflect and think why the trajectory failed, and then summarize what lessons you have learned or strategies to prevent the failure in the future."

Example from Table 3 (Insight generated from LiveCodeBench, transferred to SWE-Bench Verified):

Title: "Create quick self-contained tests using an inline Python here-doc to validate fixes"
Description: "When making small code fixes, write a minimal tests..."
Content: "Set up a short battery of tests that cover..."

Notice that this Insight encodes a behavioral pattern (write inline tests to validate fixes) without mentioning specific files, repositories, or task details. This is what enables it to transfer from a competitive coding benchmark (LiveCodeBench) to a repository-level engineering benchmark (SWE-Bench Verified) — the recipient agent on the SWE-Bench task reads the Insight, recognizes that it should validate its fix with inline tests, and adapts this general principle to the specific Django codebase it is working in.

Why Insight is the most abstract: the generation prompt imposes a double abstraction step. First, the LLM must extract patterns from the concrete trajectory (same as Summary). Second, it must explicitly rewrite those patterns to remove all task-specific references. This produces memories that the paper's embedding analysis (Figure 4) shows are "sparse and intermingled" across benchmarks — meaning they occupy similar regions of the embedding space regardless of which benchmark they came from, making them accessible to queries from any benchmark.

Retrieval mechanism: same plan-based approach as Workflow and Summary — the agent generates a coding plan, embeds it, and retrieves the top-3 most similar Insight memories.

Relationship to ReasoningBank: the paper adopts ReasoningBank's Insight format directly but applies it in a cross-domain setting. ReasoningBank originally used Insights within a single domain (memories generated from a benchmark are retrieved for tasks in the same benchmark). MTL shows that the same format works better in a cross-domain setting because the forced abstraction makes the memories inherently transferable — they were already stripped of domain-specific details, so they don't need to be adapted for new domains.

Memory Pool Construction and Indexing

After generating memories for all tasks across all benchmarks, the paper constructs cross-domain memory pools: for each target benchmark B_i and each memory format τ, the pool P_τ(B_i) contains all memories generated from all benchmarks except B_i. Formally:

$P_\tau(B_i) = \{M^{(k)}_\tau \mid t^{(k)} \notin B_i\}_{k=1}^{N_i}$

where M^{(k)}_τ is the k-th memory of format τ, t^{(k)} is the source task that generated this memory, B_i is the target benchmark being evaluated, and N_i is the total number of memories from all non-target benchmarks.

What this ensures: the target benchmark is never in the memory pool. If the agent is being evaluated on SWE-Bench Verified, the memory pool contains memories from LiveCodeBenchv6, Aider-Polyglot, TerminalBench2, ReplicationBench, and MLGym-Bench — but not SWE-Bench Verified itself. This is the defining property of cross-domain transfer: any performance gain cannot be attributed to retrieving memories from the same distribution.

Indexing: each memory is converted into a dense vector embedding using OpenAI's text-embedding-3-small model. For Trajectory memories, the task description t is embedded. For Workflow, Summary, and Insight memories — which do not include the original task description — the entire memory text (goal + commands for Workflow, s_t + s_e for Summary, i_t + i_d + i_c for Insight) is embedded. These embeddings are stored alongside the raw memory texts in a vector index.

Why embeddings instead of keyword search: the tasks across benchmarks have very different surface vocabulary. A SWE-Bench task about Django ORM aggregates and a LiveCodeBench task about competitive programming will share almost no keywords — but they may both benefit from memories about test-driven verification. Embedding similarity captures semantic relatedness (the principle of "write tests before submitting") that keyword overlap would miss. However, the paper's retrieval experiments (Section 4.4.5) reveal that even embedding-based retrieval is imperfect for cross-domain transfer, with LLM reranking and adaptive rewriting both underperforming simple embedding similarity.

Memory pool statistics: Table 2 reports that the full MTL memory pool for the Insight format contains 431 memories when evaluated across three benchmarks. The AgentKB baseline uses 5,899 memories — roughly 13.7× more — yet MTL outperforms it, demonstrating efficiency.

Retrieval Mechanism Details

At inference time, for each new task in the target benchmark, the system retrieves N = 3 memories from the cross-domain pool. The retrieval procedure differs based on memory format:

For Trajectory memories (task-based retrieval):

Embed the current task description t_current using text-embedding-3-small.
Compute cosine similarity between embed(t_current) and all stored Trajectory memory embeddings (which were also computed from their task descriptions).
Select the 3 memories with the highest similarity scores.

For Workflow, Summary, and Insight memories (plan-based retrieval):

Prompt the LLM (GPT-5-mini) to "write 4-5 sentences of coding plan to solve the given task" — this generated plan is a brief strategic outline of how the agent intends to approach the problem.
Embed the generated plan using text-embedding-3-small.
Compute cosine similarity between embed(plan) and all stored memory embeddings.
Select the 3 memories with the highest similarity scores.

Why plan-based retrieval for abstract formats: Workflow, Summary, and Insight memories encode strategic knowledge (what approach to take, what pitfalls to avoid) rather than task identity (which specific problem was being solved). The plan captures the strategic intent of the current task, which is a better match for strategic memories than the raw task description. A SWE-Bench task description like "Fix the FieldError when an Aggregate contains a window expression in django/db/models/aggregates.py" will have low embedding similarity to a LiveCodeBench memory about writing inline tests — but the plan for the SWE-Bench task ("I will first inspect the aggregate resolution logic, then add a validation check, then write a test to confirm the fix") will have high similarity to an Insight about test-driven verification. This is a crucial architectural choice that the paper does not ablate directly — we never see what would happen if plan-based retrieval were used for Trajectory or task-based retrieval for Insight.

Injection into the agent: the 3 retrieved memories are converted to natural language and prepended to the agent's system prompt. The exact prompt format is not reproduced in the paper, but the case studies (Table 3, Table 5) show that the agent is instructed to reference the memories in its reasoning — e.g., "I will use Memory Item 2 (use an inline Python here-doc for safe, atomic edits and quick verification) to modify django/db/models/sql/query.py." The memories function as optional guidance, not as mandatory constraints; the agent can choose to follow them, adapt them, or ignore them based on its assessment of relevance.

Formal Modeling of the Abstraction-Transfer Relationship (Appendix C)

The paper provides a mathematical framework to explain why higher-abstraction memories transfer better. This model is presented in Appendix C and is not used to derive any quantitative predictions; it serves as a conceptual grounding for the empirical findings.

Memory embedding decomposition. Each memory m is associated with an embedding vector e(m) that is decomposed into two orthogonal components:

$e(m) = z_{\text{inv}}(m) + z_{\text{sp}}(m)$

where z_inv(m) is the domain-invariant component — the part of the embedding that captures meta-knowledge applicable across all coding tasks (e.g., "write tests before submitting," "inspect the evaluation criteria first"). This component does not change when the task domain changes. And z_sp(m) is the domain-specific component — the part that captures details specific to the source task (e.g., the exact file paths in a Django repository, the specific R language syntax for writing files). This component is misaligned with any target task from a different domain.

What it computes: a decomposition that separates transferable knowledge from non-transferable noise. The key assumption is that the two components are additive and statistically independent — an idealized simplification, but one that captures the core intuition.

Why this decomposition: it provides a formal language for discussing why Trajectory memories (which preserve many domain-specific details) underperform Insight memories (which are explicitly stripped of domain-specific references). Trajectory has a large z_sp component and a small z_inv component; Insight has a small z_sp component and a large z_inv component.

Abstraction level definition. The abstraction level A(m) of a memory is defined as the proportion of the embedding's norm that comes from the invariant component:

$A(m) = \frac{\|z_{\text{inv}}(m)\|^2}{\|z_{\text{inv}}(m)\|^2 + \|z_{\text{sp}}(m)\|^2}$

where ||·||^2 denotes the squared L2 norm (the sum of squared components).

What it computes: a scalar between 0 and 1, where 0 means the memory is entirely domain-specific (no transferable knowledge) and 1 means it is entirely domain-invariant (pure meta-knowledge). The squaring in the norm means that the ratio is sensitive to the magnitude of each component — a memory with a small invariant component and a large specific component will have A close to 0, while one with balanced components will have A around 0.5.

Why this form: the squared norm ratio is a standard way to quantify the relative importance of one component in an additive decomposition. Alternatives like ||z_inv|| / (||z_inv|| + ||z_sp||) would also work but would be less sensitive to differences when one component dominates. The squared version amplifies the contrast between low-abstraction and high-abstraction memories, which matches the empirical finding that the difference between Trajectory and Insight is qualitative, not just quantitative.

Utility of a retrieved memory. For an unseen target task x (represented as an embedding e(x)), the utility U(x, m) of retrieving memory m is modeled as a trade-off:

$U(x, m) \propto \langle e(x), z_{\text{inv}}(m) \rangle - \langle e(x), z_{\text{sp}}(m) \rangle$

where ⟨·,·⟩ denotes the dot product (cosine similarity scaled by magnitudes) between the task embedding and each component.

What it computes: the net benefit of using memory m for task x. The first term ⟨e(x), z_inv(m)⟩ is the transferable guidance — how well the memory's meta-knowledge aligns with the task's requirements. The second term ⟨e(x), z_sp(m)⟩ is the domain mismatch penalty — how much the memory's domain-specific details are misaligned with the task, potentially leading the agent astray. When U > 0, the memory helps; when U < 0, it hurts (negative transfer).

Why this additive trade-off: the subtraction form captures the paper's core empirical finding: high-abstraction memories have small z_sp (so the mismatch penalty is near zero) and large z_inv (so the guidance term dominates), leading to positive utility. Low-abstraction memories have large z_sp (so the mismatch penalty is large) and small z_inv (so the guidance is weak), leading to negative utility when the penalty exceeds the guidance. This explains the Trajectory negative transfer cases in Table 5: the memory's z_sp component (R-language syntax, specific file paths) caused a large mismatch penalty that outweighed any guidance.

Proposition 1 (Abstraction-Transfer Tradeoff). Under the assumptions that (1) embeddings have bounded total norm (i.e., increasing z_inv necessarily decreases z_sp because the total capacity is fixed) and (2) z_sp acts as misaligned noise for cross-domain tasks (i.e., ⟨e(x), z_sp(m)⟩ is effectively random with zero mean for tasks from a different domain), the expected utility E[U(x, m)] strictly increases with the abstraction level A(m).

What it claims: higher abstraction monotonically improves expected transfer performance. This is not a theorem the paper proves — it's a proposition that formalizes the empirical pattern observed in the experiments: Insight (highest abstraction) > Summary > Workflow > Trajectory (lowest abstraction). The model does not predict the magnitude of the improvement, only the direction.

Why this modeling matters: it converts the qualitative insight "abstraction helps transfer" into a quantitative framework that could, in future work, be used to design memory generation procedures that explicitly maximize A(m) — for example, by training a memory generator to minimize the domain-specific component z_sp while preserving z_inv. The paper does not take this step, but the framework provides the conceptual foundation for doing so.

Experimental Configuration and Design Choices

Benchmarks and sampling. The paper evaluates on 6 coding benchmarks, each representing a distinct task domain:

LiveCodeBenchv6 (Jain et al., 2024): function-level competitive programming problems with hidden test cases.
Aider-Polyglot (Gauthier, 2024): multi-language code editing tasks from the Aider benchmark.
SWE-Bench Verified (Jimenez et al., 2024): repository-level software engineering bug fixes requiring multi-file edits in real-world codebases.
TerminalBench2 (Merrill et al., 2026): command-line interface tasks requiring complex shell interactions.
ReplicationBench (Ye et al., 2025): scientific code generation for reproducing results from astrophysics research papers.
MLGym-Bench (Nathani et al., 2025): machine learning research tasks involving model development, hyperparameter tuning, and experiment management.

For each benchmark, 100 tasks are randomly sampled if the total exceeds 100 (Section 3.2.1). This uniform sample size ensures that no benchmark dominates the memory pool simply by having more tasks.

Evaluation metric: Pass@3. All results report Pass@3 scores — the fraction of tasks for which at least one of three independent agent runs (with different random seeds) succeeds. This is a standard metric in the coding agent literature that accounts for the stochasticity of LLM inference (different runs may produce different trajectories due to sampling). The paper also reports Pass@1 in Appendix A (Table 8) for completeness, showing that the MTL gains are consistent across both metrics, though Pass@1 improvements are smaller (1.9% average for GPT-5-mini with Insight vs. 3.7% for Pass@3).

Why Pass@3: the agent's environment interactions are inherently noisy — the same prompt can lead to different bash commands, which can lead to different observations, which can lead to different subsequent reasoning. Pass@3 gives the agent multiple attempts to succeed, measuring not just its median performance but its ability to eventually find a correct solution. This is appropriate for a memory system because one of the key benefits of memory is providing guidance that increases the probability of successful actions in any given run.

Memory pool sizes. For the main MTL experiments with GPT-5-mini, the memory pool for each target benchmark contains memories from the remaining 5 benchmarks. The exact number of memories varies by format because not all trajectories produce well-formed memories in every format (the LLM may fail to generate valid JSON, or a trajectory may be too short to extract a meaningful workflow). The paper reports in Table 2 that the Insight memory pool used for the three-benchmark comparison contains 431 memories total. With 5 source benchmarks × ~100 tasks each, the maximum possible pool size would be ~500 memories per format, suggesting that memory generation succeeds for most but not all trajectories.

Why top-3 retrieval: the number of retrieved memories N = 3 is a design choice that balances informativeness against context window constraints. More memories would provide more guidance but would also consume more of the agent's context window (which is finite and must also accommodate the system prompt, task description, and interaction history). The paper does not ablate N to find the optimal number — this remains an open question for future work.

Cross-validation and fair comparison logic. To ensure fair comparison between zero-shot and MTL conditions, the paper constructs the memory pool such that it contains no memories from the target benchmark. This means that any performance difference between zero-shot (no memory) and MTL (memory from other benchmarks) is attributable solely to cross-domain transfer, not to within-domain memory effects. The comparison with self-evolving baselines (Table 2) uses the same three benchmarks (LiveCodeBenchv6, SWE-Bench Verified, ReplicationBench) for all methods, with three independent runs per method to compute Pass@3 reliably.

Model consistency. All components use GPT-5-mini: the coding agent, the memory generator, the LLM judge, and the plan generator for retrieval. This eliminates confounding from cross-model distribution shift in the main experiments. The cross-model transfer experiments (Section 4.4.4) deliberately introduce such shift to test robustness, and they find that cross-model MTL still helps but underperforms same-model MTL, confirming that model-specific biases in memory generation do affect transferability.

Memory generation: offline vs. online. Memory generation is performed entirely offline — all trajectories are collected first, then memories are generated, then the memory pool is constructed. This means the memory pool is static during evaluation; the agent does not add new memories to the pool as it solves tasks. This is a deliberate simplification to isolate the transfer effect: if the pool grew during evaluation, it would be unclear whether performance gains came from cross-domain memories or from newly added in-domain memories. The paper acknowledges this as a limitation implicitly by not studying online memory accumulation, which would be a natural extension for deployment.

Analysis Framework: How Transfer Benefits Were Categorized

The paper's analysis of why memories help (Section 4.2.1, Figure 3) uses a systematic categorization procedure:

Identifying transfer-success cases: the authors collect all instances where the agent fails in the zero-shot setting but succeeds when MTL with Insight memory is applied. These are the instances where transferred memory makes the difference between failure and success.
LLM-based categorization: GPT-5 is prompted to categorize how the transferred memory contributed to the successful outcome. The LLM assigns each case to one of 10 categories (listed in Appendix D, Table 10), such as "Iterative Workflow Discipline," "Test Driven Verification," "Environmental Adaptation," "Anti-Pattern Avoidance," and "Algorithmic Strategy Transfer."
Frequency analysis: the percentage of cases assigned to each category is computed and visualized in Figure 3. The top categories are Iterative Workflow Discipline (15.0%), Test Driven Verification (14.5%), Anti-Pattern Avoidance (14.4%), and Input Validation & Robustness (10.4%). Algorithmic Strategy Transfer accounts for only 5.5%.

Design choice: LLM categorization vs. human annotation. Using an LLM for categorization is fast and scalable but risks confirmation bias — the same model that performed the task is now evaluating how it used the memory. The paper does not report inter-annotator agreement or validate the LLM categorizations against human judgments. This is a limitation, but a reasonable trade-off given the exploratory nature of the analysis: the goal is to identify patterns, not to produce a definitive causal account.

Interpretation of Figure 3: the dominance of meta-knowledge categories over algorithmic transfer confirms Core Finding 2 — transferred memories help primarily by supplying procedural and behavioral guidance, not by providing reusable code or algorithms. This explains why abstraction matters: procedural guidance (e.g., "write a test before submitting," "check the evaluation criteria first") generalizes across domains, while algorithmic strategies are often domain-specific and hence account for only a small fraction of transfer benefits.

Isolation of the Abstraction Effect Within a Single Format

The paper's most controlled test of the abstraction-transfer relationship is the within-format comparison in Section 4.3.3 (Table 4). This experiment is crucial because it rules out the alternative explanation that the format itself (Insight vs. Trajectory) causes the performance difference, rather than the abstraction level.

Procedure:

Take all Insight memories generated for the cross-domain pool.
For each Insight memory, prompt an LLM to infer the original task solely from the memory content (without seeing the original task description).
Measure the similarity between the inferred task and the ground-truth original task (presumably using embedding cosine similarity, though the exact metric is not specified beyond "measure the similarity").
Sort memories by this similarity score. High-similarity memories are "task-specific" — the LLM can reconstruct the original task from the memory content, meaning the memory retains domain-specific information. Low-similarity memories are "task-agnostic" — the LLM cannot reconstruct the original task, meaning the memory has successfully abstracted away domain-specific details.
Partition the memories into the top 30% (most task-specific, least abstract) and bottom 30% (most task-agnostic, most abstract).
Construct two separate memory pools — one using only task-specific Insights, one using only task-agnostic Insights — and evaluate the agent on three benchmarks with each pool.

Why this controls for format: both pools use the exact same memory format (Insight: title + description + content). The only difference is the content — whether the memory retains recognizable task-specific details or not. Any performance difference can therefore be attributed to abstraction, not to format-level artifacts like memory length, structure, or retrieval mechanism.

Results (Table 4): task-agnostic Insights outperform task-specific Insights by an average of 1.1 percentage points across the three benchmarks (LCB: +0.6%, SWEB: +1.0%, RepliB: +1.5%). This is a smaller effect than the full format comparison (Insight vs. Trajectory: +7.8% on ReplicationBench), which makes sense — the within-format comparison only varies abstraction level, not format, and the difference between "most abstract Insight" and "least abstract Insight" is smaller than the difference between Insight and Trajectory.

What this proves: the paper's Core Finding 3 — "More abstract and generalized memory representations yield higher transfer effectiveness" — holds even when controlling for format, strengthening the causal claim that abstraction itself drives transferability.

Memory Pool Scaling Analysis

Section 4.4.3 investigates how MTL performance scales with two variables: memory pool size and number of source domains. These experiments test whether the benefits of MTL are primarily driven by having any cross-domain memories (a binary effect) or by having more and more diverse memories (a continuous scaling effect).

Pool size scaling:

Start with the full cross-domain Insight memory pool.
Randomly subsample memories at ratios of 1/4, 2/4, and 3/4 of the original size.
Evaluate the agent on three benchmarks (LiveCodeBenchv6, SWE-Bench Verified, ReplicationBench) using Pass@1 (not Pass@3 — this experiment uses the finer-grained Pass@1 metric to capture smaller differences).
Report the ΔPass@1 relative to the zero-shot baseline.

Results (Figure 6, left): the average ΔPass@1 across benchmarks increases monotonically from approximately +0.5% at 1/4 pool size to approximately +2% at full pool size. The relationship is roughly linear. This demonstrates that MTL is not just a binary "having cross-domain memory helps" effect — it scales with the amount of memory available, consistent with the idea that a larger pool increases the probability of retrieving memories with high-quality meta-knowledge for any given task.

Source domain scaling:

Vary the number of benchmarks used to construct the memory pool from 0 (zero-shot baseline) to 9 (the maximum available, presumably including some benchmarks beyond the 6 used in the main experiments — the paper does not specify what the additional 3 domains are).
Evaluate on the same three benchmarks with Pass@1.

Results (Figure 6, right): the average ΔPass@1 increases from 0 (0 domains, zero-shot) to approximately +2% at 5 domains, with a slight further increase to approximately +2.2% at 9 domains. The marginal benefit of additional domains diminishes but remains positive. This suggests that diversity matters — more domains increase the variety of meta-knowledge in the pool, making it more likely that some memory will be relevant to any given target task.

Why these scaling results matter: they establish that MTL is not just a curiosity of specific benchmark pairs but a general phenomenon that improves with scale. This has practical implications: as self-evolving agents accumulate more experience across more diverse tasks, the value of cross-domain memory transfer should increase, not saturate. It also suggests that the paper's reported 3.7% average improvement is a lower bound — larger memory pools (e.g., from production deployments with thousands of tasks) would likely yield larger gains.

4. Key Insights and Innovations

Innovation 1: Reframing Memory-Based Self-Evolution from Single-Domain Learning to Cross-Domain Transfer

Before this paper, the dominant assumption in memory-augmented coding agents was that memories should be generated and retrieved within the same task domain — a single benchmark, a single task type, a single distribution of problems. This assumption was so ingrained that it functioned as background architecture rather than an explicit design choice; papers on self-evolving agents (ReasoningBank by Ouyang et al., 2025; AWM by Wang et al., 2024c; ReMe by Cao et al., 2025) built their memory pipelines around in-domain retrieval without questioning whether out-of-domain memories might be valuable. The paper's core conceptual move is to treat this assumption as a hypothesis to be tested, not a constraint to be accepted.

The significance of this reframing extends beyond the performance gains it produces (3.7% average improvement, Table 1). It changes what "memory utilization" means for coding agents. In the single-domain paradigm, memory is a cache of task-specific experience: the agent remembers how it solved similar problems before and re-applies those solutions. In the cross-domain paradigm that MTL introduces, memory becomes a repository of transferable procedural knowledge: the agent remembers how to operate effectively — how to verify fixes, how to navigate unfamiliar codebases, how to avoid common environmental pitfalls — and applies this operational knowledge regardless of what specific problem it's solving.

This reframing is conceptually analogous to the shift in machine learning from task-specific feature engineering to learned representations that transfer across tasks. Just as representation learning showed that features learned for one task could benefit others, MTL shows that agent experiences generated for one type of coding problem can benefit others — provided the experiences are encoded at the right level of abstraction. The field had implicitly assumed that coding experiences were domain-specific; MTL provides systematic evidence that this assumption is false and unnecessarily constraining.

The paper doesn't just argue for this reframing abstractly — it demonstrates its practical superiority against strong baselines. Table 2 shows MTL outperforming ReasoningBank (in-domain Insight memories only) by 2.9 percentage points and AgentKB (unified pool without mechanistic analysis) by 1.7 percentage points on the same three-benchmark subset, despite using roughly 13.7× fewer memories than AgentKB. This efficiency gap — better performance with an order of magnitude less memory — is the empirical signature of the reframing's validity: cross-domain meta-knowledge is not just a supplement to in-domain experience but a more efficient source of guidance.

Innovation 2: Abstraction as the Mechanism of Transferability — Not Format, Not Domain Similarity

The paper's most intellectually distinctive finding is that abstraction, not format or surface similarity, is the primary driver of cross-domain memory transfer effectiveness. This is a diagnostic contribution: it identifies which property of a memory makes it transferable, which in turn provides a design principle for memory generation (prioritize abstraction) and explains why prior approaches produced mixed or domain-bound results.

The cleanest evidence for this is the within-format experiment in Table 4. By taking the same Insight memory format and partitioning memories into task-specific and task-agnostic subsets — using an LLM's ability to reconstruct the original task from the memory content as a proxy for specificity — the paper isolates abstraction from all format-level confounds (memory length, structure, retrieval embedding quality). Task-agnostic Insights consistently outperform task-specific ones across three benchmarks (+1.1% average), even though both subsets use identical formatting. This is a controlled experiment that establishes a causal direction: higher abstraction → better transfer.

What makes this a genuine innovation rather than an obvious observation is that it contradicts a plausible alternative hypothesis: that transfer effectiveness depends on domain similarity — i.e., memories from competitive coding should transfer better to competitive coding-like tasks than to repository-level engineering tasks. If domain similarity were the key factor, Trajectory memories might outperform Insight memories when the source and target domains are closely related, because Trajectory preserves more potentially relevant detail. The paper's results show the opposite: even on benchmarks where the surface task structure is similar, Insight dominates Trajectory because the abstraction level is what matters. The negative transfer cases in Table 5 and Appendix B (Table 9) provide vivid examples of why: concrete memories act as brittle anchors, causing the agent to blindly imitate specific commands or patterns that are incompatible with the new environment.

The formal model in Appendix C (Proposition 1) elevates this from an empirical observation to a conceptual framework. By decomposing memory embeddings into domain-invariant (z_inv) and domain-specific (z_sp) components, and modeling utility as U ∝ ⟨e(x), z_inv⟩ − ⟨e(x), z_sp⟩, the paper provides a mathematical language for discussing the abstraction-transfer tradeoff. The subtraction form captures why low-abstraction memories can cause negative transfer: when the domain-specific component's mismatch penalty exceeds the invariant component's guidance, the net utility becomes negative. This framework doesn't produce numerical predictions, but it reifies "abstraction" from a vague design principle into a quantifiable property of memory representations — one that future work could explicitly optimize.

Innovation 3: Meta-Knowledge as the Carrier of Transfer Value — Not Domain-Specific Skills

Prior work on memory-augmented agents operated under an implicit assumption that the value of memory lies in its content similarity to the target task: a memory about fixing a Django ORM bug helps with future Django ORM bugs because it contains relevant code patterns, file locations, or framework-specific knowledge. The paper's categorization analysis in Figure 3 directly contradicts this assumption and replaces it with a fundamentally different account.

By analyzing every case where MTL turned a zero-shot failure into a success, and categorizing how the transferred memory contributed, the paper finds that algorithmic strategy transfer — the direct reuse of programming knowledge, algorithms, or code patterns — accounts for only 5.5% of total gains. The dominant categories are procedural and behavioral: Iterative Workflow Discipline (15.0%), Test Driven Verification (14.5%), Anti-Pattern Avoidance (14.4%), and Input Validation & Robustness (10.4%). These are not domain-specific skills; they are meta-knowledge — knowledge about how to act in a coding environment, not what code to write for a particular problem.

The case study in Table 3 crystallizes this finding. The Insight memory transferred from LiveCodeBench to a SWE-Bench task does not provide any Django-specific guidance; it provides a behavioral heuristic: "create quick self-contained tests using an inline Python here-doc to validate fixes." The zero-shot agent fails because it makes a code change without verifying it. The MTL agent succeeds because it follows the memory's procedural guidance — it writes an inline test, validates its fix, and only then considers the task complete. The crucial point is that this memory could have come from any benchmark where test-driven verification proved useful; its source domain is irrelevant to its value.

This finding has significant implications for how memory-based agents should be designed. If the primary value of memory lies in procedural meta-knowledge rather than domain-specific content, then memory generation pipelines should prioritize extracting and preserving these procedural patterns — even at the cost of discarding domain-specific code details. The Insight format's explicit instruction to "not mention specific files or details, but rather focus on the generalizable insights" (Appendix E, Figure 11) operationalizes this principle, and its superior performance validates it. This is a design principle with causal force: it tells practitioners not just that Insight works better than Trajectory (a format comparison), but why it works better (because it preserves meta-knowledge while stripping domain-specific noise), which enables principled improvement rather than format-shopping.

Innovation 4: Negative Transfer as a Systematic, Diagnosable Failure Mode — Not Random Noise

Most papers on memory-augmented agents report average performance gains and treat individual failures as statistical noise. This paper takes the opposite approach: it treats negative transfer — cases where cross-domain memory hurts performance — as a first-class phenomenon worthy of systematic analysis. This is a diagnostic contribution that changes how the field should think about memory reliability.

The paper identifies three distinct negative transfer mechanisms (Section 4.4.1), each with different implications for system design:

Domain-mismatched anchoring: structurally irrelevant but superficially similar memories act as misleading anchors, introducing incorrect assumptions that divert the agent from the task's actual logic. The Appendix B case study (Table 9) shows this vividly: a Workflow memory about writing R-language files using heredoc syntax leads the agent to blindly apply the same pattern to a C++ project, overwriting existing files without checking their structure.
False validation confidence: verification-oriented memories create a false sense of certainty, leading to self-confirming loops where agents rely on superficial checks rather than formal evaluation criteria, missing critical specifications.
Misapplied best-practice transfer: successful patterns are transferred indiscriminately, overriding task-specific semantics and causing rigid adherence to familiar workflows that violate new task requirements. The second Appendix B case study shows an agent distorting a memory about pre-flight verification (checking that datasets and checkpoints exist before running expensive experiments) into a justification for quick, low-quality completion.

What makes these categories innovative is not the specific labels but the framework they imply for memory system design. Each failure mode corresponds to a different systemic weakness: wrong retrieval (domain-mismatched anchoring → better retrieval needed), wrong confidence calibration (false validation confidence → better verifier integration needed), and wrong adaptation (misapplied best practices → memory rewriting or context-sensitive application needed). This decomposition transforms negative transfer from an amorphous "sometimes memories don't work" problem into a set of tractable engineering challenges, each with distinct solution directions. The retrieval method experiments in Section 4.4.5 (Table 7) — which show that LLM reranking and adaptive rewriting both underperform simple embedding similarity — demonstrate that these challenges are non-trivial and that naive solutions can make things worse, underscoring the value of the diagnostic framework.

Innovation 5: Memory Transfer Scales with Pool Diversity — An Inference-Time Scaling Law

The paper demonstrates that MTL effectiveness scales with both memory pool size and number of source domains (Section 4.4.3, Figure 6), establishing what amounts to an inference-time scaling law for memory-augmented agents. This is a fundamental finding because it shows that cross-domain memory transfer is not a threshold effect (where having any cross-domain memory provides a fixed benefit) but a continuous scaling effect (where more and more diverse memory yields larger gains, with diminishing but still positive marginal returns).

The pool size experiment shows a roughly monotonic increase in ΔPass@1 from approximately +0.5% at 1/4 pool size to approximately +2% at full pool size across three benchmarks. The domain count experiment shows gains increasing from 0 domains (zero-shot baseline) to +2% at 5 domains, with a slight further increase at 9 domains. These results have counterintuitive implications: adding memories from a domain that is very different from the target domain still helps, because diversity increases the probability that some memory in the pool encodes relevant meta-knowledge. The agent doesn't need every retrieved memory to be helpful; it needs at least one of the top-N retrieved memories to contain applicable guidance, and a larger, more diverse pool increases that probability.

This scaling behavior parallels the empirical observation in pretraining scaling laws that model performance improves predictably with more data and more diverse data — but applied at inference time via memory rather than at training time via parameter updates. The paper does not fit a parametric scaling law (the sample sizes are too small), but the qualitative pattern is clear and has immediate practical consequences: self-evolving agent systems should accumulate memories across as many diverse domains as possible, because the value of the memory pool compounds with scale and diversity. This provides theoretical justification for the kind of large-scale memory aggregation that AgentKB pioneered, while explaining why AgentKB's massive pool (5,899 memories) underperformed MTL's smaller but more abstract pool (431 memories): diversity matters, but abstraction matters more, and the two must be jointly optimized.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on 6 coding benchmarks, each representing a distinct task domain. LiveCodeBenchv6 (Jain et al., 2024) provides function-level competitive programming problems with hidden test cases. Aider-Polyglot (Gauthier, 2024) contains multi-language code editing tasks. SWE-Bench Verified (Jimenez et al., 2024) consists of repository-level software engineering bug fixes requiring multi-file edits in real-world codebases. TerminalBench2 (Merrill et al., 2026) tests command-line interface tasks with complex shell interactions. ReplicationBench (Ye et al., 2025) requires scientific code generation for reproducing astrophysics research papers. MLGym-Bench (Nathani et al., 2025) covers machine learning research tasks involving model development and hyperparameter tuning. For each benchmark, 100 tasks are randomly sampled if the total exceeds 100 (Section 3.2.1), ensuring no benchmark dominates the memory pool through raw task count.
Base model(s). The primary base model is GPT-5-mini, used consistently across all components: the coding agent's reasoning, memory generation, the LLM judge for success/failure labeling, and the plan generator for retrieval (Section 3.2.2). The paper also validates transferability across two additional model families: DeepSeek V3.2 (Liu et al., 2025) and Qwen3-Coder-480B-A35B-Instruct (Yang et al., 2025; Cao et al., 2026), which are evaluated both as target models receiving transferred memories and as source models generating memories for cross-model transfer experiments (Section 4.4.4). The choice of GPT-5-mini as the primary model is pragmatic — it is representative of strong contemporary LLMs — rather than being justified by specific architectural properties.
Metrics. The primary metric throughout is Pass@3 — the fraction of tasks for which at least one of three independent agent runs succeeds (Section 3.2.1). This accounts for the stochasticity of LLM inference, since different runs may produce different trajectories due to sampling. The paper reports Pass@1 results in Appendix A (Table 8) as a secondary metric, noting that MTL gains are consistent across both metrics though Pass@1 improvements are smaller in absolute terms (e.g., 1.9% average for GPT-5-mini with Insight vs. 3.7% for Pass@3). Task success is determined using each benchmark's native evaluation protocol (e.g., running unit tests for SWE-Bench Verified, checking output correctness for LiveCodeBench).
Baselines. The paper compares against three distinct baselines. (1) Zero-shot: the coding agent without any memory — evaluated on each benchmark with the same Pass@3 protocol. This is the primary baseline in Table 1 and establishes the floor for all MTL variants. (2) ReasoningBank (Ouyang et al., 2025): a self-evolving method that generates Insight memories from in-domain trajectories and retrieves them for tasks within the same benchmark. The comparison in Table 2 uses 97 in-domain memories on the subset of three benchmarks (LiveCodeBenchv6, SWE-Bench Verified, ReplicationBench). (3) AgentKB (Tang et al., 2025): a unified memory pool approach that aggregates 5,899 memories from heterogeneous task types (coding, web, general reasoning) and retrieves from this pool for software engineering tasks. This baseline is also evaluated on the three-benchmark subset in Table 2.
Generation budget / compute accounting. The paper does not measure compute in FLOPs or wall-clock time. Instead, the memory pool size (number of stored memories) serves as the implicit resource measure, with retrieval cost held constant at top-N = 3 across all experiments (Section 3.2.2). Memory generation is performed offline and its cost is not amortized over evaluations. The memory pool scaling experiments in Section 4.4.3 (Figure 6) explicitly vary pool size at ratios of 1/4, 2/4, 3/4, and full, measuring performance as a function of memory availability. The number of source domains is also varied from 0 to 9 to measure diversity effects. Notably, the retrieval mechanism itself (embedding computation + cosine similarity top-3) has negligible cost compared to generation, making memory pool size the dominant resource variable.
Cross-validation / statistical protocol. The paper employs a leave-one-benchmark-out protocol for cross-domain evaluation: when testing on benchmark B_i, the memory pool contains all memories from all other benchmarks but excludes B_i entirely (Section 3.1.2). This ensures that any performance difference between zero-shot and MTL is attributable solely to cross-domain transfer, not to within-domain memory effects. For the comparison with self-evolving baselines (Table 2), each method is evaluated over three independent runs with different random seeds to compute Pass@3 scores robustly. The paper does not report confidence intervals, standard errors, or statistical significance tests for any comparison, which limits the ability to assess whether observed differences (e.g., 3.7% average gain) are statistically reliable given the 100-task per-benchmark sample size.

Main Quantitative Results

Cross-Domain Memory Transfer Performance (Table 1, Appendix A Table 8)

The headline result from Table 1 is that Memory Transfer Learning with Insight memories improves average Pass@3 by 3.7% over zero-shot across six benchmarks using GPT-5-mini (0.523 → 0.560). The per-benchmark breakdown reveals substantial heterogeneity:

LiveCodeBenchv6: Insight achieves 0.930 vs. zero-shot 0.910 (+2.0%). Trajectory actually outperforms Insight slightly here (0.940), the only benchmark where the least-abstract format leads.
Aider-Polyglot: All MTL variants are within 0.02 of zero-shot (0.470), with Insight at 0.470 (0.0% change) and Trajectory at 0.490 (+2.0%). This benchmark shows the weakest transfer effect.
SWE-Bench Verified: Insight achieves 0.770 vs. zero-shot 0.730 (+4.0%). Trajectory, Workflow, and Insight all reach 0.770, while Summary reaches 0.760.
TerminalBench2: Insight achieves 0.360 vs. zero-shot 0.315 (+4.5%). Trajectory degrades performance to 0.270 (−4.5%), the clearest negative transfer case in the main results.
ReplicationBench: Insight achieves 0.189 vs. zero-shot 0.111 (+7.8%), the largest absolute gain. This is notable because the zero-shot baseline is low — suggesting MTL helps most when the base agent struggles.
MLGym-Bench: Insight achieves 0.750 vs. zero-shot 0.667 (+8.3%), the largest percentage gain. Trajectory degrades performance to 0.583 (−8.4%), a swing of 16.7 percentage points between the least-abstract and most-abstract formats on this benchmark.

The consistent pattern across four of six benchmarks is Insight ≥ Summary ≥ Workflow ≥ Trajectory, aligning with the abstraction-transfer hypothesis. The two exceptions are LiveCodeBench (Trajectory slightly ahead) and Aider-Polyglot (all formats roughly equal), suggesting that benchmark-specific factors — perhaps task surface similarity enabling effective trajectory reuse, or tasks too diverse for any format to provide consistently relevant guidance — modulate the abstraction effect.

Cross-model validation (Table 1, lower sections). MTL with Insight memories also benefits alternative model families, though with smaller magnitudes:

DeepSeek V3.2: Average improvement of 2.6% (0.542 → 0.568). The largest gains are on SWE-Bench Verified (+6.0%, from 0.530 to 0.590) and MLGym-Bench (+8.3%, from 0.583 to 0.667). Aider-Polyglot shows a −1.0% degradation (0.590 → 0.580).
Qwen3-Coder-480B-A35B-Instruct: Average improvement of 1.8% (0.483 → 0.501). Gains are smaller and more uniform, with TerminalBench2 showing the largest (+3.4%, from 0.292 to 0.326) and ReplicationBench and MLGym-Bench showing no change (0.211 and 0.583 respectively).

The cross-model results demonstrate that MTL generalizes beyond the primary model family, though the effect size varies. The paper does not analyze why DeepSeek benefits more than Qwen3-Coder, but one hypothesis is that model-specific reasoning styles interact with the transferred meta-knowledge — a model that already exhibits strong test-driven verification behavior, for example, would benefit less from memories that reinforce this pattern.

Pass@1 results (Appendix A, Table 8). The Pass@1 metrics show the same abstraction ordering but with compressed magnitudes: GPT-5-mini with Insight achieves 0.454 average Pass@1 vs. zero-shot 0.435 (+1.9%). The gap between Pass@1 and Pass@3 gains (+1.9% vs. +3.7%) suggests that memory helps not just by improving the average solution quality but by increasing the probability of eventually finding a correct solution across multiple attempts — consistent with the meta-knowledge mechanism (procedural guidance improves exploration efficiency rather than guaranteeing success on every attempt).

Comparison with Self-Evolving Methods (Table 2)

On the three-benchmark subset where self-evolving baselines are evaluated, MTL with Insight achieves an average Pass@3 of 0.630, compared to:

Zero-shot: 0.584 (+4.6 percentage points)
ReasoningBank: 0.601 (+2.9 percentage points over ReasoningBank)
AgentKB: 0.613 (+1.7 percentage points over AgentKB)

Per-benchmark, MTL leads on LiveCodeBenchv6 (0.930 vs. ReasoningBank 0.920 and AgentKB 0.920) and SWE-Bench Verified (0.770 vs. 0.750 and 0.720), while AgentKB leads on ReplicationBench (0.200 vs. MTL 0.189 and ReasoningBank 0.133). The ReplicationBench result is notable: AgentKB's massive pool (5,899 memories vs. MTL's 431) provides better coverage for this specific benchmark, suggesting that for certain domains, raw memory quantity can partially compensate for lower per-memory quality.

The key efficiency claim — "MTL uses only 431 memories yet achieves the highest average performance" — is supported, but the margin is small (+1.7% over AgentKB on a 3-benchmark subset), and the paper does not report whether this difference is statistically significant given three runs per method. The practical implication — that curating for abstraction is more efficient than aggregating massive raw pools — is directionally correct but the strength of evidence is moderate.

Meta-Knowledge as the Primary Transfer Mechanism (Figure 3, Table 10)

The categorization analysis in Figure 3 quantifies how transferred Insight memories contribute to success. Across all instances where MTL turned a zero-shot failure into a success, the top five contribution categories are:

Iterative Workflow Discipline: 15.0% — guiding the agent to follow structured step-by-step processes (inspect → edit → run → verify) rather than attempting risky one-shot solutions.
Test Driven Verification: 14.5% — encouraging the creation of inline tests, reproduction scripts, or smoke tests before submission.
Anti-Pattern Avoidance: 14.4% — acting as cautionary guardrails against known failure modes (blind overwrites, guess-based outputs, unverified assumptions).
Input Validation & Robustness: 10.4% — handling edge cases, data normalization, and defensive parsing.
Environmental Adaptation: 9.5% — navigating system constraints, build tools, and OS-level idiosyncrasies.

Algorithmic Strategy Transfer — the direct reuse of specific code patterns, algorithms, or data structures — accounts for only 5.5% of total contributions. This is the key evidence for Core Finding 2: transferred memory helps primarily by supplying procedural meta-knowledge, not domain-specific programming content. The remaining categories distribute across Interaction Protocol Adherence (8.5%), API & Interface Compliance (8.1%), File and Syntax Management (7.8%), Repository Exploration Tactics (6.4%), and a small residual.

The LLM-based categorization methodology (Appendix D, Table 10) provides operational definitions for each category. For example, "Iterative Workflow Discipline" applies "when memories reinforced the pattern of making small changes and checking them immediately," while "Anti-Pattern Avoidance" applies "when the agent explicitly avoided actions that caused failures in retrieved memories." The paper does not report whether a second annotator (human or LLM) verified these categorizations, so the percentages should be interpreted as approximate distributions rather than precise measurements.

The case study in Table 3 concretely illustrates the meta-knowledge mechanism. On a SWE-Bench Verified task involving Django aggregate validation, the transferred Insight from LiveCodeBench — "Create quick self-contained tests using an inline Python here-doc to validate fixes" — provides behavioral guidance (write a test before considering the fix complete) rather than Django-specific knowledge. The zero-shot agent fails because it makes a code change without verification; the MTL agent explicitly references the memory in its reasoning ("I will use Memory Item 2... to modify...") and succeeds by following the procedural pattern.

Abstraction-Transfer Correlation (Figures 4-5, Table 4)

The core empirical finding linking abstraction to transferability comes from three converging lines of evidence:

Format-level ordering (Table 1). Across all six benchmarks, the average Pass@3 improvement over zero-shot follows the ordering: Insight (+3.7%) > Summary (+2.3%) > Workflow (+1.5%) > Trajectory (+1.1%). This ordering is consistent with the abstraction hierarchy established in Section 3: Insight is the most abstract (explicitly stripped of domain-specific references), Summary is moderately abstract (paraphrases actions in natural language but may retain task context), Workflow is less abstract (extracts key commands but preserves them verbatim), and Trajectory is the least abstract (raw command-observation pairs).

Embedding space analysis (Figures 4-5). The t-SNE visualizations in Figure 4 show that task embeddings (used for Trajectory retrieval) form distinct benchmark-level clusters — each color (representing a benchmark) occupies a relatively compact, well-separated region. Workflow embeddings show weaker but still visible clustering. Summary embeddings are more intermingled. Insight embeddings are the most "sparse and intermingled," with points from different benchmarks occupying overlapping regions of the space. Quantitatively, Figure 5 confirms this pattern: the Davies-Bouldin Index (DBI), which measures cluster separation (lower = more separated), increases from Trajectory (3.09) to Workflow (4.02) to Summary (4.47) to Insight (6.50). The Local Inverse Simpson's Index (LISI), which measures local mixing (higher = more mixing), increases from Trajectory (1.70) to Workflow (2.33) to Summary (2.34) to Insight (4.00). Higher DBI and LISI for more abstract formats mean weaker domain-specific clustering and stronger cross-domain mixing — exactly what one would expect if abstraction strips away domain identity while preserving transferable content.

Within-format isolation (Table 4). By partitioning Insight memories into task-specific (top 30% by task reconstruction similarity) and task-agnostic (bottom 30%) subsets, the paper isolates abstraction from format-level confounds. Task-agnostic Insights outperform task-specific Insights by +1.1% on average across the three-benchmark subset: LiveCodeBenchv6 +0.6% (0.887 → 0.893), SWE-Bench Verified +1.0% (0.617 → 0.627), ReplicationBench +1.5% (0.067 → 0.082). This is a smaller effect than the full format comparison (Insight vs. Trajectory on ReplicationBench: +7.8%), which makes sense — both subsets use the same Insight format, so the abstraction range within the format is narrower than the range across formats. The existence of a within-format effect, however, strengthens the causal claim that abstraction itself (not format-specific confounds like memory length or structure) drives transfer effectiveness.

Negative Transfer Analysis (Section 4.4.1, Appendix B Table 9)

The paper identifies three distinct categories of negative transfer by analyzing instances where zero-shot succeeded but MTL failed (Section 4.4.1):

Domain-mismatched anchoring: "Structurally irrelevant but superficially similar memories act as misleading anchors," introducing incorrect assumptions that divert reasoning from core task logic. The Appendix B case study (Table 9) shows a concrete example: a Workflow memory about creating R-language source files using heredoc syntax (cat <<'EOF' > solution.txt) is retrieved for a C++ task. The agent blindly follows the file-writing pattern, overwriting existing C++ files without checking their original structure or namespaces — a failure caused by misapplying a language-specific pattern to an incompatible environment.
False validation confidence: "Verification memories can create a false sense of certainty," leading to "self-confirming loops where agents rely on superficial checks instead of formal criteria." The second Appendix B case study illustrates this: an Insight memory about pre-flight verification of datasets and checkpoints (intended to prevent wasting time on broken experiments) is distorted by the agent into a justification for a quick, low-quality training run ("I will perform a quick, low-cost training run... to keep this as a short smoke test rather than a full long run"). The memory's valid guidance about checking prerequisites is semantically distorted into a shortcut rationale.
Misapplied best-practice transfer: "Successful patterns are sometimes transferred indiscriminately, overriding task-specific semantics," causing "procedural over-engineering and rigid adherence to familiar workflows that violate new task requirements." The paper describes this as a failure of adaptation — the pattern itself is valid, but its application context is misjudged.

The paper attributes these failures primarily to "wrong memory retrieval and failed adaptation of the retrieved memory to the new task" and suggests that "advanced memory retrieval methods" and "better memory adaptation methods, such as memory rewriting" could mitigate them. However, the retrieval method experiments in Section 4.4.5 (Table 7) show that LLM reranking and adaptive rewriting both underperform simple embedding similarity, indicating that these are non-trivial challenges.

Memory Pool Scaling (Section 4.4.3, Figure 6)

The scaling experiments use Pass@1 (not Pass@3) to provide finer granularity for detecting small improvements.

Pool size scaling (Figure 6, left). As the Insight memory pool is randomly subsampled from 1/4 to full size, the average ΔPass@1 across three benchmarks increases approximately monotonically:

1/4 pool: ~+0.5%
2/4 pool: ~+1.0%
3/4 pool: ~+1.5%
Full pool: ~+2.0%

The relationship is roughly linear, with no evidence of saturation at the full pool size of 431 memories. This suggests that larger pools would likely yield further gains, though the paper does not test this beyond the available data.

Source domain scaling (Figure 6, right). Varying the number of benchmarks used as memory sources from 0 to 9 shows:

0 domains (zero-shot): baseline
2 domains: ~+0.5%
5 domains: ~+2.0%
9 domains: ~+2.2%

The marginal benefit of additional domains diminishes — the jump from 2 to 5 domains provides most of the gain, with 9 domains adding only a small further improvement. This pattern suggests that diversity matters substantially at small pool sizes but the most common types of transferable meta-knowledge are covered by a moderate number of diverse domains. The paper does not identify which specific domain combinations are most complementary, leaving open the question of whether certain benchmark pairs transfer better than others.

Interpretation for Core Finding 5: The scaling results support the claim that "the effectiveness of Memory Transfer Learning scales with the size of the memory pool and the number of domains," but the effect sizes are modest (+0.5% to +2.2% in ΔPass@1). The practical implication is that accumulating cross-domain memories provides cumulative benefits, but the per-memory or per-domain marginal gain is small — large pools are needed to realize substantial absolute improvements.

Cross-Model Memory Transfer (Section 4.4.4, Table 6)

Table 6 evaluates whether memories generated by one model benefit a different model at inference time. Pass@1 results on the three-benchmark subset:

Source Model	Target Model	Avg. Pass@1	Δ vs. Zero-shot
(Zero-shot)	GPT-5-mini	0.515	—
DeepSeek V3.2	GPT-5-mini	0.518	+0.3%
Qwen3-Coder	GPT-5-mini	0.528	+1.3%
GPT-5-mini	GPT-5-mini	0.543	+2.8%
(Zero-shot)	DeepSeek V3.2	0.486	—
GPT-5-mini	DeepSeek V3.2	0.501	+1.5%
DeepSeek V3.2	DeepSeek V3.2	0.511	+2.5%
(Zero-shot)	Qwen3-Coder	0.402	—
GPT-5-mini	Qwen3-Coder	0.413	+1.1%
Qwen3-Coder	Qwen3-Coder	0.413	+1.1%

Three patterns emerge. First, cross-model transfer always helps relative to zero-shot — every cross-model pair shows positive Δ, ranging from +0.3% (DeepSeek → GPT-5-mini) to +1.5% (GPT-5-mini → DeepSeek). This supports the meta-knowledge hypothesis: if transferred value lies in model-agnostic procedural guidance, memories from any model should provide some benefit. Second, self-generated memories consistently outperform cross-model memories — the same-model rows (bolded) always achieve the highest Pass@1. The paper attributes this to "model-specific biases" in memory generation: each model encodes experiences in slightly different ways, and the retrieving model best understands its own encoding patterns. Third, transfer from stronger to weaker models appears directionally effective: GPT-5-mini memories improve DeepSeek by +1.5% and Qwen3-Coder by +1.1%, while DeepSeek memories improve GPT-5-mini by only +0.3%. The paper does not analyze this asymmetry, but it is consistent with a stronger model generating higher-quality (more abstract, more generalizable) memories that benefit weaker models more than the reverse.

Retrieval Method Comparison (Section 4.4.5, Table 7)

Table 7 compares three retrieval strategies for Insight memories on the three-benchmark subset using Pass@3:

Retrieval Method	LCB	SWEB	RepliB	Avg.
No Memory (zero-shot)	0.910	0.730	0.111	0.584
LLM Reranking	0.920	0.730	0.144	0.598
Adaptive Rewriting	0.920	0.760	0.144	0.608
Embedding Similarity	0.930	0.770	0.189	0.630

LLM Reranking retrieves 20 candidate memories by embedding similarity, then prompts the LLM to select the 3 most helpful for the given task. This underperforms simple embedding similarity on all three benchmarks, with the largest gap on ReplicationBench (0.144 vs. 0.189). The paper's explanation — "the required knowledge is difficult to anticipate in dynamic, multi-step agent settings" — suggests that the LLM cannot reliably judge, at the outset of a task, which memories will prove useful during execution.

Adaptive Rewriting retrieves memories by embedding similarity, then prompts the LLM to rewrite them to better align with the target task. This improves over reranking on SWE-Bench Verified (0.760 vs. 0.730 for reranking, matching zero-shot) but still trails simple embedding similarity (0.770). On LiveCodeBench and ReplicationBench, rewriting matches reranking.

The counterintuitive finding — that more sophisticated retrieval strategies degrade rather than improve performance — is a genuinely informative negative result. It suggests that cross-domain retrieval is fundamentally different from standard retrieval-augmented generation: the relevance of a memory for an agentic task depends on the interactive dynamics of the solving process, which are not predictable from the task description alone. Simple embedding similarity may work precisely because it is "dumb" — it retrieves memories that are broadly semantically related, and the agent can adapt or ignore them during execution. Attempts to optimize retrieval (reranking, rewriting) introduce a second layer of LLM judgment that can filter out useful memories or rewrite them in ways that introduce new misinterpretations.

Ablation Studies and Robustness Checks

Memory format abstraction ordering (Table 1, all benchmarks): The Pass@3 results consistently show Insight ≥ Summary ≥ Workflow ≥ Trajectory across four of six benchmarks (SWE-Bench Verified, TerminalBench2, ReplicationBench, MLGym-Bench). The two exceptions — LiveCodeBenchv6 (Trajectory 0.940 vs. Insight 0.930) and Aider-Polyglot (all within 0.02 of zero-shot) — suggest that the abstraction benefit may be modulated by benchmark characteristics. LiveCodeBench's tasks may have sufficient surface similarity that concrete command traces are directly reusable, while Aider-Polyglot's high variance (all results near 0.47) may indicate that no memory format provides consistently relevant guidance on this benchmark.

Within-format abstraction isolation (Table 4): Task-agnostic Insights outperform task-specific ones by +1.1% on average across three benchmarks, confirming that abstraction helps even when format is held constant. The effect is largest on ReplicationBench (+1.5%), the benchmark where zero-shot performance is lowest (0.067 task-specific vs. 0.082 task-agnostic), consistent with the pattern that MTL helps most when the base agent struggles.

Cross-model transfer (Table 6): Self-generated memories outperform cross-model memories in all three model pairs tested (GPT-5-mini, DeepSeek V3.2, Qwen3-Coder). The GPT-5-mini → GPT-5-mini advantage over DeepSeek → GPT-5-mini is +2.5 percentage points (0.543 vs. 0.518), a meaningful gap. This validates that model-specific memory generation biases exist and affect transferability, though cross-model transfer still provides positive gains over zero-shot in all cases.

Memory pool size scaling (Figure 6, left): Performance improves roughly linearly with pool size from 1/4 to full on three benchmarks, measured in ΔPass@1. The paper does not ablate the retrieval depth (N = 3) — it is possible that larger pools would benefit from retrieving more than 3 memories, but this interaction is not tested.

Source domain count scaling (Figure 6, right): Marginal returns diminish after 5 domains, suggesting that the most common types of transferable meta-knowledge are covered by moderate diversity. The paper does not report which domain combinations are most complementary, leaving open the question of whether certain source-target benchmark pairs transfer better than others (e.g., does MLGym-Bench memory help SWE-Bench more than LiveCodeBench memory does?).

Retrieval method comparison (Table 7): Simple embedding similarity outperforms both LLM Reranking and Adaptive Rewriting across all three benchmarks, with the largest margin on the hardest benchmark (ReplicationBench: embedding 0.189 vs. rewriting 0.144). This is a robustness check in the negative direction: more sophisticated retrieval does not help and can hurt, confirming that cross-domain retrieval is a non-trivial challenge.

Pass@1 vs. Pass@3 consistency (Table 1 vs. Appendix A Table 8): The abstraction ordering and MTL benefit direction are preserved across both metrics. The magnitude compression from Pass@3 to Pass@1 (e.g., GPT-5-mini Insight: +3.7% Pass@3 vs. +1.9% Pass@1) is expected and suggests that memory helps both by improving per-attempt quality and by increasing the probability that at least one of three attempts succeeds.

Number of benchmarks for self-evolving comparison (Table 2): The comparison with ReasoningBank and AgentKB is only on 3 of the 6 benchmarks (LiveCodeBenchv6, SWE-Bench Verified, ReplicationBench). The paper does not explain why the other three benchmarks (Aider-Polyglot, TerminalBench2, MLGym-Bench) are excluded, limiting the generalizability of the claim that MTL outperforms self-evolving methods.

Single-retrieval-depth design (all experiments): All experiments use N = 3 retrieved memories. The paper does not ablate this parameter — we do not know whether retrieving 1, 5, or 10 memories would change the relative ordering of formats or the magnitude of MTL gains. This is a notable absence because negative transfer could potentially be mitigated by retrieving fewer memories (reducing the chance of pulling in a misleading one) or by retrieving more (diluting the influence of any single bad memory).

Offline-only memory pool (no online accumulation): The memory pool is static — memories are generated once from initial trajectories and never updated during evaluation. The paper does not test online memory accumulation, where the agent adds its own new experiences to the pool as it solves tasks. This limits ecological validity: in a real deployment, the memory pool would grow over time, potentially changing the transfer dynamics.

Critical Assessment

Claim: "Memory Transfer Learning improves average performance by 3.7%"

What was actually tested: The 3.7% figure (Table 1, GPT-5-mini, Insight vs. zero-shot, average across 6 benchmarks) is based on 100 tasks per benchmark with Pass@3 as the metric. The per-benchmark variance is substantial: gains range from 0.0% (Aider-Polyglot) to +8.3% (MLGym-Bench). The average is pulled up by two benchmarks with large gains (ReplicationBench +7.8%, MLGym-Bench +8.3%); the median gain across benchmarks is approximately +4.25% (between SWE-Bench's +4.0% and TerminalBench2's +4.5%).

Genuine strengths: The result is replicated across three model families (GPT-5-mini, DeepSeek V3.2, Qwen3-Coder) with consistent directionality, and across both Pass@3 and Pass@1 metrics. The cross-model transfer experiments (Table 6) provide additional evidence that the effect is robust to model choice, though magnitude varies.

Weaknesses: No confidence intervals or significance tests are reported. With 100 tasks per benchmark, a Pass@3 difference of 3.7 percentage points on average means roughly 2-4 additional tasks succeeded out of 100, depending on the benchmark. Without variance estimates, we cannot assess whether differences of this magnitude are statistically reliable or could arise from sampling noise. The paper also does not report whether the same 100 tasks are used for zero-shot and MTL evaluation; if different random task subsets were used, sampling variance could account for some of the observed differences.

Conditionality: The 3.7% average masks extreme heterogeneity. On Aider-Polyglot, MTL provides essentially no benefit (0.0% change). On MLGym-Bench, it provides an 8.3% gain. Practitioners cannot expect a uniform 3.7% improvement; the actual gain depends heavily on the target benchmark. The paper does not provide a way to predict which benchmarks will benefit most from MTL without running the experiment.

Claim: "The primary transferable value lies in meta-knowledge, not task-specific code"

What was actually tested: The categorization analysis (Figure 3) uses GPT-5 to label how transferred memories contributed to success, finding that 5.5% of contributions involve algorithmic strategy transfer while the remaining ~94.5% involve various forms of procedural guidance.

Genuine strengths: The case study in Table 3 provides a concrete, interpretable example of meta-knowledge transfer that aligns with the quantitative categorization. The embedding space visualizations (Figure 4) independently support the claim: if memories primarily transferred task-specific code, Insight embeddings would cluster by benchmark (since task-specific code is benchmark-specific); the fact that they are "sparse and intermingled" suggests the preserved information is domain-invariant, consistent with meta-knowledge.

Weaknesses: The categorization is performed by GPT-5, the same model used for the agent. There is no human validation of the category assignments, no inter-annotator agreement reported, and no second LLM used for verification. The categories themselves (Appendix D, Table 10) overlap substantially — "Iterative Workflow Discipline" and "Test Driven Verification" and "Anti-Pattern Avoidance" could all apply to the same memory contribution, and the LLM's assignment to one category rather than another is subjective. The 5.5% algorithmic transfer figure should be interpreted as an approximate estimate, not a precise measurement.

The paper also does not report the total number of instances analyzed for Figure 3. If only a small number of tasks showed zero-shot→MTL success transitions (which is plausible given the modest average gains), the percentage breakdown is based on a small sample and individual misclassifications could meaningfully shift the distribution.

Claim: "Abstraction dictates transferability; high-level insights generalize well, whereas low-level traces often induce negative transfer"

What was actually tested: Three converging lines of evidence: (1) format-level Pass@3 ordering (Insight > Summary > Workflow > Trajectory across 4/6 benchmarks), (2) embedding space analysis (DBI and LISI metrics showing weaker domain clustering for more abstract formats), (3) within-format isolation (task-agnostic Insights outperform task-specific Insights by +1.1% on 3 benchmarks).

Genuine strengths: The within-format experiment (Table 4) is the strongest evidence because it controls for format-level confounds. The embedding space analysis provides convergent validity from a completely different methodology (unsupervised clustering metrics rather than downstream task performance). The consistent ordering across multiple benchmarks and models reduces the likelihood that the pattern is an artifact of a specific dataset or model.

Weaknesses: The within-format effect is small (+1.1% average on 3 benchmarks) and may not be statistically significant — the paper provides no variance estimates. The task-specific vs. task-agnostic partitioning uses an LLM to reconstruct the original task from the memory content and measure similarity; this proxy for "abstraction" is itself noisy, and errors in the similarity measurement would attenuate the observed effect.

More fundamentally, the paper's claim is that abstraction causes transferability, but the evidence shows only a correlation between abstraction (operationalized as format type or task reconstruction difficulty) and transfer performance. An alternative explanation is that Insight memories are simply shorter (less context window consumption) or better formatted (title + description + content structure is easier for the agent to parse) than Trajectory memories, and these format-level properties rather than abstraction per se drive the performance difference. The within-format experiment partially addresses this by holding format constant, but it does not control for memory length — task-agnostic Insights may be systematically shorter than task-specific ones.

The "negative transfer" part of the claim is supported by specific benchmarks (Trajectory on TerminalBench2: −4.5%, on MLGym-Bench: −8.4%) and case studies (Table 5, Appendix B Table 9), but the paper does not systematically analyze when negative transfer occurs. Is it more common for certain benchmark pairs? For certain types of tasks within a benchmark? The three failure categories (Section 4.4.1) are descriptive but not predictive — we cannot look at a memory and a task and determine whether negative transfer will occur.

Claim: "Transfer effectiveness scales with the size of the memory pool and the number of domains"

What was actually tested: Pool size scaling from 1/4 to full on 3 benchmarks (Figure 6, left) and domain count scaling from 0 to 9 on 3 benchmarks (Figure 6, right), both using Pass@1.

Genuine strengths: The monotonic improvement with pool size is clean and consistent with the proposed mechanism (larger pools → higher probability of retrieving useful memories). The domain count experiment provides a practical guideline: 5 diverse domains capture most of the available gain.

Weaknesses: The Pass@1 differences are small — the full range from 1/4 pool to full pool is approximately +1.5 percentage points in ΔPass@1. Without variance estimates, we cannot distinguish this from noise. The pool size experiment uses random subsampling; the paper does not report whether results vary across different random seeds for subsampling, which they likely do given the small absolute differences. The domain count experiment presumably adds benchmarks in some order (the paper does not specify which benchmarks are added at each step), and the results may depend on which specific domains are included — adding a highly relevant domain might provide a larger boost than adding a less relevant one.

The claim that transfers scale with pool size is supported directionally, but the practical significance is unclear: doubling the pool size from 1/4 to 2/4 yields approximately +0.5% ΔPass@1, which may not justify the computational cost of generating, embedding, and storing the additional memories in a real deployment.

Missing experiments that would have strengthened the paper

Benchmark-pair transfer matrix: The paper never reports which source benchmarks transfer best to which target benchmarks. A 6×6 transfer matrix (row = source, column = target, cell = Pass@3 when only that source's memories are used) would reveal whether certain benchmark pairs have particularly strong or weak transfer, providing actionable guidance for memory pool construction.
Retrieval depth ablation: All experiments use N = 3 retrieved memories. Testing N = 1, 5, 10, and 20 would reveal whether more (or fewer) memories change the abstraction ordering or the magnitude of negative transfer.
Human validation of memory categorization: The Figure 3 percentages and the within-format task-specific/task-agnostic partitioning both rely on LLM judgments. A human evaluation on a subset of cases would calibrate the reliability of these LLM-based analyses.
Statistical significance testing: With 100 tasks per benchmark, it is straightforward to compute confidence intervals for Pass@3 differences (e.g., via bootstrap). Their absence makes it impossible to distinguish real effects from sampling noise, particularly for the smaller effects (e.g., within-format +1.1%, cross-model +0.3% to +1.5%).
Memory format interaction with retrieval method: The paper uses task-based retrieval for Trajectory and plan-based retrieval for the other three formats. An ablation that tests plan-based retrieval for Trajectory (or task-based retrieval for Insight) would disentangle format effects from retrieval method effects — currently they are confounded.
Online memory accumulation: The static memory pool is a significant simplification relative to real deployment. An experiment where the memory pool grows as the agent solves tasks (adding new memories from the target benchmark as it goes) would test whether cross-domain transfer benefits persist or are supplanted by in-domain memories as they become available.
Larger-scale validation: The memory pool scaling experiments stop at 9 domains and ~431 memories (for the 3-benchmark subset). Testing with larger pools (e.g., aggregating memories from dozens of benchmarks or thousands of tasks) would test the extrapolation of the linear scaling trend and reveal whether saturation eventually occurs.

Summary of What the Experiments Do and Do Not Establish

The experiments convincingly establish that cross-domain memory transfer provides positive average benefits for coding agents, with the effect size depending on memory format, target benchmark, and base model. The evidence for the abstraction mechanism is strong in a correlational sense — more abstract formats consistently outperform less abstract ones — but the causal claim that abstraction causes better transfer (as opposed to abstraction being correlated with other beneficial format properties like brevity or structure) is only partially supported by the within-format experiment, which shows a small effect.

The meta-knowledge dominance claim (Figure 3) is intuitively plausible and supported by qualitative case studies, but the quantitative evidence (LLM-based categorization with no human validation, unknown sample size, overlapping categories) is weaker than the paper's confident presentation suggests. The scaling claims are directionally supported but the effect sizes are small enough that statistical noise cannot be ruled out.

The paper's most robust contribution is the systematic demonstration that memory format matters enormously for cross-domain transfer — the performance gap between Insight and Trajectory on individual benchmarks can exceed 16 percentage points (MLGym-Bench: +8.3% vs. −8.4%). This is a practically significant finding even if the precise mechanisms (abstraction vs. format confounds) are not fully isolated: practitioners building memory-augmented coding agents should use abstract, generalized memory formats if they want their agents to benefit from cross-domain experience.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation Is Not Amortized in the Headline Efficiency Gains

The assumption or constraint. The paper's entire framework depends on estimating prompt difficulty before deciding how to allocate memory resources. The method used to construct difficulty-dependent compute-optimal policies requires generating 2048 samples per question and averaging PRM scores or ground-truth correctness across them — a procedure that is orders of magnitude more expensive than the test-time compute budgets being optimized. The authors are transparent about this gap:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is already known, without including the cost of acquiring that knowledge. In a realistic deployment, the total cost would be difficulty estimation + memory-augmented inference, and the former could easily dominate. If estimating difficulty for a single question requires generating 2048 samples, then a system that purports to achieve best-of-256 performance with only 64 generations has actually consumed 2048 + 64 = 2112 generations — making the claimed efficiency illusory. The 4× figure should be understood as an upper bound on achievable efficiency under the assumption of free difficulty labels, not a realized deployment gain.

What evidence exists in the paper. The paper explicitly flags this limitation in Section 3.2 and mentions it as a "key avenue for future work," but never quantifies the difficulty estimation cost in the same units as the test-time compute budgets being studied. The per-question cost of 2048 samples is not compared against the generation budgets on the x-axes of any figure (which typically range from 1 to 512 generations). The reader cannot determine at what point the difficulty estimation cost renders the adaptive allocation strategy net-negative relative to a uniform best-of-N baseline.

Mitigation status. Not addressed. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8), but no such model is developed or evaluated. The predicted difficulty bins (using PRM scores rather than ground-truth labels) eliminate access to oracle answers but do not reduce the computational cost — they still require 2048 samples per question. An adaptive difficulty estimation approach (start with few samples, estimate difficulty, then allocate remaining budget) is mentioned as future work but not explored.

All Results Are on a Single Benchmark Family (MATH) with a Single Model Family (PaLM 2-S*)

The assumption or constraint. Every experiment — the search analysis, the revision analysis, the difficulty-dependent allocation, the FLOPs-matched comparison — uses the MATH benchmark (Hendrycks et al., 2021) with PaLM 2-S* as the base model. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but provide no evidence from other model families or task domains.

The consequence. The paper's key findings may not generalize. Several aspects of the results could be model-specific or benchmark-specific in ways that affect the practical recommendations:

PRM quality and over-optimization behavior: The difficulty-dependent search patterns (beam search degrades on easy problems, helps on medium problems) depend on the PRM's calibration properties, which are a function of PaLM 2-S*'s output distribution. A model with different error patterns (e.g., more confident but less accurate, or vice versa) could produce different over-optimization thresholds and therefore different compute-optimal strategies.
Revision model trainability: The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities. The paper uses edit-distance-based pairing to construct revision training data — a technique that may work differently (or not at all) with models that have different in-context learning behaviors.
Domain specificity of the difficulty pattern: The finding that difficulty-dependent allocation provides 4× gains over best-of-N may be specific to mathematical reasoning, where difficulty is relatively well-defined (multi-step symbolic deduction) and verifiers can be trained on structure-rich step-by-step solutions. The paper does not test whether the same patterns hold for code generation (where correctness is binary and step-level scoring may be harder), logical reasoning, or open-ended generation.

Additionally, the MATH benchmark consists exclusively of problems with ground-truth answers that can be evaluated with exact string matching. This enables both oracle difficulty estimation (pass@1) and PRM training (Monte Carlo rollouts that check final answer correctness). Many real-world tasks — code generation, dialogue, creative writing — lack such clean correctness signals, and the entire difficulty estimation and verifier training pipeline would need to be redesigned for those settings.

The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation for strategy selection (~50 per fold per bin), means the compute-optimal policy is selected based on roughly 50 examples per difficulty bin. This is a small sample for strategy selection, and the selected strategies may not be robust. The paper does not report confidence intervals on the compute-optimal scaling curves (Figures 4, 8, 9), making it difficult to assess whether small differences between strategies (e.g., beam search vs. best-of-N at a particular budget and difficulty level) are reliable or could reverse with a different test set split.

What evidence exists in the paper. The single-benchmark, single-model-family nature is stated in Section 4 but justified only by the authors' belief in model representativeness. There is no ablation testing whether results hold on a different benchmark (e.g., GSM8K for math reasoning, HumanEval for code generation) or a different model family (e.g., GPT-4, LLaMA, Gemini).

Mitigation status. Not addressed as a limitation. The paper does not claim generalization beyond MATH/PaLM 2-S* but also does not caution readers about the specificity of the findings. The discussion in Section 8 focuses on future extensions (combining search and revisions, better verifiers, self-improvement loops) rather than on validating the current results across domains.

The 14× Larger Model Baseline in the FLOPs-Matched Comparison Is Not Compute-Optimally Trained

The assumption or constraint. In the FLOPs-matched comparison (Section 7), the paper scales only model parameters when increasing pretraining compute, holding training data fixed — following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal approach of scaling both parameters and data equally (Hoffmann et al., 2022). The authors acknowledge this explicitly:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling both parameters and data) would likely outperform a parameter-only-scaled model at the same total training budget. By using a weaker pretraining baseline, the paper may overstate how much test-time compute can substitute for pretraining compute. The headline finding that "a smaller model with test-time compute can outperform a 14× larger model" (Section 7, Figures 1 and 9) may shrink or reverse against a properly compute-optimal larger model.

Furthermore, the 14× larger model uses only greedy decoding in the comparison — no majority voting, no best-of-N, no search. This is an asymmetric comparison: the smaller model gets compute-optimal test-time scaling (combining search, revision, and difficulty-based allocation), while the larger model gets no test-time compute at all. A fairer comparison would give the larger model at least a modest test-time compute budget (e.g., best-of-8 with majority voting), which would substantially strengthen the baseline. The paper's approach stacks the deck in favor of test-time compute by giving it all the inference-time optimization while giving pretraining none.

What evidence exists in the paper. The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 show test-time compute outperforming the larger model on easy-to-medium problems at low R values, but these results are conditional on the specific (non-optimal) pretraining baseline. The paper does not report what a Chinchilla-optimal 14× larger model would achieve, nor does it give the larger model any test-time compute budget.

Mitigation status. Acknowledged but not addressed. The paper flags the parameter-only scaling choice and defers compute-optimal pretraining comparisons to future work, but does not discuss how this choice might affect the interpretation of the FLOPs-matched results. The comparison is presented as evidence of a training-inference tradeoff without adequate caveats about the baseline strength.

Verifier Over-Optimization Is a Hard Ceiling That the Compute-Optimal Policy Mitigates But Does Not Solve

The assumption or constraint. The paper identifies verifier over-optimization as the primary bottleneck preventing unbounded improvements from test-time compute: beam search degrades easy-problem performance at high budgets (Figure 3, right); lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left); and qualitative examples show search producing degenerate outputs (repetitive low-information steps, overly short solutions; Appendix M, Figures 29, etc.) that score highly under the PRM.

The compute-optimal policy works around this bottleneck by routing easy problems away from aggressive search (using best-of-N where the verifier is reliable) and applying search only to medium-hard problems (where the verifier signal has room to provide genuine guidance). But it does not eliminate the bottleneck. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling — the beam search curves in Figure 3 flatten and sometimes decline well before the budget is exhausted.

The consequence. The compute-optimal approach is fundamentally bounded by verifier quality, and improving the verifier would likely shift the entire difficulty-dependent strategy landscape. If a more robust PRM were available — one that remained calibrated under aggressive optimization — then beam search or lookahead search might be optimal for easy problems as well, potentially changing the compute-optimal policy qualitatively. The paper's specific recommendations (use best-of-N on easy problems, beam search on medium problems) are specific to the PRM quality achievable with the Monte Carlo rollout training procedure; they are not universal principles.

More practically, this means that the 4× efficiency gain is not a fixed property of compute-optimal allocation but rather a function of the current verifier's reliability. As verifiers improve (through better training data, adversarial robustness, ensemble methods, or architectural advances), the efficiency ceiling will rise, and the current paper's results will become a lower bound. The paper does not explore this sensitivity or provide guidance on how verifier quality affects the optimal policy.

What evidence exists in the paper. Figure 3 (right) is the clearest evidence: easy-problem accuracy under beam search decreases with increasing budget, while medium-problem accuracy increases, and hard-problem accuracy is flat near zero. Figure 3 (left) shows lookahead search underperforming all methods at the same budget despite being the most powerful optimizer. The qualitative failure modes are shown in Appendix M. However, the paper does not ablate verifier quality — we do not know what the compute-optimal policy would look like with a better PRM, or how much the 4× efficiency figure would improve if the PRM were more robust.

Mitigation status. The paper identifies the problem (Section 8: "improving verifier robustness is the key bottleneck") but does not address it experimentally. No experiments test alternative PRM training procedures (e.g., adversarial training on search-generated solutions, ensemble verification, KL-constrained search). The over-optimization ceiling is documented but left as a barrier for future work.

Sequential Revision Strategies Introduce Latency That Is Ignored in the "Generations" Cost Model

The assumption or constraint. The paper measures test-time compute in "generations" — the number of complete solutions sampled — which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial: each revision depends on the previous one, meaning a chain of 64 sequential revisions takes roughly 64× the wall-clock time of 64 parallel independent samples, assuming sufficient hardware to run the parallel samples simultaneously.

This matters because the compute-optimal policy on easy problems (Section 6.2, Figure 7 right) strongly favors purely sequential revisions — a chain of many revisions with little or no parallel sampling. The optimal sequential-to-parallel ratio for easy questions (bin 1) has performance essentially flat across all ratios (around 90–92%), but the fully sequential configuration would take far longer in wall-clock time than the fully parallel configuration, even though both use the same number of generations.

The consequence. For latency-sensitive applications — interactive coding assistants, real-time debugging, any deployment where users wait for responses — the sequential-heavy strategies favored by the compute-optimal policy on easy problems may be impractical regardless of their accuracy advantages. A user waiting 30 seconds for a sequential chain of 64 revisions (each requiring a full LLM forward pass) may prefer a parallel strategy that produces an answer in 2 seconds with slightly lower accuracy. The paper's cost model (generations) implicitly assumes that all generations have equal value regardless of when they become available, which is true for throughput-oriented batch processing but false for latency-sensitive interactive use.

The FLOPs-matched comparison (Section 7) compounds this issue: the 14× larger model with greedy decoding has a latency of exactly one forward pass, while the smaller model with compute-optimal test-time scaling may require dozens or hundreds of sequential forward passes. In a latency-matched comparison (rather than FLOPs-matched), the larger model could be substantially more competitive.

What evidence exists in the paper. The paper never discusses latency, wall-clock time, or the serial dependency structure of sequential revisions. The "generations" metric is defined in Section 5.3 as the universal unit of test-time compute, with no mention of how it maps to real time. The sequential-to-parallel ratio sweep in Figure 7 shows the trade-off space but only reports accuracy, not latency.

Mitigation status. Not addressed. The paper does not acknowledge latency as a limitation, does not report wall-clock times for any experiment, and does not discuss the throughput-latency tradeoff inherent in sequential vs. parallel allocation. This is a significant omission for a paper whose practical recommendations (use sequential revisions on easy problems) have direct latency implications.

The Revision Model Has a ~38% Correct-to-Incorrect Reversion Rate That the Selection Mechanism Only Partially Mitigates

The assumption or constraint. The revision model is trained exclusively on trajectories where all in-context answers are incorrect, followed by a correct target answer. This training regime teaches the model to revise incorrect answers into correct ones but provides no signal for what to do when the current answer is already correct. As a result, at inference time, the model sometimes "revises" a correct answer into an incorrect one — a phenomenon the paper quantifies:

"approximately 38% of correct answers get converted back to incorrect ones" (Section 6.1)

The consequence. This reversion problem fundamentally limits the effectiveness of long sequential revision chains. If 38% of correct answers are lost at each revision step, then a chain of length N has a compounding risk of losing correct solutions that were found at earlier steps. In the worst case, a correct answer at step 3 might be revised to incorrect at step 4, then revised to a different incorrect answer at step 5, and so on — the chain can drift away from correctness.

The paper mitigates this with a selection mechanism: majority voting or verifier-based selection across the entire chain of revisions, picking the best answer from any point in the chain rather than always taking the last revision. This ensures that a correct answer at step 3 is not lost even if later steps degrade it. However, this mitigation has two weaknesses. First, it requires storing every intermediate answer in the chain, increasing memory usage (this is trivial for 64 revisions but matters at larger scales). Second, and more importantly, the selection mechanism must correctly identify which revision in the chain is correct — and this identification itself relies on the same (imperfect) verifier or majority-voting mechanism that the revision model is trying to improve upon. If the verifier misranks a later incorrect revision above an earlier correct one, the correct answer is still lost.

More fundamentally, the reversion problem means the revision model is wasting compute: 38% of its revision steps actively destroy value by turning correct answers into incorrect ones. If these wasted steps could be avoided — for example, by training the model to recognize when no revision is needed and output a special "DONE" token — the same compute budget would produce substantially better results.

What evidence exists in the paper. The 38% figure is reported in Section 6.1 but is not derived from a dedicated experiment — it appears to come from analyzing revision chains and counting how often a correct answer at step k becomes incorrect at step k+1. The paper does not report how this rate varies with chain position (does the reversion rate increase or decrease deeper into the chain?) or with problem difficulty (are easy problems more or less susceptible to reversion than hard problems?). The ReST^EM experiment (Appendix K, Figure 16) provides additional evidence that revision training is fragile: attempting to optimize the revision model with on-policy RL-style training caused performance to degrade substantially with sequential revisions.

Mitigation status. Partially addressed. The chain-wide selection mechanism (majority voting or verifier) mitigates the loss of correct answers but does not prevent the generation of incorrect revisions in the first place. The paper acknowledges the problem and its training-data cause but does not propose a solution beyond the selection mechanism. A more principled fix — such as including correct→correct trajectories in the training data, or training a separate "stop criterion" that determines when further revisions are unlikely to help — is not explored.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper transforms memory-based self-evolving agents from a single-domain optimization problem into a cross-domain generalization challenge. The field's default assumption—that memories are most valuable when retrieved within the same task domain that generated them—was so deeply embedded that it functioned as architecture rather than hypothesis. Prior work (ReasoningBank by Ouyang et al., 2025; AWM by Wang et al., 2024c; ReMe by Cao et al., 2025) built retrieval pipelines around in-domain similarity without questioning whether out-of-domain memories might be valuable, while AgentKB (Tang et al., 2025) demonstrated cross-domain benefit without explaining its mechanism. MTL provides both the existence proof and the mechanistic account: cross-domain memory does help, and it helps primarily through procedural meta-knowledge rather than through task-specific code reuse.

This is a diagnostic reframing rather than a paradigm shift. The paper does not introduce a new agent architecture, a new training objective, or a new memory format. Instead, it changes how we think about memory value: the question shifts from "is this memory from a similar task?" to "does this memory encode transferable behavioral guidance?" This reframing has direct design implications—it tells practitioners to generate memories that strip away domain-specific details (Insight format, with its explicit "do not mention specific files or details" instruction) rather than preserving them (Trajectory format)—but it does so by explaining why abstraction works (meta-knowledge transfer) rather than simply showing that it works (a format comparison).

The paper's most significant conceptual contribution is reconciling an apparent contradiction in the self-evolving agent literature. On one hand, several papers demonstrated that memory helps agents—ReasoningBank showed Insight memories providing within-domain gains, AWM showed workflow extraction improving web agents, and ReMe showed memory refinement outperforming static memory pools. On the other hand, the assumption that memory is domain-specific implied that these gains were inherently bounded: each domain required its own memory generation effort, and experiences from one benchmark were worthless for another. MTL resolves this by showing that the value of memory is not domain-specific at all—the same Insight memory generated from a competitive coding task can help fix a Django bug or train an ML model, because the transferred value lies in procedural knowledge (how to test, how to verify, how to navigate environments) that transcends task boundaries. This explains why ReasoningBank's in-domain approach underperformed MTL (Table 2: 0.601 vs. 0.630 average Pass@3): it was leaving the most transferable part of its own memories—the meta-knowledge—underutilized by restricting retrieval to in-domain contexts.

The paper also redirects research attention in two specific ways. First, it makes memory abstraction a first-class design target. Before this work, memory format was an implementation detail—some papers used trajectories, some used summaries, some used insights, but there was no systematic understanding of how format quality interacted with transfer effectiveness. The paper's demonstration that abstraction level (Insight > Summary > Workflow > Trajectory) predicts transfer performance across benchmarks and models establishes abstraction as a design principle rather than a stylistic choice. Future memory generation pipelines will need to justify their abstraction level, and the Insight format's explicit generalization instruction ("do not mention specific files or details") provides a concrete recipe for achieving high abstraction.

Second, it makes negative transfer a diagnosable phenomenon rather than an amorphous failure mode. By decomposing negative transfer into three categories—domain-mismatched anchoring, false validation confidence, and misapplied best-practice transfer (Section 4.4.1)—the paper provides a diagnostic vocabulary that enables systematic improvement. Each category corresponds to a different engineering challenge: retrieval quality (fix anchoring by retrieving better memories), confidence calibration (fix false validation by integrating verifier feedback), and adaptation fidelity (fix misapplication by rewriting memories for the target context). The retrieval experiments in Table 7—which show that naive LLM reranking and adaptive rewriting both underperform simple embedding similarity—demonstrate that these are non-trivial problems requiring genuine research advances, not simple engineering fixes. This negative result is as informative as the positive MTL results: it tells the field that cross-domain retrieval is fundamentally harder than standard retrieval-augmented generation and needs dedicated algorithmic development.

The paper's emphasis on meta-knowledge transfer (Figure 3, Table 3) has a subtle but important implication for the self-evolution research agenda. If the primary value of memory lies in procedural guidance rather than domain-specific code, then memory generation should prioritize extracting and preserving how the agent solved a task (its strategy, verification habits, environmental adaptation) rather than what the agent produced (the specific bash commands, file edits, or code snippets). This inverts the natural tendency in trajectory-based memory systems, which preserve the concrete actions and discard the reasoning. The paper's Trajectory format—which strips reasoning and keeps only commands and observations—is the empirical worst performer because it discards the meta-knowledge and preserves only the domain-specific surface forms that cause brittle anchoring.

The scaling results (Section 4.4.3, Figure 6) establish what amounts to an inference-time memory scaling law: more diverse memories from more domains produce monotonically improving transfer performance. This has practical consequences for how memory systems should be designed. Self-evolving agents should accumulate memories across as many diverse coding domains as possible, because the value of the pool compounds with diversity. The diminishing marginal returns after 5 domains suggest that the most common meta-knowledge patterns are covered relatively quickly, but the continued small improvements out to 9 domains indicate that niche patterns (specific environmental adaptation tricks, rare anti-patterns) continue to provide value. This scaling behavior also explains why AgentKB's massive pool (5,899 memories) underperformed MTL's smaller, more curated pool (431 memories, Table 2): raw quantity without abstraction control adds noise that dilutes the meta-knowledge signal.

Follow-Up Research This Work Enables

Benchmark-pair transfer matrix to identify complementary coding domains. The paper evaluates MTL by aggregating memories from all non-target benchmarks into a single pool, but never reports which specific source-target benchmark pairs transfer best. A natural follow-up would construct a 6×6 transfer matrix where each cell (i, j) reports Pass@3 on benchmark j when using only memories from benchmark i. This would reveal whether certain benchmark pairs have particularly strong transfer (e.g., does MLGym-Bench memory help SWE-Bench more than LiveCodeBench memory does?) or whether all cross-domain pairs contribute roughly equally. The paper's Figure 3 categorization (Iterative Workflow Discipline 15%, Test Driven Verification 14.5%, Anti-Pattern Avoidance 14.4%) provides hypotheses: if the dominant transfer categories are environmental and procedural, then benchmarks sharing runtime environments (Linux shells, Python ecosystems) should transfer better than those differing in environment. A strong follow-up would test this by comparing transfer between Python-heavy benchmarks (MLGym-Bench, ReplicationBench) vs. transfer to multi-language benchmarks (Aider-Polyglot) where environment mismatch is higher.

Online memory accumulation with dynamic abstraction control. The paper's memory pool is static—generated once from initial trajectories and never updated during evaluation. In a real deployment, the agent would generate new memories continuously as it solves tasks, and these new memories could be from the target domain itself. A critical open question is whether cross-domain transfer benefits persist when in-domain memories become available, or whether in-domain memories dominate and the cross-domain pool becomes irrelevant. A direct extension would test a deployment where the agent starts with only the cross-domain Insight pool, solves tasks on the target benchmark, generates new Insight memories from its own trajectories, adds them to the pool, and measures whether performance continues to improve (suggesting cross-domain memories provide foundational benefits that in-domain memories build on) or plateaus (suggesting cross-domain memories are only valuable in the absence of in-domain experience). The paper's pool size scaling results (Figure 6, left: +0.5% at 1/4 pool to +2.0% at full pool) predict that adding in-domain memories should yield further gains, but the magnitude and whether they might replace rather than supplement cross-domain memories is unknown.

Training a dedicated memory abstraction model to maximize A(m). The formal model in Appendix C defines abstraction level A(m) as the ratio of domain-invariant to total embedding norm, and Proposition 1 states that expected transfer utility increases with A. This suggests a concrete training objective: fine-tune a memory generator to maximize A(m) while preserving retrieval utility. The current Insight format achieves abstraction through a prompting instruction ("do not mention specific files or details"), but this is a blunt instrument—the LLM may over-abstract (losing useful meta-knowledge) or under-abstract (retaining harmful domain-specific anchors). A follow-up could train a memory generation model with a multi-task loss: (1) a reconstruction loss that ensures the memory preserves meta-knowledge (e.g., can the memory be used to predict the success/failure of a related trajectory?), (2) an adversarial loss that penalizes domain-specific information (train a domain classifier on memory embeddings and maximize its error), and (3) a transfer utility loss measured on held-out cross-domain tasks. The paper's within-format experiment (Table 4: task-agnostic Insights outperform task-specific ones by +1.1%) provides the proof of concept that within a single format, more abstract memories transfer better—a trained abstraction model should be able to push this further, potentially closing the gap between Insight and an even more abstract, purpose-optimized format.

Retrieval methods designed for agentic, multi-step transfer. The paper's most surprising negative result is that LLM reranking and adaptive rewriting both underperform simple embedding similarity for cross-domain memory retrieval (Table 7). The authors attribute this to the difficulty of anticipating memory relevance in "dynamic, multi-step agent settings"—the LLM cannot judge, at the outset of a task, which memory will prove useful during execution because usefulness depends on the interactive trajectory of the solving process. This suggests that retrieval should be dynamic: rather than retrieving memories once at the start, the agent should retrieve memories at each step based on the current state of the environment and the partial solution so far. A concrete experiment would modify the agent loop to: (1) at each step, embed the current observation (the shell output from the last command) plus the current reasoning, (2) retrieve the top-1 memory whose embedding is most similar to this state embedding, (3) inject only that memory into the next reasoning step. This "step-wise retrieval" would test whether the right memory can be surfaced at the right moment—e.g., a memory about test-driven verification appearing right when the agent is about to submit code, rather than at the beginning when the agent is still exploring the repository. The ReMe framework (Cao et al., 2025) introduced step-wise retrieval in a single-domain setting; extending it to cross-domain transfer would be a natural synthesis.

Human validation study of the meta-knowledge categorization. The Figure 3 analysis categorizing memory contributions into 10 types (Iterative Workflow Discipline 15%, Test Driven Verification 14.5%, etc.) is performed entirely by GPT-5, the same model used as the agent. The categories overlap (a memory about writing inline tests to validate fixes could be classified as Test Driven Verification, Iterative Workflow Discipline, or Anti-Pattern Avoidance), and there is no human validation or inter-annotator agreement reported. A human study with expert software engineering annotators would: (1) validate or re-weight the category distribution—if humans assign "Algorithmic Strategy Transfer" to 15% of cases rather than 5.5%, the entire meta-knowledge dominance claim would need revision; (2) identify whether the LLM systematically misclassifies certain types of contributions, providing a calibration baseline for future automated analyses; (3) surface whether there are additional contribution categories that the LLM's pre-defined taxonomy missed. This study would be relatively low-cost (annotate 50–100 transition cases from the zero-shot→MTL success set) and would substantially strengthen or qualify Core Finding 2.

Cross-language and cross-framework transfer stress test. All memories in the paper are generated from coding tasks that share a common infrastructure (Linux shells, predominantly Python, standard Unix tools). The paper argues that this shared infrastructure is what makes cross-domain transfer possible—but what happens when the infrastructure differs? A stress test would generate memories from Python-based ML tasks (MLGym-Bench) and attempt to transfer them to tasks in fundamentally different environments: R-language statistical computing tasks, JavaScript/Node.js web development tasks, or low-level systems programming in C/Rust where the interaction patterns differ substantially. The meta-knowledge hypothesis predicts that some transfer should still occur (test-driven verification is language-agnostic, iterative workflow discipline applies regardless of language), but the magnitude should be smaller because environmental adaptation meta-knowledge (how to manage dependencies, how to interpret compiler errors) becomes irrelevant. A null or negative transfer result for cross-language transfer would establish a boundary condition on the meta-knowledge hypothesis and suggest that MTL's benefits are environment-specific, not truly domain-agnostic.

Practical Applications and Downstream Use Cases

Shared memory pools for open-source coding agent communities. The paper's cross-model transfer results (Table 6: GPT-5-mini → DeepSeek V3.2 improves Pass@1 by +1.5%, GPT-5-mini → Qwen3-Coder improves by +1.1%) demonstrate that Insight memories are partially model-agnostic—a stronger model's experiences can benefit weaker models. This enables a practical deployment model: a community of coding agent developers maintains a shared, curated Insight memory pool generated from high-performance models (e.g., GPT-5-mini or better) solving diverse benchmarks. Developers deploying open-source coding agents (Qwen3-Coder, DeepSeek, LLaMA-based agents) can retrieve from this shared pool to bootstrap their agents' performance without generating their own expensive experience trajectories. The paper shows that self-generated memories outperform cross-model memories (GPT-5-mini → GPT-5-mini: +2.8% vs. GPT-5-mini → Qwen3-Coder: +1.1%, Table 6), so the shared pool would be a starting point that individual agents supplement with their own domain-specific experiences. The memory pool scaling results (Figure 6) suggest that the value of such a shared pool would compound as more diverse coding tasks are added—a community pool covering dozens of benchmarks and thousands of tasks could substantially exceed the 3.7% average gain reported in the paper.

Cost-efficient memory generation for specialized coding agents. Organizations deploying coding agents for specific domains (e.g., financial modeling code, scientific computing, internal DevOps) face a cold-start problem: their agents have no in-domain experience to draw on. The paper's results provide a cost-efficient solution: generate Insight memories from publicly available coding benchmarks (LiveCodeBench, SWE-Bench, MLGym-Bench) to create a cross-domain meta-knowledge pool, then use this pool to bootstrap the agent on the specialized domain. The 431-memory Insight pool in Table 2 achieves a 4.6% average Pass@3 gain over zero-shot across three benchmarks—a meaningful improvement that requires no in-domain experience generation. The specialized domain's own experience can then be added incrementally as the agent solves real tasks, creating a hybrid pool where cross-domain memories provide foundational procedural guidance (testing, verification, environment adaptation) and in-domain memories provide specialized knowledge. This two-stage deployment—cross-domain bootstrap followed by online domain-specific accumulation—is directly supported by the pool size scaling results: starting with the cross-domain pool (equivalent to some fraction of full pool size) and adding in-domain memories should follow the upward scaling trajectory in Figure 6.

Automated memory quality filtering for large-scale experience repositories. The paper's finding that task-specific Insights underperform task-agnostic Insights (Table 4: +1.1% average gain) provides a practical filtering criterion for memory pool maintenance. Organizations that accumulate large volumes of agent experience over time (thousands of trajectories per day in a production coding assistant) face a memory curation problem: not all generated memories are equally valuable, and storing low-quality, overly specific memories wastes storage and retrieval bandwidth while potentially causing negative transfer. The paper's task-reconstruction test—prompt an LLM to infer the original task from the memory content and measure similarity—can be deployed as an automated filter: memories where the original task can be easily reconstructed are too domain-specific and should be discarded or rewritten to increase abstraction. In a production system, this filter could run continuously as new memories are generated, ensuring that the memory pool trends toward higher average abstraction over time, which the paper's results predict will increase transfer effectiveness. The filter is cheap (one LLM call per memory for reconstruction, one embedding similarity computation) and does not require ground-truth labels, making it practical for deployment.

When to Prefer This Method

The paper explicitly compares MTL against two alternatives: single-domain self-evolving methods (ReasoningBank) and large-scale unified memory pools without abstraction control (AgentKB). The results in Table 2 establish a clear tradeoff favoring MTL when memory efficiency matters and when the target domain has limited in-domain experience:

Prefer MTL over single-domain self-evolving methods when: the agent operates across multiple distinct coding task types and you want to amortize memory generation cost across domains. ReasoningBank achieves only +1.7% over zero-shot on the three-benchmark subset using 97 in-domain memories, while MTL achieves +4.6% using 431 cross-domain memories (Table 2). The MTL advantage comes from accessing a larger and more diverse memory pool—but critically, single-domain methods retain an advantage when in-domain experience is abundant and cross-domain transfer is weak (e.g., Aider-Polyglot in Table 1, where MTL shows 0.0% gain, suggesting that cross-domain memories provide no relevant guidance for this benchmark).
Prefer MTL over large unified pools (AgentKB-style) when: memory abstraction control is possible and storage/retrieval efficiency matters. AgentKB uses 5,899 memories (13.7× more than MTL's 431) to achieve slightly lower average performance (0.613 vs. 0.630 on the three-benchmark subset, Table 2). The efficiency gap means MTL is preferable when memory storage is constrained or when retrieval latency scales with pool size. However, AgentKB's lead on ReplicationBench (0.200 vs. MTL 0.189) suggests that massive pools with low abstraction control can excel when the target domain benefits from sheer memory quantity—a tradeoff the paper does not fully characterize but which points to a hybrid approach where a large raw pool is filtered post-hoc for abstraction.
Prefer the Insight format over concrete formats (Trajectory, Workflow) when: the target domain differs substantially from source domains in surface task structure (programming language, framework, file layout). The Trajectory negative transfer cases (MLGym-Bench: −8.4%, TerminalBench2: −4.5%, Table 1) show that raw execution traces are actively harmful when environments differ. The Insight format's explicit abstraction instruction mitigates this, and the +16.7 percentage point swing between Trajectory (−8.4%) and Insight (+8.3%) on MLGym-Bench quantifies the risk of using low-abstraction formats for cross-domain transfer. Concrete formats may still be valuable when source and target domains share infrastructure (e.g., the Trajectory slight edge on LiveCodeBenchv6 in Table 1, 0.940 vs. 0.930 for Insight), but this is the exception rather than the rule—the paper's results across 6 benchmarks show Insight outperforming Trajectory on 4, tying or slightly trailing on 2.
Prefer cross-model memory transfer when: a stronger model is available for memory generation but a weaker model must be deployed (e.g., for cost or latency reasons). Table 6 shows GPT-5-mini memories improving DeepSeek V3.2 by +1.5% and Qwen3-Coder by +1.1% in Pass@1. The gains are smaller than same-model transfer but consistently positive, making cross-model transfer a viable bootstrapping strategy when deploying open-source models with limited experience generation budgets. The asymmetry—GPT-5-mini → weaker models provides larger gains than the reverse (DeepSeek → GPT-5-mini: only +0.3%)—suggests that memory quality (driven by source model capability) matters for transfer, and that generating memories with the strongest available model is always preferable, even when those memories will be used by a different model.