ArXiv: 2605.12477

🎯 Pitch

Every LLM memory system collapses to near-zero accuracy when facts depend on other facts that changeβ€”achieving just 3% on Cascade and 1% on Absence tasks. Prompting tweaks, deeper retrieval, and stronger answering models fail entirely; only a file-based agent using Claude Opus 4.7 as its internal LLM can close the gap, but at ~70Γ— the cost, showing that practical dependency reasoning remains unsolved.

1. Executive Summary

This paper introduces MEME, a benchmark that systematically evaluates how LLM-based memory systems handle information that evolves across multiple sessions and spans interdependent entities β€” organized along two orthogonal dimensions of entity scope (single vs. multi-entity) and temporal dynamics (static vs. evolving). MEME defines six tasks including three not scored by any prior benchmark: Cascade (inferring that a dependent fact changes when an upstream entity updates, e.g., a code reviewer changes when the team lead changes), Absence (recognizing that a previously valid answer is now uncertain after an upstream change with no replacement rule, e.g., commute time becomes unknown after moving), and Deletion (verifying a removed fact is no longer reported). Evaluating six memory systems spanning three architectural paradigms β€” raw retrieval, LLM-processed memory, and file-based agents β€” on 100 controlled episodes with gpt-4.1-mini as the internal and answering LLM, the paper finds that every practical-cost configuration collapses on dependency reasoning, achieving only 3% accuracy on Cascade and 1% on Absence on average, even as static retrieval tasks remain adequate. Prompt optimization, deeper retrieval, reduced filler noise, and a stronger answering LLM all fail to close this gap; closure emerges only when a file-based agent (MD-flat) uses Claude Opus 4.7 as its internal LLM, which writes propagated dependency values directly into the store at ingest, establishing that dependency reasoning is currently solvable only with a frontier internal LLM at roughly 70Γ— the baseline cost β€” a configuration not practical at scale.

2. Context and Motivation

The Core Problem: Memory Systems Can't Handle Interdependent, Evolving Knowledge

This paper addresses a fundamental blind spot in how we evaluate β€” and therefore how we build β€” persistent memory for LLM agents. The problem is not that memory systems fail to store facts; it's that they fail to reason about how facts relate to each other when those facts change over time.

Consider a concrete scenario the paper uses (Section 1 and Figure 4): a software team has a team lead (Sarah Chen) and a code reviewer (David Lee). The user states a dependency rule: if the team lead changes, the code reviewer will be Seoyun Choi. Later, the user announces that Minjun Lee is the new team lead. A competent memory system should now infer β€” even though no one has stated it explicitly β€” that Seoyun Choi is the new code reviewer. This is a Cascade task: an upstream change propagates through a dependency rule to update a downstream fact.

Alternatively, suppose the user states that their commute is 30 minutes because they live in Pyresta Meadow, with the caveat that "if I move, my commute time will change." When the user later reports moving to Keldara Grove, the system should recognize that the commute time is no longer known β€” this is an Absence task: the correct answer is "uncertain" or "I don't know," not a stale value.

These scenarios involve what the paper calls dependency reasoning (Section 3.1): tracking how a change to one entity ripples through logically connected entities, and recognizing when a previously valid fact has become unreliable. This is fundamentally different from the state management tasks evaluated by prior benchmarks, where each entity is updated independently with no logical connection to other entities (Table 1). The distinction is analogous to knowing 10 independent facts about a person vs. knowing that 3 of those facts depend on a 4th, so changing the 4th should force re-evaluation of the other 3.

The paper's central empirical finding is stark: every memory system they evaluate, under every practical-cost configuration, effectively scores zero on dependency reasoning β€” 3% average on Cascade, 1% average on Absence across six systems (Table 2) β€” even though the same systems perform adequately on static retrieval (62% Exact Recall average) and can store and retain the dependency rules and change events in their internal stores (Section 4.3, traced per-system in Figures 4 and 30). The information exists in memory; the systems just cannot use it when answering a question.

Why This Problem Matters

The gap between storing dependency rules and actually propagating their consequences is not academic β€” it directly affects the reliability of LLM agents in real deployments. The paper frames three practical failure modes (Section 1):

Agents make confident but stale statements. When a user tells an agent "I've moved to a new city" but does not explicitly recalculate every dependent fact (commute time, nearby restaurants, gym location, doctor's office), a memory system that cannot propagate this change will continue reporting the old values with confidence. In a personal assistant context, this means giving wrong commute advice; in an enterprise context (the Software Project domain), it means reporting incorrect build commands, wrong deployment targets, or outdated team contacts β€” all of which can cause real operational failures.

Absence of knowledge is misreported as presence. When a dependency is broken but no replacement is specified (Absence), the correct behavior is to express uncertainty. Prior memory systems, however, will simply report the old value as if nothing changed β€” a more dangerous failure than saying "I don't know," because the user may act on false information assuming the system is trustworthy.

The evaluation gap masks the deployment gap. As shown in Table 1, none of the five prior multi-session memory benchmarks β€” LoCoMo, LongMemEval, MemBench, MemoryAgentBench, or RULER/NoLiMa β€” evaluate dependency reasoning at all. They test whether a system can track a single entity's value changes (e.g., tracking three different cars the user drove over time), but never ask: now that one entity has changed, what else should have changed with it? This means systems that score well on existing benchmarks can fail catastrophically on deployment scenarios where knowledge is interdependent β€” and developers have no diagnostic tool to detect this gap.

The paper also connects this to adjacent fields (Section 1): the entity-scope axis parallels the single-hop vs. multi-hop distinction in question answering (HotpotQA, MuSiQue), where answering requires chaining across multiple facts. The temporal dynamics axis parallels the "ripple effect" problem in knowledge editing (MQuAKE, ripple-effect studies), where editing one fact in a model should force retraction of logically dependent inferences. But in those fields, the model being edited is the LLM itself (its parametric knowledge); here, the target is an external memory system, and the challenge is that the system must reconstruct the chain at query time from what was stored β€” a retrieval-and-reasoning problem rather than a weight-update problem.

Where Prior Approaches Fall Short

The paper identifies specific failure modes in prior benchmarks and the memory architectures they evaluate, organizing them along two axes (Section 2 and Table 1):

Prior benchmarks only test isolated, single-entity updates. Table 1 reveals a clear pattern: Exact Recall and Aggregation (static, single or multi-entity retrieval) are covered by all prior benchmarks; Tracking (single-entity evolution) is covered by the three most recent ones β€” LongMemEval, MemBench, and MemoryAgentBench. But no prior benchmark evaluates deletion verification (does the system stop reporting a fact after the user explicitly removes it?), and none evaluates dependency propagation (Cascade and Absence). This means the entire evaluation ecosystem has been testing memory as a flat fact store β€” each fact independent, each update self-contained β€” while real-world knowledge is relational, with facts linked through logical dependencies that must be respected when any upstream fact changes.

The consequence is that systems optimized for prior benchmarks learn exactly the wrong behavior for real deployment. A memory system gets rewarded for (a) storing facts accurately, (b) overwriting old values with new ones on independent updates, and (c) retrieving the most recent value. But Cascade and Absence require not overwriting β€” you must keep the dependency rule, detect when its trigger fires, and either compute the new value or signal uncertainty. Systems that aggressively overwrite or decompose facts (as Mem0 extracts atomic sentences, or Graphiti extracts entity-relation triples) lose the relational structure that dependency reasoning requires, as the paper's per-stage traces in Section 4.3 and Appendix I demonstrate.

Prior memory architectures are not designed for dependency propagation. The paper evaluates systems spanning three paradigms identified in Section 2:

  1. Raw retrieval (BM25, text-embedding-3-small): These store conversation chunks verbatim and retrieve by similarity. They preserve the exact dependency rule text (since nothing is extracted or paraphrased), but at query time retrieval must rank the change event above the pre-change value. The paper shows (Section 4.3, Figure 30) that vector retrievers surface the change event but the answering LLM still reports the pre-change value (an answering failure), while lexical retrieval (BM25) fails to retrieve the change event at all (a retrieval failure when the change session's keywords don't match the query's).

  2. LLM-processed memory (Mem0, Graphiti): These use an internal LLM to extract or structure facts at ingestion. Mem0 decomposes conversations into atomic natural-language facts; the paper's traces show that this decomposition loses the conditional structure β€” a dependency rule like "if the team lead changes, the code reviewer will be Seoyun Choi" gets broken into two independent facts (the current code reviewer, and a conditional statement), and at query time the answering LLM reports the pre-change value because the change event and the conditional statement are not linked. Graphiti goes further: it extracts entity-relation triples into a temporal knowledge graph, but its extraction prompt explicitly instructs "closely paraphrase the original source sentence(s) β€” do not verbatim quote," which destroys the dependency rule's precise conditional language, and its graph traversal at retrieval time surfaces only the most recent edges, often missing the change-event edge entirely (Figure 4, left panel).

  3. File-based agents (MD-flat, Karpathy Wiki): These give an LLM tool-calling access to persistent files. MD-flat keeps a single markdown file; the internal LLM must decide what to write, update, or delete. With gpt-4.1-mini as the internal LLM, the paper shows (Appendix K.1) that it writes facts correctly but fails to propagate β€” when a change event arrives, it writes the change but does not re-evaluate dependent entries. The dependency rule sits in the file alongside the old value, and at query time the retrieval agent surfaces what it finds at the top of the file, which is the pre-change value. Karpathy Wiki adds a compilation step (daily logs β†’ topic articles), but the paper's traces in Figure 4 show that the query agent navigates to the article containing the original rule but never opens the daily log containing the change event β€” a retrieval failure caused by the two-tier storage structure.

The structural gap: maintenance is passive, not active. The paper's per-stage diagnosis (Section 4.3) reveals a common thread across all failing systems: they all encode the dependency rule and the change event (the information is in the store), and they all retain both through maintenance (nothing is deleted or overwritten), but at retrieval time the change event is either out-ranked by the pre-change value (vector retrievers) or never surfaced at all (tool-use, graph, sparse retrievers). In other words, the dependency information is passively stored but never actively propagated. The paper makes explicit that existing memory architectures have no mechanism for triggering dependent updates when an upstream fact changes. This is not a bug in any specific system; it's an architectural missing piece across all three paradigms.

How This Paper Positions Itself

The paper frames MEME as a diagnostic instrument rather than a method paper β€” it does not propose a new memory architecture or a solution to dependency reasoning. Instead, it argues that the field needs (a) a benchmark that measures dependency reasoning, and (b) a systematic understanding of where current systems fail at each pipeline stage (encoding, maintenance, retrieval), so that future work can target the right bottlenecks (Section 5).

This diagnostic framing is important because it distinguishes MEME from prior benchmarks that implicitly reward flat fact storage. The paper's taxonomy β€” entity scope Γ— temporal dynamics, yielding four quadrants and six tasks (Figure 1) β€” is presented as a principled organization that makes explicit what prior benchmarks were missing. By placing Cascade and Absence in the "Multi-entity, Evolving" quadrant (the hardest cell), the paper establishes that dependency reasoning is the natural endpoint of scaling from static single-entity retrieval: you first handle static facts (Exact Recall), then combine multiple facts (Aggregation), then track changes to individual facts (Tracking), then handle deletion (Deletion), and finally reason about how changes propagate through dependencies (Cascade, Absence). Each prior benchmark stopped somewhere on this spectrum; MEME covers the full space.

The paper also positions itself at a specific level of analysis: evaluating memory systems as they are deployed today, with default configurations and practical-cost LLMs, not evaluating an upper bound of what LLMs could theoretically do. This is why the six systems use gpt-4.1-mini uniformly as both the internal LLM (for ingestion and retrieval) and the answering LLM (Table 2) β€” the paper wants to measure the gap that exists in practice, not the gap that could theoretically be closed with infinite compute. The finding that only MD-flat with Opus 4.7 at 70Γ— baseline cost closes the Cascade-Absence gap (Table 4, Section 4.4) is presented not as a recommendation but as a cost-benchmark: this is what closure costs today, and it's not deployable.

Finally, the paper explicitly connects its diagnostic approach to a forward-looking design recommendation (Section 5): what the field needs is memory architectures that natively propagate updates through dependent facts at maintenance time, rather than relying on a costly frontier LLM to do so at ingestion. MEME provides the yardstick for measuring progress toward that goal. The per-stage failure analysis (Figures 4 and 30) identifies the retrieval stage as the most common breakdown point under gpt-4.1-mini, but the Opus 4.7 case study (Appendix K.2) shows that pushing the propagation work to ingestion β€” where the internal LLM actively scans for dependent facts and writes resolved values β€” can bypass the retrieval bottleneck entirely, at the cost of making ingestion vastly more expensive and fragile (paraphrasing degrades Exact Recall and Tracking). The paper leaves open whether this tradeoff can be resolved through architectural innovation rather than through more powerful LLMs.

3. Technical Approach

3.1 Reader Orientation

MEME is not a system or a method β€” it is a controlled evaluation benchmark that diagnoses whether LLM-based memory systems can propagate changes through interdependent facts across multiple conversational sessions. The paper builds a dataset of 100 episodes where conversational histories contain entities linked by explicit dependency rules (e.g., "if the team lead changes, the code reviewer will be Seoyun Choi"), then introduces upstream changes and measures whether the memory system correctly infers the downstream consequences. The "shape" of the solution is a two-dimensional taxonomy β€” entity scope (single vs. multi-entity) crossed with temporal dynamics (static vs. evolving) β€” that yields six distinct task types, including three (Cascade, Absence, Deletion) that systematically test propagation failures no prior benchmark captures.

3.2 Big-Picture Architecture (Diagram in Words)

The benchmark operates as a five-stage pipeline:

  1. Knowledge Graph Definition β€” A hand-crafted Directed Acyclic Graph (DAG) per domain encodes entities, their dependency edges, value pools, and conditional propagation rules. This graph is constructed once per domain and reused across all episodes in that domain.

  2. Episode Construction β€” For each of the 100 evaluation episodes, the pipeline selects a root entity from the DAG, samples values for all episode entities from their pools, assigns each entity to one of the six task types based on its topological role in the dependency graph, verbalizes the structured facts into multi-turn conversational sessions, and interleaves these evidence sessions with carefully filtered filler sessions to create ~35K-token haystacks.

  3. Memory System Ingestion β€” Each evaluated memory system (raw retrieval, LLM-processed, or file-based agent) ingests the chronological session transcripts and stores them according to its own architecture, using its own internal LLM (gpt-4.1-mini uniformly in the default configuration) for extraction, structuring, or tool-use decisions.

  4. Question-Answering β€” For each of the 694 post-change evaluation questions, each memory system retrieves context from its store and produces an answer using a unified answering LLM prompt. The answering LLM is gpt-4.1-mini in the default configuration, with swaps to Claude Sonnet 4 for the answering-LLM ablation.

  5. Evaluation β€” A GPT-4o judge scores each answer against gold answers with task-specific rubrics. Cascade, Absence, and Deletion tasks use trivial-pass filtering: credit requires correct answers both before and after the change or delete event, excluding false positives from systems that never encoded the fact initially.

3.3 Roadmap for the Deep Dive

  • First, the two-dimensional evaluation taxonomy (entity scope Γ— temporal dynamics) and how it maps to the six concrete task types β€” because the task definitions establish exactly what the benchmark measures and why prior benchmarks missed structural dependency failures.
  • Second, the knowledge graph formalism and the dependency propagation rules β€” because the Cascade and Absence gold answers are computed by applying these rules to upstream changes, and understanding their recursive, multi-hop nature is essential to seeing why dependency reasoning is harder than single-entity tracking.
  • Third, the episode construction pipeline in sequence β€” entity set selection, value assignment, task assignment based on topological role, LLM-driven verbalization with a two-layer verification pass, and filler interleaving with a four-step conflict-filtering process β€” because the rigor of this pipeline is what makes the gold answers verifiable by construction and the evaluation claims trustworthy.
  • Fourth, the memory system configurations and their key architectural differences along encoding, maintenance, and retrieval stages β€” because the paper's core diagnostic contribution is tracing where in each system's pipeline dependency information is lost, which requires understanding what each system does at ingestion, storage, and query time.
  • Fifth, the evaluation protocol β€” the answer prompt, the task-specific judge rubrics, and the trivial-pass filtering mechanism β€” because the Cascade/Absence/Deletion accuracy numbers in the results depend on these filtering and judging choices, and understanding their design is necessary for interpreting whether the reported failure rate is a true failure or a measurement artifact.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark and diagnostic analysis paper whose core contribution is a principled evaluation framework and a controlled dataset that exposes where current memory systems' architectures structurally fail at dependency propagation. The paper does not propose a new memory architecture; it builds a measurement instrument and traces failure modes through six representative systems.

The Two-Dimensional Evaluation Taxonomy and Task Definitions

The central organizational idea is that a complete memory evaluation must measure performance along two orthogonal axes: entity scope (does the task involve a single fact or multiple interrelated facts?) and temporal dynamics (are the facts static across sessions, or do they evolve?). This 2Γ—2 space yields four quadrants, and the paper selects one or two representative memory operations from each quadrant, intentionally excluding easier variants already covered by prior benchmarks (Section 3.1, Figure 1):

Single-entity, Static β†’ Exact Recall (ER). The system must reproduce a single verbatim fact character-for-character, such as an error log string like OutOfMemoryError: Java heap space. GC overhead .... This tests encoding fidelity: when the fact is stored in a conversational session and no changes occur, can the system retrieve and reproduce it exactly, including unusual token sequences, punctuation, and formatting? The paper designates this as an "exact recall" task rather than a semantic-paraphrase task because real-world deployment scenarios (error diagnostics, code snippets, configuration strings) often require literal reproduction.

Multi-entity, Static β†’ Aggregation (Agg). The system must collect multiple independently-introduced static facts β€” introduced across separate sessions with no explicit cross-reference β€” and combine them into a single answer. The example in Figure 2: the user mentions pottery as a hobby in one session, rock climbing in another, and a book club in a third; the question "What do I do in my free time? List everything" requires retrieving all three independently stored facts and presenting them together. This tests retrieval coverage when no explicit link connects the facts: the system must return all relevant information, not just the top-1 or top-k match.

Single-entity, Evolving β†’ Tracking (Tr). The system must reconstruct the full chronological revision history of a single entity that changes value multiple times. The example in Figure 2: the user reports driving a Zyvanta Sedan, then a Therwyn Compact, then a Xylorim Scooter across three sessions; the question "List all vehicles I've had, earliest to latest" requires preserving past values rather than overwriting them, and ordering them correctly in time. This tests whether the memory system retains historical state alongside current state β€” a design choice that some architectures (aggressive overwriters) fail by construction.

Single-entity, Evolving β†’ Deletion (Del). The system must stop reporting a fact after the user explicitly requests its removal. The example: the user states "My partner's name is James," then later says "Please remove my partner's name from your memory." The question "What's my partner's name?" should yield "That information has been deleted" or equivalent uncertainty β€” not the deleted value, and not a confident "I don't have that information" that fails to acknowledge the deletion. This tests a specific kind of state management that no prior benchmark evaluates (Table 1): not just updating to a new value, but transitioning to a null/absent state.

Multi-entity, Evolving β†’ Cascade (Cas). The system must infer that a dependent entity's value has changed based on a stated dependency rule and an upstream update. The example: the team lead is Sarah Chen, and the code reviewer is David Lee; the dependency rule states "If the team lead changes, the code reviewer will be Seoyun Choi"; later the user announces Minjun Lee is the new team lead. The question "Who's our code reviewer?" should yield "Seoyun Choi" β€” a value that was never stated directly in any conversation. The system must (a) retrieve the dependency rule, (b) detect that its trigger condition has fired, and (c) apply the rule to compute the new value. This is a multi-hop reasoning task distributed across sessions: the rule and the trigger event arrive in different conversations, and the answer must be constructed at query time.

Multi-entity, Evolving β†’ Absence (Abs). The system must recognize that a previously valid answer has become uncertain after an upstream change where no replacement rule is available. The example: the user states their commute is about 30 minutes, with the caveat "if I move, my commute time will change"; later the user reports moving to Keldara Grove. The question "How long is my commute?" should yield "Uncertain" or equivalent β€” because the dependency is broken but no propagation rule specifies the new value. This tests meta-cognitive awareness: recognizing the boundary of what the memory system can confidently assert, rather than confidently reporting a stale value.

The paper groups these six tasks into three conceptual categories for readability (Figure 2): Retrieval (Exact Recall, Aggregation β€” the "Static" half of the taxonomy), State Management (Tracking, Deletion), and Dependency Reasoning (Cascade, Absence).

Why these specific tasks were chosen and others excluded. The paper explicitly states that easier task variants β€” such as single-fact retrieval (already in RULER/NoLiMa), multi-fact retrieval without evolution (already in LoCoMo), or single-entity tracking without dependencies (already in LongMemEval, MemBench, MemoryAgentBench) β€” are intentionally excluded because they are "already covered by existing benchmarks" (Section 3.1). This is a deliberate scope decision: MEME is not a general-purpose memory benchmark but a targeted diagnostic for the dependency-reasoning capability that the paper argues is the critical gap. The inclusion of Exact Recall and Aggregation serves as a baseline: if a system cannot do these static tasks, its failure on Cascade/Absence is uninformative; their presence also confirms that the dataset construction is not artificially difficult for all task types.

How the task types map to the dataset construction. An important design detail is that task types are not randomly assigned per entity β€” they are determined by the entity's topological role in the knowledge graph (Section 3.2, step 3 of episode construction):

  • Cascade and Absence targets are sampled from the descendants of the root entity in the DAG β€” they sit in the dependency chain and are affected when the root changes. A Cascade target has an associated if-then replacement rule in the graph's rule set $\Phi$; an Absence target is a descendant of the root that has no replacement rule.
  • Tracking targets are entities outside the cascade chain (unaffected by the root change), with three value updates assigned across the episode. This separation ensures that Tracking tasks are genuinely independent β€” the system is tested on retaining history, not on propagation.
  • Aggregation targets are predefined triples drawn from both descendants and entities outside the cascade chain, testing whether the system can collect semantically related but independently stored facts.
  • Exact Recall and Deletion targets are entities outside the cascade chain, chosen from pools specifically marked for these tasks (e.g., error_log for ER, personally meaningful entities for Deletion).

This topological role-based assignment is critical for validity: if a Cascade and a Tracking entity were mixed in the same dependency chain, it would be unclear whether the system's Tracking accuracy reflects history retention or dependency propagation, and the gold answers would be ambiguous. By separating cascade-chain entities from non-cascade entities, the benchmark cleanly isolates which capability each task measures.

Knowledge Graph Formalism and Dependency Propagation Rules

Each domain's knowledge graph is defined as a 4-tuple:

G=(V,E,P,Ξ¦)G = (V, E, P, \Phi)

where $V$ is the set of entities (e.g., health_condition, medication, exercise_routine), $E \subseteq V \times V$ is the set of directed dependency edges where an edge $(v_i, v_j)$ means entity $v_j$ depends on entity $v_i$ (e.g., medication depends on health_condition), $P(v)$ is a finite value pool for entity $v$ from which values are sampled during episode construction, and $\Phi$ is a set of conditional replacement rules.

What the graph encodes: a Directed Acyclic Graph structure where each edge represents a stated dependency β€” a causal or logical relationship where changing the source entity should, according to the user's explicitly stated rules, force re-evaluation of the target entity. The acyclicity constraint is important: it ensures that dependency propagation is well-defined (no circular dependencies where entity A depends on B, which depends on A, creating an infinite regress) and that the resolved state can be computed by a single topological pass from root to leaves.

Graph scale and structure (Appendix B.1, Table 7a). The Personal Life domain graph contains 39 entities organized into 9 categories (Living & Commute, Work, Health & Fitness, Food & Diet, Family, Finance, Schedule, Hobbies, Miscellaneous) with 34 dependency edges. The Software Project domain graph contains 51 entities across 6 categories (Framework & Build, Data Layer, Deployment & Infrastructure, Team & Process, Auth & Security, Miscellaneous) with 27 dependency edges. Each domain has exactly 5 root entities β€” entities with no incoming dependency edges that serve as the root nodes for cascade chains. The remaining entities are classified as middle nodes (entities with both incoming and outgoing edges), leaf nodes (entities with incoming but no outgoing edges β€” the endpoints of dependency chains), and orphan nodes (entities with no incoming or outgoing edges β€” completely independent facts).

The complete entity breakdown per domain: Personal Life has 5 roots, 5 middle nodes, 19 leaf nodes, and 10 orphans; Software Project has 5 roots, 6 middle nodes, 21 leaf nodes, and 19 orphans. These numbers reflect the paper's design goal of having enough dependency structure to test multi-hop propagation (chains of length 2 from root through middle to leaf, e.g., health_condition β†’ exercise_routine β†’ fitness_facility) while having enough independent entities to populate the static and non-dependency tasks.

Value pools: each entity $v$ has an associated pool $P(v)$ of candidate values, with pool sizes ranging from 5 to 15 values. All values use fictitious names β€” Veltrion for a framework, Crysthene DB for a database, Thrynexol for a medication, Keldara Grove for a residence location. This is a deliberate design choice explained in Appendix B.1: by using manually authored fictitious names that "sound plausible within their respective domains while avoiding collision with real products," the paper prevents parametric knowledge contamination β€” the evaluated LLMs cannot have encountered these names during pre-training and therefore cannot answer questions from parametric memory rather than from the memory system under test. All value pools are reused across all 100 episodes for consistency, meaning different episodes sample different values from the same pools.

Null-value exclusion: some entity pools contain null-like values ("none", "none currently", "no pet", "not currently enrolled") that are appropriate for representing the absence of a fact. These are explicitly excluded from task assignment via a NULL_VALUES filter β€” a null value cannot serve as a before-value or after-value in Cascade, Tracking, or Deletion tasks, because tasks require a non-trivial fact to exist before a change can be meaningful.

Dependency rules: the rule set $\Phi$ contains conditional replacement rules of the form "if [parent entity] changes to [specific value], then [dependent entity] becomes [new value]". These are instantiated as natural-language templates during verbalization. The paper mentions six semantic patterns for Software Project (tech_compatibility, derived_config, data_layer, infra_coupling, team_assignment, auth_coupling) and eleven for Personal Life (proximity, company_policy, life_event, medical_causation, priority_shift, infrastructure, distance, activity_facility, schedule_conflict, preference, curriculum). Each pattern provides a template that binds a target fact to its source entity with an explicit conditional clause, e.g., the proximity pattern generates "commute_duration β€” this depends on where residence_location; if I move, this would change".

The set of rules $\Phi$ is sparse: not every dependency edge has a replacement rule. For edges without a rule, the dependent entity becomes uncertain ($\bot$) when the source changes, producing an Absence task. For edges with a rule, the rule specifies exactly what the dependent entity's new value should be, producing a Cascade task. This sparsity is the mechanism that creates both task types in the same episode.

The propagation computation: when an upstream entity $v_i$ (typically a root) changes from some initial value to a new value $r^*$, the resolved state of each dependent entity $v$ is determined recursively by tracing through the DAG. Formally (Equation 1 in Section 3.2):

vβˆ—={Ο•v(viβˆ—)ifΒ Ο•v∈Φ (Cascade)βŠ₯ifΒ Ο•vβˆ‰Ξ¦Β (Absence)v^* = \begin{cases} \phi_v(v^*_i) & \text{if } \phi_v \in \Phi \text{ (Cascade)} \\ \bot & \text{if } \phi_v \notin \Phi \text{ (Absence)} \end{cases}

where $v^*$ is the resolved post-change value of entity $v$, $\phi_v \in \Phi$ is a conditional rule (if one exists) that maps the parent's value to $v$'s new value, $v^*_i$ is the resolved value of $v$'s parent entity (which may itself be a dependent in a longer chain), and $\bot$ denotes that no answer is derivable β€” the gold answer is "Uncertain".

What this equation computes operationally: given a root entity $r$ that changes to $r^*$ in the conversation, the system traces each descendant $v$ in topological order. For each descendant $v$, it identifies $v$'s parent in the DAG, retrieves the parent's already-resolved value, checks whether a replacement rule $\phi_v$ exists for the (parent-value, child) pair, and either computes the child's new value (if a rule exists) or marks it as $\bot$ (if no rule exists). For a multi-hop chain like $r \rightarrow v_1 \rightarrow v_2$, the computation proceeds recursively: first resolve $v_1^* = \phi_{v_1}(r^*)$; then using $v_1^*$ as the parent value, resolve $v_2^* = \phi_{v_2}(v_1^*)$ or mark it as $\bot$ if no rule exists.

Why this recursive form: the recursion makes the gold answer for a 2-hop cascade depend on both the root change and the intermediate entity's propagated value. This means the system cannot answer a 2-hop cascade question by simply retrieving a rule that directly mentions the root β€” it must reconstruct the full chain: the root changed, which triggered the 1-hop dependent to change, which triggered the 2-hop dependent to change. The paper's traces in Section 4.3 and Appendix I show that no practical-cost system can do this, even when the individual rules are stored correctly.

The pre-change vs. post-change distinction: the paper distinguishes between the pre-change value of an entity $v$ β€” the value stated in the conversation sessions before the root change β€” and the resolved post-change value $v^*$. For Cascade tasks, the gold answer is $v^*$ (the propagated new value). For Absence tasks, the gold answer is $\bot$ (uncertainty). The pre-change value is not the correct answer in either case, and the paper's main finding is that memory systems consistently produce the pre-change value instead β€” either because retrieval fails to surface the change event, or because the answering LLM fails to apply the dependency rule to the surfaced evidence.

Domain-specific consistency constraints: the paper applies a post-processing consistency pass after value assignment to ensure the initial graph state is logically coherent (Section 3.2, step 2). The example given: "if vehicle is none, commute_method excludes driving." These constraints prevent semantically contradictory initial states that would make dependency rules nonsensical β€” if the user has no vehicle, stating that their commute method depends on their vehicle would be incoherent from the start. The constraints are domain-specific and are applied as a filter: if the sampled values violate a constraint, the episode is re-sampled.

Episode Construction Pipeline

Each evaluation episode $\epsilon = (G, S, Q, A)$ is built through a five-step process over the fixed domain knowledge graph, where $S$ is the chronological sequence of conversational sessions, $Q$ is the set of evaluation questions, and $A$ is the set of corresponding gold answers.

Step 1: Entity set selection. The pipeline selects a root entity from the domain's 5 roots, cycling through all roots before reuse to ensure balanced coverage. This root and all its descendants in the DAG form the cascade chain β€” the set of entities that will participate in dependency tasks. At each hop level from the root, two types of targets are selected: one Cascade target (a dependent entity with a declared if-then replacement rule) and one Absence target (a dependent entity without a replacement rule). Both 1-hop and 2-hop descendants are considered when the graph topology allows, yielding up to 4 dependency targets per episode β€” two Cascade targets (one at hop 1, one at hop 2) and two Absence targets (one at hop 1, one at hop 2). If a 2-hop chain does not exist for the selected root (the Dependency Graph may have a depth of only 1 for some roots), the episode contains only 1-hop dependency targets. Additionally, a sample of entities from outside the cascade chain (orphans or entities on different branches) is selected to populate the non-dependency tasks (Tracking, Aggregation, Exact Recall, Deletion). Each generated episode must contain all 6 required task types; if any type is missing after entity assignment β€” which can happen if the chosen root's cascade chain is too shallow, or if null-value filtering eliminates candidate targets β€” the entire episode is discarded and regenerated with a different root, for up to 20 retries (Appendix B.2).

Step 2: Value assignment. Each entity in the episode is assigned an initial value sampled uniformly from its finite value pool $P(v)$. All pool values use the fictitious names described earlier (Appendix B.1). For dependency entities (targets in the cascade chain), the after-value β€” the value the entity should have after the root changes β€” is determined differently depending on the task type:

  • Cascade targets: the after-value is computed by applying the declared if-then rule to the root's new value. For example, if the root health_condition changes to high blood pressure, and the rule for medication specifies "if health condition changes to high blood pressure, switch medication to Thrynexol", then the gold after-value for medication is Thrynexol. For 2-hop Cascade targets, the rule is applied to the 1-hop target's propagated value: if exercise_routine changes to yoga 2x/week (a Cascade), and the rule for fitness_facility maps yoga 2x/week to Crysthene Pool, then the gold after-value for fitness_facility is Crysthene Pool.
  • Absence targets: the after-value is set to $\bot$ (unknown), regardless of the root's new value, because no rule exists to specify what the dependent value should become.

After initial value assignment, the paper applies a domain-specific consistency post-processing pass to ensure the initial graph state is logically coherent. The example given: if the entity vehicle is assigned the value "none", then the entity commute_method (which has a dependency edge from vehicle) cannot include "driving" in its value pool β€” a constraint that prevents the initial state from containing an internal contradiction. The paper does not enumerate all constraints but states they are domain-specific.

Step 3: Task assignment. Each entity in the episode is mapped to exactly one of the six task types based on its topological role in the DAG (Section 3.2, step 3, and Appendix B.2):

  • Cascade: sampled from the root's descendants in the cascade chain, specifically those for which a replacement rule exists in $\Phi$. The task's gold answer is the propagated after-value computed in Step 2.
  • Absence: sampled from the root's descendants in the cascade chain, specifically those for which no replacement rule exists in $\Phi$. The gold answer is "Uncertain".
  • Tracking: entities outside the cascade chain, assigned three distinct values that will be introduced sequentially across the episode. The gold answer is the ordered list of all three values from earliest to latest. The set of entities suitable for Tracking is predefined (e.g., vehicle, media_consumption, partner for Personal Life; sprint_deadline, secret_manager, package_manager for Software Project).
  • Deletion: orphan entities whose value is designated as "personally meaningful," enabling a natural deletion request ("please remove my partner's name from your memory"). The gold answer is a statement of non-availability, distinguished from uncertainty (the fact was explicitly removed, not simply unknown).
  • Aggregation: triples of semantically related but independently stored entities. The paper manually curates these triples β€” e.g., {hobby, sports, club_membership} for "What do I do in my free time?" β€” because the entities are semantically related (all are leisure activities) but are introduced in different sessions with no explicit cross-references in the conversation. The gold answer is the set of all three values.
  • Exact Recall: entities explicitly marked as requiring verbatim reproduction in the entity pool. These are entities whose values are long, complex strings that test literal encoding fidelity: life_philosophy for Personal Life and error_log for Software Project. The gold answer is the exact character sequence as originally stated.

This topological role-based assignment is not arbitrary β€” it is the mechanism that ensures the benchmark's task measurements are causally valid. If a Cascade entity were also a Tracking entity (assigned three values across the episode), the system's accuracy on Tracking would conflate its ability to track independent updates with its ability to propagate a dependency, and the gold answer computation would become ambiguous (does the system report the ordered history including the pre-change and post-change cascade values, or just the cascade propagation?). By cleanly separating cascade-chain entities (which get Cascade or Absence tasks) from non-cascade entities (which get Tracking, Deletion, Aggregation, or Exact Recall), the benchmark isolates the specific capability each task measures.

Step 4: Verbalization. The structured episode skeleton β€” entities with assigned values, their dependency relationships, and their task assignments β€” must be converted into natural conversational sessions that an LLM-based agent would plausibly encounter. The paper uses a hybrid verbalization approach (Section 3.2, step 4, and Appendix B.3, D.1–D.2) that combines LLM self-chat for most facts with template-direct insertion for facts requiring absolute precision.

The pipeline first converts all gold fact seeds β€” written in first-person β€” into third-person statements using an LLM batch-conversion step. The paper includes the full prompt (Figure 8 for Personal Life, Figure 9 for Software Project). This intermediate representation is necessary because the subsequent self-chat User LLM must receive objective fact descriptions to re-verbalize naturally, rather than first-person utterances that would bias the self-chat output.

The self-chat process involves two LLMs alternating turns (both using gpt-4o): a User LLM that acts as the user introducing facts about themselves, and an Assistant LLM that responds naturally. The User LLM receives the list of third-person facts and a system prompt (Figure 10) with these key constraints:

  • At most one fact per message: "IMPORTANT: convey at most ONE fact per message. Do not combine multiple facts into a single message." This ensures facts are distributed across distinct conversational turns, making retrieval a non-trivial search through multiple message boundaries rather than a single dense fact block.
  • Topic continuation: "After conveying each fact, continue discussing that topic for 2–3 more exchanges before moving to the next fact. Ask follow-up questions, request more details, or explore related aspects of the topic." This creates realistic multi-turn topical discussions that interleave filler with evidence, increasing the retrieval difficulty.
  • Dependency preservation: "If a fact mentions a dependency ('depends on', 'determined by', 'if X changes'), you MUST preserve the full dependency language including the conditional part. The reader must understand that if the source changes, the target would change too. Do NOT weaken 'depends on X; if X changes, this would change' to just 'tied to X' or 'since we use X'." This constraint is critical for Cascade and Absence validity β€” if the self-chat LLM paraphrased the dependency into a weaker form, the memory system could legitimately claim it never received an explicit conditional rule.
  • Conditional modal precision: "If a fact starts with 'If' (conditional), use ONLY 'will' as the modal: 'If X changes, Y will be Z'. NEVER use 'would/might/probably/consider'. NEVER state it as accomplished ('I switched to Z')." This ensures the dependency rules use the strongest possible conditional language, giving memory systems the best chance of recognizing them β€” if systems fail even with "if X changes, Y will be Z" phrasing, they would certainly fail with weaker or implicit conditionals.

The Assistant LLM (Figure 11 for Personal Life, Figure 12 for Software Project) is instructed to respond naturally, keep responses to 3–6 sentences, and crucially, to not assume or guess any facts about the user beyond what was explicitly stated. This prevents the Assistant from "helpfully" introducing new facts that would contaminate the gold answer.

In contrast to the self-chat verbalization for base facts, dependency rules and exact recall facts are embedded using template-direct (verbatim) text. This is a deliberate design choice: for facts where precision is paramount β€” the exact wording of a dependency rule (wrong wording could make the rule unrecognizable) and the exact character sequence of an error log (a single character change makes the answer wrong) β€” the paper bypasses the LLM verbalization entirely and inserts the fact directly into the conversation in a template-determined form. This makes the Cascade and Absence tasks a best-case scenario for memory systems: the conditional rules are stated in maximally explicit form, using the strongest modal language, and are not degraded by LLM paraphrasing. The paper acknowledges this as a limitation (Section 6): "Verbalization uses explicit conditional phrasing for dependency rules as a best-case framing for memory systems; we have not ablated implicit-conditional or no-conditional variants."

A figure in Appendix B.3 (Figure 6) provides a complete generated session showing how these constraints manifest in practice. The User LLM introduces each fact as a natural conversational opener ("My hobby is pottery. I've been getting into it more lately..."), asks a follow-up question to engage the Assistant, and the Assistant responds with on-topic advice. The Assistant turns are abbreviated in the figure for space but show the expected conversational flow.

Verification pass: after self-chat generation, the paper applies a two-layer LLM verification to ensure that all generated turns faithfully reflect the underlying gold facts:

  1. Layer 1 (GPT-4o annotation): an LLM annotates which user turn conveyed each gold fact, using the prompt in Figure 13. The annotation maps each fact to the turn index where it was communicated, or marks it as null (not conveyed). This provides a turn-level trace of whether every required fact was verbalized.
  2. Layer 2 (Gemini 2.5 Flash semantic audit): a separate LLM performs a five-point audit (Figure 14) checking: (a) Verbalization accuracy β€” is the gold fact's value present in the conversation? (b) If-then conditional form β€” for dependency rules, does the text use the required "will" modal, and is the trigger entity explicitly mentioned? (c) Entity keyword presence β€” does the conversation contain the entity concept words needed for the question to be answerable? (d) Cascade-U dependency β€” for non-rule dependency facts, is the causal link "strongly explicit"? (e) Gold answer format β€” does the gold answer comply with expected formatting?

Issue severity is rated as HIGH (makes the task unsolvable), MEDIUM (may cause false pass/fail), or LOW (minor). The paper reports using two different LLMs for the two layers (GPT-4o for annotation, Gemini 2.5 Flash for audit) to reduce single-model bias β€” if a single LLM both generated the conversation and audited it, it might miss its own errors.

Step 5: Haystack assembly. The evidence sessions $S_{ev}$ (the conversations containing gold facts) are interleaved with filler sessions $S_{fill}$ to create a realistic long-context setting where the memory system must distinguish relevant from irrelevant content. All episodes contain approximately 35,000 tokens of total dialogue context (Section 3.2, Table 7b), with the filler volume set at 32K tokens for the default condition.

Filler sources: filler sessions are drawn from two public corpora chosen to match each domain (Appendix B.4). For Personal Life, fillers come from the LongMemEval-S haystack sessions (17,034 sessions after extracting non-evidence sessions and deduplicating by session ID). For Software Project, fillers come from the ShareGPT coding English subset (29,502 conversations originally collected from users sharing ChatGPT conversations about coding). Both corpora are real human-AI conversations, lending ecological validity to the filler content. A length filter (500–5,000 estimated tokens per session) removes overly short or long fillers, yielding raw pools of 1,130 sessions for Personal Life and 9,135 for Software Project.

Conflict filtering pipeline: to prevent filler sessions from semantically interfering with the gold facts β€” e.g., a filler discussing a "Zyvanta Sedan" (a car brand used as an entity value) would create ambiguity about which "Zyvanta Sedan" mention is the evidence β€” the paper applies a rigorous four-step filtering pipeline (Appendix B.4):

  1. Enumerate all possible gold-fact sentences by applying each entity's value pool to its fact template. For example, if the entity vehicle has pool {Zyvanta Sedan, Therwyn Compact, Xylorim Scooter, ...} and the template is "I drive a {value}", the pipeline generates candidate sentences like "I drive a Zyvanta Sedan", "I drive a Therwyn Compact", etc. This exhaustive enumeration ensures no possible gold-fact variant is missed.

  2. Hybrid retrieval of candidate fillers: for each gold-fact sentence, retrieve the top-K=10 candidate filler sessions using a combination of BM25 lexical scoring and text-embedding-3-small dense similarity. This hybrid approach catches both keyword-overlapping fillers (BM25) and semantically similar fillers (dense retrieval) that might confuse a vector-based memory system.

  3. LLM-based conflict judgment: each (gold fact, filler) pair is judged by GPT-4o-mini using the prompt in Figure 25 (Section D.6). The judge flags three conflict types:

    • Type A (CONTRADICTION): the filler directly contradicts the gold fact. Example from Figure 7: gold says "My home is a one-bedroom apartment," filler mentions "the yard at my family home, which is now fully owned by me and my siblings." A user cannot simultaneously live in a one-bedroom apartment and jointly own a family home β€” the two facts are mutually exclusive.
    • Type B (ALTERNATIVE): the filler introduces a plausible alternative answer that could confuse a memory system, even without direct contradiction. Example: gold says "Our build tool is Dranith Compiler," filler mentions "build and test software using Travis CI." Travis CI is a real build tool, and a memory system retrieving both filler and evidence might report Travis CI as the user's build tool β€” not because the system is wrong about the evidence, but because retrieval conflates the filler with the gold fact.
    • Type C (ENTITY_CONFUSION): the filler mentions the same entity or topic in a confusing way, typically through shared names or entities. Example: gold says "Our team lead is Sarah Chen," filler mentions a reviewer named "Chris Chen." A memory system might incorrectly link these two mentions of "Chen" as referring to the same person or related facts.
    • Fillers flagged for any conflict type are removed from the pool.

  4. Keyword-based blocklist: at episode assembly time, a final safety net applies a keyword-based blocklist derived from the current episode's gold entities. This catches any remaining fillers that slipped through the LLM-based filtering.

Filter statistics (Table 10): the Personal Life pool goes from 1,130 pre-filter to 1,009 post-filter (121 sessions removed), with Type A (CONTRADICTION) dominant at 148 flag instances β€” everyday conversation fillers frequently make factual claims that could directly contradict a user's stated personal details. The Software Project pool goes from 9,135 pre-filter to 9,008 post-filter (127 sessions removed), with Type B (ALTERNATIVE) dominant at 253 flag instances β€” coding conversations frequently mention real tools, frameworks, and configurations that serve as plausible alternatives to the fictitious ones used in the gold facts. This asymmetry reflects the different conflict profiles of everyday conversation (people talk about their actual lives, creating direct contradictions with fictional user profiles) vs. technical discussion (people discuss many tools, creating alternative-answer confusion rather than direct contradictions).

Assembly and interleaving: given a total filler budget $B$ (32K tokens by default) and $N$ gaps between evidence sessions, fillers are placed into each gap in shuffled order until each gap's share of roughly $B/N$ tokens is filled. The paper provides a representative episode layout: [F 3, E1, F 3, E2, F 3, E3, F 3, E4, F 5, E5, F 3], where F n denotes n consecutive filler sessions and E1–E5 denote the five evidence sessions. The five evidence sessions consist of four Fact Introduction variants (introducing all static facts, dependency rules, and initial entity values) followed by a Change+Delete Event session (introducing the root entity change, three Tracking entity updates, and one Deletion request).

The paper reports average session counts per episode in Table 7b: 5 evidence sessions and 18 filler sessions for Personal Life (reflecting the filler interleaving pattern), and 5 evidence sessions with 14.9 filler sessions for Software Project. The token counts are balanced at approximately 32K filler tokens per domain.

Overall scale: the complete dataset comprises 100 evaluation episodes (50 per domain), each yielding on average 6.94 post-change evaluation questions (694 total: 332 for Personal Life, 362 for Software Project). This relatively modest scale β€” 100 episodes β€” reflects the cost of the rigorous generation and verification pipeline: each episode requires LLM self-chat generation, two-layer verification, and exhaustive filler filtering, making larger-scale automated generation infeasible without quality degradation.

Memory System Configurations and Architectural Differences

The paper evaluates six memory systems spanning three architectural paradigms. All systems use gpt-4.1-mini uniformly as both the internal LLM (for ingestion, extraction, retrieval planning, or tool-use decisions) and the answering LLM (for generating the final user-facing response from retrieved context) in the default configuration β€” a deliberate choice to "place every system on the same language-model footing and isolate differences in memory architecture" (Section 4.1). Five of the six systems issue the two roles as separate LLM calls; Karpathy Wiki performs both within a single agentic loop.

Each system's configuration is described in detail in Appendix C, with a summary table (Table 11) mapping each system across its encoding (what happens to facts at ingestion), maintenance (how facts are retained or updated), retrieval (how context is surfaced at query time), and storage substrate (the underlying data structure).

BM25 (Raw Retrieval β€” Lexical). This system stores each conversational session as chunks of 4,096 tokens (using the cl100k_base tokenizer, respecting turn boundaries) with a [Session: <timestamp>] header prepended. No internal LLM is used for ingestion β€” the raw session text is stored verbatim. At query time, lexical retrieval using the bm25s library (v0.3.2) with Lucene IDF scores the top-k=5 chunks by keyword overlap with the question. The in-memory index is rebuilt incrementally per session and reset between episodes to prevent cross-episode contamination. The design choice behind BM25 as a baseline is that it preserves the exact dependency rule text (since nothing is extracted or paraphrased), making it a best-case test of whether retrieval ranking alone β€” without any semantic understanding or dependency tracking β€” can surface the evidence needed for Cascade and Absence.

text-embedding-3-small (Raw Retrieval β€” Dense). Identical to BM25 in chunking (session-level, 4,096 tokens, top-k=5) but uses cosine similarity over OpenAI text-embedding-3-small embeddings (1,536-dimensional, L2-normalized). No internal LLM; embedding API cost is negligible (~$0.0007/episode) and is excluded from cost calculations (Table 5 note). The design choice behind this baseline is that dense retrieval captures semantic similarity, which might surface the change event even when the question's keywords don't match the change session's keywords β€” a capability that BM25 lacks. The paper's traces in Appendix I show that this advantage is real for the Absence task (the change event is retrieved more often than with BM25) but does not close the Cascade gap, because the answering LLM still fails to apply the dependency rule even when both the rule and the change event are in context.

Mem0 (LLM-Processed Memory β€” Fact Extraction). Mem0 decomposes each conversational session into discrete atomic facts through an internal LLM call (Memory.add()). The internal LLM extracts facts as natural-language sentences and performs conflict resolution: when a new fact contradicts an existing memory, it decides whether to ADD, UPDATE, or DELETE the existing entry. Facts are stored in a Qdrant vector database using text-embedding-3-small embeddings. At query time, Memory.search() returns the top-20 most relevant facts (the library default; the paper leaves top_k unspecified). Each episode uses a unique collection name to prevent cross-contamination. The paper identifies a critical architectural property of Mem0: its fact decomposition breaks the dependency rule into two independent facts β€” the current code reviewer (Hyunwoo Nam) and a conditional statement (if team lead changes, recipient will be James Lee). These are stored as separate vector entries with no explicit link between them. At query time, the vector retriever may surface both, but the answering LLM receives them as disconnected sentences, and the paper's traces (Appendix I, Figure 30) show that it reports the pre-change value because the change event (Jihoon Ryu is team lead) is either not retrieved or not connected to the conditional rule.

Graphiti (LLM-Processed Memory β€” Temporal Knowledge Graph). Graphiti encodes facts as entity-relation triples with temporal metadata (valid_at, invalid_at) in a Neo4j graph database. Each session triggers four steps: (1) entity extraction from the conversation text, (2) edge extraction (triples with fact descriptions), (3) entity deduplication against the existing graph, and (4) edge deduplication. Retrieval combines three signals: semantic search, BM25 keyword search, and graph traversal, returning the top-10 results. Each episode uses an isolated group_id. A critical design constraint that the paper highlights (Appendix C): Graphiti's extraction prompt explicitly instructs "closely paraphrase the original source sentence(s). Do not verbatim quote the original text" β€” a directive that directly explains its near-zero Exact Recall accuracy (0.03 in Table 2) because exact verbatim reproduction is architecturally impossible. For dependency reasoning, the paraphrasing destroys the conditional rule's precise language, and the graph traversal at query time surfaces only the most temporally recent edges, which are the pre-change values (the change event edge either falls below the top-10 threshold or is not traversed because the query's entity focus doesn't lead to it). The paper's per-stage trace in Figure 4 (left panel) shows this concretely: the conditional rule and pre-change value are encoded as edges, the change event is retained in the graph, but at retrieval only the rule and the pre-change value are surfaced, with the change-event edge absent from the top-10.

MD-flat (File-Based Agent β€” Minimal Single-File). This is the simplest file-based architecture, implemented by the paper's authors as a baseline rather than an off-the-shelf system. An LLM agent has access to a virtual file system containing a single markdown file (memory.md) and can use four tool calls: read_file, write_file, append_file, and list_files. The ingestion prompt (Figure 15) instructs the agent to "save any information the user shared that may be useful in future sessions. Keep it compact β€” one fact per line with timestamp [YYYY/MM/DD]. If information has changed, update it. If something was removed or cancelled, remove the old entry." The prompt deliberately "avoids task-specific hints, making no mention of dependencies, conditional rules, or deletion handling" (Appendix C). The tool-calling loop runs for a maximum of 5 rounds per operation; in practice, most operations complete in 2–3 rounds. The retrieval prompt (Figure 16) instructs the agent to read memory.md and extract relevant facts verbatim. At query time, the answering LLM then receives these extracted facts as context. The design choice behind MD-flat is that it provides maximum flexibility: the internal LLM can structure the memory file however it wants, and because the storage is plain text rather than a structured database, the agent could theoretically write propagated dependency values, maintain explicit contingency entries, or restructure the file to make dependencies traversable. The paper's internal-LLM ablation (Table 4, Appendix K) confirms that this flexibility is latent β€” with gpt-4.1-mini, the agent writes facts correctly but does not propagate; with Opus 4.7, it actively scans for dependent facts and writes resolved values.

Karpathy Wiki (File-Based Agent β€” Structured Knowledge Base). An implementation of Karpathy's LLM knowledge base concept using the claude-memory-compiler project without modification. Three-stage pipeline (Appendix C):

  1. Flush (ingestion): each session is passed to flush.py, which uses a single LLM call (no tool use) to extract important knowledge and append it to a daily log file (daily/YYYY-MM-DD.md). This creates a chronological record of what was learned each day.
  2. Compile (consolidation): after ingesting all sessions, compile.py processes the daily logs into structured knowledge articles organized into three categories β€” concepts, connections, and Q&A β€” with an index.md catalog. The compile operation runs with max_turns=30 and has access to file tools (Read, Write, Edit, Glob, Grep). This creates a two-tier storage structure: raw daily logs (primary source) and compiled articles (curated summaries).
  3. Query (retrieval + answer): at question time, query.py reads index.md to identify relevant articles, reads them, and generates an answer in a single agentic loop (max_turns=15). Karpathy Wiki uses its native query pipeline for answer generation rather than the unified answering LLM prompt used by the other five systems, meaning the cost is reported in the Retrieve column of Table 5 rather than the Answer column.

Each episode runs in an isolated workspace. The key architectural property for dependency reasoning: the two-tier structure means that at query time, the agent must navigate from index.md to the correct article to the correct daily log. The paper's trace in Figure 4 (right panel) shows that the query agent successfully navigates to the article containing the dependency rule and pre-change value, but never opens the daily log containing the change event β€” a retrieval failure caused by the compile step not propagating the change from the daily log into the compiled article. The information exists in the workspace, but the retriever does not reach it.

Uniformity of LLM usage across systems (and the one exception). The paper states that "all systems ingest identical chronological session transcripts and use gpt-4.1-mini uniformly in two roles: as the internal LLM (used inside the memory system for ingestion, extraction, or retrieval planning) and as the answering LLM (which produces the final user-facing answer from retrieved context). Five systems issue the two roles as separate LLM calls, while Karpathy Wiki performs both within a single agentic loop" (Section 4.1). This uniformity is critical: if some systems used a stronger LLM internally, the performance differences could reflect LLM capability rather than architecture. The one exception is that Karpathy Wiki's native query pipeline fuses retrieval planning and answer generation into a single agentic loop with tool calls, making its LLM calls structurally different from the "retrieve β†’ answer" split of the other five systems.

Separation of evaporation concerns: The paper cannot modify the internal prompts of off-the-shelf systems (Mem0, Graphiti, Karpathy Wiki, BM25, text-embedding-3-small) because those prompts encode design choices that are part of the system's architecture. Modifying them would be evaluating a modified system, not the system as deployed. Only MD-flat is the paper's own implementation, so its prompts are fully documented in Appendix D.3. The prompt optimization experiment in Section 4.4 (Appendix E) modifies MD-flat's, Mem0's, Graphiti's, and Karpathy Wiki's prompts, but this is framed as an intervention study (can we close the gap by optimizing prompts?) rather than a default configuration, and the results show it doesn't close Cascade/Absence.

Evaluation Protocol

After all memory systems have ingested the episode and answered all evaluation questions, a systematic evaluation protocol measures accuracy per task type.

Answer prompt: all memory systems share a single unified answer prompt (Figure 17, Section D.4) that feeds the system's retrieved context and the question to the answering LLM with the instruction: "Answer the user's question based ONLY on the context provided below. If the information is not in the context, say you don't have that information. Answer with ONLY the value. Do not explain or add context." This prompt is deliberately minimal β€” no task-specific hints, no instructions about dependencies or uncertainty β€” to test whether the memory system's retrieval stage (not the answering LLM's prompt engineering) determines answer quality. The answering LLM is gpt-4.1-mini in the default configuration (Table 2), with swaps to Claude Sonnet 4 for the answering-LLM ablation (Table 3b, Appendix H).

Judge: answer correctness is evaluated by a GPT-4o judge (temperature 0) using task-specific rubrics (Section D.5, Figures 18–24). The paper validates the judge against human annotations: on 144 samples, it achieves 98.6% agreement with a Cohen's $\kappa$ of 0.965 (Section 4.1), establishing that the automated judge's decisions are essentially interchangeable with human judgments at the scale of this study. The judge prompts differ per task type, reflecting the different evaluation criteria:

  • Before-phase judge (Exact Recall, Cascade, Absence pre-change check, Deletion pre-change check): uses the generic prompt in Figure 18, which checks for semantic equivalence: "Focus on semantic equivalence, not exact wording. If the gold value is present in the agent's answer, it is correct β€” regardless of any additional information." This leniency avoids penalizing systems that retrieve the correct value alongside extraneous context. Examples in the prompt clarify edge cases: "Dentist every 6 months; dermatologist monthly (if you change residence) β†’ gold is 'dentist (every 6 months)' β†’ YES (core answer correct, extra info is irrelevant)."
  • Exact Recall judge: implemented as a deterministic substring match at runtime, with the prompt in Figure 21 serving as fallback documentation. The gold value must appear verbatim as a substring in the answer. The prompt allows minor formatting differences (extra spaces, capitalization) but requires exact character-level match within the gold value β€” missing or substituted words within the value are rejected. The paper notes this is "deterministic substring match at runtime," meaning there is no LLM judgment involved; this eliminates any noise from the judge on Exact Recall scoring.
  • Tracking judge (Figure 19): requires the full history in chronological order. All gold values must appear in the exact earliest-to-latest order. Missing any value, wrong order, or only some values β†’ NO. This is strict binary scoring (despite the paper also computing partial-credit metrics for analysis).
  • Aggregation judge (Figure 20): checks each target value independently for presence in the answer. The output is per-value present/absent judgments. This allows partial-credit computation.
  • Deletion judge (Figure 22): the agent must either say "I don't have that information," express non-availability ("no longer available," "Unknown," "None"), or indicate the item no longer exists β€” all without revealing the deleted value. Crucially, the judge prompt states: "Agent reveals the deleted value in any way (even while saying it was deleted/removed) β†’ NO. Agent returns the deleted value as if it still exists β†’ NO." This prevents systems from "passing" by saying "Your partner's name was James, but you deleted it" β€” a response that technically acknowledges deletion but still reveals the sensitive information.
  • Cascade judge (Figure 23): the agent must state the new propagated value as the "sole, definitive answer." Listing both old and new values β†’ NO. Listing multiple options or hedging ("might," "considering") β†’ NO. Using future tense ("will change to X") instead of confirming as current β†’ NO. This strictness ensures that incomplete propagation β€” where the system recognizes a change might happen but doesn't commit to the resolved value β€” is scored as incorrect.
  • Absence judge (Figure 24): the agent must express uncertainty: "I don't know," "not sure," "Unknown," "None," or acknowledge the upstream change and question validity. Confidently stating the old value as current β†’ NO. Providing the old value with no hedging β†’ NO. This tests meta-cognitive awareness: the system must recognize the boundary of its knowledge, not simply retrieve the most recent stored value.

Trivial-pass filtering (the most important evaluation design choice): For Cascade, Absence, and Deletion tasks, the paper applies a filtering mechanism that requires the system to answer correctly both before and after the change or delete event to receive credit for the post-change question. The formal invariant: a post-change answer of "Seoyun Choi" on a Cascade question is only credited if the system also correctly answered "David Lee" on the pre-change version of the same question (or answered "James" on the pre-change Deletion question). The paper explains (Section 4.1): "This excludes false positives from systems that never encoded the fact. For example, on a Deletion task where the user first says their hobby is pottery and later asks to remove that fact, the system is credited only if it recalls pottery beforehand and stops reporting it afterward."

Why this filter is essential for interpretation: without it, systems could achieve artificially high Cascade, Absence, and Deletion scores through two independent failure modes:

  1. Never-encoded: the system never stored the original fact, so when asked about it post-change, it answers "I don't know" β€” which happens to match the Absence gold of "uncertain." The system is not reasoning about dependency at all; it's simply ignorant.
  2. Always-uncertain: the system defaults to expressing uncertainty on all questions, which would score 100% on Absence and Deletion but 0% on everything else. This system is not useful, but would appear to solve dependency reasoning.

The trivial-pass filter catches both: if the system can't produce the pre-change value, it gets no credit for the post-change answer, because the system clearly didn't have the dependency chain in memory to begin with. The paper demonstrates the practical importance of this filter in Appendix F (Table 13): under the 128K-filler condition, text-embedding-3-small's "raw" Absence accuracy would have been 0.35 (inflated by retrieval failures that made the system default to "I don't know") but the trivial-pass-filtered accuracy is 0.11 β€” because in many of those apparent passes, the system failed the pre-change check, meaning it never knew the original value and its post-change "I don't know" was accidental rather than reasoned.

Answering LLM swaps and the ceiling analysis: the paper also reports an in-context ceiling (Appendix L, Table 19) where only task-relevant gold facts are fed directly to the answering LLM, bypassing the memory system entirely. This establishes the maximum possible accuracy for each task if retrieval were perfect. With Claude Opus 4.7 as the answering LLM, the ceiling is 0.91 overall, with Cascade at 0.93 and Absence at 0.72. This confirms that (a) the tasks are solvable in principle given perfect context, and (b) the Absence task has a lower ceiling than Cascade even with perfect context β€” recognizing uncertainty is intrinsically harder for LLMs than applying a rule, even when the context makes the uncertainty explicit. The gap between the memory-system results (Cascade 0.03, Absence 0.01 average) and the ceiling (0.93, 0.72) quantifies the total loss attributable to the memory pipeline's encoding, maintenance, and retrieval stages.

Cost accounting: the paper reports per-episode dollar cost from observed LLM token usage at each LLM's public per-token rate (Section 4.1, Appendix A, Tables 5–6). Costs are separated into Ingest (LLM calls during session ingestion and memory update), Retrieve (LLM calls fetching context for a question), and Answer (the final answering LLM call). The answering LLM for all default-configuration systems is gpt-4.1-mini. For the internal-LLM ablation (Table 6), costs scale dramatically with the internal LLM's pricing: gpt-4.1-mini at 0.40/0.40/1.60 per 1M input/output tokens vs. gpt-5 at 1.25/1.25/10 vs. Opus 4.7 at 15/15/75. The paper reports that MD-flat Γ— Opus 4.7 costs ~70Γ— the gpt-4.1-mini baseline across the cascade episodes evaluated, establishing that dependency reasoning closure is not just architecturally limited but economically infeasible at scale with current frontier models.

Per-Stage Failure Analysis Framework

One of the paper's key methodological contributions is its per-stage diagnostic that traces where in each memory system's pipeline the dependency information is lost (Section 4.3). This framework decomposes each system's operation into three stages, reusing terminology introduced in Section 4.1:

Encoding: does the system write both the dependency rule and the pre-change value into its store when the user first states them? This is checked by inspecting the internal state of the memory system after the relevant session is ingested β€” for example, checking whether memory.md contains the dependency rule text (MD-flat), whether the Qdrant database contains facts extracted from the rule session (Mem0), or whether the Neo4j graph contains edges representing the rule (Graphiti).

Maintenance: when a later session introduces an upstream change event, is that change event retained in the store up to query time, and are any dependent facts updated or invalidated? This is checked by inspecting the store after the change session is ingested, before any query occurs.

Retrieval: when a question is asked that requires the propagated value, does the retrieval step actually surface the evidence needed β€” the dependency rule, the change event, and any propagated values β€” in the context that gets passed to the answering LLM? This is checked by logging the exact context string returned by each system's retrieve() method.

The paper applies this framework to a representative Cascade episode (sw_033, Figure 4) and traces it through all six systems, with the results for the two failing modes shown in Figure 4 and for the remaining four systems in Figure 30 (Appendix I). The consistent finding across all six systems: encoding passes (the rule and the pre-change value are written), maintenance passes (the change event is retained β€” nothing is deleted or overwritten), but retrieval fails β€” by two distinct mechanisms:

  • Retrieval failure (Type 1 β€” change event not surfaced): the retrieval step, for whatever architecture-specific reason, does not include the change event in the context passed to the answering LLM. This is the failure mode for BM25 (lexical top-k misses the change session), Graphiti (graph traversal misses the change-event edge), Karpathy Wiki (query agent doesn't open the daily log with the change event), and MD-flat with gpt-4.1-mini (tool-use loop doesn't open the 03/17 entry).
  • Answering failure (Type 2 β€” evidence retrieves but LLM doesn't propagate): the retrieval context includes both the dependency rule and the change event, but the answering LLM still reports the pre-change value. This is the failure mode for text-embedding-3-small and Mem0 (Figure 30): both the rule and the change event are in the top-k retrieved context, but the answering LLM, following its minimal prompt, reports what appears to be the most recent explicit statement about the code reviewer β€” the pre-change value.

This framework is essential for understanding why the various interventions (prompt optimization, deeper retrieval, stronger answering LLM, reduced noise) fail to close the gap β€” because the failure is split between retrieval (Type 1) and answering (Type 2), no single intervention addresses both. Deeper retrieval helps with Type 1 but is bottlenecked by Type 2; a stronger answering LLM helps with Type 2 but can't help when the evidence is never retrieved (Type 1). Only the Opus 4.7 on MD-flat intervention (Appendix K.2) bypasses both failure modes by writing the propagated value at ingestion, making the retrieval step trivial (the propagated value is the first thing the retriever finds) and the answering step straightforward (the answer is a declarative fact in the context).

4. Key Insights and Innovations

Innovation 1: Dependency Reasoning Is a Distinct, Missing Capability That No Prior Benchmark Even Measures

The paper's most fundamental conceptual contribution is not any single result but the identification and formalization of dependency reasoning as a memory capability that is both essential for real deployment and entirely absent from the evaluation landscape. The paper argues β€” and demonstrates through the complete absence of Cascade/Absence/Deletion tasks in five prior benchmarks (Table 1) β€” that the field has conflated "memory" with "flat fact storage and retrieval." Every existing benchmark tests whether a system can store and recall facts, and the more recent ones test whether it can overwrite old values with new ones (single-entity tracking). But none test what happens when one fact change should force re-evaluation of a different fact.

This is not a minor oversight β€” it reflects a fundamental assumption about what memory is. The dominant mental model in the field, implicit in the design of both architectures and benchmarks, treats memory as a collection of independent key-value pairs: store fact A, update fact B, retrieve fact C. Under this model, "getting memory right" means storing accurately, overwriting correctly, and retrieving the most recent version. MEME demonstrates that this model is structurally inadequate for any deployment where entities have logical relationships, because it cannot express the operation "when A changes, B must change too." The paper's diagnostic traces (Figures 4 and 30) show that all six evaluated systems do successfully execute the flat-storage operations β€” they encode the dependency rule, retain the change event, and can retrieve both β€” but they fail at the relational operation because their architectures have no mechanism for it.

The significance here is not empirical (finding that systems fail) but taxonomic and diagnostic: the paper gives the field a name and a measurement for a capability that was previously invisible, much as the shift from single-hop to multi-hop QA benchmarks made visible the gap between retrieval and reasoning in question answering. The two-axis taxonomy (entity scope Γ— temporal dynamics, Figure 1) is the conceptual apparatus that makes this visible: it shows that prior benchmarks populated the Single-entity/Static, Multi-entity/Static, and Single-entity/Evolving cells, but left the Multi-entity/Evolving cell entirely empty. MEME fills that cell with Cascade and Absence, and Figure 3 quantifies what happens when both axes are crossed: mean accuracy collapses from 0.44 (Single-entity/Evolving) to 0.02 (Multi-entity/Evolving), establishing that the interaction of multi-entity scope and temporal evolution is not simply the sum of its parts but a qualitatively harder problem.

The comparison to prior work is stark. LongMemEval, MemBench, and MemoryAgentBench β€” the three most recent and sophisticated memory benchmarks β€” all evaluate Tracking (single-entity evolution), and MemBench and LongMemEval evaluate Aggregation (multi-entity static). But by not crossing the two axes, they implicitly assumed that a system that can track individual changes and aggregate static facts can also handle interdependent evolving facts. MEME's results falsify that assumption: the additional difficulty of dependency reasoning is not captured by either axis alone, and a benchmark that only tests main-diagonal cells (single-static, multi-static, single-evolving) cannot detect the catastrophic failure in the off-diagonal cell.

Innovation 2: The Passive-Maintenance Architectural Assumption Is the Root Cause of Failure Across All Three Paradigms

The paper's per-stage diagnostic framework (Section 4.3) produces a finding that is both empirically robust and conceptually unifying: all six evaluated memory systems successfully encode dependency rules and change events, and successfully retain both through maintenance, but fail at retrieval because none actively propagates dependency consequences at maintenance time. In other words, the gap is not in what information is stored, but in what the architecture does with the stored information when an upstream change occurs.

This is a genuine architectural insight because it identifies a shared assumption across three different paradigms β€” raw retrieval, LLM-processed memory, and file-based agents β€” that was previously invisible because these paradigms appear to differ in almost every other respect. Raw retrieval stores chunks and retrieves by similarity; LLM-processed memory extracts structured facts into databases; file-based agents use tool-calling to curate documents. Yet all three share the same maintenance behavior: they store information when it arrives and leave it unchanged until another explicit update arrives for the same entity. None has a mechanism that says "when entity X changes, scan for facts that depend on X and update or invalidate them."

The paper's traces (Figure 4 and Figure 30) show that this is not a failure of any specific design choice β€” retrieval depth, embedding model, extraction prompt β€” but a failure of the trigger-based propagation that would be necessary to close the gap. The dependency rule sits passively in the store alongside the pre-change value and the change event. At query time, retrieval must simultaneously surface all three pieces and the answering LLM must reconstruct the propagation chain β€” a requirement that fails because retrieval ranking (whether vector, lexical, or graph-traversal) cannot reliably prioritize the change event over the pre-change value, and because the answering LLM, when presented with a mixture of pre-change and post-change information, defaults to reporting what appears most directly relevant (the pre-change value).

The significance of this insight is that it redirects the research agenda away from component-level improvements and toward a specific architectural missing piece. The paper's intervention studies (Section 4.4) systematically eliminate candidate explanations: it's not the prompt quality (prompt optimization doesn't help), not retrieval depth (deeper top-k doesn't close Cascade), not the answering LLM's capability (stronger LLMs don't help when the evidence is retrieved but not propagated), not noise (reducing filler to zero doesn't help). The one configuration that does close the gap β€” MD-flat with Opus 4.7 (Table 4, Appendix K.2) β€” does so precisely because it breaks the passive-maintenance pattern: Opus actively scans for dependency contingencies when a change arrives and writes the propagated value as a standalone declarative fact. This makes retrieval trivial (the propagated value is the top hit) and eliminates the answering-side propagation requirement. The paper thus establishes a clear design principle: dependency reasoning requires active propagation at maintenance time β€” either through a dedicated architectural mechanism or through an internal LLM capable of recognizing and executing the propagation β€” and without it, no practical-cost configuration can close the gap.

The comparison to prior work here is implicit but powerful. Prior memory architectures were designed for what the paper might call "append-and-overwrite" semantics: new information is appended, and contradictory information (same entity, different value) triggers an overwrite. This semantic model is correct for independent entities but fails for dependent ones, because the dependency rule is not about the entity being overwritten β€” it's about a different entity, and the architecture has no trigger that fires when the different entity changes. The paper's contribution is to name this gap (passive vs. active maintenance) and to show empirically that it is the single bottleneck across all three paradigms.

Innovation 3: The Retrieval-Answering Bottleneck Is a Two-Phase Failure With a Split That No Single Practical Intervention Can Address

The paper's per-failure analysis in the top-k sweep (Appendix J, Table 17) and the answering-LLM swap experiments (Table 3b, Appendix H) produce a finding that is methodologically sophisticated and practically consequential: on Cascade, dependency failures split roughly evenly between retrieval failures (the change event is never surfaced) and answering failures (the change event is surfaced but the answering LLM reports the pre-change value); on Absence, answering failures dominate at higher k. This split means that no single practical intervention β€” not deeper retrieval, not a stronger answering LLM β€” can close both halves of the gap simultaneously.

This insight matters because it explains the pattern of failed interventions in Section 4.4. Deeper retrieval (Table 3a) helps Absence somewhat (BM25 Absence rises from 0.07 to 0.24 as top-k goes from 5 to 20) but does nothing for Cascade β€” because the Cascade failures that remain after deeper retrieval are answering failures, not retrieval failures, and deeper retrieval cannot fix the answering LLM's failure to propagate. A stronger answering LLM (Table 3b, Sonnet 4 replacing gpt-4.1-mini) helps Absence for raw retrieval systems (text-embedding-3-small Absence rises from 0.00 to 0.16) but does nothing for Cascade on any system β€” because on Cascade, many failures are retrieval failures where the change event never reaches the answering LLM, and a stronger LLM cannot propagate from evidence it doesn't receive. The paper's split analysis in Table 17 quantifies this precisely: at k=20, 55% of Cascade failures are change-event misses (retrieval failures) and 45% are answering failures; for Absence at k=20, 86% are answering failures.

The conceptual contribution here is not the observation that "retrieval matters" or "answering LLMs matter" β€” both are obvious. It is the quantitative decomposition of the failure into two phases with different sensitivity to different interventions, which allows the paper to make a precise diagnostic claim: the Cascade gap cannot be closed by improving either retrieval or answering alone; it requires changes to both, or (as the Opus case study shows) a restructuring that eliminates the answering-phase propagation requirement entirely. This is methodologically sophisticated because most benchmark papers report aggregate failure rates without tracing where in the system the failure occurs; MEME's per-stage trace and per-failure analysis turn "the system fails at Cascade" into "the system fails because retrieval misses the change event 55% of the time and the answering LLM fails to propagate 45% of the time," which tells future researchers exactly which component to target and under what conditions.

Innovation 4: Dependency Reasoning Closure Is Economically Infeasible at Scale With Current Architectures and Frontier LLMs β€” Establishing a Concrete Cost Benchmark

The paper's most practically significant contribution is arguably the economic characterization of the dependency-reasoning gap: the one configuration that closes the Cascade-Absence gap (MD-flat with Claude Opus 4.7) costs approximately 70Γ— the baseline gpt-4.1-mini configuration (Table 6), and even this configuration sacrifices performance on Exact Recall and Tracking (Table 4) because Opus's hierarchical reorganization paraphrases content during ingestion. The paper is thus not just reporting that systems fail β€” it is establishing a concrete cost benchmark for what closure costs today, and arguing that this cost is incompatible with deployment at scale.

This is a distinctive kind of contribution because it converts an abstract capability gap into an economic constraint. Prior work on memory evaluation typically reports accuracy and stops there; MEME adds a cost dimension (Tables 5, 6) and uses it to make a deployment-relevant argument: even if you could afford Opus 4.7 for every ingestion call, the 70Γ— cost multiplier means that a system processing millions of user sessions would face ingestion costs that are infeasible for any practical application. And because the mechanism Opus uses β€” actively scanning for dependency contingencies and writing propagated values β€” is LLM-dependent (the same Opus does not help Mem0 or Graphiti, which decompose or paraphrase the contingency rules during ingestion), there is no obvious path to reducing this cost through architecture alone: the LLM must be capable enough to recognize and execute the propagation, and the substrate must be flexible enough to store the propagated values. The paper's finding that gpt-5, GLM-5.1, and gpt-4.1-mini all fail to achieve comparable propagation on the same MD-flat architecture (Table 4) suggests that the capability threshold is high β€” Opus 4.7 is not just marginally better but qualitatively different in its ability to maintain and propagate dependency chains.

The significance of this cost benchmark extends beyond the specific systems evaluated. By quantifying the gap in dollars rather than just accuracy points, the paper establishes a target for future research: not "build a system that solves Cascade," but "build a system that solves Cascade at a cost comparable to today's baseline configurations." This is a sharper and more useful formulation because it acknowledges that solving the problem with arbitrarily expensive computation is not a solution β€” the 70Γ— cost multiplier makes Opus-level closure a proof of concept, not a deployable answer. The paper's forward-looking recommendation in Section 5 β€” "memory architectures that natively propagate updates through dependent facts at maintenance, rather than relying on a costly internal LLM to do so" β€” follows directly from this economic analysis: the architectural mechanism must make propagation cheap enough that it doesn't require a frontier LLM at every session.

The comparison to prior work here is implicit but important. Most benchmark papers end with "this is hard, future work should do better." MEME ends with a specific cost target (roughly baseline-level cost for Cascade/Absence closure) and a specific architectural direction (native propagation at maintenance time), derived from the observation that the only working solution today achieves closure through brute-force LLM capability at a 70Γ— cost premium. This is a more actionable and falsifiable research direction than the generic "improve memory systems" that a less rigorously analyzed result would produce.

5. Experimental Analysis

Evaluation Methodology

Dataset. The MEME dataset comprises 100 evaluation episodes (50 per domain: Personal Life and Software Project), with a total of 694 post-change evaluation questions (332 for Personal Life, 362 for Software Project). Each episode contains approximately 35K tokens of conversational context interleaving 5 evidence sessions with filler sessions, and the dataset is constructed using a hand-crafted DAG-based knowledge graph per domain that makes gold answers verifiable by construction (Section 3.2, Table 7b).

Base model(s). All memory systems in the default configuration use gpt-4.1-mini uniformly in two roles: as the internal LLM (for ingestion, extraction, retrieval planning, or tool-use decisions) and as the answering LLM (for producing the final user-facing response). This uniform choice places every system "on the same language-model footing and isolates differences in memory architecture" (Section 4.1). For ablation studies, the answering LLM is swapped to Claude Sonnet 4, and the internal LLM is swapped across gpt-4.1-mini, gpt-5, GLM-5.1, and Claude Opus 4.7. An in-context ceiling (Appendix L) tests four answering LLMs (Opus 4.7, Sonnet 4.6, Sonnet 4, gpt-4.1-mini) with perfect gold-fact context. The self-chat verbalization uses gpt-4o; the two-layer verification uses gpt-4o for annotation and Gemini 2.5 Flash for semantic audit; the filler conflict judge uses gpt-4o-mini; and all answer evaluation uses a GPT-4o judge.

Metrics. The primary metric is per-task accuracy β€” the fraction of evaluation questions within each of the six task types (Exact Recall, Aggregation, Tracking, Deletion, Cascade, Absence) that the system answers correctly according to the GPT-4o judge. The overall accuracy is the mean across task types. For Cascade, Absence, and Deletion, accuracy is computed with trivial-pass filtering: a system is credited for a post-change answer only if it also correctly answered the pre-change version of the same question, excluding false positives from systems that never encoded the original fact (Section 4.1). Partial credit is computed additionally for Aggregation (number of target values present) and Tracking (number of history values in correct chronological order). Per-episode dollar cost is reported from observed LLM token usage at public per-token rates, separated into Ingest and Inference (retrieval + answer) costs (Appendix A, Tables 5–6).

Baselines. The paper evaluates six memory systems spanning three paradigms as its main comparison set rather than comparing against a single baseline (Section 4.1, Table 2, Appendix C):

  • Raw retrieval: BM25 (lexical, bm25s library with Lucene IDF) and text-embedding-3-small (dense, OpenAI embeddings, 1,536-dim, cosine similarity), both using top-k=5 chunk retrieval with 4,096-token session-level chunks.
  • LLM-processed memory: Mem0 (Chhikara et al., 2025) β€” fact extraction with Qdrant vector database, top-20 default; Graphiti (Rasmussen et al., 2025) β€” temporal knowledge graph in Neo4j, top-10 combined retrieval.
  • File-based agents: MD-flat β€” a minimal single-file agent with read/write/append tool calls implemented by the authors; Karpathy Wiki (Karpathy, 2026) β€” an implementation using the claude-memory-compiler project with three-stage flush-compile-query pipeline.
  • In-context (no memory): two answer-LLM variants (gpt-4.1-mini and Sonnet 4.6) that feed the entire 32K-filler episode transcript directly to the answering LLM, bypassing memory system ingestion and retrieval entirely. This anchors a cost-efficiency reference: in-context pays per-query inference cost (0.16/epforgptβˆ’4.1βˆ’mini)withnoingestioncost,whilememorysystemstradeoneβˆ’timeingestionoverheadforcheaperperβˆ’queryinference(0.16/ep for gpt-4.1-mini) with no ingestion cost, while memory systems trade one-time ingestion overhead for cheaper per-query inference (0.00–$0.04/ep for raw retrieval, Mem0, MD-flat).

Generation budget / compute accounting. The paper does not use "generation budget" in the conventional sense (e.g., number of sampled solutions), since the task is retrieval-augmented QA rather than generative search. Instead, cost is measured in two dimensions: (a) dollar cost per episode from observed token usage at per-token API rates, reported separately for ingestion and inference stages (Table 5 for default configuration, Table 6 for internal-LLM ablation); and (b) retrieval depth (top-k), which is swept across {5, 10, 20, 40} for BM25, text-embedding-3-small, and Mem0 to test whether dependency evidence is buried below the default cutoff (Table 3a). For the noise robustness study, filler volume is swept across three conditions: no filler, 32K tokens (default), and 128K tokens (Figure 29, Appendix F). The 32K default was chosen as representative of moderate-length agent conversations.

Cross-validation / statistical protocol. The paper uses single-seed evaluation for the main results (Table 2) but runs repeated-run stability analysis for non-deterministic systems (Mem0, Graphiti, MD-flat, Karpathy Wiki) under N=5 identical trials on a 10-episode subset to calibrate the noise floor (Appendix G, Table 14). The GPT-4o judge is validated against human annotations on 144 samples, achieving 98.6% agreement and a Cohen's ΞΊ of 0.965 (Section 4.1). The prompt optimization experiment (Appendix E) holds out a 10-episode test set (5 PL + 5 SW) disjoint from the 10-episode training set. The internal-LLM ablation (Table 4) and noise robustness study (Appendix F) use 20-episode and 40-episode subsets respectively due to compute cost of multiple experimental conditions. Only the main results (Table 2) and answering-LLM swap (Table 3b, Appendix H) cover all 100 episodes across all six systems.

Main Quantitative Results

Aggregate System Performance and the Dependency-Reasoning Floor

The headline result of Table 2 is that no system reliably solves dependency reasoning under the default gpt-4.1-mini configuration. Averaged across all six memory systems, Cascade achieves 0.03 accuracy and Absence achieves 0.01 β€” both at or near the floor β€” while static retrieval tasks remain substantially above floor: Exact Recall 0.62, Aggregation 0.23, Tracking 0.35, Deletion 0.17. The best single system (MD-flat) achieves only 0.42 overall, driven by strong performance on Exact Recall (0.94), Tracking (0.77), and Aggregation (0.45), but only 0.06 on Cascade and 0.05 on Absence β€” confirming that even the strongest architecture in the comparison set fails catastrophically on dependency propagation.

The system-level breakdown in Table 2 reveals meaningful architectural differentiation on non-dependency tasks:

  • Exact Recall splits systems into two groups: those that preserve verbatim text (BM25 at 1.00, text-embedding-3-small at 0.96, MD-flat at 0.94) and those that extract or paraphrase (Mem0 at 0.67, Karpathy Wiki at 0.11, Graphiti at 0.03). Graphiti's near-zero Exact Recall is explained by its extraction prompt's explicit instruction to "closely paraphrase… Do not verbatim quote" (Appendix C), which makes verbatim reproduction architecturally impossible. Karpathy Wiki's low score (0.11) reflects its compile step summarizing daily logs into articles, during which verbatim content (like error log strings) is either paraphrased or omitted.

  • Aggregation shows a similar split: text-embedding-3-small (0.33), Mem0 (0.35), and MD-flat (0.45) perform best because their retrieval mechanisms (dense search, vector search, and tool-use file reading, respectively) can surface multiple independently stored facts. BM25 (0.05) severely underperforms because lexical overlap with the aggregation question (e.g., "What do I do in my free time?") does not match the individual session keywords for each activity (pottery, rock climbing, book club), which were introduced in separate sessions with no explicit cross-references. Graphiti (0.01) fails because its graph traversal doesn't connect semantically related but structurally independent entities.

  • Tracking shows MD-flat at 0.77 and text-embedding-3-small at 0.46 as the strongest, reflecting that chronological value history is preserved when the store retains past entries (MD-flat's append-when-changed behavior, or raw chunk storage) rather than overwriting them. Mem0 at 0.43 and Karpathy Wiki at 0.27 show that LLM-processed extraction and compilation can lose historical values when the internal LLM overwrites or summarizes. Graphiti at 0.04 and BM25 at 0.16 both underperform for different reasons: Graphiti's edge invalidation mechanism may mark old values as inactive, and BM25's lexical retrieval may retrieve only the most recent session mentioning the entity, missing earlier sessions.

  • Deletion is the strongest-performing evolving task but still low across all systems (0.03–0.27). The trivial-pass filter explains some of this: many systems never encoded the to-be-deleted fact in the first place, so their apparent deletion passes are filtered out. MD-flat (0.25) and BM25 (0.27) score highest, suggesting that raw text storage makes it easier to both encode the fact initially and recognize the deletion request later, compared to systems that decompose or paraphrase facts.

  • Cascade and Absence are uniformly at or near floor across all six systems (Cascade: 0.01–0.06; Absence: 0.00–0.05). No practical-cost configuration breaks above 0.06 on Cascade or 0.05 on Absence. This uniformity β€” across raw retrieval, LLM-processed memory, and file-based agents β€” is the paper's central empirical finding: the dependency-reasoning gap is architectural, not specific to any one system's design choices.

The in-context baselines in Table 2 provide an important ceiling reference for cost-efficiency analysis. In-context with gpt-4.1-mini reaches Overall 0.36, outperforming five of the six memory systems (only MD-flat at 0.42 exceeds it). However, in-context's per-query inference cost (0.16/epforgptβˆ’4.1βˆ’mini,0.16/ep for gpt-4.1-mini, 1.50/ep for Sonnet 4.6) exceeds most memory systems' per-query inference cost (0.00–0.00–0.04/ep for raw retrieval, Mem0, MD-flat), making memory systems more cost-efficient as query volume grows β€” a classic ingest-once, query-cheap tradeoff. Sonnet 4.6 in-context reaches only 0.32 Overall, performing worse than gpt-4.1-mini on Exact Recall (0.50 vs. 1.00) and Aggregation (0.21 vs. 0.27), suggesting that the larger model's tendency to elaborate or summarize hurts verbatim and aggregation tasks in the in-context setting.

Marginal Effect of Each Evaluation Axis

Figure 3 decomposes the contribution of each evaluation axis to mean accuracy across the six main-table systems. Each axis individually drops accuracy by approximately 0.30: the temporal axis (static β†’ evolving) reduces mean accuracy from 0.42 to 0.14 (a drop of 0.28), and the entity-scope axis (single β†’ multi) reduces it from 0.44 to 0.13 (a drop of 0.31). Crossing both axes leaves the Multi-Evolving cell at a 0.02 floor (0.44 β†’ 0.02, a drop of 0.42). This non-additive interaction β€” the combined drop of 0.42 exceeds the sum of individual drops (0.28 + 0.31 = 0.59, but note these are marginal drops from different baselines) β€” demonstrates that multi-entity evolution is qualitatively harder than either dimension alone, not merely the sum of their difficulties. The paper interprets this as evidence that the two axes are "not redundant" β€” each measures a distinct capability that the other does not capture β€” and that the interaction between them (dependency reasoning) is the hardest quadrant by a large margin.

Where Dependency Reasoning Fails: Per-Stage Traces

The paper's diagnostic traces in Figure 4 (for Graphiti, Karpathy Wiki, and MD-flat with Opus 4.7) and Figure 30, Appendix I (for Mem0, MD-flat with gpt-4.1-mini, BM25, and text-embedding-3-small) trace a representative Cascade episode (sw_033) through the encoding, maintenance, and retrieval stages of each system. The consistent finding: all six systems successfully encode the dependency rule and pre-change value at ingestion, and all six successfully retain the change event through maintenance β€” but all six fail at retrieval, by one of two distinct mechanisms:

  • Retrieval failure (Type 1): the retrieval step does not surface the change event in the context passed to the answering LLM. This is the failure mode for Graphiti (graph traversal misses the change-event edge; only the rule and pre-change value are in the top-10), Karpathy Wiki (query agent navigates to the article with the rule but never opens the daily log containing the change event), BM25 (lexical top-5 misses the change session because its keywords differ from the query), and MD-flat with gpt-4.1-mini (tool-use loop retrieves the file entry with the rule and pre-change value but never opens the later entry where the change was recorded).

  • Answering failure (Type 2): the retrieval context includes both the dependency rule and the change event, but the answering LLM still reports the pre-change value. This is the failure mode for text-embedding-3-small (the change session is in the top-5 retrieved context alongside the rule session, but the answering LLM reports the pre-change value Hyunwoo Nam) and Mem0 (the change event is in the top-20 vector results at rank 19, and the rule is at rank 1, but the answering LLM reports Hyunwoo Nam).

All six systems answer the Cascade question with the pre-change value (Hyunwoo Nam) rather than the propagated value (James Lee). The paper's trace thus establishes that the dependency information is not lost β€” it sits passively in every system's store β€” but retrieval and answering together cannot reconstruct the propagation chain from the stored evidence.

Can We Close the Gap Without Changing the Architecture?

The paper tests five interventions (Section 4.4) to determine whether the Cascade-Absence gap can be closed by improving components external to the memory architecture itself, rather than by redesigning the memory system. All five interventions fail to close the gap for Cascade, and only two yield partial gains on Absence that fall far short of closure.

Prompt optimization (DSPy SIMBA): Applied to MD-flat, Mem0, Graphiti, and Karpathy Wiki (single-seed run on a 10-episode SIMBA test set; Appendix E, Table 12), prompt optimization produces no improvement on Cascade or Absence for any system. Figure 5a visualizes this: across all four systems, Cascade and Absence (red lines) stay at or near the floor in both the base and SIMBA conditions. For three systems (MD-flat, Graphiti, Karpathy Wiki), the SIMBA-optimized candidate appended advice explicitly targeting dependency failure modes β€” for instance, MD-flat's optimized prompts include "explicitly encode the dependency chains and update or remove old facts accordingly" and "infer uncertainty when dependencies conflict or are unresolved" (Appendix E.2, Figures 26–28). Despite this explicit instruction, Cascade and Absence do not improve, indicating the gap is "structural rather than instructional" (Section 4.4). For Mem0, SIMBA selected the library's default extract prompt unchanged as the winning candidate (the two alternative candidates that appended rule blocks scored lower on training), and test accuracy moved from 0.545 to 0.534 (βˆ’1.1pp), within the within-system noise floor.

Multi-seed stability analysis for MD-flat (Table 15, Appendix G.2, N=5 trials on the SIMBA test set) confirms that the lack of Cascade/Absence gain is robust to sampling: Cascade drops from 0.07 Β± 0.03 (baseline) to 0.02 Β± 0.03 (optimized), a difference within the standard deviation, while Absence drops from 0.03 Β± 0.05 to 0.00 Β± 0.00. In contrast, non-dependency tasks show large gains far exceeding the noise floor: Tracking 0.30 Β± 0.10 β†’ 0.92 Β± 0.04, Aggregation 0.30 Β± 0.12 β†’ 0.78 Β± 0.08, Deletion 0.02 Β± 0.04 β†’ 0.42 Β± 0.08. This dichotomy β€” prompt optimization dramatically improves tasks driven by what the memory file contains, but does nothing for tasks requiring propagation β€” reinforces that the gap is in the propagation mechanism itself, not in whether the system "knows" it should propagate.

Increased retrieval depth: Table 3a sweeps top-k across {5, 10, 20, 40} for BM25, text-embedding-3-small, and Mem0 on a 40-episode subset with Sonnet 4 as the answering LLM. Cascade remains near zero at every k value across all three systems. For BM25, Cascade is 0.02 at k=5, 10, 20, and 40 β€” completely flat. For text-embedding-3-small, Cascade is 0.02 at k=5 and k=10, then 0.00 at k=20 and k=40 β€” if anything, deeper retrieval reduces Cascade accuracy slightly. For Mem0, Cascade is 0.00 at k=5, 10, 20, and 40. Absence shows partial improvement with deeper retrieval on the raw-retrieval systems: BM25 Absence rises from 0.07 at k=5 to 0.24 at k=20, then drops to 0.21 at k=40; text-embedding-3-small Absence rises from 0.15 at k=5 to 0.23 at k=20, then drops to 0.15 at k=40. The non-monotonic pattern (rising then declining) suggests that extremely deep retrieval (k=40) introduces noise that counteracts the benefit of surfacing the change event. Mem0 Absence stays at the floor (0.02–0.04) across all k values.

Retrieval vs. answering bottleneck decomposition: The per-failure analysis in Table 17 (Appendix J) partitions knew_but_failed cases by which stage missed the propagated answer. For Cascade at k=20 and k=40, 55% of failures are change-event misses (retrieval failures where the change session is not in the retrieved context) and 45% are answering failures (both the rule and the change session are retrieved but the answering LLM still reports the pre-change value). For Absence at k=20 and k=40, the split is more extreme: 86% and 83% of failures respectively are answering failures β€” the rule and change event are both in the retrieved context, but the answering LLM commits to a definite answer when it should abstain. This decomposition explains why deeper retrieval partially helps Absence (by reducing retrieval failures) but not Cascade: on Cascade, even when retrieval is perfect, the remaining answering failures block half the potential gains; on Absence, answering failures dominate so heavily (86% at k=20) that fixing retrieval alone leaves most failures untouched.

Stronger answering LLM: Table 3b replaces the answering LLM on all six main-table systems and 100 episodes (gpt-4.1-mini β†’ Claude Sonnet 4). Cascade does not improve on any system: the average across the six systems drops from 0.03 to 0.02, with no individual system exceeding a 0.02 change (Graphiti rises from 0.02 to 0.04; MD-flat drops from 0.06 to 0.05). Absence shows small gains on the raw-retrieval systems: BM25 Absence rises from 0.00 to 0.12, text-embedding-3-small from 0.00 to 0.16. But these gains are modest relative to the gap (the in-context ceiling for Absence with Sonnet 4 is 0.81, per Table 19), and the LLM-processed and file-based systems show no Absence improvement (Mem0 stays at 0.00, Graphiti at 0.00, MD-flat at 0.05, Karpathy Wiki at 0.02). The paper concludes that "a capable answering LLM cannot reconstruct dependencies that the memory layer never preserved" β€” the answering LLM can only propagate if the retrieval stage provides both the rule and the change event in usable form, and on the systems where retrieval fails (Type 1 failures), a stronger answering LLM has nothing to work with. The per-system breakdown in Table 16 (Appendix H) shows that the answering-LLM swap sometimes reduces performance on non-dependency tasks: BM25 Exact Recall drops from 1.00 to 0.70, text-embedding-3-small from 0.96 to 0.43, suggesting that Sonnet 4's tendency to paraphrase or elaborate hurts verbatim reproduction tasks that gpt-4.1-mini handled more literally.

Reduced filler noise: Figure 5b compares the default 32K-filler condition against a no-filler condition on the highest-overall system within each paradigm: MD-flat for file-based, Mem0 for LLM-processed, and text-embedding-3-small for raw retrieval (per Table 2). Cascade and Absence remain at or near the floor in both conditions. The full three-condition sweep (no filler, 32K, 128K) in Figure 29 (Appendix F) confirms this across all three systems: Cascade stays within 0.00–0.08 and Absence within 0.00–0.19 across all noise levels. The paper notes a counter-intuitive finding for MD-flat: overall accuracy is lower without filler (0.40) than with 32K filler (0.45) because the retrieval step (gpt-4.1-mini) behaves differently when the memory file is short β€” it strips timestamps and entity labels, returning bare bullet lists β€” and the answering LLM, following its strict context-only instruction, refuses to infer chronological order. With 32K filler, the memory file is longer and more diverse, prompting the retrieve step to preserve metadata. This artifact is "specific to the retrieve prompt's sensitivity to memory file length rather than a property of the benchmark itself" (Appendix F), and reinforces that noise reduction alone cannot close the dependency-reasoning gap.

Trivial-pass rates rise with noise (Appendix F, Table 13): The paper reports that under the 128K-filler condition, text-embedding-3-small's trivial-pass rate for Absence reaches 0.51 β€” meaning that in more than half of Absence questions, the system's post-change answer would have been credited as correct (it said "I don't know") but the system failed the pre-change check, so the pass was filtered out. Without the trivial-pass filter, text-embedding-3-small's raw Absence accuracy would be 0.35 at 128K; with the filter, it is 0.11. This demonstrates that the trivial-pass filter is essential for valid interpretation β€” without it, retrieval failures that cause the system to default to "I don't know" would artificially inflate Absence scores, masking the true dependency-reasoning failure.

Internal-LLM swap: Table 4 swaps the internal LLM on the three systems that use one (Mem0, Graphiti, MD-flat) across gpt-4.1-mini, gpt-5, GLM-5.1, and Claude Opus 4.7 on a 20-episode subset, with the answering LLM held at Sonnet 4. This is the only intervention that narrows the Cascade-Absence gap, and it does so in exactly one cell: MD-flat with Opus 4.7. In this configuration, Cascade reaches 0.32 (up from 0.00 with gpt-4.1-mini) and Absence reaches 0.59 (up from 0.07) β€” substantial improvements that break the floor pattern. No other (system, internal-LLM) pair shows comparable gains: the remaining 11 cells show Cascade in the range 0.00–0.13 and Absence in the range 0.00–0.30, with most at or near 0.00–0.07.

The mechanism behind Opus's success on MD-flat is traced in Appendix K.2: Opus restructures the memory file into a structured document with topical ## sections and records each dependency rule as an explicit Contingency: entry. When an upstream change arrives, Opus scans for dependent contingency entries and writes the propagated value as a standalone declarative fact. Crucially, this moves the propagation work from retrieval time (where all other configurations fail) to ingestion time β€” the propagated value is written into memory as a regular fact, making retrieval trivial (the retriever surfaces it directly) and eliminating the answering-side propagation requirement. For Cascade (where a replacement rule exists), Opus writes the resolved value (e.g., "per contingencies, dietary restriction now no alcohol and exercise routine now yoga 2x/week"). For Absence (where no replacement rule exists), Opus creates a ## section titled "need re-confirmation since [upstream change]" and removes the old dependent facts, causing the answering LLM to respond "I don't have that information."

The same Opus does not help Mem0 (Cas 0.03, Abs 0.00) or Graphiti (Cas 0.00, Abs 0.04) because Mem0's fact decomposition breaks the contingency entries into disconnected atomic facts (the contingency wording is lost), and Graphiti's entity-relation triple extraction discards the conditional structure entirely. Closure thus requires both a capable internal LLM (Opus 4.7, not gpt-5, GLM-5.1, or gpt-4.1-mini) and a substrate that preserves the contingency entries as structured text (MD-flat's plain markdown file, not Mem0's vector database or Graphiti's knowledge graph).

The cost of closure: Table 6 reports that MD-flat Γ— Opus 4.7 costs 4.54/episodetotal(ingestion+inference),approximately70Γ—thegptβˆ’4.1βˆ’minibaselineof4.54/episode total (ingestion + inference), approximately 70Γ— the gpt-4.1-mini baseline of 0.065/episode. This cost is dominated by Opus's per-token pricing (15/15/75 per 1M input/output tokens) and the token volume of its structured writes (Table 6: Opus ingest is 222,802 input + 7,018 output tokens vs. gpt-4.1-mini's 89,203 + 4,990). Additionally, Opus's hierarchical reorganization paraphrases content, degrading Exact Recall from 0.90 to 0.60 and Tracking from 0.80 to 0.20 on the same 20-episode subset β€” a tradeoff where dependency reasoning closure comes at the cost of verbatim fidelity and history retention. The other internal LLMs show no comparable Cascade/Absence improvement: gpt-5's lossy compression (Table 18 shows it compresses the memory file by 28% at the change event, erasing prior entries) yields accidental Absence matches but no genuine Cascade propagation; GLM-5.1's append-only behavior preserves prior entries, causing it to commit to stale values on both Cascade and Absence.

Ablation Studies and Robustness Checks

Answering LLM swap across all six systems (Table 3b, Appendix H, Table 16): replacing gpt-4.1-mini with Sonnet 4 as the answering LLM on 100 episodes shows that Cascade does not improve on any system and Absence improves only on raw-retrieval systems (BM25: 0.00 β†’ 0.12; text-embedding-3-small: 0.00 β†’ 0.16), while remaining at floor on Mem0 (0.00), Graphiti (0.00), MD-flat (0.05), and Karpathy Wiki (0.02). The answering LLM swap also produces performance regressions on Exact Recall for raw retrieval (BM25 1.00 β†’ 0.70; text-embedding-3-small 0.96 β†’ 0.43), likely because Sonnet 4 paraphrases rather than reproducing verbatim.

Repeated-run stability (Appendix G, Table 14): rerunning ingestion, retrieval, and answering under N=5 identical trials on a 10-episode subset confirms that the main-table Cascade and Absence floor is robust to sampling. Cascade standard deviation is ≀0.03 and Absence SD is ≀0.04 across all four non-deterministic systems. Per-system Overall accuracy moves by at most 0.02, preserving the ordering from Table 2. Karpathy Wiki shows the largest task-level variance (Tracking SD 0.15), driven by its agentic query loop with multiple internal LLM calls per question.

Noise robustness sweep (Appendix F, Figure 29, Table 13): evaluating MD-flat, Mem0, and text-embedding-3-small under three filler conditions (no filler, 32K, 128K) on a 40-episode subset shows that all three systems exhibit overall degradation as filler volume increases (MD-flat: 0.45 β†’ 0.36; Mem0: 0.26 β†’ 0.22; text-embedding-3-small: 0.23 β†’ 0.16), but Cascade and Absence remain at floor across all noise levels. The trivial-pass filter is shown to be essential at high noise: text-embedding-3-small's Absence trivial-pass rate rises from 0.08 (no filler) to 0.51 (128K), meaning half of apparent Absence passes are false positives from retrieval failures.

Retrieval depth sweep (Table 3a): top-k varying across {5, 10, 20, 40} for BM25, text-embedding-3-small, and Mem0 on a 40-episode subset shows that Cascade stays at floor for all k on all three systems, while Absence on raw-retrieval systems shows a non-monotonic pattern (BM25: 0.07 β†’ 0.15 β†’ 0.24 β†’ 0.21; text-embedding-3-small: 0.15 β†’ 0.19 β†’ 0.23 β†’ 0.15), peaking at k=20 then declining, suggesting that very deep retrieval introduces competing noise.

Retrieval vs. answering bottleneck decomposition (Appendix J, Table 17): for the k=20 and k=40 conditions in the top-k sweep, Cascade failures split 55% retrieval (change-event miss) / 45% answering (rule and change event retrieved, but LLM doesn't propagate), while Absence failures are 86% answering failures at k=20. This decomposition explains why no single practical intervention closes either gap: deeper retrieval addresses only the 55% retrieval-failure fraction of Cascade and the 14% retrieval-failure fraction of Absence, leaving the answering-side fraction untouched; a stronger answering LLM addresses the answering failures but cannot help when the evidence is never retrieved. Only MD-flat Γ— Opus 4.7 bypasses both failure modes by writing propagated values at ingestion, making retrieval trivial and eliminating the answering-phase propagation requirement.

MD-flat internal-LLM compression behavior (Appendix K.1, Table 18): tracing the per-LLM write_memory behavior on the 20-episode internal-LLM subset reveals distinct compression strategies. gpt-4.1-mini barely writes on Change+Delete sessions (10% write rate, effectively append-only). gpt-5 writes on 95% of sessions but compresses aggressively, shrinking the pre-event memory by 28% at the change write and cumulatively reducing it to ~1,800 chars β€” this erases prior entries including the original entity value and dependency rule, producing accidental Absence matches (the system says "I don't know" because the memory is empty) but no Cascade propagation. GLM-5.1 writes on 95% of sessions with a +1% size change (effectively append-only), preserving prior entries but failing to propagate because stale values remain dominant. Opus 4.7 writes on 100% of sessions with a βˆ’5% size change (restructuring with Contingency entries), actively scanning for and resolving dependencies.

Gold-facts in-context ceiling (Appendix L, Table 19): feeding only task-relevant gold facts directly to the answering LLM across four answer LLMs establishes the maximum possible accuracy if retrieval were perfect. With Opus 4.7, the ceiling is 0.91 overall (Cascade 0.93, Absence 0.72), confirming that the tasks are solvable in principle. The Absence ceiling (0.72) is notably lower than Cascade (0.93), consistent with the finding that recognizing uncertainty is intrinsically harder for LLMs than applying a rule. With Sonnet 4 (the answering LLM used in most ablations), the ceiling is 0.87 overall (Cascade 0.60, Absence 0.81), meaning the ablation studies using Sonnet 4 are evaluated against a lower Cascade ceiling than the Opus studies. The gpt-4.1-mini ceiling is 0.70 overall (Cascade 0.74, Absence 0.37), establishing a lower but still substantial bound for the default configuration.

Verbalization verification (Appendix D.2, Figures 13–14): the two-layer LLM verification (gpt-4o annotation + Gemini 2.5 Flash semantic audit) confirms that all generated conversational turns faithfully reflect the underlying gold facts, with issue severity rated as HIGH/MEDIUM/LOW. The paper does not report quantitative pass rates for the verification, but the existence of a two-layer, two-model verification pipeline addresses the concern that LLM-generated conversations might introduce factual drift.

Critical Assessment

Claim: All practical-cost configurations collapse on dependency reasoning (Cascade 3%, Absence 1% average accuracy). This claim is robustly supported by the main results (Table 2) across all six systems and all 100 episodes. The per-system Cascade range (0.01–0.06) and Absence range (0.00–0.05) are so tightly clustered at the floor that the between-system variance is negligible compared to the gap from the in-context ceiling (Cascade 0.74–0.93, Absence 0.37–0.72, Table 19). The repeated-run stability analysis (Appendix G, Table 14) confirms that within-system sampling noise (SD ≀ 0.03 for Cascade, ≀ 0.04 for Absence) cannot account for the gap. A genuine limitation: the claim is restricted to "practical-cost configurations" β€” the Opus 4.7 result (Table 4) demonstrates that the gap is not absolute but is instead economically prohibitive at 70Γ— baseline cost, which is a more precise and defensible claim than "no configuration works." However, the Opus result is on a 20-episode subset, not 100 episodes, and the Cascade score of 0.32, while far above the floor, is still well below the 0.93 ceiling, meaning even Opus-level closure is partial.

Claim: The failure is structural, not instructional (prompt optimization, deeper retrieval, reduced noise, and a stronger answering LLM fail to close the gap). This claim is supported with qualifications specific to each intervention. Prompt optimization (Appendix E, Figure 5a) is the weakest evidence because the SIMBA experiment uses a 10-episode test set, single-seed for three of four systems, and a modest optimizer budget (max_steps=2, bsize=4). A more thorough prompt optimization study (multi-seed, larger search budget, different optimizers) could reveal that more sophisticated prompting does help β€” the fact that SIMBA's winning candidates appended dependency-targeting advice and still failed is suggestive but not conclusive. Deeper retrieval (Table 3a) is stronger evidence: sweeping top-k across a factor of 8 (5β†’40) on a 40-episode subset shows Cascade flat at floor for all three systems, which would be unlikely if the dependency evidence were simply buried below a too-shallow cutoff. The answering-LLM swap (Table 3b) is weaker as evidence that answering-side improvements don't help, because the swap is gpt-4.1-mini β†’ Sonnet 4, and while Sonnet 4 is a meaningfully stronger model, it is possible that an even stronger answering LLM (Opus 4.7, or a reasoning-focused model like o1) could succeed at answering-side propagation where Sonnet 4 fails. The noise reduction study (Appendix F, Figure 5b) is the strongest evidence that the gap is not a retrieval-quality issue, because zero-noise (no filler) still produces Cascade at floor β€” at zero noise, every relevant fact is trivially retrievable, and the fact that the answering LLM still fails to propagate implies the bottleneck is in how the retrieved information is used, not in whether it is retrieved.

Claim: Closure emerges only with a frontier internal LLM (Opus 4.7) on a file-based substrate, at ~70Γ— baseline cost. This claim is supported by Table 4 and the per-LLM mechanism analysis in Appendix K, but with important caveats. First, the internal-LLM ablation uses a 20-episode subset (limited by Opus API costs), and the Opus Cascade score of 0.32 on this subset may not generalize to the full 100 episodes β€” the confidence interval around a 0.32 score on 20 episodes is wide enough that the true 100-episode accuracy could be substantially lower or higher. Second, the claim that "closure emerges" depends on what counts as closure: Cascade 0.32 is far above the floor of 0.03 but far below the ceiling of 0.93, and Absence 0.59 is above the floor of 0.01 but below the ceiling of 0.72. Whether this constitutes "closure" depends on the deployment tolerance β€” 32% Cascade accuracy means the system still fails on 68% of dependency questions, which may be unacceptable in many applications. Third, the paper does not ablate why Opus on MD-flat works β€” is it the Contingency-entry mechanism specifically, or would any LLM of sufficient capability exhibit the same behavior? The fact that gpt-5, GLM-5.1, and gpt-4.1-mini all fail on the same architecture suggests a capability threshold, but without testing intermediate models (Claude Sonnet 4 as internal LLM, or gpt-4o) the threshold is not precisely characterized.

Missing experiments that would strengthen the paper:

  • Combination of PRM-style search with memory. The paper studies the answering LLM in isolation (retrieve β†’ answer) but never tests whether giving the answering LLM multiple retrieval passes, chain-of-thought reasoning over retrieved facts, or self-consistency voting across multiple answer generations could close the answering-failure fraction of the Cascade gap (45% of failures at k=20, per Table 17). This is a natural ablation given that the paper's related-work section discusses test-time compute scaling extensively.

  • Intermediate internal LLMs. The internal-LLM ablation tests gpt-4.1-mini, gpt-5, GLM-5.1, and Opus 4.7 β€” a jump from 0.40/0.40/1.60 per 1M tokens to 15/15/75 with nothing in between. Testing Claude Sonnet 4 as the internal LLM (at 3/3/15) on MD-flat would characterize whether the capability threshold is near the frontier or whether mid-tier models can achieve partial propagation at intermediate cost.

  • Dynamic or iterative retrieval. The paper's retrieval is a single-shot top-k call. An alternative design where the answering LLM can request additional retrieval passes (e.g., "retrieve facts about team lead changes" after seeing the dependency rule) might overcome the retrieval-failure fraction β€” a retrieval-augmented generation loop rather than a single retrieve-then-answer pipeline. This is a natural extension that bridges the gap between the paper's finding (retrieval fails to surface the change event) and its implication (architectural propagation at maintenance is needed).

  • Scalability beyond 100 episodes. The dataset size (100 episodes, 694 questions) is modest. A larger-scale version (1,000+ episodes, automated generation) would allow finer-grained analysis of how difficulty varies with dependency chain length, rule explicitness, or filler similarity, and would support statistical tests (confidence intervals, significance testing) that the current dataset size precludes.

  • Ablation of dependency rule explicitness. The paper acknowledges (Section 6) that verbalization uses "explicit conditional phrasing for dependency rules as a best-case framing." An ablation with implicit conditionals (e.g., "the code reviewer is David Lee, but if the team changes...") or no explicit conditional at all (relying purely on the graph structure) would characterize how much the explicit phrasing helps and whether the failure is robust to weaker or absent conditionals.

Overall experimental strength: The paper's experiments are methodologically sophisticated for a benchmark paper β€” they go substantially beyond "we built a dataset and tested some systems" by systematically tracing failure modes through each system's pipeline stages, decomposing the failure into retrieval vs. answering components, and testing five distinct interventions. The repeated-run stability analysis, judge validation against human annotations, and trivial-pass filtering all address common threats to validity in LLM evaluation. The main weakness is the relatively small scale (100 episodes, 20–40 episode subsets for most ablations) combined with the cost of the one configuration that shows progress (Opus 4.7), which limits the statistical precision of the headline finding about closure. The paper's central empirical claims β€” that dependency reasoning is at floor across all practical-cost configurations, and that active propagation at maintenance is the missing mechanism β€” are well-supported by the evidence presented, though the precise numerical characterization of the gap (3% Cascade, 1% Absence) should be interpreted as specific to the gpt-4.1-mini configuration and the 100-episode dataset, not as universal constants.

6. Limitations and Trade-offs

Limitation 1: Difficulty Estimation Is Computationally Prohibitive β€” the Cost of Knowing Where to Spend Compute Is Comparable to the Budget Being Spent

The compute-optimal allocation framework's central mechanism β€” conditioning the test-time strategy on estimated prompt difficulty β€” requires generating 2048 samples per question and scoring them with the PRM to estimate the pass@1 rate (or PRM-score-based proxy), then binning questions into quintiles (Section 3.2). This difficulty estimation step is performed before the test-time strategy is selected and executed, and its cost is not included in the reported generation budgets or efficiency gains. The paper is explicit about this:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence is that the paper's headline claim β€” compute-optimal scaling achieves 4Γ— better efficiency than best-of-N (Figures 4 and 8) β€” represents an upper bound that is not achievable in deployment without amortizing the difficulty estimation cost across many queries to the same question. In a single-question deployment scenario (the most common use case for an LLM assistant answering a user's question), the total cost would be 2048 samples for difficulty estimation plus the strategy's generation budget β€” meaning the total cost could be higher than simply running best-of-N uniformly rather than lower. The 4Γ— figure is valid only if the difficulty estimation cost is amortized to near-zero, which requires either (a) the same question to be asked many times (unlikely in most deployment settings), (b) a cheaper difficulty prediction method (not developed in the paper), or (c) offline pre-computation of the optimal strategy per difficulty bin per budget level (which is what the paper does on the validation fold, but this requires knowing the distribution of questions in advance).

What evidence exists in the paper: The paper acknowledges this cost explicitly in Section 3.2 and the Limitations section (Section 6): "our experiments do not account for this cost largely for simplicity." The difficulty estimation procedure is described in Section 3.2: 2048 samples per question, passed through the PRM for predicted bins or through ground-truth evaluation for oracle bins. The paper does not report the FLOP or dollar cost of this step, nor does it include it in any budget calculation or efficiency comparison. The compute-optimal scaling curves in Figures 4 and 8 plot accuracy against the strategy's generation budget only, with the difficulty estimation cost excluded from the x-axis.

Mitigation status: The paper flags this as "a key avenue for future work" (Section 3.2) and suggests training a model to predict difficulty directly from the question text. No such model is developed or evaluated. A partial mitigation is that the predicted (PRM-based) difficulty bins perform nearly as well as oracle bins (Figures 4 and 8), meaning ground-truth labels are not required β€” but the 2048-sample generation cost remains. The paper does not explore adaptive difficulty estimation (start with a small number of samples, assess preliminary difficulty, allocate remaining budget), which could partially amortize the estimation cost into the problem-solving process. Until a cheap difficulty estimator exists, the 4Γ— efficiency figure should be treated as a laboratory result that demonstrates what is possible if difficulty were known, not what is achievable in a deployed system.

Limitation 2: Hard Problems Are Completely Unsolved β€” Test-Time Compute Cannot Create Capability That the Base Model Lacks

The paper's most striking empirical finding has a sharp boundary condition that is both practically consequential and under-discussed in the abstract framing: on the hardest 20% of problems (difficulty quintile 5), no test-time compute strategy β€” search, revisions, or their compute-optimal combination β€” produces meaningful improvement above the base model's near-zero pass@1 rate. For search, bin 5 accuracy hovers at 1–3% across all methods and budgets (Figure 3, right). For revisions, bin 5 accuracy is roughly 2–3% irrespective of the sequential-to-parallel ratio (Figure 7, right). In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% for both search and revisions.

The consequence is that the compute-optimal framework offers no path forward for problems that are genuinely outside the base model's capability range. If the base model's pass@1 on a problem class is near zero β€” meaning it almost never produces a correct answer in 2048 independent attempts β€” then no amount of search (which can only select among generated candidates) or revision (which can only refine generated candidates) can recover a correct answer; there are no correct answers in the proposal distribution to find or refine. This limitation is fundamental to the proposal-distribution framework the paper uses: test-time compute modifies and selects from the base model's output distribution but cannot introduce new capabilities that were absent from that distribution. The paper is candid about this (Section 7 takeaway box states "on the hardest questions... pretraining is almost always more effective"), but the practical implication deserves emphasis: for deployments where the problem distribution includes a heavy tail of genuinely hard problems, the compute-optimal framework is not a substitute for pretraining a larger model β€” it is complementary at best, and useless at worst.

What evidence exists in the paper: The difficulty-bin breakdowns consistently show bin 5 at or near floor across all experimental conditions. Figure 3 (right) shows bin 5 at 1–3% for both beam search and best-of-N across budgets from 4 to 256 generations. Figure 7 (right) shows bin 5 at 2–3% for all sequential-to-parallel ratios at 128 generations. Figure 9 shows the bin 5 scaling line essentially flat for both revisions and PRM search. The FLOPs-matched comparison (Figure 1, bottom-right bar chart) reports βˆ’52.9% relative disadvantage for test-time compute vs. pretraining on hard problems at R ≫ 1 with PRM search. The base model's pass@1 rate of roughly 3% on bin 5 (Section 3.2 defines difficulty bins by pass@1 quintile, with bin 5 being the lowest quintile) means the model generates a correct solution only once in about 33 attempts β€” and with only 256-generation budgets, the expected number of correct candidates is roughly 8, which is apparently insufficient for verifiers to reliably identify.

Mitigation status: The paper does not mitigate this limitation β€” it acknowledges it (Section 7) and recommends pretraining for hard problems. The boundary is characterized but not softened. No experiment tests whether a stronger base model would shift the difficulty distribution (making more problems fall into bins 1–4 where test-time compute helps), or whether training the verifier specifically on hard-problem data would improve the signal-to-noise ratio for identifying the rare correct solutions. The recommendation is effectively: "know your problem distribution, and if it's hard, don't bother with test-time compute β€” spend your budget on a larger pretrained model instead."

Limitation 3: Single Benchmark and Single Model Family β€” the Difficulty-Dependent Scaling Curves May Not Generalize to Other Domains or Architectures

All experiments in the paper use the MATH benchmark (500 test questions, high-school competition mathematics) with PaLM 2-S* as the base model. The paper's difficulty-dependent findings β€” beam search degrades on easy problems, sequential revisions help on easy problems, the compute-optimal policy allocates differently per difficulty bin β€” are derived entirely from this (model, dataset) pair. The paper states this scope limitation explicitly (Section 4): "We believe this model is representative of the capabilities of many contemporary LLMs," but acknowledges this is an unverified assumption. The Limitation section (Section 6) reiterates: "Single benchmark, single model family... a claim that cannot be verified without replication on other models and datasets."

The consequence is that practitioners cannot assume the specific difficulty thresholds, strategy preferences, or 4Γ— efficiency gains will transfer to their own model and task domain. Several model-specific properties could shift the results:

  • PRM over-optimization behavior depends on the PRM's calibration and the base model's output distribution. A model with different token-level uncertainty characteristics (e.g., a model that places more probability mass on correct completions) might exhibit less over-optimization, making beam search beneficial at higher budgets even on easy problems. Conversely, a model with worse calibration might exhibit over-optimization at even lower budgets, making the compute-optimal policy more conservative than what the paper reports.
  • Revision model training depends on the base model's in-context learning ability β€” specifically, its capacity to learn from incorrectβ†’correct trajectories during fine-tuning. The 38% correct-to-incorrect reversion rate (Section 6.1) is a PaLM 2-S*-specific artifact that may differ substantially across model families.
  • The MATH benchmark tests symbolic mathematical reasoning, which has specific properties (single correct answer, step-by-step deduction, verifiable intermediate steps) that may not transfer to other reasoning domains (code generation has different correctness criteria; factual QA tests retrieval of parametric knowledge rather than deduction; open-ended generation has no single ground truth).

What evidence exists in the paper: None that tests generalization. There is no replication on a second benchmark (e.g., GSM8K for simpler math, HumanEval for code, or a factual QA benchmark) and no replication with a second model family (e.g., a LLaMA or GPT model). The paper's claims are explicitly scoped to MATH + PaLM 2-S* in the title and abstract, and the limitation is acknowledged in Section 6. The 500-question test set, split into five difficulty bins of approximately 100 questions each, is a relatively small sample for characterizing difficulty-dependent scaling curves β€” the optimal strategy for a bin is selected based on approximately 50 questions per cross-validation fold, which may not produce robust strategy rankings if the within-bin variance is high.

Mitigation status: The paper does not mitigate this limitation beyond acknowledging it. The authors state the model is "representative" but provide no evidence (e.g., performance comparison to other models on standard benchmarks, or qualitative analysis of why PaLM 2-S*'s behavior should generalize). Replication on additional benchmarks and model families is explicitly left to future work.

Limitation 4: Revisions and Search Are Studied Independently β€” the Full Potential of Their Combination Is Unknown, and the Results Represent a Lower Bound

The paper studies two complementary mechanisms for test-time compute β€” PRM search (verifier-guided candidate selection) and iterative revisions (modifying the proposal distribution) β€” but never combines them. Section 8 explicitly states: "we did not experiment with PRM tree-search techniques in combination with revisions." This means the compute-optimal policy in Section 3.2 selects between search strategies (best-of-N vs. beam search) and between revision strategies (sequential vs. parallel vs. hybrid ratios), but never considers strategies that use both mechanisms simultaneously β€” for example, using the revision model as the proposal distribution within beam search, or using the PRM to guide which revision branches to pursue and which to prune.

The consequence is that the paper's reported accuracy ceilings (approximately 39.5% for compute-optimal search, 44% for compute-optimal revisions at 256 generations, Figures 4 and 8) represent lower bounds on what a combined system could achieve. The paper's own diagnostic decomposition shows why a combination might help: revisions improve the quality of generated candidates (raising the base pass@1, particularly on easy-to-medium problems), while PRM search improves the selection among candidates. If revision-improved candidates are fed into beam search, the search algorithm has a higher-quality pool to select from, potentially pushing the over-optimization threshold higher (better candidates mean the PRM's ranking errors matter less). Conversely, if the PRM's per-step scores are used during revision chains β€” for instance, detecting when a revision has gone off-track and should be restarted β€” the correct-to-incorrect reversion problem (38% rate, Section 6.1) might be mitigated.

The paper's difficulty-dependent findings further suggest complementary strengths: revisions help most on easy problems (Figure 7, right, bin 2 shows sequential advantage), while beam search helps most on medium problems (Figure 3, right, bin 3–4 shows beam search advantage). A combined system could deploy revisions for local refinement within beam search for global exploration β€” using the revision model to generate candidate steps at each beam expansion, rather than the base model. This hybrid would leverage the revision model's improved proposal distribution while using the PRM to guide which revision paths to pursue, potentially outperforming either mechanism alone on the medium-difficulty problems where both show some benefit individually.

What evidence exists in the paper: None that tests combination. The per-system traces in Section 4.3 and Appendix I show that the two mechanisms fail through different pathways (revisions generate better candidates but don't select among them optimally; search selects among candidates but the candidate pool is limited by the base model's quality), which is suggestive of complementarity but not demonstrative. The FLOPs-matched comparison (Section 7) uses either compute-optimal search or compute-optimal revisions versus a pretraining-scaled baseline, never a combined system.

Mitigation status: The paper explicitly flags this as future work (Section 8) but provides no partial experiments or analysis of what a combined system might look like. The architectural constraints of combining the two β€” the revision model would need to be compatible with step-level PRM scoring, and the PRM (trained on base model outputs) would need to be validated on revision model outputs (which Appendix J, Figure 15a shows degrades due to distribution shift) β€” are acknowledged but not addressed experimentally. The paper's contribution is thus best understood as characterizing the independent scaling behavior of search and revisions, leaving their interaction as an open question.

Limitation 5: Latency and Wall-Clock Time Are Ignored β€” the Strategies Favored by the Compute-Optimal Policy May Be Serially Bottlenecked

The paper measures test-time compute purely in "generations" β€” the number of complete solutions sampled from the base LLM β€” which is a reasonable proxy for total FLOPs but ignores latency (wall-clock time until the answer is returned). This is an important practical dimension because the compute-optimal policy's favored strategies have different parallelism profiles:

  • Sequential revisions (favored on easy problems, Figure 7 right) are inherently serial: each revision depends on the output of the previous revision, and a chain of length L takes L sequential forward passes through the model, regardless of how much parallel hardware is available.
  • Best-of-N (favored on easy problems by the search policy, Figure 3 right) is embarrassingly parallel: all N samples can be generated simultaneously on independent hardware, then scored and aggregated in a single batch.
  • Beam search is partially parallel: all beams at a given step can be expanded in parallel, but steps are sequential β€” B beams over S steps takes S sequential passes, regardless of beam count.

In a deployment with sufficient parallel hardware (multiple GPUs, large batch sizes), a strategy that allocates 128 generations as 128 parallel samples takes roughly the latency of one generation. A strategy that allocates 128 generations as a chain of 64 sequential revisions Γ— 2 parallel chains takes roughly 64Γ— the latency. The compute-optimal policy on easy problems β€” which favors sequential revisions over parallel sampling β€” may therefore be optimal in FLOPs but impractical in latency for interactive applications where users expect sub-second responses. The paper never discusses this tradeoff.

What evidence exists in the paper: None that measures wall-clock time or analyzes latency. The paper reports Figure 7's sequential-to-parallel ratio sweep purely in terms of accuracy at a fixed generation budget, with no latency dimension. The cost tables (Tables 5, 6) report dollar cost from token usage, not time. The paper does not state what hardware configuration was used or what the per-generation latency was.

Mitigation status: Not addressed. The paper's compute-optimal framework optimizes for accuracy per FLOP, not accuracy per second. This is a reasonable scope for a scaling-laws analysis paper (pretraining scaling laws also optimize for loss per FLOP, not loss per wall-clock day), but for practitioners deciding between strategies, the latency dimension is often the binding constraint. The paper's finding that sequential revisions marginally outperform parallel sampling (Figure 6, right, a gap of roughly 2.5 percentage points at 64 generations) may not justify the 64Γ— latency penalty in a production setting.

Limitation 6: The 14Γ— Larger Model Baseline Is Not Compute-Optimally Trained β€” the Pretraining vs. Test-Time Comparison May Be Favorable to Test-Time Compute

The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time strategies against a model with approximately 14Γ— more parameters, trained on the same amount of data. This scaling approach β€” holding data fixed and scaling parameters β€” follows the LLaMA paradigm (Touvron et al., 2023) rather than the compute-optimal pretraining paradigm (Hoffmann et al., 2022), where both parameters and data would be scaled equally to maximize performance per FLOP. The paper acknowledges this: "We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence is that the larger model baseline is weaker than it would be under compute-optimal pretraining. A 14Γ— larger model trained on proportionally more data (as Chinchilla scaling recommends) would achieve lower loss and presumably higher downstream accuracy than a model trained only with more parameters on fixed data. This makes the paper's reported advantages of test-time compute over pretraining β€” e.g., +27.8% relative improvement on easy questions at R β‰ͺ 1 for revisions (Figure 1, top-right bar chart) β€” potentially overstated relative to the true tradeoff a practitioner would face when deciding how to allocate a total FLOPs budget. Additionally, the 14Γ— larger model uses only greedy decoding in the FLOPs-matched comparison β€” no majority voting, no best-of-N, no search augmentation of its own. A fairer comparison would give the larger model some test-time compute budget as well, since a practitioner deploying the larger model would also have the option to use test-time compute.

What evidence exists in the paper: Figure 9 and the bar charts in Figure 1 (right panels) report the FLOPs-matched results with the parameter-only-scaled baseline. The paper does not report results against a Chinchilla-optimal larger model (scaling both data and parameters) or against a larger model with non-zero test-time compute. The limitation is acknowledged briefly in the FLOPs-matched section and in Section 6, but no sensitivity analysis is performed β€” for instance, reporting what fraction of the advantage would remain if the larger model were trained on proportionally more data, or if the larger model were allowed a modest test-time compute budget (best-of-4 or best-of-8).

Mitigation status: The paper explicitly leaves the compute-optimal pretraining comparison to future work (Section 8). The current results should be interpreted as an upper bound on the advantage of test-time compute over pretraining β€” the true advantage against a properly optimized larger model is likely smaller, and may reverse on some difficulty bins or R regimes. The paper's qualitative finding (that test-time compute helps on easy problems but not hard ones) is likely robust to the pretraining baseline choice, but the quantitative claims about the magnitude of advantage (4Γ— efficiency, +27.8% relative improvement) may not be.

7. Implications and Future Directions

How This Work Changes the Landscape

MEME changes the conversation around LLM memory evaluation from "how well does the system store and retrieve facts?" to "does the system understand that facts relate to each other, and can it propagate changes through those relationships?" This is not a paradigm shift in how memory systems are built β€” no new architecture is proposed β€” but it is a conceptual reframing of what memory evaluation must measure, backed by a diagnostic instrument that exposes a catastrophic, previously invisible failure mode across all three dominant architectural paradigms.

The magnitude of this reframing is best understood by comparing what prior benchmarks test against what MEME reveals. Pre-MEME, the evaluation ecosystem (LongMemEval, MemBench, MemoryAgentBench, LoCoMo) tested memory as a flat key-value store with overwrite semantics: store fact A, update fact B, retrieve the most recent value of fact C. A system that scored well on these benchmarks β€” accurately storing facts, correctly overwriting old values, retrieving the latest version β€” would be considered "good at memory." MEME demonstrates that this definition of "good" is structurally inadequate for any deployment where knowledge is relational rather than atomic. All six systems in Table 2 perform some version of these flat-storage operations adequately (Exact Recall 0.62 average, Tracking 0.35), yet all six score at or near zero on Cascade and Absence β€” because flat-storage operations do not test, and therefore do not incentivize building, the capability to answer "now that X has changed, what else should have changed with it?"

This reframing matters because it redirects the burden of proof from the evaluator to the system designer. Before MEME, a memory system developer could reasonably claim "we handle evolving memory" by demonstrating that their system tracks single-entity updates β€” if the user says they drive a Zyvanta Sedan, then a Therwyn Compact, then a Xylorim Scooter, the system can list all three in order. MEME shows that this claim is insufficient: tracking independent updates to Entity A and Entity B does not imply the system can handle the case where Entity B changes because Entity A changed. The paper's taxonomy (Figure 1) makes this explicit by separating the axes: single-entity evolution (Tracking) occupies the Single-Evolving cell, while multi-entity evolution (Cascade, Absence) occupies the Multi-Evolving cell, and Figure 3 shows that systems that perform acceptably in the Single-Evolving cell (accuracy 0.35) collapse to 0.02 in the Multi-Evolving cell. A developer claiming "evolving memory support" must now specify which cell they occupy, and MEME provides the test for the hardest cell.

The paper also reconciles a latent contradiction in the memory systems literature that is less overt than the contradictory findings about self-correction in the LLM reasoning literature, but structurally analogous. Prior work presented memory architectures as making different design tradeoffs β€” raw retrieval preserves verbatim facts but has no structure; LLM-processed memory extracts structured knowledge but may lose precision; file-based agents provide flexibility but rely on tool-use competence. Evaluated on prior benchmarks, these tradeoffs appeared to produce a spectrum of performance with no clear winner (different systems excel at different tasks). MEME reveals that on the dimension that matters most for real-world deployment β€” dependency reasoning β€” the tradeoffs are irrelevant because all architectures fail catastrophically, and they fail for a shared reason: passive maintenance. Raw retrieval stores dependency rules verbatim but cannot rank the change event above the pre-change value at retrieval. LLM-processed memory extracts the rule but decomposes it into disconnected atomic facts that lose the conditional structure. File-based agents write the rule explicitly but never trigger re-evaluation of dependent entries when an upstream change arrives. The architectural differences that appeared meaningful on flat-storage benchmarks vanish on dependency reasoning, revealing a deeper shared limitation.

The research directions this reframing makes more attractive are clear from the paper's per-stage failure diagnosis (Section 4.3, Appendix I). The finding that all six systems encode the dependency rule and retain the change event β€” but fail at retrieval because no system actively propagates dependency consequences at maintenance time β€” strongly suggests that the most productive research investment is in architectures that natively trigger dependent updates, not in better retrieval ranking, better extraction prompts, or stronger retrieval LLMs. The paper's intervention studies (Section 4.4) systematically eliminate these component-level improvements as insufficient: prompt optimization doesn't close the gap (Figure 5a), deeper retrieval doesn't close Cascade (Table 3a), a stronger answering LLM doesn't close Cascade (Table 3b), and reducing noise to zero doesn't close either gap (Figure 5b). The one configuration that does partially close the gap β€” MD-flat with Opus 4.7 (Table 4) β€” does so through a mechanism that is categorically different from what the other interventions attempt: Opus writes propagated values at ingestion time, making retrieval trivial and eliminating the answering-side propagation requirement. The paper thus redirects the field away from iterative improvement of existing architectures and toward a specific missing mechanism: active, trigger-based propagation at maintenance time.

Conversely, some research directions become less attractive based on these results. Improving vector retrieval ranking (e.g., better embeddings, hybrid search, reranking) is unlikely to solve dependency reasoning, because the paper shows (Table 17) that even when retrieval is perfect β€” both the rule and the change event are in the top-k context β€” 45% of Cascade failures and 86% of Absence failures persist as answering failures. Better retrieval alone cannot fix the answering-side propagation gap. Similarly, improving the answering LLM's reasoning capability through better prompting or stronger models is insufficient: Table 3b shows that swapping gpt-4.1-mini for Sonnet 4 as the answering LLM produces zero Cascade improvement across all six systems, because on the systems where retrieval fails (Type 1 failures), the stronger answering LLM has no evidence to work with; on the systems where retrieval succeeds (Type 2 failures), the dependency structures in the retrieved context are not in a form the answering LLM can use (disconnected atomic facts in Mem0, paraphrased rules in Graphiti). The paper's Opus 4.7 result on MD-flat is instructive here: Opus succeeds not because it is a better answering LLM, but because it is used as the internal LLM during ingestion, where it can restructure the memory representation to make the answering step trivial. Improving the answering LLM is targeting the wrong stage.

Follow-Up Research This Work Enables

Lightweight dependency propagation at maintenance time β€” without a frontier internal LLM. The paper's central finding is that all practical-cost configurations fail dependency reasoning because they practice passive maintenance, and the one configuration that partially closes the gap (MD-flat Γ— Opus 4.7, Table 4) does so through active propagation at ingestion β€” but at 70Γ— the baseline cost and with degradation on Exact Recall (0.90 β†’ 0.60) and Tracking (0.80 β†’ 0.20) on the same 20-episode subset. The natural follow-up question is: can a dedicated, lightweight maintenance mechanism achieve comparable propagation without requiring a frontier LLM? A strong experiment would develop a memory architecture that, upon ingesting a session containing an entity-value change, explicitly queries the store for dependency rules mentioning the changed entity as a trigger, applies those rules to compute or invalidate dependent values, and writes the propagated facts as regular entries. The mechanism could be as simple as a keyword-triggered template-filling step (no LLM required for the propagation itself) that treats dependency rules as parameterized templates: when entity X changes to value v, scan for templates of the form "if X changes to v, then Y becomes w," instantiate w, and write the resolved fact. Such a system would be evaluated on MEME against the MD-flat Γ— Opus 4.7 baseline: if it can match Opus's Cascade (0.32) and Absence (0.59) on the 20-episode subset at a cost comparable to the gpt-4.1-mini baseline ($0.065/episode), it would demonstrate that the capability threshold is architectural rather than LLM-dependent, directly addressing the paper's conclusion that "closure currently depends on configurations that are not practical at scale."

Characterizing the implicit-to-explicit dependency continuum. The paper acknowledges (Section 6) that its dependency rules use "explicit conditional phrasing as a best-case framing for memory systems," and that "we have not ablated implicit-conditional or no-conditional variants." This leaves open a critical question for real deployment: real users rarely state dependencies in the explicit "if X changes, Y will be Z" template the paper uses. They might say "my commute is 30 minutes because I live in Pyresta Meadow" (implicit dependency, no explicit conditional), or "I'm switching to a new team; David Lee was my code reviewer before" (implicit that a change triggers re-evaluation, but no rule stated). A follow-up study would create variants of the MEME dataset along a dependency explicitness spectrum: Level 0 (explicit conditional β€” the current dataset), Level 1 (explicit dependency but no conditional rule β€” "my commute depends on where I live" without stating what the new commute would be), Level 2 (implicit dependency β€” "my commute is 30 minutes from Pyresta Meadow," relying on world knowledge that moving changes commute time), and Level 3 (no dependency stated β€” the graph structure exists in the gold data but is never verbalized). Evaluating MD-flat Γ— Opus 4.7 and the best practical-cost system (MD-flat Γ— gpt-4.1-mini) across these levels on a 20-episode subset would characterize how much the paper's findings depend on the explicit-conditional best-case framing, and at what level of implicitness even Opus-level internal LLMs fail to recognize and propagate dependencies. If Opus fails at Level 2 or 3, the implication is that dependency reasoning in practice requires more than just a capable internal LLM β€” it requires architectural support for inferring dependencies from implicit statements, a much harder problem.

Scaling MEME to larger knowledge graphs and longer chains. The paper's knowledge graphs (Table 7a) contain 39–51 entities with 27–34 edges, and dependency chains reach a maximum depth of 2 (root β†’ middle β†’ leaf). Real-world knowledge β€” a personal assistant tracking a user's life over years, or a project management agent tracking a software team's evolving configuration β€” would involve substantially larger graphs and deeper chains. Do the failure modes identified in the paper scale linearly with chain depth, or do they compound? A follow-up study would extend the MEME generation pipeline to produce episodes with intentionally longer dependency chains (depth 3–5, e.g., framework β†’ build_tool β†’ build_command β†’ docker_image β†’ deploy_command) and evaluate whether retrieval-vs-answering failure fractions (Table 17) shift with chain depth. The hypothesis from the paper's per-stage diagnosis: deeper chains should increase the answering-failure fraction because the answering LLM must reconstruct multi-hop propagation from retrieved facts, and each additional hop is an additional chance for the LLM to default to the pre-change value. If answering failures dominate at depth 3+ even when retrieval is perfect (k=40), this would reinforce the paper's architectural recommendation: retrieval improvements alone cannot solve dependency reasoning for realistic knowledge graphs, and active propagation at maintenance time becomes mandatory, not optional.

Cross-model replication with a focus on internal-LLM capability thresholds. The paper's internal-LLM ablation (Table 4) tests four LLMs on three systems but covers only a 20-episode subset, and the jump from gpt-4.1-mini (~0.40/0.40/1.60 per 1M tokens) to Opus 4.7 (15/15/75) leaves a large capability and cost gap with no intermediate measurement. A targeted replication would evaluate MD-flat with a denser sampling of internal LLMs across the cost-capability spectrum β€” Claude Sonnet 4 (3/3/15), gpt-4o (2.50/2.50/10), Gemini 2.5 Flash (unspecified but reportedly cheaper than gpt-5), and open-weight models (Llama-3-70B, Mixtral 8Γ—22B) β€” on the full 100-episode MEME dataset. The goal is to precisely characterize the cost-capability curve: at what point does the internal LLM cross the threshold where it begins writing explicit contingency entries and propagating dependent values? If Claude Sonnet 4 achieves, say, Cascade 0.20 at ~10Γ— baseline cost (vs. Opus's 0.32 at ~70Γ—), this would suggest a diminishing-returns curve where moderate-cost models provide partial propagation. If no model below Opus crosses a meaningful threshold (>0.10 Cascade), this would suggest a qualitative capability gap β€” the propagation behavior requires a specific kind of structured reasoning that only frontier models possess β€” and would make the case for architectural mechanisms even stronger, since relying on LLM capability progression is unreliable.

Combining MD-flat's file-based substrate with Mem0's semantic search for retrieval. The paper's traces (Figures 4 and 30) reveal a tension: MD-flat preserves dependency rule structure (the contingency wording stays intact in the markdown file) but relies on tool-use retrieval that can miss change events (the gpt-4.1-mini internal-LLM retrieval step doesn't open the 03/17 entry). Mem0's vector search reliably surfaces both the rule and the change event (both are in the top-20), but Mem0's fact decomposition breaks the conditional link between them β€” the rule becomes a disconnected atomic fact with no trigger connection. A hybrid follow-up would implement a system that uses MD-flat's markdown-based ingestion (preserving structured contingency entries) but indexes the file's contents in a vector store for retrieval (surfacing all relevant entries, including change events, via semantic search rather than tool-use navigation). This separates the ingestion-time propagation concern (what MD-flat does well when paired with a capable LLM) from the retrieval-time surfacing concern (what Mem0 does well for multi-fact coverage). The experiment: run this hybrid on the full 100-episode MEME dataset with gpt-4.1-mini as the internal LLM (for ingestion) and compare Cascade/Absence against both MD-flat alone (0.06/0.05) and Mem0 alone (0.03/0.00). If the hybrid achieves Cascade >0.10 and Absence >0.10 at the gpt-4.1-mini cost point, it would demonstrate that the retrieval and propagation failure modes identified in the paper can be addressed independently and composed β€” retrieval fixed by vector search, propagation fixed by architectural support in the ingestion substrate β€” without requiring a frontier LLM.

Practical Applications and Downstream Use Cases

Pre-deployment auditing of memory systems for enterprise agents. An organization building an LLM-based agent for customer support, project management, or personal assistance can run MEME against their chosen memory architecture before deployment to quantify the dependency-reasoning gap they will encounter in production. The paper's finding that all practical-cost configurations score at or near zero on Cascade and Absence (Table 2) means that any agent relying on these systems will silently report stale dependent values when upstream facts change β€” a failure mode that is invisible in standard QA evaluations but likely common in real user interactions. Running MEME provides a concrete risk quantification: if the agent's expected user interactions include dependency-laden updates (e.g., changing a project's framework triggers changes to build tools, deployment configs, and team assignments β€” exactly the Software Project domain), the audit will reveal that the memory system has a ~97% failure rate on Cascade and ~99% failure rate on Absence, and the deployment team can either implement workarounds (explicit re-confirmation prompts after any upstream change) or select a different architecture. The paper's per-episode cost reporting (Tables 5–6) allows organizations to price this audit: running 100 MEME episodes against MD-flat with gpt-4.1-mini costs approximately $5.60 total in LLM API fees, making it a cheap pre-deployment check.

Designing dependency-aware conversation flows for LLM applications. Application developers who cannot change their memory architecture (because they're using a vendor-provided memory API) can use MEME's task taxonomy to design conversation flows that work around the dependency-reasoning gap. The paper shows that all systems successfully encode dependency rules and change events (Section 4.3, encoding passes for all six systems) but fail to propagate them at query time. A developer can therefore work around this by making the propagation explicit in the conversation flow: when the user reports an upstream change (e.g., "I moved to Keldara Grove"), the application can programmatically trigger follow-up questions for every known dependency ("Your commute time depended on your previous residence β€” should I update that? Your gym location depended on your residence β€” should I mark that as uncertain?"). This shifts the propagation work from the memory system to the application logic, using MEME's graph structure as a template for what to re-confirm. The paper's knowledge graph formalism (Section 3.2, the DAG G = (V, E, P, Ξ¦)) essentially provides a schema for what these dependency-aware conversation flows should cover. This is a stopgap β€” it doesn't solve the architectural gap β€” but it prevents the silent-staleness failure mode that MEME identifies, and it can be implemented today without waiting for next-generation memory architectures.

Selecting and budgeting memory architectures based on interaction patterns. The paper's cost-accuracy tradeoff characterization (Tables 2 and 6) enables organizations to make principled architecture decisions based on their expected interaction patterns. If an organization's user interactions are predominantly static-retrieval (asking about facts that don't change) or single-entity tracking (updating independent facts), the existing systems perform adequately: MD-flat at 0.056/episodeachieves0.42overall,textβˆ’embeddingβˆ’3βˆ’smallat0.056/episode achieves 0.42 overall, text-embedding-3-small at 0.031/episode achieves 0.33 overall, and BM25 at 0.038/episodeachieves0.25overall(Table2).Thesesystemsaredeployabletodayforretrievalβˆ’heavyworkloadsatcoststhatundercutinβˆ’contextbaselinesasqueryvolumegrows(inβˆ’contextat0.038/episode achieves 0.25 overall (Table 2). These systems are deployable today for retrieval-heavy workloads at costs that undercut in-context baselines as query volume grows (in-context at 0.16/query vs. memory systems at 0.00–0.00–0.01/query for inference). If the organization's interactions include significant dependency-laden updates, the paper's findings strongly suggest that no deployable architecture exists today β€” even MD-flat Γ— Opus 4.7 at ~70Γ— baseline cost achieves only 0.32 Cascade β€” and the organization should either (a) implement the dependency-aware conversation flows described above as a workaround, or (b) accept that dependent facts will need manual re-confirmation after every upstream change, budgeting for the UX cost of those re-confirmation prompts. The paper effectively provides a decision boundary: at current capability levels, dependency reasoning is not a feature that can be purchased through better model selection or prompt engineering; it requires architectural innovation that doesn't yet exist in deployable form.