MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models

🎯 Pitch

Long-context vision-language models and memory-augmented agents both fail MEMLENS—for opposite reasons. Direct visual grounding degrades as conversations lengthen, while memory compression destroys the fine-grained visual details needed to answer, proving neither approach alone handles multimodal conversational memory.

1. Executive Summary

This paper introduces MEMLENS, a benchmark for multimodal long-term memory in large vision-language models (LVLMs) and memory-augmented agents, comprising 789 questions across five memory abilities—information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal—at four standardized context lengths (32K–256K tokens). Evaluating 27 LVLMs and 7 memory-augmented agents, the work identifies complementary failure modes: long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade as conversations grow (e.g., information extraction and knowledge update drop ~20% and ~12% from 32K to 128K), whereas memory agents remain length-stable but lose visual fidelity under storage-time compression through caption-based or embedding-based encoding that discards fine-grained visual cues (e.g., counts, attributes, spatial relations). Multi-session reasoning caps most systems below 30% accuracy, and multi-session reasoning accuracy reaches only 44.06% for the strongest model at 32K, establishing that cross-session aggregation over three to eight evidence sessions remains the shared capability ceiling. An image-ablation study confirms that solving MEMLENS genuinely requires visual evidence: removing evidence images collapses two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images, demonstrating that neither long-context attention nor retrieval-based memory alone suffices for multimodal conversational memory when problems demand cross-modal evidence retrieval.

2. Context and Motivation

The Core Problem: No Benchmark Exists for Multimodal Conversational Memory

The fundamental gap this paper identifies is deceptively simple: no existing benchmark provides a length-controlled comparison of long-context LVLMs and memory-augmented agents on questions that genuinely require visual evidence from multi-session conversations. This absence matters because the field has developed two competing architectural approaches to memory—long-context models that attend directly over full conversation histories, and memory-augmented agents that compress, index, and retrieve past content—without any standardized way to compare them on the task they both claim to solve.

The nature of this gap becomes clear when examining what existing benchmarks do measure (Table 1). Current long-context multimodal benchmarks such as MMLongBench [11] and MMLongBench-Doc [13] test how well vision-language models retrieve information from long documents or needle-in-a-haystack scenarios with interleaved images. These benchmarks establish valuable context-length scaling protocols, but their inputs are documents or static multimodal sequences—not the incrementally accumulating, timestamped multi-session conversations that characterize real-world agent deployments. An LVLM that excels at finding a specific paragraph in a 128K-token document may fail completely when the same information is distributed across sessions separated by weeks of simulated conversation.

Conversational memory benchmarks restore the multi-session structure but almost uniformly discard the visual modality. LongMemEval [14], the most comprehensive text-only conversational memory benchmark with 500 questions and five memory abilities across up to 1.5 million tokens, deliberately omits images. MemoryAgentBench [15] tests retrieval, test-time learning, long-range understanding, and selective forgetting—but again, only over text. This creates a dangerous research blind spot: the community can measure how well agents remember textual facts across sessions but has no instrument for assessing whether they preserve the visual evidence that real multimodal conversations contain.

When multimodal conversational benchmarks do include images, they fail to enforce cross-modal dependency. LoCoMo [16] and Mem-Gallery [17] embed images in persona-grounded dialogue, but the paper notes that "most of their questions admit a text-only shortcut, rendering the visual modality redundant" (Section 2). A model can answer "What did the user eat for breakfast?" from the textual description "I had toast and eggs" without ever examining the accompanying photo—making the benchmark a text-only memory test in disguise. The paper's cross-modality ablation (§3.4) quantifies the severity of this issue: on MEMLENS, where cross-modal dependency is structurally enforced, removing evidence images collapses frontier models from ~90% to below 2% accuracy. This validates that prior benchmarks testing "multimodal memory" were not actually testing memory for visual evidence.

Why This Problem Matters

The practical stakes are substantial because memory is not a luxury feature—it is a capability boundary for deploying LVLMs in sustained real-world interactions. When an LVLM serves as a personal assistant over weeks or months, it encounters several memory demands simultaneously:

Recall: the assistant must retrieve facts shared in previous sessions—a user's food preferences mentioned three weeks ago, a project deadline discussed with a shared screenshot, a medication name written on a pharmacy label photo.
Integration: it must aggregate information across sessions—summing expenses from receipts photographed at different times, counting distinct items mentioned across disparate conversations, or determining whether the "new plant" in today's session matches the one described in a session from last month.
Temporal reasoning: it must sequence events correctly—knowing that the user started a job after moving to a new city, not before, where the evidence for each fact arrives in different sessions with implicit timestamp signals (session metadata) or explicit visual cues (clock faces, calendar photos).
Knowledge update: it must track evolving user attributes—preferences that change across sessions, corrections to earlier statements, plans that get revised—and recognize that the current state is the answer, not an outdated one.
Calibrated abstention: it must decline to answer when evidence is absent—refusing rather than hallucinating when asked about a detail from a conversation that never actually happened.

These are not speculative requirements; they describe the everyday operation of any persistent conversational assistant. Yet without a benchmark that forces joint visual-textual reasoning across sessions at controlled context lengths, researchers lack the diagnostic tools to determine which memory architecture actually works for which memory ability. A system might achieve high text-only retrieval scores while silently corrupting visual evidence through lossy compression. Another system might maintain perfect visual fidelity in short conversations but collapse when context grows. Without MEMLENS, these failure modes are invisible to standardized evaluation.

The theoretical significance extends beyond benchmarking. The paper reveals that long-context attention and retrieval-based memory exhibit orthogonal, complementary failure modes (§4.3): LVLMs lose to context length, while memory agents lose to lossy multimodal compression at storage time. Because each architecture covers only one axis, scaling along that axis leaves the other failure mode untreated. This finding has direct architectural implications—it suggests that the next generation of memory systems should combine long-context attention with structured multimodal retrieval rather than scaling either component in isolation. The benchmark thus serves not only as an evaluation tool but as a discovery instrument that reveals how current approaches fail, pointing toward the hybrid architectures that the results motivate.

Where Prior Approaches Fall Short

The paper identifies cascading limitations across the two prevailing approaches, with each limitation compounding the next to create the gap that MEMLENS fills.

Long-context multimodality is benchmarked on documents, not conversations. Benchmarks like MMLongBench [11] and MM-NIAH [12] have established core protocols for measuring how context length affects multimodal retrieval. They show that some LVLMs maintain performance up to 128K tokens on document-style inputs with interleaved images, while others degrade sharply. However, these findings do not transfer to conversational settings. In a document, the temporal structure is linear and static—page 7 follows page 6 in a predictable way. In a multi-session conversation, sessions are timestamped non-uniformly, evidence is distributed across arbitrary temporal positions, and the model cannot rely on spatial layout cues (headers, captions, section breaks) that aid document retrieval. A model trained and evaluated on long documents may develop retrieval strategies that exploit document-specific regularities without ever developing the more general memory operations that conversations require.

Conversational memory benchmarks exist but jettison vision. Text-only benchmarks like LongMemEval [14] provide comprehensive coverage of memory abilities—their five-task taxonomy (information extraction, multi-session reasoning, temporal reasoning, knowledge update, answer refusal) directly inspired MEMLENS's structure. However, by discarding images entirely, these benchmarks treat memory as a single-modality problem where the challenge is purely textual retrieval and reasoning. In reality, users share photos, screenshots, receipts, and documents within conversations, and the information in those images is often not recoverable from the surrounding text. A real-world assistant that can only remember text but forgets every image its user has ever shared would be severely degraded—yet the research community had no way to measure this degradation because no benchmark tested it.

Multimodal conversational benchmarks exist but don't require visual evidence. LoCoMo [16] and Mem-Gallery [17] represent important steps toward multimodal conversational evaluation, but they fail on a crucial design criterion: visual necessity. When a question can be answered from text alone, the visual modality becomes decorative rather than essential, and the benchmark measures text-only memory despite its multimodal surface appearance. The paper's image-ablation study (§3.4) provides causal evidence for this concern: when evidence images are removed from MEMLENS questions, frontier models collapse below 2% accuracy. On LoCoMo, a comparable ablation would likely show much smaller drops because the visual content is redundant with the text. This means prior "multimodal conversational memory" benchmarks were not actually measuring whether models remember visual evidence—they were measuring text-only memory on conversations that happened to contain images.

No benchmark compares both approaches on the same questions. Perhaps the most operationally significant gap is the absence of head-to-head comparison. Long-context benchmarks evaluate LVLMs but not memory-augmented agents. Memory-agent benchmarks evaluate agents but not long-context LVLMs. Each community has developed its own evaluation ecosystem with incompatible inputs, metrics, and difficulty levels. Researchers building memory-augmented agents cannot answer basic questions like "Does my pipeline outperform a long-context model of equivalent size?" or "At what context length does long-context attention become inferior to retrieval?" This fragmentation obscures the practical tradeoff that deployers face: given a multimodal conversation of a certain length, should they use a model with a large context window or a memory-augmented architecture?

Context length is not standardized across modalities. A subtle but important design flaw in several prior long-context benchmarks is inconsistent token accounting. Images consume different numbers of tokens depending on the model's visual encoder resolution, dynamic tiling strategy, and token compression scheme. When benchmarks report a "128K context" without specifying how image tokens are counted, the actual information content varies across models in ways that confound length comparisons. MEMLENS adopts the cross-modal token-counting scheme from MMLongBench [11] (§3.2), which aligns text and vision tokens, ensuring that a 128K context means the same total information budget regardless of how individual models allocate that budget between modalities.

How This Paper Positions Itself

The paper frames all memory approaches through a unifying comparative lens: length-controlled evaluation of long-context LVLMs alongside memory-augmented agents on questions that require cross-modal evidence retrieval. This framing is explicit in Table 1, which systematically compares MEMLENS against nine existing benchmarks along six dimensions (type, maximum context length, multimodality, multi-session structure, length control, comparative evaluation, and memory task coverage). MEMLENS is the only entry that satisfies all six criteria.

The positioning is deliberately diagnostic rather than prescriptive. The paper does not argue that LVLMs are superior to agents or vice versa. Instead, it constructs a benchmark that forces both approaches to grapple with the same multimodal evidence under the same length constraints, then lets the results reveal where each approach excels and fails. The finding that the two exhibit complementary failure modes (§4.3)—LVLMs degrading with length, agents losing visual fidelity—emerges from the evaluation rather than being assumed a priori. This diagnostic stance distinguishes MEMLENS from benchmarks designed to advocate for a particular architectural direction.

The paper also positions itself relative to the needle-in-a-haystack paradigm. Prior benchmarks like MM-NIAH [12] and Multimodal NIAH [20] use the needle-in-a-haystack format to test whether models can locate a single piece of evidence in long contexts. MEMLENS extends this paradigm in two ways. First, questions require multiple needles across multiple modalities: an MSR question might require the model to retrieve three evidence sessions, each containing both a textual fact and an evidence image, and aggregate across them. Second, the evidence is contextually embedded rather than anomalously inserted: evidence sessions are generated using the same pipeline and templates as haystack sessions (§3.2), and a post-hoc classifier achieves only marginally above-chance accuracy when distinguishing evidence from haystack text (§G.8). This makes retrieval genuinely difficult—agents cannot locate evidence by surface-level anomaly detection—and tests whether models can identify relevant content through semantic understanding rather than statistical shortcuts.

The cross-modality validation (§3.4) serves as a causal anchor for the benchmark's claims. By demonstrating that removing evidence images collapses model performance from ~90% to below 2%, the paper establishes that MEMLENS questions genuinely require visual evidence. This is not a correlational claim ("models that use images perform better") but a causal one ("images cause performance; removing them destroys it"). The causal design preempts the criticism that the visual modality is decorative rather than essential—a criticism that applies to prior multimodal conversational benchmarks but not to MEMLENS.

Finally, the paper positions its five-ability taxonomy as a functional decomposition of conversational memory rather than an arbitrary categorization. Information extraction tests single-session retrieval; multi-session reasoning tests cross-session aggregation; temporal reasoning tests joint understanding of timestamps, session boundaries, and visual temporal cues; knowledge update tests current-state tracking through evolving attribute chains; and answer refusal tests epistemic calibration. Together, these five abilities span the retrieval–reasoning–update space that any persistent conversational assistant must navigate. The correlation analysis (§4.3) validates this design empirically: the five abilities show low cross-type correlation, confirming that they measure distinct capabilities rather than redundant variants of a single memory skill.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

MEMLENS is a benchmark construction and evaluation system — not a model or an agent, but a carefully engineered dataset of 789 multimodal questions together with a protocol for comparing long-context LVLMs and memory-augmented agents on equal footing. The core problem it solves is enabling apples-to-apples measurement of multimodal long-term memory: given a long multi-session conversation history with interleaved images and a question that requires visual evidence to answer, can a system locate the relevant past content and reason over it correctly? The "shape" of the solution is a four-stage construction pipeline that generates conversation histories with structurally embedded multimodal evidence, instantiated at four standardized context lengths (32K, 64K, 128K, 256K tokens) under a cross-modal token-counting scheme, yielding 789 evaluation instances spanning five distinct memory abilities.

3.2 Big-picture architecture (diagram in words)

The MEMLENS system has two major halves — construction (building the benchmark) and evaluation (scoring models against it) — connected by a quality-control pipeline.

Construction half (four stages):

Multimodal Session Simulator: Takes a hierarchical topic ontology (§C.2) as input, retrieves web images per topic, and runs a two-model dialogue simulation (GPT-5.1 as user, Gemini-3-Pro as assistant) to produce multi-turn conversations with interleaved images. Output: a pool of multimodal sessions with structurally aligned image–text pairs, each ~10 turns long with ~1.5 images per session on average.
Question-Answer Generator: Samples topics from the ontology, generates background paragraphs with named entities, abstracts one entity via image-dependent anaphora (e.g., "Golden Gate Bridge" → "the bridge shown in <image>"), and produces (question, answer, evidence facts) triples using Gemini-3-Pro. Output: candidate evaluation questions whose answers require cross-modal reasoning.
Evidence Session Wrapper: Takes each needle fact from Stage 2 and wraps it into a complete multi-turn session, generated through the same pipeline as haystack sessions so that evidence is contextually embedded rather than trivially locatable. Output: evidence sessions indistinguishable in style and structure from background filler.
Conversation History Assembler: Inserts evidence sessions into timestamp-ordered sequences of haystack sessions at random positions (with type-specific ordering constraints), pads with text-only filler sessions from ShareGPT and UltraChat to maintain a fixed text-per-image ratio, and compiles each question into four standardized input lengths. Output: 789 evaluation instances, each instantiated at 32K, 64K, 128K, and 256K tokens.

Evaluation half (two components):

Model Runner: Feeds the assembled conversation history to either a long-context LVLM (direct context-window processing, with images inlined as <image> tokens) or a memory-augmented agent (with architecture-specific input adapters: text-only agents receive BLIP-2 captions in place of images; M3-Agent receives per-session composite images; M2A and M3C receive original interleaved inputs). Both receive the same question text and the same conversation history, modulo the adapter transformation.
Judge Module: Scores model outputs against reference answers using LLM-as-Judge (Qwen3-VL-235B-A22B-Instruct as the canonical judge) with task-specific grading criteria, validated against both GPT-5.4-mini cross-judging (κ = 0.93) and three-annotator human consensus (κ = 0.86).

Quality gate: Between Stages 2 and 3, a combined automated-filter-plus-three-round-human-review pipeline reduces ~20K candidate questions to the final 789, enforcing cross-modal necessity, naturalness, and difficulty calibration.

3.3 Roadmap for the deep dive

First, the formal problem specification (§C.1) — what exactly is an evaluation instance in MEMLENS, and what must a correct system do? This establishes the mathematical structure before any implementation details.
Second, the topic ontology (§C.2) — what topics drive the generation, how are they organized, and why this matters for benchmark diversity. This is the generative backbone shared across all question types.
Third, the question construction pipeline (§C.4) — how background paragraphs, evidence images, entity abstraction, and question generation produce cross-modal evaluation items. This is the central engine of the benchmark, and understanding its four-step flow is prerequisite to understanding any specific memory ability.
Fourth, the evidence session and conversation history assembly (§C.8) — how evidence is embedded without surface-level shortcuts, how length control is achieved, and what constraints preserve temporal consistency. This is where the benchmark becomes genuinely difficult for current systems.
Fifth, the evaluation protocol (§B and §E.2) — how models are run, how agents are adapted to multimodal inputs, how judging works, and how judge reliability is validated. This closes the loop from construction to measurement.
Sixth, the memory abilities as instantiated in the pipeline — a walkthrough of each of the five question types (IE, MSR, TR, KU, AR), showing how the general pipeline specializes for each, what visual skill each isolates, and what modality constraints are enforced.

3.4 Detailed, sentence-based technical breakdown

This paper is primarily a benchmark construction and empirical analysis paper whose core idea is that evaluating multimodal long-term memory requires a length-controlled testbed where questions demand cross-modal evidence retrieval, and where both long-context LVLMs and memory-augmented agents can be compared under identical conditions. The technical contribution lies in (a) a four-stage data construction pipeline that enforces cross-modal dependency through entity abstraction and contextual embedding, (b) a cross-modal token-counting scheme that enables fair length comparisons, and (c) a standardized evaluation protocol that accommodates both architectural classes through well-defined input adapters.

Formal Problem Specification

An evaluation instance in MEMLENS is formally a 4-tuple $(S, q, I, a)$ , where:

$S = [(t_1, M_1), (t_2, M_2), \ldots, (t_N, M_N)]$ is a sequence of $N$ timestamped multi-turn sessions, with $t_1 < t_2 < \ldots < t_N$ . Each session $M_i$ is itself a sequence of user and assistant turns, interleaving text segments and images at arbitrary positions within the turn sequence.
$V(S)$ denotes the set of all images appearing anywhere in $S$ .
$I \subseteq V(S)$ is the subset of images that carry answer-critical visual information — information not recoverable from the textual content of $S$ alone, and without which the correct answer cannot be determined. For image-essential questions (§3.4), $I$ is non-empty and the answer is unrecoverable without every element of $I$ . For image-supportive questions, $I$ confirms or disambiguates a textual fact. For text-sufficient questions (all AR items and a subset of MSR items), $I$ may be empty.
$q$ is a natural-language query targeting one of five memory abilities, phrased so that it does not explicitly name the entity that has been abstracted (e.g., "What time did I meet Emma?" rather than "What time did I meet Emma at the Golden Gate Bridge?").
$a$ is the gold answer — either a content answer (a short phrase, a number, a date, a binary choice, a ranked ordering) or the literal string NOT_MENTIONED for answer-refusal items.

What a correct system must do: Given $S$ (the full multi-session conversation history with interleaved images) and $q$ (the question), the system must (i) localize the relevant evidence sessions $S_{\text{ev}} \subseteq S$ that contain the facts and images $I$ needed to answer $q$ , and (ii) ground its answer in cross-modal reasoning over the joint visual–textual content of $S_{\text{ev}}$ . This two-step requirement — localization followed by cross-modal reasoning — is what makes MEMLENS a memory benchmark rather than a pure perception or reasoning benchmark. The haystack sessions $S_{\text{hay}} = S \setminus S_{\text{ev}}$ serve as distractors: they are topically related to the evidence but contain no answer-relevant information, so the system must not be misled by their semantic similarity.

Why this formulation: The key property enforced by this structure is that the question cannot be reduced to a text-only retrieval task. Because entity abstraction removes the named entity from the textual context and replaces it with an image-dependent anaphor (e.g., "the bridge shown in <image>"), the first hop of reasoning — which real-world entity is being discussed — requires visual recognition that no text-only system can perform. This is what the cross-modality ablation (§3.4) validates empirically: when $I$ is removed, leaving only the textual content of $S$ , frontier models collapse below 2% accuracy. The localization step (finding $S_{\text{ev}}$ within $S$ ) tests memory capacity, while the reasoning step tests cross-modal integration — and neither alone suffices.

Topic Ontology: The Generative Backbone

The topic ontology is the structured knowledge source that drives all content generation in MEMLENS. It is organized into three tracks, each covering a distinct perceptual regime that the benchmark must exercise:

Track 1 — Identification (~40% of topics): Recognizing real-world entities. This track decomposes into five sub-attributes: retail and commerce (supermarket shelves, pharmacy aisles, street food carts), home objects and belongings (kitchen appliances, bookshelves, jewelry layout, pet accessories), vehicles and mobility (personal cars, scooters, bicycle helmets, license plates), urban environment and landmarks (street signs, shopfront names, building numbers, public swimming pools), and workspace and institutions (office kitchens, gym equipment, library shelves, pharmacy windows). Each sub-attribute contains roughly 20 topic titles, and each topic title is expanded into ~30 fine-grained subtopics — for example, "supermarket shelves" might expand to "international food aisle," "dairy section," "bulk bins," etc.

Track 2 — Experience (~40% of topics): Everyday activities and lifestyle moments. Seven sub-attributes span dining and food activities (casual restaurants, cafes, street food walks, baking experiments), fitness and outdoor recreation (indoor climbing, badminton, hiking, beach outings), social and community gatherings (house parties, karaoke rooms, board game cafés, volunteer shifts), cultural and entertainment outings (art museum visits, concerts, theme park rides, cosplay meetups), routines and errands (morning bathroom routines, grocery top-up trips, salon visits, DIY home repair), travel and mobility outings (weekend road trips, hotel check-ins, coastal promenades), and home life and indoor hobbies (morning balcony coffee, evening reading corner, hobby painting, pottery class).

Track 3 — Document (~20% of topics): Text-rich artifacts where dense text and spatial layout carry information. Seven sub-attributes cover receipts and bills (grocery receipts, monthly utility bills, parking garage receipts), tickets, passes, and itineraries (event tickets, transit passes, hotel booking confirmations), schedules and calendars (weekly schedules, lunch menus, fitness class timetables), contracts and application forms (rental leases, job offer letters, gym membership contracts), personal records and logs (budget sheets, fitness tracker logs, to-do lists, recipe cards), subscriptions and digital records (streaming billing emails, online shopping screenshots, social media content calendars), and health and service records (prescriptions, clinic appointment cards, warranty cards, repair invoices).

What this ontology enables: Each question type draws from a dedicated subset of these topics (§C.2), with ~100 topics per answerable ability type, yielding roughly $4 \times 100 \times 30 = 12,000$ fine-grained subtopics as the generative seed space. This breadth ensures that questions span diverse visual regimes — recognizing a landmark, reading a receipt, parsing a clock face, identifying a plant species — rather than over-fitting to a narrow object-recognition domain. The document track is particularly important because it tests OCR and layout understanding, skills that current LVLMs handle differently from natural-image recognition, and that memory agents with caption-based storage might lose entirely (since BLIP-2 captions do not transcribe text from document images).

Why three tracks: The separation into identification, experience, and document tracks is not arbitrary — it corresponds to distinct failure modes. Identification images test whether the system can recognize what an entity is; experience images test whether it can interpret what is happening in a scene; document images test whether it can read structured information. A model that excels at landmark recognition but fails at reading a receipt, or vice versa, would show uneven performance across question types, and the three-track ontology makes such asymmetries detectable.

Question Construction Pipeline (The Central Engine)

The question construction pipeline is a four-step process that produces evaluation questions with structurally enforced cross-modal dependency. This pipeline is shared across all four answerable question types (IE, MSR, TR, KU), with type-specific adaptations in the final question-generation step.

Step 1 — Background generation. For each question, a topic is sampled from the hierarchical ontology, and Gemini-3-Pro generates a background paragraph of three to five sentences grounded in that topic. The generator is prompted to include at least two named entities drawn from recognizable real-world referents — landmarks, commercial products, biological species, cultural institutions — whose visual appearance is distinctive and web-searchable. For example, a paragraph about San Francisco landmarks might mention the Golden Gate Bridge, Alcatraz Island, and Fisherman's Wharf. The paragraph provides the factual context that the eventual question will target, and it introduces the named entity that will be abstracted in Step 3.

Step 2 — Entity selection and image retrieval. From the background paragraph, one named entity is selected as the visual anchor for the question. The selection criteria are: (a) the entity must be visually identifiable in a photograph; (b) it must be sufficiently specific that a web image search returns relevant results (a generic concept like "tree" would retrieve too many diverse candidates, making the evidence image ambiguous); and (c) it must be critical to at least one fact stated in the paragraph — the fact whose retrieval the question will test.

A text query derived from the entity name is issued to a web-crawling pipeline using iCrawler, which retrieves a batch of candidate photographs from public web image search (not commercial stock aggregators). Each candidate is scored by a three-channel relevance filter:

CLIP (ViT-L/14): scores image–text alignment between the query and the image, with a threshold of 0.30 — a candidate must exceed this threshold to pass.
SigLIP (ViT-SO400M): independently scores image–text alignment with a different pretraining objective.
Text–text cosine: a BLIP-2 caption of the candidate image is encoded into a text embedding, and its cosine similarity to the query embedding is computed as a third relevance signal.

A candidate must exceed the CLIP threshold and at least one of the two secondary channels to pass. The highest-scoring candidate that also passes a negative-content filter (rejecting watermarks, stock-photo logos, copyright overlays, and resolution artifacts) and a global URL deduplication check (ensuring no two questions in MEMLENS share the same source image) is selected as the evidence image. For DocVQA-style images (receipts, menus, forms), an additional multimodal judge (GPT-4V) inspects the image for watermark presence.

Why multi-channel scoring: CLIP and SigLIP have complementary failure modes. CLIP is strong on object-level alignment but can be fooled by text-like patterns in images. SigLIP, trained with sigmoid loss rather than softmax, is more discriminative on fine-grained distinctions. The text–text cosine channel provides a third signal that is robust to visual noise — even if the image is blurry or poorly lit, a caption might capture the semantic content. Requiring at least two of three channels to agree reduces the false-positive rate compared to relying on CLIP alone.

Step 3 — Entity abstraction. The surface form of the selected entity is masked in the background paragraph and replaced with a natural anaphor that references the evidence image. This proceeds in two stages.

First, the entity is classified into one of 55 semantic categories spanning places (museums, parks, restaurants, bridges, temples), organizations (companies, foundations, institutes), objects (books, paintings, vehicles, instruments), and generic fallbacks. Second, a replacement phrase is sampled from a type-aware dictionary of ~170 entries, with each category providing three to four paraphrase variants. For instance, "Golden Gate Bridge" (category: bridge) might be replaced with "the bridge shown in <image>," "that bridge we discussed <image>," or "this crossing <image>." For "Portland Art Museum" (category: museum), replacements include "the gallery I visited, shown in <image>" and "that museum in the photo <image>."

Why this mechanism enforces cross-modal dependency: After replacement, the background paragraph no longer names the entity. The anaphor is deliberately under-specified: "the bridge" could refer to any of thousands of bridges worldwide, and only the evidence image resolves the reference. A text-only system reading "What time did I meet Emma at the bridge shown in <image>?" has no way to determine which bridge, and therefore cannot locate the correct answer in the conversation history. The cross-modal dependency is structural — built into the grammar of the question — rather than incidental.

For Knowledge Update (KU) questions, dictionary-based replacement is insufficient because the evolving attribute chain tracks concrete items within a single category (e.g., successive favorite fruits: apples → kiwis → mangoes → dragonfruit). In this case, an LLM rewrites each evidence fact with a short sensory or visual descriptor of at most five words that does not name the category. For instance, "blood orange" might become "this tangy round thing <image>" and "blueberries" might become "these tiny purple spheres <image>." The descriptor is constrained to be plausible for multiple items within the same category, preserving the ambiguity needed for cross-modal dependency — a text-only system seeing "this tangy round thing" cannot determine whether it refers to an orange, a grapefruit, or a pomelo.

Step 4 — Question generation and quality verification. The abstracted background paragraph, the evidence image, and the original entity name are provided to Gemini-3-Pro, which generates a (question, answer) pair together with one or more atomic evidence facts. The generation prompt enforces two constraints:

Cross-modal constraint: the question must be answerable only when both the image and the surrounding text are available. This means the image provides the entity identity (which bridge, which product, which plant), while the text provides the propositional content (the meeting time, the price, the preference).
Self-containment constraint: the answer must be derivable from the evidence facts without requiring external knowledge beyond the provided context. This prevents questions that depend on parametric knowledge (e.g., "In what year was the bridge built?" when the conversation never mentions the construction date).

For Multi-Session Reasoning (MSR) questions, a three-layer anti-leakage defense prevents the entity identity from being recoverable from text alone:

Anti-leakage prompt rules: the generation prompt explicitly forbids the model from naming the entity in any fact text, and requires that descriptive adjectives cannot confirm or contradict the visual identity (e.g., "tiny" cannot appear if the image shows a large dog).
Rule-based pre-filter: a deterministic check scans fact texts for the entity name or close synonyms, rejecting any candidate that leaks the answer.
LLM text-only judge: a separate GPT-5.1 call, given only the question text without images, attempts to answer the question. If the judge can confidently determine the answer from text alone, the candidate is rejected.

All generated questions then pass through the automated filtering and human review pipeline described in §3.3 — the rule-based pre-filter, the LLM judge for parametric-knowledge solvability, and three rounds of human annotation — which collectively reduce the initial pool of ~20K candidates to the final 789 questions.

Per-subtype generation routes. The four answerable question types follow different generation paths, each tailoring the visual skill under test while preserving the shared entity-abstraction mechanism:

IE – Entity two-hop (5 visual subtypes, n = 120): Gemini-3-Pro generates a two-hop chain where the first hop resolves the entity from the evidence image, and the second hop retrieves a property of that entity from the surrounding text. The five subtypes vary the visual skill required for the first hop: disambiguation (distinguishing similar-looking entities, e.g., different bridge types), alignment (matching an image to a textual description of a specific element), counting (enumerating items in the image), spatial reasoning (locating objects relative to each other), and arithmetic (computing from visually presented numbers, e.g., reading a price tag).
IE – PrevInfo (3 subtypes, n = 126): The question asks about a visual detail from an image shared in an earlier conversation session — not the current session, forcing multi-session recall. The three subtypes correspond to the image source: a screenshot of a chat interface, an app or web interface, or a natural photograph. Entity abstraction is applied to the session reference rather than the entity itself (e.g., "the screenshot I sent you on Tuesday <image>").
MSR (3 subtypes, n = 143): Facts are distributed across three to eight sessions, testing cross-session aggregation. Arithmetic (n = 50) requires summing prices or quantities; Counting (n = 46) requires enumerating entities matching a criterion; Entity Resolution (n = 47) requires determining whether two cross-session references denote the same entity, either by counting distinct entities or by yes/no identity matching (e.g., "Is the new bird the same species as Rio?").
TR (2 subtypes, n = 194): Duration Comparison (n = 91) derives two durations from session timestamps and visual cues (clocks, calendars) and asks which is longer. Temporal Grounding (n = 103) bundles two operations: order ranking (n = 24, sort events chronologically) and date extraction (n = 79, answer "When did X happen?" in YYYY/MM/DD format). Three generation modes cross with these operations — Mode B renders the temporal cue itself as a visual artifact (clock face, calendar page, receipt timestamp), Mode C pairs an entity image with explicit textual dates, and Mode D pairs an entity image with session-level timestamps that serve as implicit temporal anchors.
KU – Update (n = 116): A 4-fact atomic chain is generated: each fact updates a user attribute (e.g., favorite drink changes from tea → coffee → matcha → espresso martini). The question asks for the current state, requiring the model to locate all four updates and identify the latest, distinguishing it from outdated values.

Evidence Session Construction and Conversation History Assembly

The evidence session construction and conversation history assembly stages are where MEMLENS becomes genuinely difficult for current systems. The design goal is to embed evidence so that it is contextually indistinguishable from haystack content while remaining recoverable through semantic understanding — ruling out surface-level shortcuts that would inflate retrieval scores without testing memory.

Evidence session construction. Each atomic evidence fact from the question construction pipeline is wrapped into a complete multi-turn session using the same generation pipeline as haystack sessions. The user model (GPT-5.1) generates turns that embed the needle fact indirectly — for instance, to embed the fact "I started a new job last month," the user turn might open by asking about updating tax withholding and mention the job change incidentally later, rather than foregrounding it as the topic. The assistant model (Gemini-3-Pro) generates responses following a three-part template: acknowledge what the user shared (1–2 sentences), provide substantive information or perspective (150–250 words, covering 1–2 key points), and end with a knowledge-oriented follow-up question rather than a personal or social one (e.g., "Would you like tips on X?" not "Do you prefer X or Y?").

A six-stage validator chain screens each generated turn before it is admitted:

Rule-based length check: turns must fall within a target word-count range to maintain conversational naturalness.
Photo-directive check: if the evidence fact carries an evidence image, the user turn must include the <image> token placed adjacent to the corresponding textual mention, preserving unambiguous image–text co-reference.
N-gram leakage check: a deterministic scan rejects turns containing the original named entity or close synonyms, ensuring the abstraction is not accidentally reversed.
Semantic leakage check: a separate LLM call evaluates whether the turn's text, independent of images, reveals information that should only be available through the image.
Ambiguity preservation check: for KU and MSR questions, the validator verifies that the turn does not textually disambiguate what the image is supposed to encode.
End-marker verification: ensures the turn is grammatically complete and follows the assistant template structure.

Any turn that fails any check is regenerated with adjusted constraints. This validator chain is what makes evidence sessions structurally indistinguishable from haystack sessions — they share the same generator, the same style constraints, and the same quality gates, and the validator actively suppresses any surface-level signal that might distinguish them.

Haystack session curation. Haystack sessions are multimodal conversations generated from the same pipeline but grounded on sampled ontology topics rather than evidence facts. They are curated to be contextually related to the evidence topic but uninformative for the question — for instance, if the evidence concerns the Golden Gate Bridge, a haystack session might discuss other San Francisco landmarks (Fisherman's Wharf, Alcatraz) without mentioning the bridge itself. Haystack sessions never include answer-relevant details.

Conversation history assembly. For each question, evidence sessions are inserted into a timestamp-ordered history of haystack sessions, with positions chosen uniformly at random — except for type-specific ordering constraints:

MSR and KU questions: the relative order of evidence sessions is preserved, since the answer depends on the temporal sequence of facts (a four-step knowledge update chain, or a cross-session counting task). Randomizing their order would destroy the temporal structure that the question probes.
TR questions: each evidence session is assigned a timestamp that strictly precedes the question's reference date, so that temporal reasoning over session dates remains internally consistent.

Haystack sessions are then added to reach the target context length. To prevent evidence positions from being revealed by image clustering (evidence images might form a denser cluster in the timeline than haystack images), the assembler maintains a fixed text-per-image ratio across the entire history. Text-only filler sessions from ShareGPT and UltraChat — real user–AI conversations, not synthetically generated — are inserted to pad the context while keeping the image density uniform.

Length control and cross-modal token counting. MEMLENS adopts the cross-modal token-counting scheme from MMLongBench, where one image is counted as the equivalent of ~2,000 text tokens regardless of the model's actual visual encoder resolution. This ensures that a "32K context" means the same total information budget across different LVLM architectures, avoiding the confound where models with larger visual encoders consume more tokens per image and therefore have shorter effective text contexts. The four standardized lengths (32K, 64K, 128K, 256K) are achieved by varying the number of haystack and filler sessions:

At 32K: ~14 sessions, ~20 images
At 64K: ~32 sessions, ~46 images
At 128K: ~63 sessions, ~92 images
At 256K: ~93 sessions, ~138 images

The 256K length is tested only for memory-augmented agents, since most open-weight LVLMs do not natively support contexts beyond 128K.

Session indistinguishability validation. To verify that evidence sessions carry no exploitable stylistic fingerprint, the paper trains two binary classifiers to distinguish evidence from haystack sessions on text features alone (with <image> tokens removed and speaker labels normalized). A TF-IDF logistic regression model (unigrams and bigrams, 10,000 features, sublinear term frequency) achieves 56.71% ± 1.20% accuracy with ROC-AUC of 0.59 under five-fold cross-validation with question-grouped splits. A fine-tuned DeBERTa-v3-base classifier (512-token truncation, three epochs) achieves 57.92% ± 0.85% with ROC-AUC of 0.55. Both results sit only marginally above the 50% chance floor with small effect sizes (Cohen's h < 0.16), indicating that evidence sessions carry negligible stylistic signal for localization. A follow-up vocabulary ablation progressively removes the top K most discriminative n-grams from the TF-IDF feature set, and accuracy remains between 54.77% and 56.71% across the entire sweep — the weak signal is diffuse rather than concentrated in identifiable markers.

Why this matters: If evidence sessions could be located by surface-level cues (unusual vocabulary, distinct syntactic patterns, anomalous turn lengths), a simple keyword-matching or anomaly-detection approach would inflate retrieval scores without testing genuine memory. The indistinguishability validation confirms that retrieval difficulty is driven by semantic relevance rather than statistical shortcuts, making MEMLENS a genuine test of content-based memory.

Evaluation Protocol

The evaluation protocol is designed to accommodate both long-context LVLMs and memory-augmented agents under comparable conditions, while respecting each architecture's native input format.

LVLM evaluation. For 27 LVLMs spanning closed-source APIs (GPT-5.4, Claude Sonnet 4.5, Gemini-3.1-Pro) and open-source families (Kimi-K2.5, Qwen3.5, GLM-4.6V, Gemma3, Qwen3-VL, and others — full roster in Appendix B), the full conversation history is provided as input with images inlined as <image> tokens in the standard interleaved format of each model. The model receives the prompt:

Provide answers based on the given conversation history. If the question
cannot be answered based on the given conversation, respond with
"Insufficient information".

Conversation:
{context}

Directly output the answer with no extra output.
Question Date: {question_date}
Question: {question}

The {context} placeholder is replaced with the full assembled conversation history for the target context length. Generation length is set to 2,048 tokens for direct models and 16,384 for thinking-mode models (to accommodate reasoning traces). LVLMs are evaluated at 32K, 64K, and 128K; most do not natively support 256K. Open-weight models are served locally with vLLM v0.17–0.18 using FlashAttention-2 on 8×A100-80GB nodes with tensor parallelism for 128K inputs; API models use provider endpoints with 4–8 concurrent threads.

Agent evaluation. Seven memory-augmented agents are evaluated on a stratified 195-question canonical subset (~1/4 of the full benchmark, preserving per-type proportions to within 0.2 percentage points) rather than the full 789-question set, because agent pipelines are substantially slower than direct LVLM inference (M2A takes roughly 60× longer per question).

The agents divide into three multimodal pipelines and four text-only pipelines, each with its own input adapter:

M3-Agent: native video LVLM backbone (Qwen2-VL-7B, RL-trained). Does not natively accept interleaved image–text sequences; each session is rendered as a composite image, and the agent receives sessions as a sequence of composite images. At answer time, the agent retrieves relevant session composites; raw pixels from original images are not available at query time.
M2A: native LVLM backbone (Qwen3-VL-8B) with dual-layer SQLite + SigLIP2 memory. Original images enter the memory store at write time, but at answer time the agent retrieves stored embeddings — not raw pixels.
M3C: native LVLM backbone (Qwen2-VL-2B, LoRA-adapted) for session retrieval. Original images enter the memory store at write time; at answer time, embedding-based retrieval recovers memory entries rather than raw images.
Mem0: text-only LLM backbone (Qwen3-8B, also evaluated with gpt-4.1-mini and Qwen2.5-7B for backbone ablation) with FAISS vector store. At write time, every evidence image is replaced by a BLIP-2 generated caption; at answer time, the agent works with captions only — pixel information is lost entirely. Adaptation: images → BLIP-2 captions input.
MemOS: text-only LLM backbone (Qwen3-8B, also evaluated with Qwen2.5-7B) with layered memory architecture. Same BLIP-2 captioning pipeline as Mem0.
MemAgent-7B: text-only LLM backbone (Qwen2.5-7B, RL-trained) with recurrent sliding-window memory. Same BLIP-2 captioning pipeline.
Memory-T1: text-only LLM backbone (Qwen2.5-3B, RL-trained) with BM25 text retrieval. Same BLIP-2 captioning pipeline.

Agents are evaluated at all four context lengths (32K, 64K, 128K, 256K) because their memory architectures are length-invariant by construction.

Why adapters are not normalized: The paper explicitly does not normalize input formats across agents because the adapter is part of the system being evaluated. Released checkpoints assume the input format their authors trained on, and any uniform substitute would either degrade architectures that depend on caption-only memory (Mem0, MemOS, MemAgent-7B, Memory-T1) or block agents whose backbones cannot accept interleaved input (M3-Agent). The reported deficits relative to direct LVLMs therefore conflate adapter-induced visual information loss with retrieval and reading quality — but this is the correct comparison for understanding how currently deployable systems perform.

LLM-as-Judge scoring. The primary metric is LLM-as-Judge accuracy with Qwen3-VL-235B-A22B-Instruct as the canonical judge (thinking disabled). The judge receives the question, the reference answer, and the raw model output, and emits a binary correct/incorrect verdict under task-specific grading criteria.

The judge system prompt encodes a final-answer extraction policy robust to long thinking traces and circular reasoning:

The judge must extract the final answer from the model output, ignoring intermediate reasoning unless the output lacks a clear concluding answer statement.
If the output contains both a refusal and a guessed answer, the judge scores the final committed answer.
If the output shows circular reasoning — repeatedly revisiting the same evidence and flip-flopping between answers with no clear final commitment — the judge assigns 0 points.
If the output is entirely a reasoning trace with no extractable answer, the judge assigns 0 points even if the correct answer appears somewhere within the reasoning.

The judge prompt includes three universal worked examples (circular reasoning → 0 points, redundant but committed → score normally, reasoning trace with no answer → 0 points) that apply across all task types.

Task-specific grading criteria are appended at runtime for each of the eight reporting subtypes (IE, MSR Yes/No, MSR Counting, MSR Arithmetic, TR Duration Comparison, TR Order Ranking, TR Date Extraction, KU, AR). For Information Extraction, the criterion is:

Assign 1 point if the student's response contains the core information from the standard answer. Minor wording differences are acceptable, but the essential information must be present and correct. Assign 0 points if the core information is missing, contradicted, too vague, refused, or incorrect.

For AR questions, the judge maps any prediction in a canonical refusal-phrase set ("Insufficient information", "NOT_MENTIONED", "cannot be determined", etc.) to a successful abstention; the gold answer stored in the dataset is the literal string NOT_MENTIONED, which is a different string from the surface refusal phrase the model is instructed to emit.

The pipeline tail-truncates outputs beyond 6,000 characters, auto-zeros responses exceeding 500 parsed words (to catch degenerate verbose outputs that never produce an answer), and instructs the judge to reject circular reasoning. Each judge call is cached in a SQLite store keyed by question, reference, prediction, and judge identifier, so that re-runs over the same outputs remain deterministic.

Judge validation. Judge reliability is validated through two independent checks:

Cross-family agreement: a stratified sample of 800 items (from 73,784 total judge calls, ~1.08% of the population) is re-evaluated with GPT-5.4-mini as an independent judge. The sample combines 200 random, 250 targeted (on hard cells where models disagree), and 350 GPT-only extended items to ensure coverage across model family, context length, question type, and judge score. Item-level agreement reaches 96.40% with Cohen's κ = 0.93, and the model-level Spearman rank correlation is ρ = 0.97 (p < 10⁻⁶). The mean per-model accuracy delta between judges is 3.70%, and judge choice does not reorder the leaderboard.
Human consensus: three annotators independently labeled 484 items; disagreements were resolved to consensus. Against these consensus labels, the Qwen3-VL-235B judge reaches 93.60% raw agreement with Cohen's κ = 0.86. Errors are leniency-biased: 29 false positives versus 2 false negatives, meaning the judge credits borderline answers more often than it rejects correct ones. The per-question-type breakdown shows the largest false-positive counts on Information Extraction (11, partial match on short factual answers), Answer Refusal (10, hedge phrases credited as refusal), and Knowledge Update (5, verbose-correct vs. literal mismatch).

A format-dependent bias correction addresses a specific leniency pattern: the judge evaluates very short answers (1–3 words) more leniently. The correction adjusts false positives for six affected models; all reported scores use corrected values.

Deterministic typed-accuracy audit. Seven of the nine reporting subtypes admit deterministic normalization independent of the LLM judge: MSR Counting (integer match), MSR Arithmetic (currency-normalized scalar), MSR Yes/No (Yes/No), TR Order Ranking (tuple of session indices), TR Duration Comparison (A/B label), TR Date Extraction (multi-format date canonicalization), and AR Answer Refusal (canonical refusal-phrase set). A rule-based rescoring of all 32K runs across 34 evaluation rosters (12,234 deterministic items) yields item-level agreement with the LLM judge of 93.6% — in line with the 93.6% raw agreement on the human-consensus subset. The LLM judge is systematically lenient (5.4% judge-false-positives vs. 1.0% judge-false-negatives), with the largest leniency channels on TR Order Ranking (12.6% J-FP, partial credit on near-correct tuples) and TR Date Extraction (8.6% J-FP, format-flexible date matching). At the model-leaderboard level, the Spearman rank correlation between LLM and deterministic scores is ρ = 0.78, the top-5 models are preserved under both metrics, and the top-10 sets overlap on 7 of 10 entries. The three rosters that drop out of the top-10 under deterministic rescoring all produce verbose justifications that the LLM judge credits but the rule-based check rejects, consistent with the format-dependent leniency pattern.

Why this validation matters: The multi-layered validation (cross-family, human-consensus, deterministic-typed) is necessary because LLM-as-Judge evaluation is vulnerable to three confounds: family bias (the judge favoring outputs from its own model family), format bias (leniency on certain answer formats), and task-specific misinterpretation (e.g., crediting a partial match as correct when the task requires exact identity). The cross-family test eliminates family bias as a concern (the Qwen judge's leniency gap on Qwen-family outputs vs. non-Qwen outputs differs by only +0.33%, well below the 3% practical-significance threshold). The human-consensus test bounds the absolute error rate at ~6.4%. The deterministic audit confirms that the leniency is format-dependent rather than task-dependent, and that it does not reorder the top of the leaderboard.

Memory Abilities: How the Pipeline Specializes for Each Type

Each of the five memory abilities in MEMLENS is instantiated through a specialization of the shared construction pipeline, targeting a distinct cognitive operation while preserving the cross-modal dependency mechanism.

Information Extraction (IE) — Single-session retrieval with visual grounding (n = 246). IE questions test the most fundamental memory operation: recall a specific fact from a single evidence session. The question targets one evidence session embedded in the full haystack, and the model must retrieve that session and extract the relevant fact. This is the baseline memory test — if a system cannot perform single-session retrieval, it will fail on all more complex abilities.

IE – Entity (n = 120): Two-hop reasoning through entity abstraction. The first hop is the visual hop: the model sees the anaphor "the bridge shown in <image>" in the question, locates the evidence image in the conversation history, and identifies the entity (Golden Gate Bridge). The second hop is the textual hop: once the entity is identified, the model retrieves the associated property from the surrounding text (e.g., the meeting time, the toll price, the architectural detail). The five visual subtypes vary the first-hop skill: disambiguation (is it this specific bridge or a similar one?), alignment (does this image match the description of "the circular seal in the top right corner"?), counting (how many cables are visible?), spatial reasoning (where is the logo relative to the text?), and arithmetic (what total does the price tag show?). A representative question: "What phrase does that message spell out?" where the message is a vanity license plate visible only in the evidence image.

IE – PrevInfo (n = 126): Recall a visual detail from an image shared in an earlier session. The evidence image is in Session A; the question is asked in Session B (or later), and the model must remember what it saw. The three subtypes correspond to the image source — a screenshot of a chat interface, an app or web interface, or a natural photograph — because these image types exercise different visual skills (UI parsing vs. natural-image recognition). A representative question: "What color is the engine bay painted?" where the engine bay photo was shared three sessions ago and the textual context never describes the color.

Multi-Session Reasoning (MSR) — Cross-session aggregation (n = 143). MSR extends retrieval to aggregation: the model must locate three to eight evidence sessions, extract the relevant facts from each, and combine them into a coherent answer. This tests whether memory supports reasoning over distributed evidence, which is a harder requirement than single-session recall.

MSR – Arithmetic (n = 50): Sum or compute over prices, quantities, or durations scattered across sessions. At least one operand is visible only in an image (e.g., a price tag, a receipt total, a number visible in a screenshot), so text-only aggregation would produce incorrect results. A representative question: "How much total have I spent on German language board games this month?" where one game's price is stated textually and another's is visible only on the game box in a photo.

MSR – Counting (n = 46): Count entities matching a criterion across sessions. The entities are identified by their evidence images — the model must visually determine whether each session's image depicts a qualifying entity, then tally the count. A representative question: "How many educational board games do I currently have for the kids?" where each game appears in a different session as a photo, and the model must count the ones tagged as educational.

MSR – Entity Resolution (n = 47): Determine whether two cross-session references denote the same entity, either by counting distinct entities or by yes/no identity matching. One session shows a named entity ("my Monstera Deliciosa called Monty"), and another session shows an ambiguously referenced entity with an image ("this creepy Shining-room plant <image>"). The model must compare the visual identities to answer yes/no. The anti-leakage defense is critical here: the textual description of the vague reference must not reveal whether it matches the named entity, so the image is the only discriminating signal.

Temporal Reasoning (TR) — Joint temporal–visual reasoning (n = 194). TR tests whether the model can jointly process heterogeneous temporal signals — session timestamps, natural-language date expressions, and visual temporal artifacts (clock faces, calendar pages, receipt timestamps) — together with entity evidence images.

TR – Duration Comparison (n = 91): The model must derive two time intervals from the conversation history and compare their durations. The endpoints of each interval come from a mixture of sources: some are explicit textual dates ("I moved to New York on June 9, 2009"), some are session-level timestamps ("Session date: 2019/03/01"), and some are embedded in visual artifacts (a boarding pass photo showing a flight date, a clock image showing a specific time). The model must normalize all these sources into comparable durations and select which is longer. A representative question: "Which duration is longer: my time living in New York City vs. the time I spent working on my freelance writing project?" where the NYC start date is textually stated, the NYC end date is on a boarding pass image, and the freelance project dates are drawn from session metadata.

TR – Temporal Grounding (n = 103): Two operations. Order ranking (n = 24) requires sorting a set of events chronologically based on their occurrence dates, which are distributed across textual statements, session timestamps, and visual cues. Date extraction (n = 79) requires extracting a specific date and formatting it as YYYY/MM/DD, where the date might be textually stated, inferable from a session timestamp paired with a relative expression ("last Tuesday"), or visible only in a calendar or clock image.

Knowledge Update (KU) — Current-state tracking through evolving chains (n = 116). KU tests whether the model can track an evolving user attribute across a chain of four successive updates and report the current state rather than an earlier, superseded value. Each update fact is anchored by a different evidence image — for instance, the user's favorite fruit changes from apples (image: red apple) → kiwis (image: sliced kiwifruit) → mangoes (image: yellow mango) → dragonfruit (image: pink dragonfruit). The question asks "What is my favorite fruit now?" and the model must identify the most recent image in the chain and recognize the fruit it depicts. If the model retrieves an outdated image (e.g., the apple from two updates ago), it will answer incorrectly.

Entity abstraction is applied to the attribute category with sensory descriptors rather than dictionary-based replacement: instead of naming "apple," the fact describes "this round red thing <image>." The descriptor is constrained to be plausible for multiple items in the same category, so text alone cannot determine which specific fruit is referenced. A correct answer requires locating all four update images, determining their temporal order (preserved in the conversation assembly), and reporting only the final state.

Answer Refusal (AR) — Epistemic calibration (n = 90). AR is a calibration check, not a core memory retrieval task. Each AR question is constructed from an answerable instance by deliberately removing all evidence facts from the conversation history — the question remains in the history, but no supporting information is present anywhere. A correct model must decline to answer rather than hallucinate a plausible response. The gold answer is the literal string NOT_MENTIONED, and the judge credits any prediction in the canonical refusal-phrase set as correct.

Why this design isolates memory from perception: If the model locates the evidence and answers correctly on IE, MSR, TR, and KU questions, but also answers correctly on AR questions (where no evidence exists), it is likely hallucinating rather than genuinely retrieving. A model that scores high on the four answerable types but low on AR might be producing plausible guesses that happen to align with the gold answers, inflating its apparent memory performance. AR thus serves as a necessary calibration check — it validates that good performance on answerable questions reflects genuine evidence retrieval rather than lucky hallucination.

4. Key Insights and Innovations

The paper's most distinctive intellectual contribution is not the benchmark itself but the design methodology that makes it valid: enforcing cross-modal dependency through entity abstraction at the grammatical level, so that the visual modality is structurally necessary rather than decoratively present. This is a fundamentally different approach from how prior multimodal conversational benchmarks handled images.

Prior work—LoCoMo, Mem-Gallery—embedded images in dialogue but constructed questions that were answerable from text alone. The field's implicit assumption was that if images are present in the input and the model can access them, the benchmark is "multimodal." The paper shows this assumption is empirically false and proposes a causal alternative: rather than hoping models use images, force the dependency by removing the textual path. The entity abstraction mechanism (§3.4 in the Technical Approach) does exactly this—replacing "Golden Gate Bridge" with "the bridge shown in <image>" makes text-only reasoning impossible because the anaphor is deliberately under-specified.

What makes this a conceptual advance rather than an engineering trick is the causal validation: the image-ablation study (§3.4, Table 3) doesn't just show that models perform worse without images (which would be a correlation), but that removing images collapses performance from ~90% to below 2%—a near-total causal effect. This is the first benchmark in the conversational memory space to provide this kind of causal evidence that its questions genuinely require visual reasoning, and it retroactively reveals that prior multimodal conversational benchmarks were measuring text-only memory under multimodal disguise. The causal framing—"if images cause performance, removing them should destroy it"—is a methodological standard that future multimodal benchmarks should adopt.

The significance extends beyond this paper: the entity-abstraction pipeline generalizes to any domain where visual evidence carries information not recoverable from text. A code-generation benchmark could abstract function names behind image-dependent references (a screenshot of the output rather than the function name); a medical QA benchmark could replace drug names with pill photos. The design pattern—identify an entity, replace its surface form with a visual anaphor, validate the causal dependency—is a template for constructing genuinely multimodal evaluation.

Innovation 2: The Orthogonal Failure Mode Diagnosis—Long-Context LVLMs and Memory Agents Fail on Different Axes

The paper's central empirical finding is not that one architecture outperforms the other, but that the two approaches exhibit complementary, orthogonal failure modes: LVLMs degrade with context length, while memory agents lose to lossy multimodal compression at storage time (§4.3, Figure 4a). This is a diagnostic insight that restructures how the field should think about memory architectures.

Before this work, the research community largely operated under an implicit scalar assumption: one approach is "better" than the other, and the goal is to find which one. Long-context proponents argued that retrieval is unnecessary when the model can attend over the full history; memory-agent proponents argued that long contexts are computationally expensive and attention degrades with length. Both positions made sense within their own evaluation ecosystems—long-context benchmarks showed length-dependent degradation, agent benchmarks showed length-invariant retrieval—but no one had tested both on the same multimodal task.

The paper's side-by-side comparison reveals that the scalar assumption is wrong. LVLMs and agents don't occupy different points on a single quality axis; they occupy different quadrants in a 2D space defined by length stability and visual fidelity. LVLMs have high visual fidelity (pixel-level access to evidence images) but poor length stability (IE drops ~~20% from 32K to 128K). Agents have excellent length stability (~~±7% across 32K–256K) but poor visual fidelity (caption-based storage discards fine-grained visual cues). The consequence is that neither approach alone can solve the full benchmark—each covers one dimension, and the uncovered dimension is where performance collapses.

This insight is more than a taxonomic observation. It has direct architectural implications: the field should stop trying to scale context or improve retrieval in isolation and instead build hybrid systems that combine long-context attention with structured multimodal retrieval. The paper doesn't implement such a system, but the failure-mode diagnosis makes the design requirement precise: the hybrid needs attention-like access to pixel-level evidence (for visual fidelity) combined with retrieval-like index structures (for length stability). The diagnosis is the contribution; the architecture follows from it.

The evidence for this diagnosis is systematic: Figure 4a shows the aggregate degradation curves; the error analysis (§4.3, Figure 4b) shows that ~90% of IE and KU errors are visual, not reasoning, confirming that evidence retrieval—not comprehension—is the bottleneck; and the agent analysis (§4.3) shows the largest LVLM–agent gaps on visually grounded types (IE, KU), consistent with lossy storage being the primary agent failure mode. This convergence of aggregate trends, per-type breakdowns, and error taxonomies makes the diagnosis robust.

Innovation 3: Multi-Session Reasoning as the Shared Capability Ceiling—Revealing the Limits of Current Memory Architectures

The paper identifies multi-session reasoning (MSR) as the hardest memory ability across all evaluated systems, with even the strongest model (Kimi-K2.5) reaching only 44.06% at 32K and most systems falling below 30% (§4.2, Figure 2, MSR panel). This is not merely a difficult sub-task—it is the shared capability ceiling that neither architecture can surmount, and understanding why reveals a fundamental limitation of current approaches.

The significance of this finding lies in the retrieval-bounded diagnosis: the paper shows that MSR is not intrinsically too hard for current reasoning capabilities, but rather that models fail to collect the necessary evidence sessions from the full conversation history. The oracle-retrieval diagnostic (§G.7) provides the key evidence: when frontier models receive the ground-truth evidence sessions with their associated images (bypassing the haystack entirely), MSR accuracy reaches 90–100%—near-perfect cross-session aggregation. This means the reasoning operations themselves (counting, summing, identity matching across 3–8 sessions) are within current model capabilities. The bottleneck is evidence collection: models cannot locate all the required sessions in a 32K–128K history dense with topically related distractors.

This diagnosis reframes the problem from "MSR is hard" to "MSR is retrieval-bounded." It is not a reasoning challenge that requires better models; it is a retrieval challenge that requires better memory architectures. The implication is that scaling model size or reasoning capability alone will not solve MSR—the bottleneck is upstream of reasoning, in the localization step that any memory system must perform first.

What makes this a genuine insight rather than an obvious observation is that it disambiguates retrieval failure from reasoning failure using a controlled intervention (oracle evidence delivery). Without the oracle diagnostic, one might conclude from the low MSR scores that current models simply cannot aggregate information across sessions—a reasoning deficit. The diagnostic shows this conclusion is wrong: they can aggregate perfectly when given the right evidence; they just cannot find the evidence. This is a sharp, causal finding that changes the research priority from "improve cross-session reasoning" to "improve cross-modal evidence retrieval at length."

The finding also connects to the orthogonal-failure-mode diagnosis above: if MSR is retrieval-bounded, then the LVLM degradation with length (which makes evidence harder to locate) and the agent storage-time compression (which makes evidence harder to recognize) both exacerbate the MSR ceiling through different mechanisms. Neither approach addresses the core problem—reliably collecting distributed multimodal evidence—and this explains why MSR remains the ceiling even as other abilities improve.

Innovation 4: Memory-Agent Post-Training Erodes Abstention—A Negative Finding with Architectural Implications

The paper documents a striking negative result: post-training on memory agent backbones through RL or LoRA for memory management substantially degrades answer refusal (AR) behavior, with frozen-backbone agents preserving 68–77% AR accuracy while fine-tuned agents collapse to 9–22% (§4.3, Figure 2, AR panel). This is not a metric regression—it is a structural property of how current memory-agent training is designed.

The mechanism is straightforward but the implication is profound. Current RL and SFT fine-tuning for memory agents optimize answer correctness and retrieval success, with no signal that refusing an unanswerable question is a correct action. As a result, the fine-tuned backbone learns that producing an answer (any answer) is rewarded, while abstention receives no reinforcement—and abstention behavior degrades accordingly. The paper quantifies this with a controlled backbone-matched comparison: M2A (fine-tuned Qwen3-VL-8B) reaches only 22.73% AR, while the same backbone under direct inference scores 81.82%—a ~59% gap attributable entirely to the fine-tuning process (Appendix G.6, Table 16).

What makes this a genuine innovation rather than a simple failure case is that it identifies a reward-design blind spot in the memory-agent training paradigm. The agent community has focused on optimizing memory operations (what to store, what to retrieve, how to update)—essentially treating memory as an information-management problem separable from hallucination control. This paper shows that these are not separable: optimizing memory access without jointly optimizing evidence-sensitive abstention produces agents that are better at retrieving but worse at knowing when retrieval is impossible. The consequence is that improved memory performance on answerable questions may be partly spurious—the agent might be hallucinating plausible answers (inflating apparent retrieval accuracy) rather than genuinely retrieving evidence.

The constructive implication is that future memory-agent training should include joint optimization objectives that reward abstention on unanswerable questions alongside retrieval success on answerable ones. The paper does not implement such objectives, but the negative finding makes the design requirement precise: memory management and hallucination control are coupled problems that must be solved together.

The backbone ablation (Table 21) confirms that this is not a model-scale issue—stronger backbones alone do not close the AR gap—which reinforces that the problem is in the training objective, not the model capacity. This is a fundamental limitation of current memory-agent design, and the paper provides the first systematic evidence for it in a multimodal setting.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation dataset is MEMLENS itself, comprising 789 questions across five memory abilities (Information Extraction, Multi-Session Reasoning, Temporal Reasoning, Knowledge Update, and Answer Refusal) at four standardized context lengths: 32K, 64K, 128K, and 256K tokens. Each question is instantiated as a multi-session conversation history with interleaved images and text, constructed through the four-stage pipeline described in Section 3. Questions are drawn from a stratified sample across 12,000+ fine-grained subtopics derived from a hierarchical topic ontology (§C.2), covering identification of real-world entities (~40%), everyday experiences (~40%), and text-rich documents (~20%). Evidence images are sourced from public web image search via iCrawler and filtered through a multi-channel relevance scoring pipeline combining CLIP, SigLIP, and BLIP-2 caption similarity (§C.5). The dataset is released at https://huggingface.co/datasets/xiyuRenBill/MEMLENS under CC-BY-4.0 for annotations and permissive source-site licenses for images (§C.7).

Base model(s). The evaluation spans 27 LVLMs across three categories. Closed-source API models include GPT-5.4, Claude Sonnet 4.5, and Gemini-3.1-Pro. Major open-source families include Kimi-K2.5 (1T parameters, MoE), the Qwen3.5 family (122B-A10B MoE through 2B dense), the Qwen3-VL family (235B-A22B MoE through 2B dense, in both Instruct and Thinking modes), GLM-4.6V (106B) and GLM-4.5V (106B), Gemma3 (27B, 12B, 4B), Phi4-Multimodal (5.6B), Cosmos-Reason2-8B, and Nemotron-Nano-12B. The full roster appears in Table 13 (Appendix G.1). Models are chosen to span current frontier capabilities across scale (2B to 1T+), architecture (dense vs. MoE), and training regime (standard instruction-tuning vs. thinking-mode reasoning). Seven memory-augmented agents are evaluated: three multimodal pipelines (M3-Agent, M2A, M3C) and four text-only pipelines (Mem0, MemOS, MemAgent-7B, Memory-T1). Agent backbones range from Qwen2-VL-7B to Qwen3-8B with varied training (RL, LoRA, or frozen), as detailed in Appendix B.

Metrics. The primary metric is LLM-as-Judge accuracy (%), computed as the fraction of questions for which the model's output is judged correct by Qwen3-VL-235B-A22B-Instruct against the gold reference answer. The judge emits binary 0/1 verdicts per question under task-specific grading criteria (§F.1). This is reported overall (n = 789 for LVLMs, n = 195 for agents on the canonical subset) and per-type (IE, MSR, TR, KU, AR). The judge is validated against GPT-5.4-mini on a stratified 800-item sample (κ = 0.93, Spearman ρ = 0.97 at the model level; Appendix E.2) and against three-annotator human consensus on 484 items (κ = 0.86, raw agreement 93.60%). A deterministic typed-accuracy audit on 12,234 closed-form items confirms 93.6% agreement with the LLM judge, with systematic leniency of ~5% absolute (5.4% judge-false-positives vs. 1.0% judge-false-negatives; Table 12). Agent subset scores are validated through bootstrap resampling (1000 iterations) on the 195-question canonical subset, yielding 95% confidence intervals of ±4.87–6.67% (Appendix G.2).

Baselines. The evaluation is explicitly comparative rather than baseline-driven: each model or agent is scored against the same benchmark, and results are compared across architectures. Within the LVLM cohort, the de facto baselines are the frontier API models (GPT-5.4, Gemini-3.1-Pro, Claude Sonnet 4.5), which establish the performance ceiling at each context length. Within agents, the frozen-backbone systems (Mem0 with Qwen3-8B, MemOS with Qwen3-8B) serve as retrieval-only baselines against which fine-tuned agent performance can be assessed. The backbone-matched comparison—M2A (fine-tuned Qwen3-VL-8B) vs. direct Qwen3-VL-8B-Instruct inference on the same 195 questions—measures the architectural cost of the memory pipeline independently of backbone quality (Appendix G.6, Table 16). No prior benchmark provides a direct comparison point because, as established in Table 1, no existing benchmark evaluates both LVLMs and memory agents on the same multimodal questions.

Generation budget / compute accounting. Test-time compute is measured by context length, not by generation budget, since LVLMs process the full conversation history in a single forward pass. The four standardized lengths (32K, 64K, 128K, 256K tokens) are calibrated using the cross-modal token-counting scheme from MMLongBench, where each image is counted as ~2,000 text tokens. This ensures that context length means the same total information budget across models regardless of their visual encoder resolution. For LVLMs, generation length is set to 2,048 tokens (direct models) or 16,384 tokens (thinking-mode models). For agents, the evaluation is wall-clock constrained rather than token-constrained—M2A takes roughly 60× longer per question than direct LVLM inference—motivating the 195-question canonical subset for agents. LVLMs are evaluated at 32K, 64K, and 128K (most do not natively support 256K); agents are evaluated at all four lengths including 256K. API model costs total approximately $4,500 USD across both pipeline construction and benchmarking; open-weight models are served locally on 8×A100-80GB nodes with vLLM v0.17–0.18 and FlashAttention-2 (Reproducibility Statement).

Cross-validation / statistical protocol. No cross-validation is used for model evaluation, since models are tested on a fixed benchmark rather than trained. The judge validation uses a two-tier sampling design: 450 items (200 random + 250 targeted on hard cells) are annotated by both human raters and GPT-5.4-mini, and an extended 350 items are re-judged by GPT-5.4-mini only, yielding 800 cross-judge items from a population of 73,784 total judge calls (~1.08%; Appendix E.2). The bootstrap confidence intervals on agent overall accuracy use 1000 resamples with replacement at the question level, percentile method, n = 195 (Appendix G.2). Statistical testing for context-length degradation uses McNemar's exact test on paired per-question binary outcomes (same 789 questions at two lengths) with Bonferroni correction across 72 transitions (α/72 = 0.0007; Appendix G.5). The cross-type Spearman correlations (Figure 5, Appendix G.5) are computed at the model level across all 34 evaluated systems.

Main Quantitative Results

Overall Performance: Length-Dependent Degradation vs. Length-Stable Compression Loss

The headline finding across 27 LVLMs and 7 agents is that no system solves the benchmark, and the two architectural classes exhibit complementary failure modes that prevent either from dominating across all context lengths and memory abilities.

LVLM aggregate performance (Table 13, Figure 2). At 32K, the top eight LVLMs fall within a narrow 6.34% band (Qwen3.5-122B-A10B at 58.68% overall, down to Qwen3-VL-30B-Instruct at 53.23%), so short-context accuracy does not cleanly separate frontier systems. The situation inverts at 128K: several open-weight leaders lose more than 13% (Qwen3.5-122B drops from 58.68% to 45.50%, a −13.18% decline; Qwen3-VL-30B-Instruct drops from 55.01% to 44.23%, a −10.78% decline), while Gemini-3.1-Pro retains 51.99% overall (a −2.11% drop from 54.10% at 32K) and degrades least of any model. GPT-5.4 drops from 52.72% to 49.56% (−3.16%), and Kimi-K2.5 drops from 54.88% to 51.99% (−2.89%). Claude Sonnet 4.5 occupies an unusual position: its overall accuracy is low (36.50% at 32K, 27.76% at 128K, ranking 23rd out of 31 at 32K), but its per-answer accuracy (PA) on attempted questions reaches 57.51% at 32K—competitive with models ranked considerably higher—behind a 50.50% refusal rate (§G.3, Table 17). This calibration-driven deficit reflects model-specific behavior rather than a benchmark fairness issue.

Agent aggregate performance (Table 14, Figure 2). Memory agents occupy a narrower and lower range. At 32K, the top four text-only systems cluster within 5% of each other: MemAgent-7B at 32.82%, Mem0 at 31.79%, MemOS at 30.26%, and Memory-T1 at 28.72%, while the three multimodal agents fall substantially lower: M3-Agent at 19.49%, M3C at 18.46%, and M2A at 15.38%. Across 32K to 256K, six of seven agents stay within ±7% overall—Mem0 at 31.79% → 30.77% (−1.02%), MemOS at 30.26% → 29.74% (−0.52%), MemAgent-7B at 32.82% → 24.62% (−8.20%, the largest agent drop), Memory-T1 at 28.72% → 29.23% (+0.51%), M3-Agent at 19.49% → 23.59% (+4.10%), and M2A at 15.38% → 16.41% (+1.03%). M3C drops from 18.46% to 16.92% (−1.54%). The flatness is genuine robustness rather than a floor effect: Jaccard overlap between adjacent context lengths on correctly answered questions exceeds the random baseline by 3.3× to 6.9× across agents (Appendix G.6).

The LVLM–agent gap. At 32K, the strongest LVLM (Qwen3.5-122B at 58.68%) exceeds the strongest agent (MemAgent-7B at 32.82%) by 25.86 percentage points. The gap is largest on visually grounded types: on IE, Qwen3.5-122B reaches 74.39% vs. Mem0 at 13.11% (−61.28%); on KU, Qwen3.5-122B reaches 49.14% vs. MemAgent-7B at 41.38% (−7.76%, the smallest agent-LVLM gap on any type, reflecting MemAgent-7B's RL training for update tracking). On MSR, the leaderboard inverts: Memory-T1 reaches 62.50% TR accuracy vs. Qwen3.5-122B at 51.55%, exploiting BM25 date matching for temporal cues while LVLMs struggle with temporal–visual integration.

Per-Type Analysis: Distinct Ability Profiles and a Shared MSR Ceiling

Information Extraction (IE): Single-session retrieval dominance and sharp length degradation. At 32K, Qwen3.5-122B leads IE at 74.39%, followed by Qwen3.5-27B at 70.33% and GPT-5.4 at 69.51%. The API tier sits in the 57–70% range; the open-weight tier spans 25–65%. At 128K, Qwen3.5-122B collapses to 43.09% (−31.30%), Qwen3.5-27B falls to 54.07% (−16.26%), and Qwen3-VL-30B-Instruct drops from 65.45% to 56.50% (−8.95%). GPT-5.4 falls to 60.16% (−9.35%), and Gemini-3.1-Pro remains at 55.79% (−1.53% from 57.32% at 32K), the most length-stable LVLM on IE. The IE-Entity subtype (two-hop visual + textual) is consistently harder than IE-PrevInfo (single-hop visual recall), with entity abstraction imposing a recognition bottleneck that degrades under growing filler (Appendix C.3, Figure 6). Agents trail dramatically on IE: Mem0 at 13.11%, Memory-T1 at 18.03%, M3-Agent at 18.03%—all below the weakest evaluated LVLM (Qwen3-VL-2B-Thinking at 26.83%), reflecting the caption-based storage's loss of fine-grained visual cues (counts, colors, spatial relations) that IE specifically probes.

Multi-Session Reasoning (MSR): The shared capability ceiling. MSR is the hardest type across all systems. At 32K, only Kimi-K2.5 (44.06%) clears 40%, with Gemini-3.1-Pro at 32.17% and GPT-5.4 at 28.18%. Most open-weight LVLMs fall below 25%: Qwen3.5-122B at 30.07%, Qwen3-VL-235B-Instruct at 18.88%, GLM-4.6V at 20.28%. At 128K, all LVLMs degrade: Kimi-K2.5 drops to 28.67% (−15.39%), Gemini-3.1-Pro to 29.37% (−2.80%), GPT-5.4 to 21.68% (−6.50%). The MSR subtypes show heterogeneous difficulty: MSR-Arithmetic is the hardest overall (top score ~35% at 32K), MSR-Counting and MSR-Entity Resolution are moderate (~25–45%), and MSR-Yes/No identity matching is easiest (~50–70%). Agent MSR performance is comparable to mid-tier LVLMs, with M3C at 25.71%, Mem0 at 25.00%, and Memory-T1 at 25.71% at 32K—the only ability where agents approach LVLM parity, likely because text-only agents can retrieve the textual portions of MSR evidence (prices, counts, entity names) without depending on the images that their caption storage has lost.

Temporal Reasoning (TR): Explicit temporal anchors enable high performance. TR is the second-easiest type after AR, with Qwen3-VL-30B-Instruct leading at 60.82% at 32K. Temporal cues—session timestamps and explicit dates—provide retrieval anchors that compensate for length: Qwen3-VL-30B-Instruct retains 52.58% at 128K (−8.24%), and GLM-4.6V retains 43.81% at 128K from 53.09% at 32K (−9.28%). The TR subtypes show asymmetric length sensitivity: Duration Comparison (A/B selection, n = 91) is relatively stable across lengths, while Temporal Grounding (order ranking and date extraction, n = 103) degrades more sharply as clocks and calendars become harder to locate among filler. Memory-T1 achieves the highest TR accuracy of any system at 62.50% at 32K and 58.33% at 256K—its BM25 date-matching retrieval directly exploits the explicit temporal signals that other agents lose through caption-based storage, and its RL objective targets temporal retrieval specifically. MemAgent-7B also reaches 62.50% at 32K and 50.00% at 256K, while the multimodal agents (M3-Agent at 29.17%, M2A at 2.08% at 32K) fall far below, demonstrating that architecture-specific temporal reasoning capacity, not just backbone quality, drives TR performance.

Knowledge Update (KU): Current-state tracking through image chains. At 32K, Kimi-K2.5 leads at 50.86%, with Qwen3.5-122B at 49.14%, Gemini-3.1-Pro at 49.14%, and GPT-5.4 at 47.41%. Most open-weight models fall in the 30–43% range. At 128K, Gemini-3.1-Pro retains 46.03% (−3.11%), Kimi-K2.5 retains 48.59% (−2.27%), and GPT-5.4 retains 43.10% (−4.31%)—KU shows the flattest LVLM length degradation among the four answerable types, likely because the four-fact update chain forms a coherent temporal sequence that resists being scattered by filler. Open-weight LVLMs degrade more sharply: Qwen3.5-27B drops from 42.24% at 32K to 38.79% at 128K (−3.45%), and GLM-4.6V drops from 43.97% to 28.45% (−15.52%). KU is the only answerable type where agents reach competitive per-answer accuracy: MemAgent-7B reaches 41.38% at 32K, close to the LVLM median, because its RL training specifically targets state tracking. However, the multimodal agents (M2A at 0.00%, M3-Agent at 6.90%) collapse, confirming that visual compression destroys the fine-grained category discrimination (which fruit is this?) that KU requires.

Answer Refusal (AR): Epistemic calibration under filler pressure. AR is the easiest type at short contexts: GPT-5.4, Gemini-3.1-Pro, Claude Sonnet 4.5, Kimi-K2.5, and Qwen3-VL-235B-Instruct all reach 97.78% at 32K. However, AR shows the steepest context-driven decline in the open-weight LVLM family: GLM-4.6V drops from 93.33% at 32K to 30.00% at 128K (−63.33%), Qwen3-VL-30B-Thinking drops from 71.11% to 60.00% (−11.11%, the best retention in the open-weight tier), and Qwen3.5-122B drops from 88.89% to 83.33% (−5.56%, the flattest decline). The API models remain robust: GPT-5.4 retains 94.44% at 128K (−3.34%), and Gemini-3.1-Pro retains 96.67% (−1.08%). Agents show the most dramatic AR divergence: frozen-backbone agents preserve abstention (Mem0 at 77.27%, MemOS at 68.18%), while fine-tuned agents collapse—MemAgent-7B at 13.64%, Memory-T1 at 9.09%, M3-Agent at 13.64%, M2A at 22.73%. The backbone-matched comparison isolates the effect: direct Qwen3-VL-8B-Instruct scores 81.82% AR; the same backbone with M2A's fine-tuning scores 22.73%.

The Orthogonal Failure Mode Diagnosis: Length vs. Fidelity

Figure 4a (per-type accuracy vs. context length, with LVLM and agent averages plotted separately) provides the aggregate signature of the complementary failure modes:

LVLM degradation: From 32K to 128K, the average LVLM accuracy drops from ~48% to ~35% overall. IE loses ~20% (from ~46% to ~26%), KU loses ~12% (from ~35% to ~23%), TR drops from ~45% to ~38%, MSR appears flat near 25% due to a floor effect, and AR drops from ~75% to ~45%. The degradation is not uniform: Gemini-3.1-Pro loses only 2.11% overall, GPT-5.4 loses 3.16%, while Qwen3.5-122B loses 13.18% and Qwen3.5-9B loses 17.75%. The retention ratio (128K accuracy / 32K accuracy) correlates with model size at ρ = 0.62 (p = 0.002; Figure 18, Appendix G.5): larger models generally hold up better at 128K, but architectural family shifts the profile even at matched scale—Qwen3.5-122B loses more than twice as much as Qwen3-VL-235B (−13.18% vs. −5.57%) despite similar active-parameter budgets.
Agent stability: From 32K to 256K, the average agent accuracy varies by less than ±3% for six of seven systems. Mem0: 31.79% → 30.77% (−1.02%). MemOS: 30.26% → 29.74% (−0.52%). Memory-T1: 28.72% → 29.23% (+0.51%). MemAgent-7B is the exception, dropping from 32.82% to 24.62% (−8.20%), driven by its small backbone (Qwen2.5-7B) degrading on retrieval-heavy types at 256K. The stability is architectural—the retrieve-then-reason pipeline is length-invariant by construction—and not a floor effect, as evidenced by bootstrap confidence intervals of ±4.87–6.67% that preserve the agent clusters across lengths (Appendix G.2).

The two failure modes are orthogonal: LVLMs lose to context length (visual fidelity high, retrieval degrades as filler accumulates), while agents lose to lossy multimodal compression at storage time (length-stable, but visual fidelity collapses to caption-level gist). Figure 4b decomposes wrong answers at 128K by modality: on IE and KU, ~90% of errors fall in the Visual category (the model fails to locate or read the evidence image); on MSR, 73% of errors are Reasoning; on TR, errors split between Mixed and Reasoning. The oracle-retrieval diagnostic (§G.7) confirms that MSR is retrieval-bounded—when evidence sessions are supplied directly, MSR reaches 90–100%—so the Reasoning errors are downstream consequences of incomplete evidence collection rather than reasoning-capacity deficits.

Error Analysis: Retrieval Fails Before Reasoning

The wrong-answer taxonomy (§G.4, Figure 16) partitions errors into seven labels, grouped into two meta-categories: near-miss (evidence located before erring) and total-miss (no correct evidence anchor). At 32K across all LVLMs, 69.44% of wrong answers are near-miss and 30.57% are total-miss. The largest individual labels are grounding failure (29.66%, the model locates the evidence region but misreads the detail), unsupported answer (23.00%, the model produces content with no evidence anchor), and computation slip (21.32%, correct operands but wrong arithmetic, concentrated in MSR and TR subtypes with numeric answers).

From 32K to 128K, unsupported answer gains +10.23 percentage points in the error distribution, while grounding failure loses −5.38% and computation slip loses −4.82% (Figure 17). This is the quantitative signature of retrieval degradation: models at 128K increasingly fabricate answers rather than partially grounding them in located evidence. The shift is sharpest on MSR Arithmetic (+18.40% unsupported, −4.90% grounding) and TR Date Extraction (+15.10% unsupported, −10.20% grounding), the two subtypes where precise evidence location is most demanding.

The visual-modality breakdown (Figure 4b) shows that IE and KU errors are overwhelmingly Visual (89% and 75%, respectively)—the model fails to locate or read the evidence image, and once the image is reached, the answer is usually extracted correctly. MSR errors are dominated by Reasoning (73%) and TR errors by Mixed (42%) and Reasoning (44%). The per-type breakdown thus confirms that the primary bottleneck is cross-modal evidence retrieval, not comprehension. When a model finds the right image, it usually extracts the right answer; the problem is finding the right image among 20–138 distractor images in a growing conversation history.

Backbone-Matched Comparison: Pipeline Cost Quantified

The backbone-matched comparison isolates the architectural cost of the memory pipeline independently of backbone quality. M2A builds on Qwen3-VL-8B-Instruct, a current-generation 8B backbone. Direct inference of Qwen3-VL-8B-Instruct on the 195-question canonical subset achieves 49.18% overall at 32K; M2A on the same questions achieves 15.38%—a 33.80% deficit (Table 16, Appendix G.6). This deficit is the cost of the memory architecture: the dual-layer SQLite + SigLIP2 pipeline with embedding-based retrieval loses 33.80% of the backbone's standalone capability.

The backbone ablation for Mem0 (Table 21) quantifies how much backbone quality matters within a fixed architecture. Under the Mem0 FAISS architecture, swapping the default Qwen3-8B for gpt-4.1-mini lifts overall accuracy from 32.50% to 43.15% (+10.65%), while substituting Qwen2.5-7B drops it to 28.50% (−4.00%), for a total spread of 14.65% across three backbones. Within MemOS, replacing default Qwen3-8B with Qwen2.5-7B lifts accuracy from 34.00% to 36.50% (+2.50%). The backbone spread (14.65% for Mem0, 2.50% for MemOS) is substantially smaller than the 33.80% pipeline cost measured on M2A, confirming that pipeline architecture dominates backbone quality as the determinant of agent performance. All backbone variants preserve context-length invariance (±3% from 32K to 256K), confirming that flatness is an architectural property independent of backbone quality.

Cross-Type Correlation: The Five Abilities Are Largely Independent

The Spearman rank correlation matrix at 32K (Figure 5) shows that the five memory abilities are largely independent, with the strongest correlation between IE and KU (ρ = 0.65) and between KU and AR (ρ = 0.59). IE and KU's correlation reflects their shared need to locate a specific evidence image before extracting the answer. MSR shows weak correlations with IE (ρ = −0.19) and AR (ρ = −0.11), confirming that cross-session aggregation measures a distinct capability from single-session retrieval. TR correlates weakly with all other types (ρ range: −0.00 to 0.33). At 128K (Appendix G.5), the IE–KU correlation strengthens to 0.74 while MSR becomes more internally correlated (within-MSρ = 0.38 vs. 0.20 at 32K), reflecting a shared-failure artifact as MSR collapses toward the floor across many models.

The subtype-level correlation matrix (Figure 6) further decomposes this structure: IE Entity and IE PrevInfo form the only near-ceiling pair (ρ = 0.87 at 32K, 0.94 at 128K), reflecting their shared retrieval skill. MSR-internal correlation is weak at 32K (mean pairwise ρ = 0.20), and TR-internal correlation stays near zero at both contexts (ρ = 0.06), indicating that duration comparison and temporal grounding test distinct skills. This heterogeneity supports evaluating all five major types separately rather than reporting a single aggregate.

Context-Length Degradation: Monotonicity and Bidirectional Churn

Statistical validation of degradation monotonicity (Appendix G.5): the paper systematically scans 27 LVLMs and 7 agents across five types and two context transitions (32K → 64K, 64K → 128K), identifying 72 transitions where accuracy nominally increases—38 among LVLMs and 34 among agents. None survive Bonferroni correction (α/72 = 0.0007). For the 38 LVLM transitions, McNemar's exact test on paired per-question binary outcomes (same 789 questions at both lengths) finds that only one reaches p < 0.05 uncorrected: Qwen3-VL-2B-Thinking MSR at 64K → 128K (+5.59%, p = 0.022, 9 flip-to-correct vs. 1 flip-to-wrong). This is a degenerate-output artifact: 8 of 9 flip-to-correct questions hit the 2,048-token generation cap at 64K but produced normal outputs at 128K—stochastic recovery in a highly unstable model (60.8% MSR degenerate rate). All 34 agent anomalies are attributable to small per-type sample sizes (n = 22–61): a single question flip produces a 2–5% swing.

Bidirectional churn underlies apparent reversals (Appendix G.5): per-question tracking across 21 LVLMs with judge data at all three context lengths reveals that context transitions induce bidirectional churn. At each step, models simultaneously lose 60–150 questions they previously answered correctly and gain 30–80 new correct answers. The observed accuracy decay is the net of these two opposing flows, not a uniform loss of signal. Of questions that flip at 32K → 64K, 38.7% flip back at 64K → 128K. Context-robust models balance most of this churn: at 64K → 128K, Gemini-3.1-Pro flips ~71 questions in each direction (an ~18% churn rate), and Kimi-K2.5 flips ~83 in each direction; the slight asymmetry yields the small net overall drops (1pp for Gemini, 2pp for Kimi) rather than a true zero-sum reversal. This balanced replacement, rather than static resistance, explains the flat accuracy trajectories of the most length-robust models.

Hard questions stay hard across context lengths (Appendix G.5): the 280 questions that are hard at 32K (solved by <20% of models) average 9.07% accuracy at 32K and 8.52% at 128K, so expanding context does not unlock them. The difficulty floor reflects intrinsic skill requirements rather than a length artifact.

Ablation Studies and Robustness Checks

Cross-modality validation (§3.4, Table 3): Supplying evidence images enables near-ceiling accuracy; removing them causes collapse. The answerability test supplies each image-essential and image-supportive question (n = 634) with its full evidence (textual facts and evidence images), bypassing the haystack entirely. GPT-5.4 reaches 93.13% overall and Gemini-3.1-Pro 89.42%. When evidence images are removed (leaving only textual evidence facts), overall accuracy collapses to 1.74% for GPT-5.4 and 1.89% for Gemini-3.1-Pro—both below 2%. The per-type breakdown shows that GPT-5.4's IE accuracy drops from 94.31% to 0.41%, MSR from 100.00% to 0.00%, TR from 96.91% to 5.15%, and KU from 75.86% to 0.00%. The residual TR accuracy without images (5.15% for GPT-5.4, 6.19% for Gemini-3.1-Pro) reflects text-sufficient TR items where dates are explicitly stated rather than embedded in clock or calendar images. The convergence of two frontier proprietary models on near-identical collapses confirms that the questions are highly multimodal—the image is causal for the answer in nearly all cases.

MSR oracle-retrieval diagnostic (§G.7): MSR accuracy reaches 90–100% when ground-truth evidence sessions are supplied directly. GPT-5.4 reaches 100.00% and Gemini-3.1-Pro 90.21% on MSR under oracle conditions, confirming that cross-session aggregation is within frontier reasoning capacity. The gap between oracle-retrieval MSR (90–100%) and full-benchmark MSR (≤44.06%) therefore confirms that MSR is retrieval-bounded rather than reasoning-bounded: models can solve the aggregation task once evidence is located but fail to collect the required three to eight evidence sessions from the full conversation history. The ~10% residual gap for Gemini-3.1-Pro suggests that a small fraction of MSR items—likely the more complex arithmetic patterns—do challenge reasoning even with perfect evidence delivery.

PRM aggregation strategy: Not applicable. The paper does not train or evaluate process reward models; this ablation does not apply to MEMLENS.

PRM vs. ORM: Not applicable. The paper does not train outcome reward models for answer selection; evaluation is based on LLM-as-Judge accuracy, not verifier-based selection.

Revision model verifier choice: Not applicable. The paper evaluates memory agents and LVLMs, not revision models with verifier-based selection. The agent analysis (§4.3, Appendix G.6) examines where in the pipeline information is lost, but this is an error-attribution analysis, not a verifier ablation.

Revision history in verifier context: Not applicable. The paper does not use revision chains or sequential revision models.

Oracle vs. predicted difficulty bins: Not applicable. The paper does not use difficulty bins for compute-optimal allocation; all questions are evaluated uniformly across models and context lengths.

Majority voting for revisions: Not applicable. The paper does not use majority voting for answer selection.

ReST^EM revision model: Not applicable. The paper does not train or evaluate revision models using ReST^EM or similar iterative refinement methods.

Memory-agent backbone ablation (Table 21, Appendix G.6): Backbone quality shifts absolute performance but does not change the qualitative conclusion that pipeline architecture dominates. Under Mem0's FAISS architecture, accuracy ranges from 28.50% (Qwen2.5-7B) to 43.15% (gpt-4.1-mini), a spread of 14.65%. Under MemOS, accuracy ranges from 34.00% (Qwen3-8B, default) to 36.50% (Qwen2.5-7B), a counterintuitive increase of 2.50% for the older-generation backbone. Per-type profiles shift markedly across backbones: in Mem0, default Qwen3-8B reaches 77.27% AR while Qwen2.5-7B achieves perfect refusal (100%), indicating that the backbone's intrinsic calibration propagates through the memory pipeline independently of retrieval quality. All backbone variants preserve length invariance (±3% from 32K to 256K), confirming that flatness is an architectural property of the retrieve-then-reason pipeline, not a function of backbone quality.

Agent input-format adapters (Table 5, Appendix B): Adapter choice bounds the visual-information ceiling. The text-only agents (Mem0, MemOS, MemAgent-7B, Memory-T1) receive BLIP-2 captions in place of images, losing pixel-level visual information at storage time. M3-Agent receives per-session composite images, losing the original interleaved structure but retaining some pixel information. M2A and M3C receive original interleaved images at write time but retrieve embedding-based memory entries at query time—raw pixels are not available at answer time. Direct LVLMs attend over original pixels within the context window. The adapter-induced information loss is not uniform: IE accuracy for text-only agents (13.11% for Mem0) is dominated by caption coarseness (fine-grained counts, attributes, and spatial relations are lost), while M2A's 14.75% IE accuracy reflects embedding-retrieval noise rather than caption loss. The paper does not normalize these adapters because each adapter is integral to the system being evaluated, and the reported deficits relative to direct LVLMs are the correct comparison for understanding how currently deployable systems perform.

Session indistinguishability validation (§G.8): Post-hoc classifiers cannot separate evidence from haystack sessions. A TF-IDF logistic regression achieves 56.71% ± 1.20% accuracy (ROC-AUC 0.59), and a fine-tuned DeBERTa-v3-base achieves 57.92% ± 0.85% (ROC-AUC 0.55), both marginally above the 50% chance floor for balanced binary classification (Cohen's h < 0.16). The vocabulary ablation progressively removing the top K most discriminative n-grams (K = 20, 50, 100, 200, 500, 1,000) leaves accuracy in a 54.77–56.71% band, indicating that the weak signal is diffuse rather than concentrated in identifiable stylistic markers. This confirms that the retrieval difficulty reported in §4.2 is not inflated by surface-level cues that models could exploit to shortcut evidence location.

Coverage vs. per-answer accuracy decomposition (Table 17, Appendix G.3): Overall accuracy conflates retrieval ability with calibration. GPT-5.4 reaches 73.15% per-answer accuracy on attempted questions but refuses 36.05% of answerable items, suppressing its headline score. Qwen3.5-122B attempts 87.70% but achieves only 62.32% per-answer accuracy. Claude Sonnet 4.5 reaches 57.51% per-answer accuracy behind a 50.50% refusal rate—competitive with higher-ranked models on attempted items but penalized by calibration-driven abstention. The coverage–accuracy trade-off is type-dependent: MSR has the lowest coverage across models (models are least confident on cross-session aggregation), while TR has the highest (temporal anchors increase confidence). At 128K, most models answer more questions than at 32K (coverage rises) despite lower accuracy, confirming that false confidence, not increased caution, is the dominant dynamic at longer contexts.

Deterministic typed-accuracy audit (Table 12, Appendix E.2): The LLM judge systematically over-credits by ~5% on closed-form answers, but leaderboard rankings are preserved. Across 12,234 deterministic items from 34 evaluation rosters at 32K, rule-based rescoring achieves 93.6% agreement with the LLM judge. The residual is leniency-biased: 5.4% judge-false-positives (deterministically wrong answers credited as correct) vs. 1.0% judge-false-negatives (correct answers rejected). The largest leniency channels are TR Order Ranking (12.6% J-FP, partial credit on near-correct tuples) and TR Date Extraction (8.6% J-FP, format-flexible date matching). The Spearman rank correlation between LLM and deterministic scores at the model level is ρ = 0.78, the top-5 models are preserved under both metrics, and the top-10 sets overlap on 7 of 10 entries. The three rosters that drop out—GLM-4.6V, Qwen3-VL-235B-Thinking, and Mem0 (GPT-4.1-mini backbone)—all produce verbose justifications that the LLM judge credits but rule-based checks reject.

Agent retrieval attribution (Figure 19, Appendix G.6): Agent failure modes split into retrieval-dominated and comprehension-dominated. For the three agents with retrieval logs (Mem0, Memory-T1, and M3C), errors are decomposed by whether evidence was retrieved (recall ≥ 0.5) before the error occurred. M3C is retrieval-bottlenecked: 78.1% of its errors occur when the LoRA session retriever never surfaces the relevant evidence (mean recall 0.26). Mem0 and Memory-T1 are comprehension-dominated: they retrieve evidence at high recall (0.82 to 0.89), yet 87–95% of errors occur after successful retrieval, indicating that the backbone cannot reason over the surfaced content. Retrieval-bottlenecked agents need better retrieval; comprehension-bottlenecked agents need stronger reading comprehension—the two failure modes call for different interventions.

Common agent failure subset (Appendix G.6): 52% of answerable questions are answered incorrectly by all four agents with per-question logs, despite architecturally disjoint retrieval pipelines. The all-wrong set is dominated by KU (69% of agent-wrong questions) and MSR (69%), with TR least affected (21%). The four agents span ColPali (M3-Agent), LoRA session retrieval (M3C), dual-layer SQLite (M2A), and BM25 (Memory-T1)—diverse retrieval paradigms that converge on the same hard questions, suggesting a shared bottleneck of current sub-10B retrieval-based memory systems rather than architecture-specific weaknesses.

Context-length degradation by decoding mode (Table 20, Appendix G.5): Thinking-mode degrades AR more steeply than Instruct mode. At 32K, Qwen3-VL-2B-Thinking reaches 87.8% AR; at 128K, it collapses to 14.4% (−73.3%), with truncated reasoning traces producing substantive answers instead of refusals. Qwen3-VL-2B-Instruct drops from 72.2% to 47.8% (−24.4%)—a smaller absolute decline but from a lower starting point. The pattern holds across the Qwen3-VL family: Thinking mode consistently shows larger absolute AR drops than Instruct mode, suggesting that explicit reasoning chains are more vulnerable to filler-induced hallucination than direct answer generation.

Critical Assessment

The experiments provide strong support for the paper's central diagnostic claims—that solving MEMLENS genuinely requires visual evidence, that LVLMs and agents exhibit complementary failure modes, that multi-session reasoning is the shared capability ceiling, and that memory-agent post-training erodes abstention. However, several claims are narrower than they might initially appear, and the experimental design has genuine limitations that bound the conclusions' scope.

Claim: Solving MEMLENS requires visual evidence (cross-modality ablation, §3.4, Table 3). Strongly supported, with an important nuance. The collapse from ~90% to below 2% when evidence images are removed is decisive: these questions are causally dependent on images. However, the ablation removes images from the evidence set but keeps the textual evidence facts—it tests whether images are necessary, not whether images are sufficient. The converse test (removing text while keeping images) is not performed, so the paper demonstrates cross-modal dependency but not cross-modal sufficiency—some questions might require both image and text, which is the intended design but not empirically verified through symmetric ablation. The 80.4% of questions whose evidence includes images is reported, but the 19.6% that are text-sufficient (all AR items plus a subset of MSR) are not ablated—this is correct because those items are, by construction, not image-essential, but it means the "requires visual evidence" claim applies to ~80% of the benchmark, not 100%. The convergence of two frontier models on near-identical collapse magnitudes (GPT-5.4: −91.39%, Gemini-3.1-Pro: −87.53%) provides strong cross-model validation that the dependency is structural rather than model-specific.

Claim: Long-context LVLMs degrade as context grows; memory agents are length-stable (Figure 4a, §4.3). Supported, but the LVLM degradation characterization is coarse—the degradation rate varies enormously by model and type, and the paper does not provide confidence intervals on the degradation curves. The aggregate LVLM curve (Figure 4a, solid line) masks that Gemini-3.1-Pro degrades only 2.11% from 32K to 128K while Qwen3.5-9B degrades 17.75%. The statement "LVLMs degrade" is true on average but does not characterize the distribution: a small number of very large, well-trained LVLMs degrade minimally, while open-weight models at smaller scales degrade sharply. The degradation is strongly type-dependent (AR drops ~30% for open-weight models, IE drops ~20%, MSR is flat due to a floor effect), so the aggregate curve averages over qualitatively different degradation dynamics. The paper acknowledges this through per-type panels (Figure 2), but the headline "LVLMs degrade as conversations grow" should be read as "most LVLMs degrade on most abilities," with the qualifier that degradation magnitude varies from negligible (Gemini-3.1-Pro on KU: −3.11%) to catastrophic (GLM-4.6V on AR: −63.33%). The agent length-stability claim is robust: bootstrapped confidence intervals confirm that the ±7% range is not artifactually tight, and the Jaccard-overlap analysis confirms that correctly answered questions are genuinely preserved across lengths rather than randomly re-drawn.

A missing experiment: the paper evaluates LVLMs at 32K, 64K, and 128K, but not at 256K—the length at which agents are most stable. This truncates the comparison at the point where the LVLM–agent divergence would be most pronounced, and the paper cannot empirically validate whether LVLMs would continue to degrade or whether some would stabilize at 128K+. The reason is practical (most open-weight LVLMs do not support 256K), but the result is that the "LVLMs degrade" claim is validated only up to 128K, and the paper cannot say whether 128K is a linear regime or a pre-asymptotic one.

Claim: Memory agents lose visual fidelity under storage-time compression (§4.3). Supported by a convergence of evidence, but the mechanism is inferred rather than directly measured. The LVLM–agent gap on visually grounded types (IE, KU) is large and consistent across agents: on IE, the best agent (M3-Agent at 18.03%) is 56.36% below the best LVLM (Qwen3.5-122B at 74.39%) at 32K. On KU, the best agent (MemAgent-7B at 41.38%) is 9.48% below the best LVLM (Kimi-K2.5 at 50.86%). The error decomposition (Figure 4b) shows ~90% of LVLM IE errors are visual, and the oracle-retrieval diagnostic shows that visual errors disappear when evidence is supplied directly—consistent with retrieval difficulty, not visual comprehension difficulty. However, the paper does not directly measure what information is lost in the captioning process for text-only agents, or what information survives in embedding retrieval for multimodal agents. A direct comparison of BLIP-2 caption accuracy vs. original image content on the evidence images, or a per-image analysis of what visual features (color, count, text, spatial relations) are preserved in captions vs. embeddings, would strengthen the "lossy compression" claim from a correlational observation to a mechanistic explanation. The backbone-matched M2A comparison quantifies the pipeline cost (33.80%) but does not decompose it into storage-time compression loss, retrieval-miss loss, and reasoning-error loss—the retrieval attribution (Figure 19) provides this decomposition for three agents, but not for M2A specifically.

Claim: Multi-session reasoning caps most systems below 30% (§4.2, Figure 2, MSR panel). Well-supported as an empirical observation, but the interpretation that this is the "shared capability ceiling" rests on the oracle-retrieval diagnostic. The diagnostic (GPT-5.4: 100%, Gemini-3.1-Pro: 90.21%) is compelling, but it is performed only on two models, on the subset of MSR questions that involve image evidence, and the gap between GPT-5.4's 100% and Gemini-3.1-Pro's 90.21% is not explained—the paper does not examine which MSR items Gemini-3.1-Pro fails on under oracle conditions, which would reveal whether the residual gap is reasoning-bottlenecked (the harder MSR-Arithmetic items) or still retrieval-bottlenecked in some edge case. Extending the oracle diagnostic to a broader set of models (particularly mid-tier open-weight LVLMs) and to all MSR subtypes would strengthen the "retrieval-bounded, not reasoning-bounded" claim, since it could be that only frontier models achieve near-ceiling MSR under oracle conditions, and mid-tier models genuinely cannot aggregate across sessions even with perfect evidence.

A related missing experiment: the oracle diagnostic supplies evidence sessions with their images. An ablation that supplies evidence sessions with images removed (text-only oracle) would test whether the retrieval bottleneck is specifically visual (locating images) or general (locating any evidence session). If models achieve near-ceiling MSR with text-only evidence, the bottleneck is visual retrieval; if they still struggle, the bottleneck is cross-session evidence collection regardless of modality.

Claim: Post-training on memory agent backbones weakens abstention (§4.3). Strongly supported by the backbone-matched comparison, but the mechanism is inferred—the paper does not examine the training objectives of the fine-tuned agents to confirm that abstention receives no reward signal. The M2A vs. direct Qwen3-VL-8B-Instruct comparison (22.73% vs. 81.82% AR) is a clean matched-backbone contrast, and the cross-agent pattern (frozen-backbone agents at 68–77% AR, fine-tuned agents at 9–22% AR) is consistent with the stated mechanism. However, the paper does not analyze the RL reward functions or SFT training data of the fine-tuned agents to verify that abstention is absent. MemAgent-7B's training objective might explicitly penalize refusal, or it might simply not reward it—the paper cannot distinguish these without access to the training pipeline. The backbone ablation (Table 21) shows that AR behavior transfers from the backbone: Mem0 with Qwen3-8B achieves 77.27% AR, while Mem0 with Qwen2.5-7B achieves 100%—the Qwen2.5-7B backbone is intrinsically more calibrated on refusal, and this propagates through the frozen memory architecture. This suggests that the "erosion" is not universal—a frozen-backbone agent can inherit strong abstention from its backbone—and that the degradation is specific to fine-tuning, not inherent to the memory-agent paradigm.

Claim: The five memory abilities are largely independent (Figure 5, §4.3). Partially supported, with weak evidence on MSR. The strongest correlations (IE–KU at ρ = 0.65, KU–AR at ρ = 0.59) are moderate, and the weak correlations (MSR with IE at ρ = −0.19, TR with AR at ρ = −0.00) support independence. However, the correlation analysis is across models, not across questions: it measures whether models that perform well on one ability also perform well on another, not whether the abilities are separable at the question-construction level. Two abilities could be structurally distinct (in their generative pipeline) but test a shared underlying capability (e.g., retrieval of multimodal evidence) that produces correlated model performance. The IE–KU correlation is consistent with a shared retrieval-capacity factor, which would mean the abilities are not "independent" in the sense of measuring fully disjoint capabilities—rather, they share a retrieval component but differ in what is done with the retrieved evidence (single-session extraction vs. temporal-state tracking). The near-zero MSR–IE correlation at 32K is the strongest evidence for independence, but this correlation becomes moderately positive at 128K (ρ = 0.31 in the full matrix, Appendix G.5), suggesting that as models degrade, a general "length-robustness" factor begins to dominate ability-specific factors.

Missing baselines and evaluation gaps. The paper evaluates 27 LVLMs and 7 agents, but several gaps remain. (1) No text-only LVLM baselines are evaluated on the full benchmark—a comparison of text-only models given BLIP-2 captions (matching the text-only agent input) against the LVLMs would isolate the contribution of visual grounding separately from model capacity. (2) The oracle-retrieval diagnostic is performed only on two frontier models and only for MSR—extending it to all answerable types and a broader model set would strengthen every per-type "retrieval-bounded vs. reasoning-bounded" claim. (3) No hybrid system (long-context LVLM + retrieval augmentation) is evaluated, so the paper's recommendation of hybrid architectures (§5, Conclusion) is motivated by the failure-mode diagnosis but not empirically validated. (4) The agent evaluation is on a 195-question subset, and while per-type proportions are preserved (differences < 0.2%), the subset is small enough that per-type n counts range from 22 (AR) to 61 (IE), limiting the statistical power for agent-specific claims. (5) The construction pipeline uses Gemini-3-Pro as the question generator and Gemini-3.1-Pro as the top-evaluated model—different versions in the same family. The paper argues that the oracle-retrieval diagnostic (GPT-5.4 leads Gemini-3.1-Pro under oracle conditions) rules out generator-familiarity bias, but a direct ablation where an independent generator (e.g., Claude Sonnet 4.5) regenerates a subset of questions and LVLMs are re-ranked on those items would close this concern conclusively.

Generalizability boundaries. All experiments are on a single benchmark (MEMLENS), with conversations generated by a specific pipeline (GPT-5.1 user, Gemini-3-Pro assistant) and questions targeting a specific distribution of visual skills (entity recognition, document reading, temporal–visual integration). The paper does not claim generalizability beyond this setting, but the findings' scope should be explicitly noted: the conclusion that memory agents lose visual fidelity depends on the BLIP-2 captioning pipeline used for text-only agents and the embedding-based retrieval used for multimodal agents—stronger captioners (e.g., GPT-4V-based description) or pixel-level retrieval (e.g., ColPali-style late interaction) might close the visual-fidelity gap substantially. The paper evaluates ColPali-based retrieval in M3-Agent, which scores 18.03% IE at 32K (vs. 74.39% for the best LVLM), suggesting that even sophisticated visual retrieval does not recover LVLM-level visual grounding, but this is a single ColPali configuration on a specific backbone—not an exhaustive test of pixel-level retrieval architectures. The conclusion that "neither approach alone solves the task" (§5) should therefore be read as a statement about currently deployed systems under the evaluated configurations, not an architectural impossibility proof.

6. Limitations and Trade-offs

6.1 The Benchmark Is Constructed from Synthetic Conversations — Naturalness Is Enforced but Not Guaranteed

The assumption or constraint. Every haystack session and evidence session in MEMLENS is generated by a two-model LLM simulation (GPT-5.1 as user, Gemini-3-Pro as assistant), with only the filler sessions drawn from real user–AI conversations (ShareGPT and UltraChat). The paper explicitly acknowledges this:

"characterizing the gap to the distribution of real long-term human–assistant interactions remains an open research question" (Appendix H)

The human-in-the-loop review pipeline (three rounds, reducing ~20K candidates to 789 questions; §3.3, §E.1) audits conversational naturalness — colloquial phrasing, turn-taking coherence, and indirect embedding of factual content — and returns stilted sessions to the generator for revision until annotators accept them as plausible exchanges. However, this review enforces perceived naturalness, not distributional match to real conversations. The session-indistinguishability validation (§G.8) shows that evidence and haystack sessions are indistinguishable to post-hoc classifiers (TF-IDF: 56.71%, DeBERTa: 57.92% — marginally above chance), but this proves internal stylistic consistency within the synthetic distribution, not external validity against real human–assistant interactions.

The consequence. Models trained on or exposed to synthetic conversational data during development may develop retrieval strategies that exploit regularities specific to LLM-generated dialogue — consistent turn lengths, predictable assistant response structures (acknowledgment → information → knowledge-oriented follow-up), or limited colloquial variation — that do not transfer to real-world conversations. A system that achieves high MEMLENS scores might underperform on actual multi-session interactions where turns are more fragmented, topics drift organically rather than following ontology-driven coherence, and users employ genuinely idiosyncratic language. The benchmark's diagnostic value — identifying which memory ability fails — would transfer (retrieval is retrieval), but the absolute difficulty calibration might not: real conversations could be easier (more distinctive vocabulary makes evidence location simpler) or harder (less structured topic flow makes semantic similarity between evidence and haystack more confusing).

What evidence exists in the paper. The paper provides evidence of internal consistency (the indistinguishability validation) and construct validity (the cross-modality ablation showing questions require visual evidence), but no evidence of external validity against real conversation distributions. No experiment compares model performance on MEMLENS against performance on a corpus of real multi-session multimodal interactions (which does not currently exist as a standardized evaluation — this is partly why MEMLENS was created). The human review targets naturalness at the turn level but does not benchmark the resulting dialogues against a held-out set of real conversations for distributional metrics (perplexity under a conversational LM, turn-length distributions, lexical diversity indices).

Mitigation status. The paper acknowledges this limitation explicitly (Appendix H) and frames it as an open research question — characterizing the gap between synthetic and natural multimodal conversations. The inclusion of ShareGPT and UltraChat filler sessions is a partial mitigation, embedding the synthetic content in authentic conversational frames, but the evidence sessions that carry the memory challenge are themselves synthetic. The paper does not propose or validate a method for measuring or closing the distributional gap. For practitioners, this means MEMLENS scores should be interpreted as a controlled diagnostic for relative comparison between architectures under uniform construction, not as an absolute predictor of deployed memory quality in production conversational assistants.

6.2 The Difficulty-Estimation Analogue Does Not Exist — There Is No Adaptive Allocation Mechanism

The assumption or constraint. MEMLENS evaluates all questions uniformly across all models and context lengths — every model receives all 789 questions at each of the four context lengths, and the reported accuracy is a flat average. There is no mechanism for difficulty estimation before inference, no adaptive budget allocation, and no routing of hard questions to a different model or pipeline. This is by design — MEMLENS is a benchmark, not a system — but it means the evaluation protocol assumes that all questions are worth the same compute, which is not how a deployed memory system would operate.

The paper does not claim to provide difficulty estimation or adaptive allocation. The construction pipeline does not assign difficulty labels to questions, and the evaluation protocol does not stratify results by question difficulty (beyond the five-type breakdown, which separates ability but not within-type difficulty). The paper's finding that "hard questions stay hard across context lengths" (Appendix G.5 — the 280 questions solved by <20% of models at 32K average 9.07% and remain at 8.52% at 128K) is a post-hoc observation, not a feature used during evaluation.

The consequence. A deployment-focused reader cannot use MEMLENS results to answer questions like: "If I have a fixed inference budget, should I spend more compute on medium-difficulty questions where retrieval helps, or save it for easy questions where a single pass suffices?" or "At what context length should I escalate from my on-device small LVLM to a cloud-based large LVLM?" These are the practical questions that a compute-optimal framework would answer, but MEMLENS provides only the uniform-length performance curves — it does not tell the practitioner how to use those curves to make allocation decisions.

The absence of difficulty estimation also means that the reported degradation curves (Figure 4a) are averages over questions of heterogeneous intrinsic difficulty. The ~20% IE drop from 32K to 128K might be driven entirely by a subset of questions whose evidence images are visually confusable with haystack images, while other IE questions are length-invariant. Without difficulty stratification, the practitioner cannot distinguish between "this model degrades on IE because it loses evidence at long contexts" (a general failure) and "this model degrades on IE because the easiest IE questions are already solved at 32K, and the remaining hard ones are disproportionately affected by length" (a compositional effect).

What evidence exists in the paper. The per-type accuracy curves (Figure 2) and the per-type error decomposition (Figure 4b) provide some stratification, but these are at the ability level, not the question-difficulty level. The observation that 280 questions are "hard" (solved by <20% of models) and remain hard across lengths (Appendix G.5) confirms that difficulty is a stable property of questions, but this analysis is retrospective — the benchmark itself does not use difficulty labels during evaluation, and the paper does not propose a method for estimating difficulty without ground-truth labels. The oracle-retrieval diagnostic (§G.7) shows that MSR questions are retrieval-bounded, which implies that their difficulty is determined by evidence-location complexity, but this is an existence proof, not a calibrated difficulty estimator.

Mitigation status. The paper does not address this limitation directly. It does not propose a difficulty-estimation method, and it does not frame the absence of adaptive allocation as a limitation — because MEMLENS is a benchmark, not a system, the uniform evaluation is appropriate for its diagnostic purpose. However, for practitioners who want to use the benchmark results to design memory systems, this limitation is consequential: the benchmark tells you that models fail on MSR, but not which MSR questions are failing, or whether a mixed strategy (skip hard MSR, invest compute in medium MSR) would improve practical utility. This is a natural difference in scope between a benchmark paper and a systems paper, but it is worth flagging because the paper's conclusions — "neither approach alone solves the task" and "motivating hybrid architectures" — implicitly assume that improved architectures will raise performance on all questions, rather than primarily on the medium-difficulty subset where incremental gains are most achievable.

6.3 The Generalization Bounds Are Narrow — Single Task Domain, Single Model Distribution, Unclear Transfer to Non-Symbolic Domains

The assumption or constraint. All experiments are conducted on a single benchmark (MEMLENS) whose questions target mathematical-style reasoning and fact extraction from synthetic multimodal conversations. The questions involve short-answer formats (entity names, prices, dates, counts, yes/no, ranked orderings) with ground-truth answers that can be deterministically evaluated. The visual content covers natural photographs, text-rich documents, and digital interfaces — but all within the context of conversational memory, not document-grounded QA, video understanding, or interactive task completion.

The evaluated LVLMs span a wide range — from 2B to 1T+ parameters, across six model families — but they are all instruction-tuned vision-language models trained on similar data mixtures (natural images, documents, screenshots, interleaved text–image sequences). The memory agents are even narrower: four of seven use text-only backbones with BLIP-2 captions, two use Qwen-family VL backbones, and one uses a Qwen VL backbone with ColPali retrieval. The paper does not claim generalizability beyond the evaluated setting:

"characterizing the gap to the distribution of real long-term human–assistant interactions remains an open research question" (Appendix H)

The consequence. Three distinct generalization gaps emerge, each with practical consequences for deployment decisions:

Domain gap — conversational memory vs. document memory vs. video memory. The finding that LVLMs degrade with context length while agents are length-stable but lose visual fidelity is specific to multi-session conversations where evidence is distributed across timestamped sessions with topical coherence. In a document-retrieval setting (e.g., MMLongBench, where evidence is a paragraph embedded in a long document), LVLMs might show better length stability because document structure provides retrieval cues (headers, sections, page breaks) absent from multi-turn dialogue. In a video-understanding setting, the temporal structure is continuous rather than session-discrete, and the nature of "memory" shifts from "what did the user say in session 3?" to "what happened at timestamp 2:15?" The orthogonal-failure-mode diagnosis (§4.3) might not transfer to these domains — it is plausible that in document settings, retrieval-based agents would approach LVLM-level visual fidelity because ColPali-style visual retrieval is more effective on document pages than on synthetic dialogue images, narrowing or eliminating the gap that MEMLENS reveals.

Model-family gap — these findings are for current-generation instruction-tuned VL backbones. The paper evaluates 34 systems, but they all belong to the instruction-tuned generation of VL models. A fundamentally different architecture — e.g., a model with native streaming memory, a model trained from scratch on interleaved multimodal sequences with explicit "recall" objectives, or a model with a dedicated episodic-memory module — might exhibit qualitatively different scaling behavior. The paper's recommendation of hybrid architectures (long-context attention + structured multimodal retrieval) is motivated by the failure-mode diagnosis of current systems, but it is possible that a sufficiently large long-context model with better attention mechanisms (sliding-window attention, sparse attention, memory-augmented attention) would degrade less at 256K+ than the current 128K curves suggest, making hybrid architectures unnecessary. The paper cannot rule this out because it evaluates only current architectures at their current size limits.

Task-format gap — MEMLENS questions have closed-form, evaluable answers. In open-ended dialogue (e.g., a user asking "What do you remember about my trip to Paris?"), the answer is a narrative summary, not a specific date or price. The retrieval challenge (locating Paris-related sessions) is similar, but the evaluation challenge is fundamentally different — LLM-as-Judge scoring of free-form narratives is much less reliable than scoring a date string. The paper's AR findings (models hallucinate rather than refuse) would likely be more severe in open-ended settings, where a hallucinated plausible detail ("You visited the Eiffel Tower on Tuesday") is harder to detect than an incorrectly extracted date. Conversely, the MSR finding that cross-session aggregation is retrieval-bounded might be less severe in open-ended settings, where approximate retrieval (finding "most" Paris sessions) suffices for a reasonable summary, as opposed to exact retrieval required for a precise sum or count. The paper's per-type error taxonomy (§G.4) is specific to closed-form answers and does not characterize what errors would look like in open-ended generation.

What evidence exists in the paper. None directly. The paper does not evaluate MEMLENS-style questions on a document benchmark, does not evaluate on a video memory benchmark, does not evaluate text-only LVLMs for comparison, and does not include open-ended generation questions. The cross-type correlation analysis (Figure 5) provides evidence that the five abilities are distinct within the closed-form domain — findings on IE do not predict findings on MSR — but this is evidence for within-benchmark independence, not cross-domain transfer.

Mitigation status. The paper does not claim generalizability beyond the evaluated setting (Appendix H frames this explicitly), and the construction of a single, comprehensive benchmark for a previously unevaluated capability is a legitimate contribution that does not require demonstrating cross-domain transfer. However, the paper's conclusions — particularly the recommendation of hybrid architectures — are stated as general architectural insights ("These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval," §5), which implies a broader validity than the experimental scope supports. A practitioner deploying a memory system for document-grounded QA should not assume that the LVLM-length-degradation finding transfers to their domain without evaluating on a document-specific memory benchmark (which does not currently exist for conversational-style queries over documents — this is a gap MEMLENS does not fill).

6.4 The Construction Pipeline Uses a Proprietary Generator from the Same Family as the Top-Evaluated Model — Intra-Family Familiarity Cannot Be Ruled Out

The assumption or constraint. The question-generation pipeline uses Gemini-3-Pro as the LLM that produces evaluation questions (§3.2, §C.4). The top-evaluated model in the leaderboard is Gemini-3.1-Pro (§4.2, Table 13). These are different versions (3-Pro vs. 3.1-Pro) within the same Gemini-3 model family. The paper assumes that the construction constraints — each question is conditioned on a fixed (entity, image, abstracted paragraph) triple under a deterministic prompt template, and is then filtered by automated text-leakage checks and human review — are sufficient to prevent the generator's style or content preferences from conferring an advantage to models from the same family.

The paper argues against this concern indirectly:

"On the answerability test with full evidence supplied (Table 3), GPT-5.4 reaches 93.13% overall versus 89.42% for Gemini-3.1-Pro; on the MSR cross-modality oracle (§G.7), GPT-5.4 reaches 100.00% versus 90.21% for Gemini-3.1-Pro. Under conditions where retrieval is bypassed — the conditions in which generator-style familiarity would most plausibly manifest — Gemini-3.1-Pro does not lead, so the 128K full-benchmark inversion is more consistent with retrieval-robustness than with intra-family familiarity bias." (Appendix H)

The consequence. If generator familiarity does confer an advantage, the effect would be subtle and difficult to detect without a controlled variable-generator experiment. It would not manifest as Gemini-3.1-Pro answering questions that other models find incomprehensible — the questions are in natural English and the answer format is straightforward. Instead, it could manifest as:

Semantic priming: Gemini-3-Pro generates evidence facts and question phrasings that are slightly more aligned with Gemini-3.1-Pro's internal representations — not in vocabulary (which is filtered for entity leakage), but in the way facts are structured (e.g., temporal ordering conventions, spatial description patterns, abstraction choices) that Gemini-3.1-Pro's training might recognize more efficiently than other models' training.
Retrieval cue alignment: The entity-abstraction phrases ("the bridge shown in <image>") are sampled from a dictionary of ~170 entries (§C.4), but the choice of which paraphrase to use, and the exact placement of the <image> token relative to the surrounding text, might reflect Gemini-3-Pro's own language modeling preferences — preferences that Gemini-3.1-Pro shares through common pretraining.
Failure mode similarity: If Gemini-3-Pro tends to generate evidence facts that are embedded in a particular conversational position (e.g., in the assistant's follow-up rather than the user's main message), and if Gemini-3.1-Pro has learned to attend to that position through shared training data, it would have a retrieval advantage that is not "familiarity" with the generator per se but with the generator's distribution.

The consequence of such an effect would be to inflate Gemini-3.1-Pro's apparent retrieval-robustness relative to models from other families (GPT-5.4, Claude Sonnet 4.5, Qwen3.5, Kimi-K2.5), making the "orthogonal failure mode" diagnosis appear more symmetric than it actually is. If the effect exists, the true LVLM degradation might be worse for non-Gemini models than reported, and the Gemini-3.1-Pro advantage at 128K might partly reflect a benchmark-specific edge rather than a general architectural superiority.

What evidence exists in the paper. The two oracle-retrieval diagnostics (Table 3, §G.7) provide indirect evidence against this concern. Under conditions where retrieval is bypassed — the model receives the ground-truth evidence directly — GPT-5.4 outperforms Gemini-3.1-Pro on both overall answerability (93.13% vs. 89.42%) and MSR (100.00% vs. 90.21%). This inversion suggests that Gemini-3.1-Pro's 128K advantage is retrieval-driven rather than comprehension-driven: when comprehension is tested in isolation, the cross-family model (GPT-5.4) is stronger. However, this is an indirect argument — it shows that the comprehension advantage is not Gemini-specific, but it does not show that the retrieval advantage is free of generator-familiarity confound. A retrieval advantage could still reflect subtle cue alignment even if comprehension is family-neutral.

The paper does not run the definitive experiment: regenerate a stratified subset of questions using an independent generator (e.g., Claude Sonnet 4.5 or GPT-5.4 with the same entity-abstraction and evidence-fact constraints) and re-rank the LVLMs on those questions. The paper acknowledges this gap:

"We did not, however, run a direct test in which an independent generator (e.g., Claude-Sonnet-4.5, GPT-5.4) regenerates a stratified subset of questions on the same input triples and the top LVLMs are re-ranked on those items; this targeted ablation would isolate any residual generator-familiarity confound and is left to future work." (Appendix H)

Mitigation status. The paper is transparent about this concern (Appendix H, "Question generator and test-taker overlap") and provides the oracle-retrieval evidence as a partial defense. The acknowledgment that the definitive ablation is left to future work is appropriate. For a practitioner reading the leaderboard, the practical implication is to treat the relative rankings of non-Gemini models (GPT-5.4, Claude Sonnet 4.5, Qwen3.5-122B, Kimi-K2.5) as robust, since any generator-familiarity effect would not apply to them, and to interpret the Gemini-3.1-Pro vs. GPT-5.4 gap at 128K cautiously — the direction (Gemini better at retrieval) is likely correct (consistent with Gemini-3.1-Pro's known strength in long-context retrieval), but the magnitude might be inflated.

6.5 Memory Agent Evaluation Is on a 195-Question Subset with Architecture-Specific Input Adapters — Agent Rankings Are Coarse and Adapter-Bound

The assumption or constraint. All seven memory-augmented agents are evaluated on a 195-question canonical subset (§G.2) rather than the full 789-question benchmark, because agent pipelines are substantially slower than direct LVLM inference (M2A is ~60× slower per question; Appendix B). The subset preserves per-type proportions to within 0.2 percentage points (Table 15), and bootstrap confidence intervals (95% CI: ±4.87–6.67% at 32K, ±4.87–6.67% at 128K; Appendix G.2) confirm that the text-only and multimodal agent clusters are statistically separable at these sample sizes.

Additionally, each agent receives a different input format (Table 5, Appendix B): text-only agents (Mem0, MemOS, MemAgent-7B, Memory-T1) receive BLIP-2 captions in place of images; M3-Agent receives per-session composite images rendered to fit its video-LVLM backbone; M2A and M3C receive original interleaved images at write time but retrieve embeddings at answer time. The paper explicitly does not normalize these input formats:

"We do not normalize this asymmetry because the adapter is part of the system being evaluated — released checkpoints assume the input format their authors trained on, and any uniform substitute would either degrade architectures that depend on caption-only memory or block agents whose backbones cannot accept interleaved input." (Table 5 caption)

The consequence. Two distinct limitations follow from the evaluation design, each affecting the interpretation of agent results:

The subset size limits the granularity of agent comparisons and per-type claims. With n = 195 overall, per-type n counts are small: n = 61 for IE, n = 35 for MSR, n = 48 for TR, n = 29 for KU, n = 22 for AR. A single question flip changes per-type accuracy by 1.6% (IE), 2.9% (MSR), 2.1% (TR), 3.4% (KU), or 4.5% (AR). This means:

The paper's claim that "six of seven agents stay within ±7% from 32K to 256K" (§4.3) cannot distinguish genuine 3–5% fluctuations from sampling noise — a 2–4 question swing driven by stochastic backbone output could produce an apparent 3–5% accuracy change. The bootstrap confidence intervals (±4.87–6.67%) bound this uncertainty at the overall level, but the per-type intervals would be wider and are not reported.
The paper's claim that "on MSR, the leaderboard inverts: Memory-T1 reaches 62.50% TR accuracy vs. Qwen3.5-122B at 51.55%" is based on n = 48 for TR on the agent subset. On the full benchmark (n = 194 for TR), the LVLM ordering might shift, and the agent–LVLM comparison on the subset might not replicate to the full set (the paper reports Spearman ρ = 0.94 between subset and full-benchmark rankings for six direct LVLMs at 32K, but this is at the model level, not the per-type level; Appendix G.2).
The "52% of answerable questions are answered incorrectly by all four agents with per-question logs" finding is based on the four agents that have per-question identifiers (M3-Agent, M3C, M2A, Memory-T1; Appendix G.6). Mem0, MemOS, and MemAgent-7B are excluded because their evaluation outputs lack per-question alignment, reducing an already small sample. The finding is suggestive but cannot distinguish whether the all-wrong set is driven by question difficulty (the same questions would defeat any agent) or by the specific configurations of these four agents (different configurations might answer some of those questions correctly).

The adapter asymmetry conflates architecture quality with input-format suitability. The paper's central agent finding — that memory agents lose visual fidelity under storage-time compression — is supported by the large LVLM–agent gap on IE and KU. However, it is unclear how much of this gap is due to lossy compression (the claimed mechanism) vs. the specific input format each agent uses:

Text-only agents (Mem0, MemOS, MemAgent-7B, Memory-T1): These systems never see pixels. Their performance on visually grounded types (IE, KU) is bounded from above by the quality of BLIP-2 captions, which are known to omit fine-grained visual attributes (exact colors, small text, spatial relations). The paper does not measure BLIP-2 caption accuracy on MEMLENS evidence images — it does not report what fraction of evidence-image content survives the captioning process. If BLIP-2 captions preserve only 30% of the question-relevant visual information, then the 13.11% IE accuracy for Mem0 is not a "memory" failure but a "perception" failure: the agent literally never receives the information needed to answer. This is correctly described as "lossy compression," but the severity of the loss is specific to BLIP-2; a stronger captioner (GPT-4V-based, or a fine-tuned captioner) might close much of the gap without changing the memory architecture at all.
M3-Agent (composite per-session images): Rendering each session as a composite image preserves pixel information but destroys the interleaved structure — the temporal ordering of images relative to text within a session is lost. For IE-Entity questions where the evidence image must be linked to a specific textual mention, this structural loss could be as damaging as caption-level information loss. The paper does not ablate M3-Agent with original interleaved input (which its video backbone does not support), so the relative contribution of composite-image degradation vs. ColPali retrieval noise to its 18.03% IE accuracy is unknown.
M2A and M3C (embedding-based retrieval at answer time): These agents have access to original pixels at write time but not at answer time. The paper's retrieval attribution analysis (Figure 19) is available only for Mem0, Memory-T1, and M3C — not for M2A, which has the lowest overall accuracy (15.38% at 32K) and the largest backbone-matched deficit (33.80% below direct Qwen3-VL-8B-Instruct). Without this decomposition, the paper cannot say whether M2A's poor performance is due to retrieval misses (the dual-layer SQLite failing to surface the right sessions), compression loss (the SigLIP2 embeddings losing visual detail), or reasoning failure (the Qwen3-VL-8B backbone failing to reason over retrieved content).

What evidence exists in the paper. The backbone-matched comparison (M2A vs. direct Qwen3-VL-8B-Instruct, Table 16) quantifies the total pipeline cost (33.80% at 32K) but does not decompose it. The backbone ablation (Table 21) for Mem0 and MemOS quantifies the backbone-quality contribution within fixed architectures but does not compare across adapter types. The retrieval attribution (Figure 19) provides the decomposition for three of seven agents but is missing for M2A, the weakest multimodal agent, and for M3-Agent, the agent with the most aggressive input transformation.

Mitigation status. The paper acknowledges the adapter asymmetry explicitly and argues that it is a feature of the evaluation design — agents are tested as they are deployed — rather than a confound to be removed. This is a legitimate methodological choice: if a deployed text-only agent would use BLIP-2 captions, evaluating it with original pixels would overestimate its real-world performance. However, for a practitioner trying to understand why agents underperform LVLMs — and whether that gap can be closed with better captioning, better retrieval, or better reasoning — the current evaluation provides limited diagnostic resolution. The backbone-matched comparison is the strongest evidence for the architectural-cost claim, but it applies only to one agent (M2A) on one backbone (Qwen3-VL-8B). The paper does not provide matched-backbone comparisons for text-only agents (e.g., evaluating a text-only LVLM given the same BLIP-2 captions as Mem0, to isolate the memory-architecture cost from the captioning cost), which would be the cleanest way to separate the "lossy compression" component from the "memory architecture" component of the agent deficit.

6.6 The Evaluation Does Not Account for Wall-Clock Latency or the Serial Dependency of Memory Operations

The assumption or constraint. The paper measures performance by accuracy at fixed context lengths, with no accounting for the time required to achieve that accuracy. This is a standard evaluation protocol for benchmarks — accuracy is the natural metric — but it obscures a practical tradeoff that is explicit in the paper's own description of agent evaluation:

"agent pipelines are substantially slower than direct LVLM inference (M2A takes roughly 60× longer per question)" (Appendix B)

The 60× multiplier is mentioned in passing to justify the 195-question subset for agents, but its implications for the paper's architectural conclusions are not explored. LVLM inference is a single forward pass over the full context (plus generation time), with latency proportional to context length and model size. Agent inference involves multiple serial steps: write-time encoding of session content (captioning, embedding, indexing), query-time retrieval (search over stored representations), and answer-time reasoning over retrieved content — each step potentially requiring a separate model call with its own latency.

The consequence. The paper's finding that agents are "length-stable" (§4.3, Figure 4a) is a statement about accuracy, not about compute cost. Achieving that length stability might require substantially more wall-clock time than LVLM inference at 256K, even if the LVLM's accuracy is lower. Consider a deployment scenario where:

Latency budget: 5 seconds per user query.
LVLM at 128K: Forward pass + generation: ~3–5 seconds for a medium-sized model on appropriate hardware. Accuracy: ~45% overall (Qwen3.5-122B: 45.50% at 128K).
Agent at 256K: Write-time encoding (amortized across queries), query-time retrieval (~1–2 seconds for FAISS/BM25 over stored content), answer-time reasoning (~2–4 seconds for backbone generation). Total: ~3–6 seconds, but with higher variance (retrieval latency depends on index size and query complexity). Accuracy: ~30% overall (Mem0 at 128K: 30.26%).

In this scenario, the agent is slower and less accurate than the LVLM — its length stability does not translate to a deployment advantage. The paper's recommendation of hybrid architectures implicitly assumes that retrieval is fast enough to be practical, but the current agent latencies (~60× slower than direct LVLM inference for M2A per question) suggest that retrieval overhead swamps any computational advantage at the lengths studied (32K–256K). At longer lengths (512K–1M tokens), the LVLM cost grows with context length while agent retrieval cost grows sublinearly (logarithmic in store size for FAISS), so agents would eventually become faster — but the paper does not evaluate at these lengths, so the crossover point is unknown.

More subtly, the serial dependency of memory-agent operations limits throughput in batch-inference settings. An LVLM processing 100 conversations of 128K each can batch them into a single forward pass (with appropriate padding and attention masking), achieving high hardware utilization. An agent processing 100 conversations must perform 100 independent retrieval operations (which are embarrassingly parallel but require separate index queries), followed by 100 independent backbone generations (which can be batched but lose the efficiency of a single large forward pass). The total FLOPs might be lower for the agent, but the wall-clock time might be higher due to lower hardware utilization, depending on the batch size and hardware configuration. The paper's FLOPs-based reasoning about "length-stability" does not capture this throughput dimension.

What evidence exists in the paper. The 60× multiplier for M2A and the agent subset justification (Appendix B) are the only latency-relevant numbers reported. No latency measurements (wall-clock time, FLOPs, memory footprint) are provided for any model or agent at any context length. No throughput analysis is performed. No comparison of agent vs. LVLM cost (in FLOPs, tokens, or seconds) is made at matched accuracy levels — which would be the deployment-relevant comparison ("how much does it cost to achieve 40% accuracy?").

Mitigation status. The paper does not address this limitation. The absence of latency analysis is typical for benchmark papers, which prioritize accuracy as the primary metric, and the paper's contribution — a diagnostic benchmark for multimodal memory — does not require latency data to be valid. However, the paper's conclusions have architectural implications that depend on latency assumptions. The recommendation of hybrid architectures implicitly assumes that retrieval can be made fast enough (low-latency indexing, sub-second query times) that the combined system's latency is dominated by the LVLM forward pass, not the retrieval step. The current agents, with their 60× slowdown, do not satisfy this assumption, and the paper does not discuss what retrieval-latency improvements would be needed for hybrid architectures to be practically deployable. For a practitioner, this means the paper's architectural recommendation should be treated as a research direction rather than a deployment prescription until latency data are available.

7. Implications and Future Directions

How This Work Changes the Landscape

MEMLENS shifts the conversation around multimodal memory from a scalar-competition framing — "which architecture is better, long-context LVLMs or memory-augmented agents?" — to an orthogonal-failure-mode diagnosis — "each approach fails on a different axis, and neither alone suffices." This is not a paradigm shift in the Kuhnian sense, but it is a diagnostic reframing with consequences for how the research community allocates effort: rather than scaling either component in isolation, the next generation of memory systems should combine long-context attention with structured multimodal retrieval. The paper earns this reframing by providing the first side-by-side, length-controlled comparison of the two architectural classes on questions that genuinely require visual evidence (§4.2, Table 1), and by decomposing the resulting performance into concrete, complementary failure signatures: LVLMs lose ~20% IE accuracy from 32K to 128K while agents lose ~61% IE accuracy at all lengths through caption-based storage compression (Figure 4a, Table 16).

The most landscape-changing consequence is that the paper retroactively reveals prior multimodal conversational benchmarks as text-only memory tests under multimodal disguise. The cross-modality ablation (§3.4, Table 3) provides causal evidence — removing evidence images collapses frontier models from ~90% to below 2% accuracy — that MEMLENS questions genuinely require visual evidence, while prior benchmarks (LoCoMo, Mem-Gallery) allowed text-only shortcuts. This finding should change how the community evaluates future multimodal memory systems: the structural necessity of visual evidence must be demonstrated through causal ablation, not assumed through the presence of images. The entity-abstraction mechanism that enforces this necessity (§C.4) — replacing named entities with under-specified visual anaphors like "the bridge shown in <image>" — provides a reusable design pattern for constructing genuinely multimodal benchmarks across domains (code generation, medical QA, document understanding).

The paper also resolves the latent tension between the long-context and memory-agent literatures by showing that both communities' prior findings were correct within their own evaluation ecosystems but incomplete in scope. Long-context benchmarks (MMLongBench, MM-NIAH) correctly identified length-dependent degradation but tested only on documents and retrieval tasks, not multi-session conversations. Memory-agent benchmarks (LongMemEval, MemoryAgentBench) correctly identified length-stable retrieval but discarded the visual modality entirely. MEMLENS demonstrates that when both approaches are tested on the same multimodal conversational task, the length-stability and visual-fidelity axes are orthogonal — neither community's findings generalize to the full problem. This reframes the debate from "long-context vs. retrieval" to "how to combine them," which is a more productive framing.

Less visibly but perhaps more consequentially for agent research, the paper provides the first systematic evidence that post-training on memory agent backbones erodes abstention behavior (§4.3, Figure 2 AR panel). The 59-percentage-point gap between M2A's fine-tuned backbone and the same backbone under direct inference on answer-refusal questions (22.73% vs. 81.82%; Appendix G.6, Table 16) is a stark negative finding that identifies a reward-design blind spot in current agent training paradigms. The implication — that memory management and hallucination control are coupled problems that must be optimized jointly — should shift how agent training objectives are constructed. The finding is robust because it is replicated across agents (frozen-backbone systems preserve 68–77% AR; fine-tuned systems collapse to 9–22%) and backbones (Table 21 confirms that backbone quality alone does not close the gap). This is a genuinely new insight enabled by MEMLENS's inclusion of answer-refusal as a calibration check alongside retrieval abilities.

Follow-Up Research This Work Enables

Hybrid LVLM–agent architectures with pixel-level retrieval. The paper's central finding — LVLMs have visual fidelity but poor length stability, agents have length stability but poor visual fidelity — directly motivates systems that combine both strengths. A concrete experiment: augment a long-context LVLM with a ColPali-style late-interaction visual retriever that indexes evidence images at storage time and surfaces relevant images at query time for re-attention by the LVLM's visual encoder. The hypothesis is that such a system would achieve agent-like length stability (since retrieval is length-invariant) while preserving LVLM-level visual fidelity (since retrieved images are attended over in pixel space, not reduced to captions or embeddings). MEMLENS provides the evaluation substrate: the system should be tested at 32K, 64K, 128K, and ideally 256K for direct comparison against both pure LVLMs (Table 13) and pure agents (Table 14). The oracle-retrieval diagnostic (§G.7) provides an upper bound: if evidence images are supplied directly, MSR reaches 90–100%, so a perfect retriever would recover most of the MSR deficit. The gap between hybrid performance and the oracle upper bound would quantify the retrieval-quality bottleneck.

Training memory agents with joint retrieval-and-abstention rewards. The finding that memory-agent fine-tuning erodes answer-refusal behavior (§4.3, AR panel) identifies a specific mechanism — RL and SFT objectives that reward answer correctness and retrieval success but provide no signal for abstention — that future work can directly address. A concrete follow-up: extend the training objective of an existing memory agent (e.g., MemAgent-7B or Memory-T1) to include a calibrated-abstention reward: +1 for correct answers on answerable questions, +1 for correct refusals on unanswerable questions, −1 for hallucinated answers on unanswerable questions. The baseline is the 9–22% AR accuracy reported for fine-tuned agents in Table 14; the hypothesis is that joint optimization would recover the frozen-backbone AR levels (68–77%) without sacrificing retrieval accuracy on answerable types. MEMLENS provides the evaluation framework because AR questions are structurally embedded in the benchmark — they are constructed from answerable questions by removing evidence, so the model never sees a syntactic signal distinguishing answerable from unanswerable items. A successful system would demonstrate that memory management and hallucination control can be jointly optimized, resolving the coupling that the paper identifies as a blind spot in current agent design.

Difficulty estimation for adaptive memory allocation. Although MEMLENS is a benchmark, not a system, the paper's finding that "hard questions stay hard across context lengths" (Appendix G.5 — 280 questions solved by <20% of models at 32K remain at 8.52% at 128K) implies that question difficulty is a stable property that could be estimated and exploited. A concrete research direction: train a lightweight difficulty classifier on the MEMLENS questions using features available at inference time — question type, number of evidence sessions required, presence of document-type images (which exercise different retrieval skills than natural photographs), and the PRM-style oracle retrieval difficulty (the rank of the evidence image in a similarity-ordered list of all images in the conversation). The goal is to predict, without ground-truth labels, whether a given question is in the "hard" bin (solved by <20% of models) or the "medium" bin (20–60%). A successful classifier would enable adaptive deployment strategies: route hard questions to a larger cloud-based LVLM, use a small on-device model for easy questions, and apply hybrid retrieval for the medium bin. MEMLENS's per-type and per-context-length accuracy tables (Table 13) provide the ground-truth difficulty labels for training such a classifier.

Extending the oracle-retrieval diagnostic to all question types and a broader model set. The paper's oracle-retrieval diagnostic (§G.7) is performed only on MSR and only on two frontier models (GPT-5.4 and Gemini-3.1-Pro). The finding that MSR reaches 90–100% under oracle conditions is strong evidence that MSR is retrieval-bounded, but extending this diagnostic to all four answerable types (IE, MSR, TR, KU) and to a representative set of smaller open-weight LVLMs would refine the "retrieval-bounded vs. reasoning-bounded" diagnosis per type and per model scale. A concrete experiment: take the top 5 LVLMs from each scale tier (large: GPT-5.4, Gemini-3.1-Pro, Kimi-K2.5; medium: Qwen3.5-27B, Qwen3-VL-30B-Instruct; small: Qwen3-VL-8B-Instruct, Phi4-Multimodal) and evaluate them under oracle evidence delivery on all 699 answerable questions. The hypothesis is that IE, KU, and MSR would all be near-ceiling under oracle conditions for frontier models (confirming retrieval-boundedness), while smaller models might show residual reasoning deficits on KU (tracking four-fact update chains) and MSR-Arithmetic (multi-operand summation) — revealing an interaction between model scale and the retrieval-vs-reasoning bottleneck that the current paper cannot decompose.

Replicating the orthogonal-failure-mode diagnosis on human-authored multimodal conversations. The paper's conversations are LLM-generated (GPT-5.1 user, Gemini-3-Pro assistant) with human review for naturalness, but the distributional gap to real human–assistant interactions is uncharacterized (Appendix H). A stress-test: collect a corpus of 200–500 real multi-session multimodal conversations (e.g., from a deployed assistant with user consent, or from a controlled study where participants interact with an LVLM over multiple sessions while sharing images) and apply MEMLENS's entity-abstraction and question-construction pipeline to generate evaluation questions on this real-conversation corpus. The goal is not to build a new benchmark but to test whether the LVLM-length-degradation and agent-visual-fidelity findings replicate on human-authored conversations, or whether synthetic conversations overestimate the difficulty of evidence retrieval (because real conversations have more distinctive vocabulary that aids retrieval) or underestimate it (because real conversations have less structured topic coherence that makes evidence harder to distinguish from filler). A null result — synthetic and human conversations yield different degradation curves — would bound the external validity of MEMLENS-based findings; a replication would strengthen the paper's architectural recommendations.

Automatic image-ablation testing as a benchmark design standard. The paper's cross-modality ablation (§3.4) — removing evidence images and observing performance collapse — is the methodological innovation that distinguishes MEMLENS from prior multimodal conversational benchmarks. Future multimodal benchmark construction should adopt this as a standard validation step: before releasing any multimodal evaluation dataset, demonstrate through image-ablation that (a) frontier models achieve high accuracy with images present (answerability check), and (b) accuracy collapses to near-chance when images are removed (necessity check). The paper provides a template: a stratified sample of n = 634 image-essential and image-supportive questions, ablated by removing evidence images while retaining textual evidence facts, scored with two frontier models from different families to rule out model-specific confounds. A benchmark that passes both checks has demonstrated genuine cross-modal dependency; a benchmark that fails the necessity check (high accuracy without images) is measuring text-only capability under multimodal disguise and should be redesigned or re-labeled.

Practical Applications and Downstream Use Cases

Selecting a memory architecture for production multimodal assistants. For teams deploying LVLM-based assistants that must maintain state across sessions — customer support systems, personal productivity tools, healthcare companions — MEMLENS provides the first direct comparison of the two prevalent architectural approaches on the specific memory abilities that deployment requires. The key deployment-facing result is Table 16: at matched context length, the strongest agent (MemAgent-7B at 32.82%) trails the weakest frontier LVLM by ~21 percentage points on the same 195-question subset, but at 256K (where no LVLM is even evaluated), agents maintain their accuracy while LVLM performance is unknown. A team deploying an assistant with expected session histories in the 64K–128K range should prefer a long-context LVLM (Gemini-3.1-Pro retains 51.99% at 128K, losing only 2.11% from 32K); a team deploying with expected histories in the 256K+ range should benchmark their LVLM at that length and, if degradation is severe, consider a memory-augmented agent with the explicit caveat that current agents lose ~61% IE accuracy relative to LVLMs through caption-based compression (Table 16: Mem0 at 13.11% IE vs. Qwen3.5-122B at 74.39%). The paper does not provide a 256K LVLM datapoint, so the crossover length where agents become preferable is unknown — but the framework for making this decision (which length, which ability, which accuracy requirement) is now available.

Diagnosing retrieval-vs-reasoning bottlenecks in deployed memory systems. For teams that have already deployed a memory-augmented agent and are observing low accuracy on a specific memory ability, MEMLENS's error taxonomy (§G.4, Figure 4b) provides a diagnostic framework for determining whether to invest engineering effort in better retrieval, better reasoning, or better visual encoding. If the deployed system shows high error rates on IE (like the 89% Visual errors reported at 128K for LVLMs), the bottleneck is evidence retrieval — the system is not locating the correct sessions or images — and investment should target retrieval quality (better embeddings, better index structures, query rewriting). If the system shows high error rates on MSR (like the 73% Reasoning errors), the bottleneck is aggregation — the system finds the evidence but cannot combine it correctly — and investment should target reasoning capability (chain-of-thought prompting, backbone model upgrade). If the system is a text-only agent (captions, no pixels) and shows low accuracy on KU (like M2A's 0.00% KU at all lengths), the bottleneck is visual compression — captions lose the fine-grained category discrimination needed to distinguish "apple" from "dragonfruit" — and investment should target pixel-level storage (migrating to a multimodal backbone or a ColPali-style visual retriever). The paper's retrieval-attribution analysis (Figure 19) provides the template for this triage: instrument the agent to log whether evidence was retrieved (recall ≥ 0.5) before each error, then partition errors into retrieval-failure and comprehension-failure bins.

Calibrated abstention as a deployment safety check for memory agents. The paper's finding that fine-tuned memory agents collapse to 9–22% AR accuracy (Table 14, AR column) provides a concrete safety test for teams deploying fine-tuned agents in production. Before deploying a memory-augmented assistant, evaluate it on a set of deliberately unanswerable questions embedded in realistic conversation histories (MEMLENS-style AR items, constructed by removing evidence from otherwise answerable instances). If the agent's AR accuracy is below ~70%, it is likely hallucinating answers when evidence is absent — a safety-critical failure mode for applications where incorrect answers have real-world consequences (medical information, financial advice, legal guidance). The paper's backbone-matched comparison provides a calibration target: the backbone's AR accuracy under direct inference (e.g., Qwen3-VL-8B-Instruct at 81.82% on AR; Table 16) is the upper bound that joint retrieval-and-abstention training should aim to recover. Teams that observe a large gap between backbone-direct AR and fine-tuned-agent AR should suspect that their training objective is eroding abstention and should add an explicit abstention-reward term. MEMLENS's AR items, combined with the paper's refusal-phrase grading rubric (§F.1), provide an off-the-shelf evaluation harness for this safety check.