Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams

🎯 Pitch

Even top LLMs like Gemini 3 Pro fail to track simple factual updates in a streaming text—accuracy plummets from 66% to 33% when facts change frequently, exposing a fundamental blindness to when knowledge becomes stale. The core struggle isn’t retrieving facts, but recognizing that an earlier state has been overridden and locking onto the correct new answer, a divide the paper reveals by disentangling update detection from state identification.

1. Executive Summary

This paper introduces ONLINE ADAPTATION TO CONTINUAL KNOWLEDGE STREAMS (OAKS), a benchmark that evaluates how large language models track and update factual knowledge as it evolves sequentially in streaming contexts. Using two datasets — OAKS-BABI (synthetic, derived from BABILong) and OAKS-Novel (human-curated from literary texts) — the authors test 14 models including Gemini 3 Pro, Qwen3 variants, and GPT-OSS under multiple inference strategies: base context concatenation, retrieval-augmented generation (RAG), and agentic memory systems (HippoRAG-V2, MemAgent, A-Mem). Even state-of-the-art models struggle, with Gemini 3 Pro reaching only 66.3% accuracy on OAKS-BABI and 75.5% on OAKS-Novel, while performance degrades sharply under frequent knowledge updates — dropping to 33.3% and 53.0% respectively — and at later time intervals as context length accumulates. Fine-grained behavioral analysis reveals that models exhibit distinct failure modes: some over-update (changing predictions unnecessarily due to contextual distraction) while others under-update (displaying inertia even when the underlying state shifts), establishing that correct state tracking requires both detecting when to update and determining what the new answer should be, with the latter proving substantially harder.

2. Context and Motivation

The Core Problem: Language Models Must Track Evolving Facts in Real Time

The fundamental challenge this paper addresses is straightforward to state but profoundly difficult to achieve: when a language model receives information sequentially, over time, can it correctly update its understanding when facts change? This is not about retrieving static facts from a fixed corpus. It is about watching a continuous stream of text — the chapters of a novel, the turns of a conversation, the sensor readings of a robot — and recognizing that "the box is on the table" from chapter 1 is no longer true in chapter 5, when a character moves it to the basement. The model must detect the update, revise its internal representation, and answer questions based on the most recent state, not a stale one.

This problem sits at the intersection of two established research areas that the paper explicitly draws from (Section 2, Appendix A). Continual knowledge learning asks whether models can incorporate new facts without forgetting old ones, typically tested by feeding a model batches of new information and checking if it can answer questions about everything it has seen (Jang et al., 2022; Liska et al., 2022; Kim et al., 2024c). Online adaptation asks whether models can adjust their behavior on the fly as inputs stream in — a setting studied extensively in video understanding (Lin et al., 2024; Xu et al., 2025a; Niu et al., 2025) but relatively neglected for text. The key distinction the authors make is that these two paradigms, despite obvious practical overlap, have never been unified into a single evaluation framework. Continual knowledge benchmarks track divergent facts — the model learns that one new fact after another — but rarely, if ever, track the same fact being revised multiple times. Online adaptation benchmarks for text focus on task-level or behavior-level changes (Zheng et al., 2025a; Wei et al., 2025), not on the granularity of individual knowledge assertions.

The gap is therefore: no existing benchmark evaluates whether a model can maintain a temporally consistent knowledge state across a long, streaming context where individual facts are revised repeatedly, requiring the model to detect transitions, discard stale information, and compose the current state from evidence distributed across non-contiguous chunks — all without access to future information and without parameter updates.

Why This Matters: Compounding Errors in Dynamic Environments

The authors motivate this gap through a series of practical deployment scenarios (Section 1) that share a common structure: information arrives incrementally, facts evolve, and acting on outdated knowledge causes compounding downstream errors.

Conversational assistants are a primary example. In long-term dialogue (Maharana et al., 2024; Wu et al., 2025), a user might gradually reveal details about their preferences, constraints, or situation. An assistant that fails to update when the user says "I've moved to a new apartment" will keep recommending restaurants near the old address. More dangerously, an assistant that fails to track when a user corrects a previous statement — "Actually, I'm allergic to penicillin, not aspirin" — could make life-threatening recommendations.

Embodied agents and robotics present a parallel challenge. A robot exploring an environment (Majumder et al., 2023; Kim et al., 2024a) encounters new properties sequentially: it might learn that a door is locked, then later discover it has been unlocked, then find a different entrance. If the robot's internal world model does not track these state changes accurately, it will plan based on a stale representation — wasting time, missing opportunities, or executing unsafe actions.

The paper emphasizes that these failure modes are not hypothetical edge cases but systematic consequences of how current evaluation paradigms work. When benchmarks test models on static snapshots (all context available at once) or offline settings (answer questions after reading everything), they hide the model's behavior during the unfolding of information. A model might achieve high accuracy on the final state of a story but have made incorrect predictions at every intermediate step — and those intermediate errors matter in any real-time deployment.

The theoretical significance is equally important. Tracking evolving facts over long horizons is a stress test for several fundamental capabilities that language models currently lack: temporal reasoning (understanding that information is valid only within a time window), state persistence (maintaining a representation that persists until explicitly contradicted), distraction robustness (not confusing similar but temporally distinct facts), and compositional inference (reasoning over multiple facts that themselves evolve at different rates). These are not simply scaled-up versions of retrieval — they require qualitatively different mechanisms that the benchmark is designed to isolate and measure.

Where Prior Benchmarks Fall Short

The paper identifies specific, enumerated limitations in prior work that prevent it from testing the capabilities described above. Table 1 in Section 2 provides a structured comparison along four axes — what the paper calls the unit of analysis, number of evaluation steps, updates per question, and whether answers are fully annotated — revealing consistent gaps.

Continual knowledge benchmarks track divergent facts, not revisions of the same fact. EvolvingQA (Kim et al., 2024c) and StreamingQA (Liska et al., 2022) both introduce new facts over time — the model learns that entity A has property X, then entity B has property Y — but rarely, if ever, update a previously established fact. As Table 1 shows, these benchmarks average only 2.0 updates per question. Since the questions themselves target isolated facts, a model can answer correctly by retrieving the most recently mentioned information even if it fails to understand that the old information has been superseded. The difference is subtle but critical: recognizing that a new fact contradicts an old one requires detecting the logical relationship between two temporally separated statements, not just retrieving the most recent one. By designing questions where the same underlying fact changes multiple times (average 4.7 updates per question in OAKS), the benchmark explicitly forces models to demonstrate this contradiction-detection capability.

Online adaptation benchmarks for text are too coarse-grained. StreamingBench (Lin et al., 2024) evaluates "streaming video understanding" at the frame level, which captures temporal dynamics at high granularity — but it operates on video, not text. Prior text-based online adaptation work focuses on tasks or behaviors rather than facts: LifelongAgentBench (Zheng et al., 2025a) evaluates whether agents can acquire new task-level skills over time; Evo-Memory (Wei et al., 2025) tests self-evolving memory for RL-like environments. These are valuable but measure something different from fine-grained knowledge tracking. As the authors note in Appendix A:

"Evaluating fine-grained knowledge updates (i.e., when and how specific facts change) is important, as failures to incorporate small but relevant updates can propagate and accumulate into larger errors when such knowledge is reused or composed in downstream tasks."

The key phrase is "propagate and accumulate." A model that makes a small tracking error on one fact does not just fail that one question — it carries an incorrect world state forward, potentially contaminating all future reasoning that depends on that state. Existing benchmarks do not measure this compounding effect because they either operate at the granularity of entire tasks (where the error signal is too coarse to diagnose the root cause) or test on short sequences (where errors do not have time to accumulate).

State-tracking benchmarks are structured and short-horizon. The paper draws on dialogue state tracking (DST) research, particularly MultiWOZ (Budzianowski et al., 2018), which requires models to maintain structured slot-value pairs (e.g., hotel-area: centre, restaurant-price: expensive) across dialogue turns. This is the closest prior work to OAKS in spirit — it tracks evolving states online — but differs in three critical ways. First, the states are highly structured (predefined slots with known possible values) rather than open-ended. Second, the horizons are short: MultiWOZ averages 14 turns, compared to OAKS's 65–78 chunks. Third, and most importantly for the paper's contribution, the questions in DST are not fully annotated at every step for every fact — you get sparse annotations at turn boundaries, making it impossible to measure whether the model maintained the correct state continuously between annotations. OAKS annotates answers at every chunk for every question, enabling the paper's core analytic contribution: measuring not just whether the model was correct at the end, but how quickly it adapted to changes, how often it was distracted within a stable phase, and how often it missed phases entirely.

Long-context benchmarks are static and single-answer. Benchmarks like BABILong (Kuratov et al., 2024), LongBench v2 (Bai et al., 2024), and NovelQA (Wang et al., 2024) present models with long documents — sometimes entire books — and ask questions that require reasoning over the full text. This tests retrieval and compositional reasoning over long contexts, which is a necessary but insufficient capability for online adaptation. The critical limitation is that these benchmarks evaluate the model after it has seen the entire context, producing a single answer. They do not test whether the model would have produced the correct answer at intermediate points in time — which, as the authors will show in their analysis (Section 6.4, Figure 3), is where models systematically fail. A model might get the final answer right by re-reading the full context and spotting the most recent evidence, but have spent most of the story carrying an incorrect belief. BABILong, in particular, serves as OAKS-B's direct ancestor — the authors repurpose its context — but they explicitly note that BABILong focuses on "static fact retrieval" while OAKS-B shifts the focus to "dynamic knowledge tracking and reasoning" (Section 3.1). The transformation is not cosmetic; it requires generating new questions, annotating answers at every interval, and designing question types that force state-change detection rather than simple retrieval.

Agentic memory systems have not been tested on fine-grained continuous updates. The paper evaluates three memory-augmented frameworks — HippoRAG-V2 (Gutiérrez et al., 2025), MemAgent (Yu et al., 2025), and A-Mem (Xu et al., 2025b) — all of which maintain and update memory incrementally. These systems are designed for exactly the kind of long-horizon, dynamically updating context that OAKS represents, and the authors' decision to include them is deliberate: if the benchmark is too easy for best-of-N retrieval, it should at least test the specialized systems. However, the paper notes that these systems were developed and evaluated under different assumptions. MemAgent, for example, is trained with rewards computed after processing all chunks, not at each interval — meaning its objective is to get the final answer right, not to maintain a continuous correct state throughout the stream. The authors observe (Section 5.3, Table 4) that these systems underperform naive RAG in aggregate, but show competitive performance specifically on the Moderate and Frequent subsets, suggesting that their mechanism is directionally useful but not yet tuned for the evaluation protocol. This finding simultaneously validates the benchmark's difficulty and reveals what the authors believe is an underexplored design space.

How This Paper Positions Itself

The paper's positioning can be understood through four explicit choices that differentiate it from prior work.

First, it unifies continual knowledge learning and online adaptation into a single framework. The authors are explicit about this synthesis:

"To the best of our knowledge, OAKS is the first benchmark to unify these two paradigms, supporting both large-scale fine-grained knowledge adaptation and stepwise online evaluation over streaming knowledge." (Section 1)

This is not merely taxonomic novelty. The unification enables analysis that was previously impossible: measuring how quickly a model adapts to new knowledge (acquisition latency), how stably it maintains knowledge once acquired (distraction susceptibility), and how these two dimensions trade off against each other. A model that adapts instantly but is constantly distracted by context is equally problematic as one that never forgets but takes too long to update. Without both paradigms in the same framework, these trade-offs cannot be characterized.

Second, it deliberately constructs questions that track individual facts across multiple revisions. The datasets are designed so that the same question is asked at every time interval, and the correct answer changes when new evidence appears. This is what enables the fine-grained behavioral analysis in Section 6: by comparing the model's prediction trajectory to the ground-truth answer trajectory across time, the authors can classify behavior into distinct patterns (Adaptability, Volatility, Stubbornness, etc. — see Table 5) and diagnose why a model's accuracy is what it is. Prior benchmarks, by asking different questions at different times or by not annotating every interval, could not perform this analysis.

Third, it deliberately excludes parameter updating, focusing purely on in-context adaptation. The authors note in the Limitations section:

"Our current analysis focuses on inference-time adaptation via incremental context accumulation. However, OAKS also serves as a valuable testbed for parametric online learning, and future work could explore how models update their internal weights to assimilate evolving knowledge."

This is a deliberate scoping choice — not a claim that parametric adaptation is unimportant — that isolates one specific capability: can models track evolving knowledge in their context window alone, without test-time training? The answer, as the results show, is that they struggle mightily, which implies that either in-context mechanisms are insufficient for this task, current architectures do not support the necessary context processing, or both. This framing also makes the benchmark accessible: it requires no training infrastructure, only inference.

Fourth, it treats the problem as one of evaluation infrastructure, not method proposal. The paper does not introduce a new model architecture, a new training procedure, or a new search algorithm. It does not claim to solve the online adaptation problem. Instead, it builds a measurement tool — the OAKS datasets, the interval-level annotation protocol, the behavioral analysis metrics (Acquisition Latency, Distraction Susceptibility, Phase Miss rate) — and uses it to produce a detailed map of where and how current models fail. This is a diagnostic contribution, not a prescriptive one, and the authors are clear about their intent: they want to enable future research by providing a shared, challenging benchmark that exposes failure modes no prior test could capture.

The intellectual lineage the paper traces is worth noting. It draws from continual learning (Jang et al., 2022; Liska et al., 2022) for the problem formulation, from state tracking (Kim and Schuster, 2023; Niu et al., 2024) for the evaluation granularity, from streaming video understanding (Lin et al., 2024; Xu et al., 2025a) for the online evaluation paradigm, and from long-context QA (Kuratov et al., 2024; Bai et al., 2024) for the dataset construction techniques. The novelty is not in any of these individual threads but in their synthesis — creating a benchmark that is simultaneously streaming (information arrives over time), fine-grained (individual facts are tracked across revisions), long-horizon (contexts span tens of thousands of tokens across dozens of chunks), and fully annotated (every interval has a ground-truth answer for every question). No prior benchmark satisfies all four criteria, and the paper argues that all four are necessary to properly evaluate online knowledge adaptation.

3. Technical Approach

3.1 Reader Orientation

This is a benchmark and evaluation paper, not a method proposal. The system being described is not a model or an algorithm but a measurement apparatus: two datasets (OAKS-B and OAKS-N), an evaluation protocol that queries models at every time interval in a streaming context, and a set of behavioral metrics that diagnose how models fail — not just how often. The core idea is that by constructing contexts where specific facts change multiple times, annotating ground-truth answers at every point in time, and asking the same questions repeatedly, you can decompose a model's tracking performance into distinct behavioral patterns — detecting when it updates knowledge appropriately, when it remains stable under distraction, when it misses transitions entirely, and when it changes predictions unnecessarily.

3.2 Big-Picture Architecture (Diagram in Words)

The OAKS evaluation pipeline has five major components:

Streaming Context — a long document (synthetic narrative or full-length novel) segmented into chunks of approximately 2,000 tokens, presented sequentially over time. Each chunk may introduce new facts or revise previously established ones.
Fixed Question Set — a set of questions designed to target facts that evolve across the stream. Each question is asked at every time interval, always with access only to chunks up to that point.
Ground-Truth Annotations — for every question at every time interval, a correct answer and supporting evidence snippet, capturing the knowledge state conditioned on all context revealed so far.
Model Under Test — an LLM (with optional RAG, agentic memory, or rolling-window context management) that produces a prediction at each interval.
Behavioral Metrics — four quantities (Accuracy, Acquisition Latency, Distraction Susceptibility, Phase Miss rate) computed from the full trajectory of predictions across time, characterizing not just correctness but the temporal dynamics of adaptation.

Information flows as follows: at time interval $t$ , chunk $c_t$ is revealed → the accumulated context $S_t = \{c_i\}_{i=1}^t$ is constructed (either by full concatenation, RAG retrieval, or memory system) → each question $q_j$ is asked against $S_t$ → the model produces prediction $p_{j,t}$ → this prediction is compared to ground truth $a_{j,t}$ → the full matrix of $(j, t)$ predictions is aggregated into accuracy and behavioral metrics.

3.3 Roadmap for the Deep Dive

First, the formal evaluation protocol — what it means to evaluate "at every interval," the precise definition of accuracy, and why this protocol differs from standard static or offline evaluation. This establishes the mathematical framework that the behavioral metrics extend.
Second, the three behavioral metrics (Acquisition Latency, Distraction Susceptibility, Phase Miss rate) — what each measures, how it is computed from the prediction trajectory, and what failure mode it isolates. These are the paper's primary analytic contribution, going beyond aggregate accuracy.
Third, the dataset construction pipeline for OAKS-BABI — the algorithm that transforms static BABILong context into streaming knowledge-tracking questions, the four question types (tracking, counting, bridge, comparison), and how answer timelines are generated.
Fourth, the dataset construction pipeline for OAKS-Novel — the human-in-the-loop process of book selection, LLM-based question drafting, human annotation, and quality control, including compensation details and filtering criteria.
Fifth, the context representation strategies — how the base, RAG, rolling-window, and combined approaches construct $S_t$ at each interval, including retrieval model choice, chunk formatting, and prompt design.
Sixth, the model configurations and inference setup — model selection rationale, thinking vs. non-thinking mode, context-length constraints, and the specific prompts and decoding parameters used.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an evaluation infrastructure paper whose core contribution is a protocol for measuring how language models track evolving knowledge online, plus two datasets and a set of behavioral diagnostics that enable fine-grained analysis of failure modes. The technical approach decomposes into the evaluation framework, the metrics, the dataset construction procedures, and the experimental design choices.

The Online Interval-Level Evaluation Protocol

The defining characteristic of OAKS's evaluation is that it measures model performance at every point in time, not just after all context has been revealed. This is what distinguishes it from standard long-context QA benchmarks and what enables the behavioral diagnostics.

Formal definition of the streaming context and question set. Each dataset instance consists of an ordered sequence of $C$ context chunks $\{c_i\}_{i=1}^C$ and a set of $Q$ questions $\{q_j\}_{j=1}^Q$ . Each chunk $c_i$ is approximately 2,000 tokens long (Section 3.1), with the exact splitting performed using the GPT-NeoX tokenizer. The chunks are revealed sequentially: at time interval $t$ , the model has access to all chunks from $i = 1$ through $i = t$ , denoted $S_t = \{c_i\}_{i=1}^t$ .

Ground-truth annotations per interval. For every question $q_j$ at every interval $t$ , there exists a ground-truth answer $a_{j,t}$ and supporting evidence $e_{j,t}$ . The answer $a_{j,t}$ represents the correct response conditioned on all knowledge accumulated up to and including chunk $c_t$ . The evidence $e_{j,t}$ is the specific sentence or passage in the current chunk $c_t$ (not in earlier chunks) that justifies the answer at that interval — although the answer itself may be determined by information from multiple earlier chunks, the evidence annotation is always taken from the most recent chunk that contains relevant information.

A contiguous sequence of intervals sharing the same ground-truth answer is called a phase $T_k$ . The number of phases per question varies: on OAKS-B and OAKS-N, the average number of answer changes per question is 4.7 (Section 3.1), meaning the average question has roughly 5–6 distinct phases. The paper stratifies questions into three subsets — Sparse, Moderate, and Frequent — based on the number of answer changes, with bin boundaries chosen to balance sample sizes while keeping questions with identical change counts in the same bin. For OAKS-B, the boundaries are 2–3 changes (Sparse), 4–5 changes (Moderate), and 6–20 changes (Frequent); for OAKS-N, they are 2–3, 4, and 5–19 changes (Table 7, Figure 4).

The prediction protocol. At each interval $t$ , the model $M$ receives the accumulated context $S_t$ and a question $q_j$ , and produces a prediction:

$p_{j,t} = M(q_j, S_t)$

where $p_{j,t}$ is the model's predicted answer string. The model does not have access to future chunks (intervals $t+1$ and beyond), nor does it receive information about its own previous predictions at earlier intervals — each interval is evaluated independently (though context accumulates).

Why evaluate at every interval? The key design choice is that the same question is asked repeatedly at every chunk, even when the answer has not changed since the previous chunk. This might seem redundant — if the answer is the same at intervals $t$ and $t+1$ , why test twice? The reason is that maintaining a correct answer under accumulating context is not trivial. As the context grows, the model must:

Remember that a previously established fact still holds (it has not been contradicted).
Avoid being distracted by superficially similar but temporally distinct information (e.g., a flashback mentioning a past state).
Not "invent" an update based on irrelevant text that happens to mention related entities.

By testing at every interval, the protocol measures not just whether the model can find the answer when prompted, but whether it can continuously maintain the correct state across an arbitrarily long stream of potentially distracting information. This is what enables the measurement of Distraction Susceptibility — without interval-level testing, you cannot distinguish between a model that makes one early error and a model that repeatedly loses and regains the correct state.

Answer extraction and matching. For OAKS-B, which uses open-ended generation, the model is instructed to output a concise answer in 1–2 words, formatted as ## Answer: short answer (Prompt 3). The answer is extracted from this template and compared to the ground truth using normalized exact-match string comparison, with several conditional allowances:

Counting-type questions accept both numeric (3) and textual (three) representations.
Before relevant information appears, counting-type questions accept both Unknown and 0.
Before valid evidence, comparison-type questions accept both Unknown and Same.

For OAKS-N, which uses multiple choice, the model must output a single character corresponding to one of the provided answer options (Prompt 4). The options always include a catch-all option: "The question cannot be answered at this point in the story" (or "We cannot answer this question at this point"), which is the correct answer for intervals before the relevant facts are first introduced. Once a valid answer option becomes available, this catch-all option is never the correct answer again.

Interval-level accuracy. The core metric is defined as:

$\text{Accuracy} = \frac{1}{Q} \sum_{j=1}^{Q} \frac{1}{C} \sum_{i=1}^{C} \mathbb{1}[p_{i,j} = a_{i,j}]$

where $\mathbb{1}[\cdot]$ is the indicator function (1 if the prediction matches the ground truth, 0 otherwise), $Q$ is the number of questions, and $C$ is the number of intervals (chunks).

What it computes: for each question, compute the fraction of time intervals where the model's prediction was correct. Then average this fraction across all questions. The result is a single number between 0 and 1 (reported as a percentage).

Why this form: this definition treats every interval as equally important, regardless of whether the answer has changed. This is deliberate — it penalizes models that are correct only intermittently (e.g., correct immediately after a state change but wrong for most of the subsequent stable phase) equally with models that miss the state change entirely. The authors' goal is to measure continuous correctness, not peak correctness or eventual correctness. An alternative would be to measure accuracy only at phase boundaries (just after state changes), but this would hide the distraction problem that the paper identifies as a primary failure mode.

Computational scale of evaluation. The number of inferences per model is $D \times C \times Q$ , where $D$ is the number of documents (12 for OAKS-B, 39 for OAKS-N), $C$ is chunks per document (65 for OAKS-B, average 77.6 for OAKS-N), and $Q$ is questions per document. The paper reports approximately 78k inferences for OAKS-B and 67k for OAKS-N per model (Appendix B.2.3), with a single Qwen3-30B run requiring approximately 125 GPU hours on Nvidia A100 GPUs. Due to this cost, results are reported from a single comprehensive run rather than multiple trials with different seeds, although the authors argue that "the sheer volume of evaluation points across thousands of unique context-question pairs provides a high degree of consistency" (Appendix B.2.3).

Behavioral Metrics: Acquisition Latency, Distraction Susceptibility, and Phase Miss Rate

The paper's central analytic contribution is a set of three metrics that decompose accuracy into distinct temporal failure modes. These metrics require the interval-level annotation protocol — they cannot be computed from a single final-answer evaluation — and they provide the diagnostic power that distinguishes OAKS from prior benchmarks.

Phase-level decomposition. All three metrics operate over phases. For question $q_j$ , there are $N_j$ distinct phases, where the $k$ -th phase $T_{k,j}$ has duration $|T_{k,j}|$ (the number of consecutive intervals sharing the same ground-truth answer). The first correct prediction within phase $T_{k,j}$ occurs at position $\tau_{k,j}$ , defined as:

$\tau_{k,j} = \min\{t \in \{1, \dots, |T_{k,j}|\} \mid p_{t,j} = a_{t,j}\}$

where $t$ indexes positions within the phase (not absolute intervals). If the model never produces the correct answer during the entire phase, $\tau_{k,j} = 0$ .

What it computes: for each phase, find the earliest relative time step (starting from 1 for the first interval of the phase) where the model's prediction matches the ground truth. If there is no such step, the phase is considered "missed" (the model never got the answer right for that entire phase).

Acquisition Latency (AL). This metric measures how many intervals the model takes to adapt after a state change:

$\text{AL} = \frac{1}{Q} \sum_{j=1}^{Q} \frac{1}{C} \sum_{k=1}^{N_j} (\tau_{k,j} - 1) \cdot \mathbb{1}[\tau_{k,j} > 0]$

where $\mathbb{1}[\tau_{k,j} > 0]$ ensures that only phases where the model eventually got the answer right contribute to the latency calculation (phases that are completely missed contribute to Phase Miss rate, not AL).

What it computes: for each phase where the model eventually reached the correct answer, count how many intervals the model spent being wrong before it first got it right. This is $\tau_{k,j} - 1$ (e.g., if the correct answer first appears at interval 3 of the phase, the latency is 2 intervals). Sum these latencies across all phases for a question, divide by the total number of intervals $C$ (for normalization), then average across all questions. The result is a number proportional to the fraction of total time the model spent "lagging behind" state changes.

Example: a question has 65 intervals and 5 phases. In phase 1, the model gets the right answer immediately (latency 0). In phase 2, it takes 3 intervals to update (latency 3). In phase 3, it gets it immediately (latency 0). In phase 4, it never gets it (not counted in AL). In phase 5, it takes 1 interval (latency 1). Total latency = 4. AL = 4 / 65 ≈ 0.062, meaning 6.2% of all intervals for this question were "lag" intervals.

Distraction Susceptibility (DS). This metric measures how often the model loses a previously acquired correct answer within a phase:

$\text{DS} = \frac{1}{Q} \sum_{j=1}^{Q} \frac{1}{C} \sum_{k=1}^{N_j} \sum_{t=\tau_{k,j}+1}^{|T_{k,j}|} d_{j,k,t}$

where the inner indicator is:

$d_{j,k,t} = \mathbb{1}[p_{t,j} \neq a_{t,j}] \cdot \mathbb{1}[\tau_{k,j} > 0]$

This counts incorrect predictions that occur after the first correct prediction within a phase — i.e., the model had the right answer, then lost it.

What it computes: for each phase where the model was correct at least once, count every interval after the first correct prediction where the model's prediction is wrong. Sum these "lapse" counts across all phases for a question, normalize by total intervals $C$ , and average across questions. The result measures the fraction of time the model spent being "distracted" — it knew the correct answer at some point but failed to maintain it.

Why this is distinct from AL: AL measures slowness in acquiring the correct state; DS measures instability in maintaining it. A model could have low AL (updates quickly) but high DS (constantly distracted), or vice versa. These are independent failure modes that aggregate accuracy alone cannot distinguish.

Phase Miss rate (PM). This metric measures how often the model fails to capture a phase entirely:

$\text{PM} = \frac{1}{Q} \sum_{j=1}^{Q} \frac{1}{C} \sum_{k=1}^{N_j} |T_{k,j}| \cdot \mathbb{1}[\tau_{k,j} = 0]$

where $\mathbb{1}[\tau_{k,j} = 0]$ identifies phases where the model never produced a single correct prediction.

What it computes: for each completely missed phase, add the phase's full duration $|T_{k,j}|$ to the miss count. Sum across all phases for a question, normalize by total intervals $C$ , and average across questions. The result is the fraction of total time the model spent being wrong because it never found the correct state for an entire phase.

Relationship between the metrics. These three metrics — plus accuracy — are not independent. Their sum, plus one additional term, reconstructs the error rate:

$1 - \text{ACC} = \text{AL} + \text{DS} + \text{PM}$

Since every incorrect interval falls into exactly one of three categories: (a) the interval is before the first correct prediction in a phase that was eventually captured (AL), (b) the interval is after the first correct prediction in a captured phase but the prediction is wrong (DS), or (c) the interval belongs to a phase that was never captured at all (PM). This decomposition is what gives the metrics their diagnostic power — when accuracy is low, the metrics reveal why: because the model is slow to update, easily distracted, or completely missing phases.

OAKS-BABI Dataset Construction

OAKS-B systematically transforms the static BABILong corpus into a streaming knowledge-tracking benchmark through a two-phase algorithmic pipeline: state extraction and temporal question synthesis (Algorithm 1 in Appendix B.1.2).

Source data. The BABILong benchmark (Kuratov et al., 2024) embeds synthetic bAbI facts — structured statements about entities moving, acquiring, and transferring objects — within long background texts drawn from the PG-19 corpus of public-domain books. The original purpose was to test retrieval under high distractor density: can a model find a specific fact ("Mary moved to the kitchen") buried in hundreds of pages of literary text?

The authors sample 12 examples from BABILong, each containing an average of 87 bAbI facts embedded within novel-length background text (Section 3.1). The key modification is that the authors do not simply reuse BABILong's questions — they generate entirely new questions designed to track state changes rather than retrieve static facts.

Step 1: Canonical entity-state extraction. Each bAbI fact is a sentence of the form (subject, verb, object, [recipient]), e.g., "Mary moved from kitchen to hallway" or "Sandra gave the box to Daniel." The algorithm parses each fact and normalizes the verb into one of four canonical verb groups $V \in \{\text{Move, Acquire, Discard, Transfer}\}$ . For each entity, the algorithm maintains a state dictionary $S[s][v_{\text{canon}}]$ that logs all observed (object, timestamp) pairs for that entity and canonical verb. For Transfer verbs, the recipient's "get" state is additionally tracked, recording (object, giver, timestamp).

This produces a temporally indexed database of every entity's location, possessions, and transaction history at every timestep in the narrative. This is the raw material from which questions are synthesized.

Step 2: Temporal question synthesis via template instantiation. For each entity in the state dictionary, the algorithm iterates over a predefined set of question templates $P$ (Table 8). There are four categories of templates, each designed to probe a different aspect of state-tracking:

Tracking questions (7% of dataset, Table 9): single-fact retrieval that undergoes frequent updates. Templates include "Where is [SUB]?" and "Who is holding [OBJ]?". These are the simplest question type in terms of reasoning complexity but the hardest in terms of update frequency — they average 8.8 state transitions per question compared to 3.7–5.7 for other types.
Counting questions (28%): require tallying actions over time. Templates include "How many times has [SUB] moved to [PLACE]?" and "How many unique objects has [SUB] picked up?". These require the model to aggregate evidence across multiple chunks — each individual movement or pickup is a separate fact — and maintain an accurate running count.
Bridge questions (30%): require multi-hop reasoning connecting temporally separated facts. Templates include "Who most recently traveled directly from [PLACE1] to [PLACE2]?" and "Where is the most recent location that [SUB] moved to after acquiring [OBJ]?". These questions force the model to compose two or more facts that are not co-located in the text.
Comparison questions (35%): require comparing aggregate quantities across entities. Templates include "Has [SUB1] or [SUB2] visited more distinct places?" and "Who picked up a greater number of distinct objects, [SUB1] or [SUB2]?". These are the most computationally demanding — the model must count actions for two different entities and then compare the results.

For each template instantiation, the algorithm retrieves the relevant action history from the state dictionary and generates a timeline: an array of length $|F|$ (one entry per fact in the narrative), where each entry is the correct answer at that point in time. A question is retained only if the timeline contains at least two distinct values — i.e., the answer changes at least once, ensuring the question actually tests state tracking rather than static retrieval. This filtering step is critical: it is what transforms BABILong from a retrieval benchmark into a tracking benchmark.

Timeline discretization. Each entry in the generated timeline corresponds to one bAbI fact, but the evaluation unit is a 2k-token chunk (which may contain multiple facts). The timeline is collapsed to chunk-level granularity: the answer for chunk $t$ is the answer from the most recent fact within that chunk or any preceding chunk. This produces the final interval-level annotation $a_{j,t}$ .

Name randomization. To prevent models from exploiting pretraining knowledge of the original bAbI entities (e.g., "Mary," "John," "kitchen"), all location and character names are replaced with randomized, non-existent strings. This ensures the task cannot be solved by recalling facts about the original bAbI world state from parametric memory.

Dataset statistics. The final OAKS-B dataset contains 1,224 questions, context split into 65 chunks of 2k tokens each (total 128k tokens), with an average of 87 embedded bAbI facts and 4.7 answer changes per question (ranging from 2 to 20). The partition into frequency-based subsets (Table 7) places 40% of questions in Sparse, 25% in Moderate, and 35% in Frequent, with bin boundaries at 3 and 5 changes.

What makes OAKS-B difficult. The synthetic nature of the bAbI facts means the ground-truth states are deterministic and logically well-defined — there is no ambiguity about whether Mary moved or what the correct count should be. The difficulty comes from three sources: (1) the facts are embedded within long literary text that serves as a distractor; (2) the same fact can be revised many times, requiring the model to recognize that old information is stale; and (3) counting, bridge, and comparison questions require composing evidence from non-contiguous chunks, which tests compositional reasoning under temporal dynamics.

OAKS-Novel Dataset Construction

OAKS-N is constructed from full-length literary novels through a human-in-the-loop pipeline that combines LLM-based initial drafting with rigorous human curation. The goal is to produce questions that test the same tracking capabilities as OAKS-B but in natural, narratively rich contexts where states evolve organically rather than through synthetic fact injection.

Book selection and preprocessing. The authors select 39 novels from adventure, mystery, and science-fiction genres — chosen because these are plot-driven genres with high density of entity state changes (characters move, acquire knowledge, change relationships, etc.). Non-content elements (titles, author names, tables of contents) are removed to prevent models from exploiting pretraining knowledge of the specific books. The main narrative text is extracted and segmented into chunks of approximately 2,000 tokens using the GPT-NeoX tokenizer. To preserve narrative coherence, sentences separated by newline characters in the original text are kept within the same chunk whenever possible — splitting a paragraph mid-sentence across chunks would make the text harder to comprehend and could artificially create "transitions" that are not real narrative boundaries.

QA draft generation with Gemini 2.5 Pro. The initial question pool is generated through a two-step prompting procedure:

Entity identification (Prompt 1): the full text of each novel is provided to Gemini 2.5 Pro with instructions to "identify the most important characters and objects, central to the narrative" and output only their names separated by |. This produces a list of key entities to focus question generation on.
Question generation per entity (Prompt 2): for each identified entity, the model generates questions that must satisfy specific criteria:
- "Each question must require synthesizing information from at least two separate parts of the novel" — ruling out single-chunk retrieval.
- Questions should target "the evolution of a character's feelings, understanding, or situation over time" or require "solving a mystery for which clues are scattered across different locations."
- Each generated question includes a set of {answer, source} pairs, where the source is a quoted piece of evidence from the text.

The key design constraint is that questions should track evolving states — the correct answer should depend on how far into the book the reader is. This is what distinguishes OAKS-N questions from standard reading comprehension questions: the answer to "What is Victor Frankenstein's primary goal?" should change as the narrative progresses from ambition to horror to revenge to regret.

Manual curation and quality control. The initial LLM-generated question pool undergoes extensive human filtering and revision, with only 55% of questions retained as high-quality (Section 3.1). The curation process involves 18 experienced freelancers hired through Upwork, all native English speakers from the US who had already read the selected books. Total annotation cost was $17,400, with compensation ranging from$ 50 to $850 per book depending on length and initial question count, at an approximate rate of$ 20 per hour with an average workload of 14 hours per book (Appendix B.1.3).

Annotators are provided with the novel text segmented into chunks and a spreadsheet containing the drafted questions and answer options. For each question at each chunk, annotators must:

Select the correct answer from the provided options (or add a new option if none are appropriate).
When an answer changes from the previous chunk, copy the exact evidence sentence from the current chunk that justifies the transition.
Flag and revise questions where multiple options appear simultaneously valid, where options are unsupported by the text, or where the question is answerable from a single chunk.

The filtering criteria remove three categories of questions (Appendix B.1.3):

(i) Questions that "merely stitched together information from multiple chunks" without requiring the tracking of state changes — these reduce to evidence search, not state tracking.
(ii) Questions that exhibit "only a single state change" without requiring multi-chunk reasoning — these are too simple for the tracking benchmark.
(iii) Questions where "answer options were unsupported by the surrounding context or appeared simultaneously within the same chunk" — these violate the requirement that exactly one option is correct at each interval.

Question reformulation for temporal uniqueness. The annotators and authors extensively revise surviving questions to enforce that exactly one answer option is valid at each chunk. This involves two types of reformulation:

Temporal anchoring: questions that ask about a state's overall evolution (e.g., "How does Mr. Darcy's assessment of Elizabeth change?") are rewritten to ask about the current state (e.g., "What is Mr. Darcy's latest assessment of Elizabeth Bennet's appearance?").
Cumulative-to-incremental reframing: questions that ask about an accumulated state (e.g., "What are the different names Erik is known by?") are rewritten to target the most recent revelation (e.g., "What term has been most recently introduced to describe the man who lives at the opera?").

Answer option design. Each question includes 5–15 answer options (average 5.5), consisting of:

The correct answer(s) at various points in the narrative.
Distractor options that are "plausible but incorrect, designed to challenge readers who haven't paid close attention to the details" (Instruction 4).
A default option: "The question cannot be answered at this point in the story," which is the correct answer for intervals before the relevant information is first introduced. Once any valid answer option becomes correct, this catch-all is never the correct answer again.
Option order is randomized in the final dataset.

Multiple-choice format justification. The authors adopt multiple-choice for OAKS-N (unlike OAKS-B's open-ended format) because natural language answers to novel-based questions have many valid surface forms — a character might be referred to as "Elizabeth," "Lizzy," "Miss Bennet," or "Eliza," making exact-match evaluation unreliable. Multiple choice constrains the answer space to predefined options, enabling consistent automatic scoring. The trade-off is that multiple-choice introduces a random baseline: with an average of 5.5 options, random guessing achieves approximately 18.6% expected accuracy (as noted in Appendix D.3). This should be kept in mind when interpreting absolute accuracy numbers — a model scoring 57.5% on OAKS-N is performing substantially above chance, but the format inherently reduces the dynamic range of measurable improvement compared to open-ended generation.

Evidence annotation. For each answer at each chunk (except when the catch-all "cannot answer" option is selected), annotators copy the exact evidence sentence from the corresponding chunk. This evidence annotation serves two purposes: (1) it enables the verification that answers are grounded in the text and (2) it supports the evidence-level analysis in Appendix E.2, where the authors examine whether models base their predictions on the correct supporting sentences.

Verification procedure. After annotation, the authors conduct a thorough verification of all annotated files, checking that: (i) each question has exactly one valid answer per chunk, (ii) the cited evidence explicitly appears in the specified chunk and correctly supports the selected answer, (iii) the evidence annotation aligns with the chunk where the answer was chosen, and (iv) each question contains at least five answer options including distractors.

Dataset statistics. The final OAKS-N dataset contains 870 questions across 39 books, with an average book length of 150,600 tokens (77.6 chunks, ranging from 26 to 286). The average number of answer changes per question is 4.7, matching OAKS-B, with a range of 2–19 changes (Figure 4). The frequency-based partition places 25% of questions in Sparse, 33% in Moderate, and 42% in Frequent (Table 7). Representative examples (Table 10) show how answers evolve across chunks for questions tracking characters' locations, opinions, and motivations in novels such as Around the World in Eighty Days, Pride and Prejudice, and Frankenstein.

Context Representation Strategies

The paper evaluates models under four context construction strategies, representing different ways of handling the constraint that context grows unboundedly over time while model context windows are finite (Section 4, Appendix C.1.3).

Base (full concatenation). All chunks from $c_1$ through $c_t$ are concatenated in order and fed to the model as a single prompt. When $t$ is large enough that the total token count exceeds the model's context limit, older chunks are truncated from the beginning — but only as necessary, not by a fixed window. The exact maximum context length used varies by model (Table 11):

Qwen3-4B/8B/30B: 262k tokens (their default context length).
Qwen2.5-7B and Qwen3-8B (when expanded via YaRN): 131k tokens (expanded from 32k default).
Qwen3-235B: 133k tokens (capped below its nominal 1M token limit due to GPU memory constraints — the full 262k context cannot fit on a single 8×H100 node).
GPT-OSS: 131k tokens.
Gemma 3: 131k tokens.
Gemini models: 1M tokens (their full advertised context window, without truncation needed for the dataset sizes).

The Base setting tests the model's native ability to process long, accumulating contexts without any retrieval or summarization — a direct stress test of long-context attention mechanisms and state-tracking within the context window.

Retrieval-Augmented Generation (RAG). Instead of concatenating all chunks, the model receives the top- $k$ most relevant chunks retrieved from the set of chunks up to $t$ . The retrieval pipeline works as follows:

Indexing: each chunk is stored in a vector index with the format "# chunk index: {chunk_idx}, context: {chunk_text}". The chunk index is included in the indexed text to allow order-aware retrieval.
Query formulation: at interval $t$ , the query is formatted as "# chunk index: {chunk_idx}, question: {question}", mirroring the indexing format to improve semantic matching.
Retrieval model: Qwen3-Embedding-0.6B (Zhang et al., 2025) performs cosine similarity search. Retrieval is restricted to chunks with indices $\leq t$ (no future information leakage).
Top- $k$ selection: unless otherwise specified, $k = 30$ chunks are retrieved and prepended to the prompt in a structured format: "- Retrieved context:\n# chunk index: {idx}, context: {text}\n...". The $k = 30$ value was chosen based on analysis in Appendix D.2 (Figure 5, Figure 6): Pass@k for the retriever approaches near-perfect recall around $k = 60$ , but downstream model accuracy peaks around $k = 30$ and degrades with more chunks (likely due to the well-documented phenomenon of LLMs struggling with long, complex retrieved contexts — the "lost in the middle" effect; Liu et al., 2024).
Combined with rolling window (RAG+RW): in the combined setting, 15 chunks are retrieved via RAG and 15 chunks are the most recent ones (rolling window), for a total of 30. The RAG+RW prompt distinguishes between retrieved and recent chunks with separate formatting: retrieved chunks are labeled "- Retrieved context:" and recent chunks are labeled "# Recent Chunks:". The question includes the current chunk index: "Current Head Index: {chunk_idx}, question: {question}" (Appendix C.1.3).

The RAG approach tests whether retrieval can effectively surface the relevant evidence for state tracking, a distinct challenge from standard QA retrieval because the same fact may appear in multiple chunks with different values — the retriever must return all relevant chunks, not just the most recent one, or the model will lack the evidence to detect state transitions.

Rolling Window (RW). The simplest alternative to full concatenation: retain only the most recent $w$ chunks, where $w = 30$ (matching the RAG chunk count for fair comparison). This is a recency-based strategy that assumes older information is less relevant — a heuristic that works well for many real-time applications but fails when the evidence for a current state was established long ago and has not been revisited.

Agentic memory systems. The paper evaluates three specialized frameworks that maintain and update a structured memory incrementally (Section 4, Appendix C.1.2), all built on Qwen2.5-7B-Instruct as the base model:

HippoRAG-V2 (Gutiérrez et al., 2025): a graph-based retrieval framework using Personalized PageRank. At each interval, the new chunk is processed and integrated into an evolving knowledge graph; retrieval traverses this graph to find relevant information.
MemAgent (Yu et al., 2025): a memory agent trained with Group Relative Policy Optimization (GRPO). It processes chunks sequentially, maintaining a compressed memory representation that is updated at each timestamp by combining the prior memory state with the newly observed chunk. MemAgent is trained with a reward signal computed after processing all chunks (not at each interval), which the authors note as a potential misalignment with OAKS's interval-level evaluation protocol.
A-Mem (Xu et al., 2025b): an agentic memory system inspired by the Zettelkasten note-taking method, organizing memory as an interconnected knowledge network through dynamic indexing and bidirectional linking. New information is linked to existing related memories, creating a web that supports both retrieval and state tracking.

These systems are evaluated to test whether purpose-built memory architectures can outperform simple context manipulation strategies, particularly on questions with frequent updates where specialized tracking mechanisms might prevent the distraction and volatility that plague vanilla concatenation.

Model Selection and Inference Configuration

Model selection rationale (Section 4, Appendix C.1). The paper evaluates 14 models spanning four dimensions of variation:

Scale: from Qwen3-4B to Qwen3-235B (and Gemini models of unspecified scale), testing whether larger models track states better.
Architecture: dense models (Qwen3-4B/8B, Qwen2.5-7B, Gemma 3-4B/27B), Mixture-of-Experts models (Qwen3-30B with 3B active parameters, Qwen3-80B with 3B active, Qwen3-235B with 22B active, GPT-OSS-20B with 3.6B active, GPT-OSS-120B with 5.1B active), and hybrid attention (Qwen3-80B uses a combination of Gated DeltaNet and Gated Attention).
Provenance: open-source (Qwen, GPT-OSS, Gemma) vs. proprietary (Gemini 2.5 Flash, Gemini 2.5 Pro, Gemini 3 Pro).
Generation mode: standard vs. "thinking" mode (extended chain-of-thought reasoning). Four configurations are tested with thinking: Qwen3-30B-Thinking, Gemini 2.5 Flash with thinking, Gemini 2.5 Pro with thinking, and Gemini 3 Pro (thinking enabled by default).

The Qwen3 family is the primary baseline because it offers multiple sizes under a single architecture, enabling clean scaling analysis. Models of comparable size but different architecture (e.g., Qwen3-30B vs. Gemma 3-27B) allow probing whether tracking ability depends on raw parameter count or architectural details.

Inference parameters (Appendix C.1.4). The default decoding configuration follows Qwen3 best practices: temperature 0.7, top-p 0.8, top-k 20, min-p 0. Some models use provider-specified configurations that differ:

Qwen3-Next-80B: temperature 0.6, top-p 0.95, top-k 20.
Qwen3-30B-Thinking: temperature 0.6, top-p 0.95, top-k 20.
Gemini 2.5 Flash and Pro: temperature 0.0, top-p 0.95, top-k 40.

All models generate up to 4,096 tokens. For Gemini 3 Pro in thinking mode, the limit is extended to 32,768 tokens because the model occasionally exhausts the 4k-token limit during extended reasoning without producing a final answer. Gemini thinking mode uses the model's default budget of up to 8,192 thinking tokens.

Context window expansion. When a model's default context length is smaller than the required accumulated context and the model supports context extension via YaRN (a positional encoding interpolation method), the maximum sequence length is expanded accordingly. Specifically, Qwen2.5-7B and Qwen3-8B are expanded from 32k to 131k tokens (Table 11). Models that do not support YaRN or for which expansion is impractical (e.g., Qwen3-235B, capped at 133k due to hardware limits) use their maximum native context length with truncation of older chunks as needed.

Prompts (Prompt 3 and Prompt 4 in Appendix C.1.5). The evaluation prompts are adapted from the original BABILong prompt and Qwen3's recommended format. They share a common task description with four rules:

Strict Contextual Grounding: base reasoning exclusively on provided text, not prior knowledge.
State Persistence: assume states persist unless explicitly changed or contradicted.
Final State Priority: answer from the most recent, final version when a state changes multiple times.
Output formatting: end every response with ## Answer: [short answer] (OAKS-B) or a single choice letter (OAKS-N).

For OAKS-B, the prompt includes an explicit "Unknown" / "Same" rule: respond "Unknown" if the answer cannot be found, "Same" for comparison questions where values are equal. For OAKS-N, the prompt includes the full list of answer options and instructs the model to select from them, with the catch-all option for pre-revelation intervals.

The output format constraint (## Answer: [CHAR]) is specifically designed to enable deterministic extraction — without it, some models produce verbose or inconsistent responses that are difficult to parse automatically. This standardization is important for a benchmark with ~78k inferences per model: manual inspection of every output is infeasible.

Hardware and software. Experiments are conducted using 4 or 8 A100 80G GPUs, 8 H100 80G GPUs, or 4 H200 140G GPUs depending on model size and availability. Inference uses vLLM (Kwon et al., 2023) for efficient serving with PagedAttention memory management.

Design Choices and Their Justifications

Why 2,000-token chunks? The chunk size balances two competing constraints: (1) granularity — smaller chunks enable finer-grained interval-level evaluation, revealing more subtle temporal dynamics; (2) computational cost — smaller chunks mean more intervals, linearly increasing the total inference count. The authors note that sentence-level granularity would be ideal (since answers change at the sentence level) but would be "computationally expensive" (Appendix B.1.1). The 2k-token size is a pragmatic compromise that keeps the total inference count manageable (65–78k per model) while still providing enough temporal resolution to observe distinct phases and transitions. This is a design choice with clear trade-offs — a future version could use smaller chunks for higher resolution at proportionally higher cost.

Why both synthetic and natural datasets? OAKS-B and OAKS-N serve complementary purposes. OAKS-B provides controlled, synthetic facts embedded in natural distractor text — the ground-truth states are unambiguous and deterministic, making it a clean test of tracking mechanics without the confound of narrative ambiguity. OAKS-N provides naturalistic state tracking in rich narrative contexts, where states evolve organically and evidence is woven into prose. The synthetic dataset isolates the tracking capability; the natural dataset tests it under realistic conditions. Additionally, OAKS-B avoids the parametric knowledge confound that affects OAKS-N — models have not been pretrained on the randomized entity names, so they cannot rely on memorized facts about the story. This allows cleaner attribution of performance to in-context tracking ability.

Why multiple choice for OAKS-N but open-ended for OAKS-B? The authors justify this asymmetry explicitly (Appendix B.1.3): natural-language answers to novel-based questions have too many valid surface forms for reliable exact-match evaluation. For OAKS-B, answers are simple entity names, locations, or numbers — the answer space is naturally constrained, and conditional equivalence rules (accepting both "3" and "three") handle ambiguity.

Why the specific behavioral metrics (AL, DS, PM)? These three metrics decompose accuracy into failure modes that correspond to intuitively distinct problems: being slow to notice changes (AL), being easily distracted after noticing (DS), and never noticing at all (PM). The decomposition is exhaustive — every incorrect interval falls into exactly one of these categories — which means the metrics provide a complete diagnostic picture. The choice of these specific metrics is driven by the paper's central observation (Section 6.1) that models exhibit two qualitatively different failure patterns: over-updating (high DS) and under-updating (high AL, PM). These metrics quantify those patterns, enabling systematic comparison across models.

Why not update model parameters? The paper deliberately restricts to inference-time adaptation only (context accumulation, no fine-tuning). The authors note this explicitly as a scope choice (Limitations), not a claim that parametric adaptation is irrelevant. The rationale is twofold: (1) testing in-context tracking isolates a specific capability that is currently underexplored, and (2) it makes the benchmark accessible — evaluating requires only inference, no training infrastructure. The authors suggest OAKS as a future testbed for parametric online learning.

Why 30 chunks for RAG? The number $k = 30$ was chosen based on an empirical sweep (Appendix D.2, Figure 6 and Figure 7): accuracy improves as $k$ increases from 1 to 30, peaks around 30, and degrades with more chunks. This inverted-U pattern is consistent with the "lost in the middle" phenomenon: feeding models too many retrieved chunks (even relevant ones) degrades their ability to locate and reason over the specific evidence needed. The combined RAG+RW configuration (15 retrieved + 15 recent) was designed to provide both relevance-based retrieval and recency-based coverage, since questions about the current state often require information from both distant and recent chunks.

Why the specific models? The model selection covers a deliberate grid: within a single family (Qwen3), four sizes (4B, 8B, 30B, 80B, 235B) enable scaling analysis; across families, comparable sizes (Qwen3-30B vs. Gemma 3-27B) probe architecture dependence; MoE vs. dense comparisons (Qwen3-30B with 3B active vs. Gemma 3-27B dense) test whether active parameter count or total capacity matters more for tracking; proprietary models (Gemini) establish a performance ceiling; and Qwen2.5-7B provides a direct predecessor comparison to Qwen3-8B, testing whether generational improvements in base model quality transfer to tracking ability. The agentic memory systems are all built on Qwen2.5-7B to isolate the contribution of the memory architecture from the base model's capability.

4. Key Insights and Innovations

Innovation 1: Interval-Level Annotations Enable a New Analytic Vocabulary for Diagnosing Knowledge Tracking Failures

The paper's most fundamental contribution is not the datasets themselves but the measurement protocol that produces them: annotating ground-truth answers at every time interval for every question, then using the resulting prediction trajectory to decompose accuracy into three distinct, exhaustively-defined behavioral failure modes — Acquisition Latency (how long the model takes to notice a state change), Distraction Susceptibility (how often it loses a previously correct state), and Phase Miss rate (how often it never finds the correct state at all).

Prior benchmarks for state tracking — whether dialogue state trackers like MultiWOZ (Budzianowski et al., 2018), continual knowledge benchmarks like StreamingQA (Liska et al., 2022), or long-context QA benchmarks like BABILong (Kuratov et al., 2024) — produce a single aggregate accuracy number or, at best, accuracy stratified by coarse categories. They cannot answer the question: when the model is wrong, is it because it updated too slowly, because it got distracted after being right, or because it never found the evidence? These are distinct engineering problems — slow updates suggest attention mechanisms failing to prioritize recent information; distraction suggests fragile state representations that degrade under accumulating context; phase misses suggest retrieval or evidence-integration failures — and treating them as undifferentiated "errors" obscures the path to improvement.

What makes this a genuine diagnostic advance rather than a minor evaluation tweak: the three metrics form an exhaustive decomposition of the error rate (1 − ACC = AL + DS + PM), meaning every incorrect interval is accounted for in exactly one behavioral category. This is not an ad-hoc selection of interesting statistics — it is a complete partition of the error space, which means the metrics can be used comparatively (Table 6: Gemma 3 and GPT-OSS have nearly identical accuracy on OAKS-B at ~37.5%, but Gemma 3's error is driven by Phase Miss rate of 30.3% while GPT-OSS's is driven by Distraction Susceptibility of 38.8%), and they can be used to track how interventions shift the error profile (enabling thinking mode on Qwen3-30B reduces Phase Miss rate from 23.2% to 10.4% while Distraction Susceptibility increases from 34.0% to 37.0%).

This is not an incremental improvement over prior evaluation — it is a conceptual reframing of what it means to measure knowledge tracking, from a scalar performance score to a vector of temporally-structured behaviors. The innovation is in recognizing that when the model is wrong matters as much as how often, and that tracking the full prediction trajectory across time enables answering that question.

Innovation 2: The Problem of Tracking Revised Facts Is Qualitatively Harder Than Retrieving Static or Divergent Facts — and Requires a Different Benchmark Design

The paper makes a subtle but critical distinction that restructures how the field should think about continual knowledge: tracking the same fact as it is revised multiple times is a fundamentally different problem from learning a sequence of divergent facts, and prior benchmarks conflated the two.

In StreamingQA (Liska et al., 2022) or EvolvingQA (Kim et al., 2024c), the model encounters a stream of new facts — entity A has property X, entity B has property Y, entity C has property Z — and is tested on whether it can recall them later. Success requires retrieval: find the most recent mention of the relevant entity and report its associated value. But if the stream instead says "entity A has property X," then later "entity A now has property Y," and later still "entity A actually has property Z," the model cannot simply retrieve the most recent mention — it must recognize that the later mentions contradict and supersede the earlier ones, and it must actively discard the stale information. This requires detecting the logical relationship between temporally separated statements, which is a reasoning capability that retrieval-centric benchmarks do not test.

The paper demonstrates that this distinction is not academic: models do struggle specifically with revision tracking. Performance drops from 42.2% on Sparse questions (few revisions) to 33.3% on Frequent questions (many revisions) on OAKS-B, and from 65.4% to 53.0% on OAKS-N (Table 2). More diagnostically, the behavioral analysis (Table 5) shows that under "No GT Transition" conditions — where the ground truth stays the same — the dominant error mode is Volatility (32.1% average across models): the model unnecessarily changes its prediction even though nothing has changed. This is exactly the failure you would expect if models confuse "new information about a related entity" with "a revision to the entity I'm tracking" — a confusion that never arises when facts only ever diverge rather than revise.

What makes this an intellectual contribution rather than just a dataset design choice: it identifies a capability — contradiction-aware temporal reasoning — that the field had implicitly assumed was tested by existing benchmarks but was not. By explicitly constructing datasets where the same fact is revised multiple times (average 4.7 updates per question), and by annotating answers at every interval to reveal when models fail, the paper isolates this capability and shows that it is a distinct bottleneck, separable from retrieval accuracy, long-context processing, or aggregate reasoning ability. This reframes the research agenda: improving continual knowledge learning is not just about better retrieval or larger context windows, but about mechanisms that explicitly track freshness — recognizing when old information has been invalidated.

Innovation 3: Over-Updating and Under-Updating Are Distinct, Model-Specific Behavioral Biases — and Thinking Mode Shifts the Bias Rather Than Eliminating It

The fine-grained transition analysis in Section 6.1 and Table 5 reveals a finding that would be invisible in aggregate accuracy comparisons: different models exhibit qualitatively different types of tracking errors, and these error types are stable enough to characterize as model-specific biases. Some models (GPT-OSS, Qwen3 non-thinking) are "over-updaters" — they change their predictions frequently, with a high rate of Volatility (changing when they should stay) but relatively low Obstinacy (staying when they should change). Other models (Gemini 2.5, Gemma 3) are "under-updaters" — they resist changing predictions, with high Obstinacy and Stubbornness but lower Volatility. These are not differences in accuracy — GPT-OSS-120B and Gemma 3-27B achieve similar accuracy on OAKS-B (37.5% vs. 37.8% in Table 15) — but they represent radically different failure surfaces with different engineering implications.

This is significant because it contradicts the implicit assumption in much of the LLM evaluation literature that errors are broadly interchangeable — that a 60% accurate model is a 60% accurate model, and the path to 70% is "make it better at the task." The paper shows that the kind of 60% matters enormously. An over-updating model would benefit from mechanisms that penalize unnecessary prediction changes (confidence thresholds, temporal smoothing, explicit state-persistence induction); an under-updating model would benefit from mechanisms that increase sensitivity to contradictory evidence (recency biasing, explicit contradiction detection). Without knowing which failure mode dominates, improvement efforts are shots in the dark.

The thinking-mode analysis sharpens this insight further. Enabling thinking mode on Qwen3-30B shifts the behavioral profile: Adaptability increases (34.3% → 39.6%), Stability increases (26.3% → 30.4%), and Obstinacy drops substantially (28.1% → 18.9%) — the model becomes both more responsive to true transitions and more stable under no-transition conditions. But Distraction Susceptibility is not reduced — in fact, for Qwen3-30B it increases slightly (34.0% → 37.0% in Table 6). Thinking mode helps the model decide when to change but does not solve the problem of losing track within a phase. This is a nuanced finding: inference-time reasoning (chain-of-thought) addresses the decision boundary problem (update vs. don't update) but not the state-maintenance problem (keep the correct answer stable under accumulating context). The paper does not merely report that thinking improves accuracy — it identifies which component of the error budget thinking addresses and which it leaves untouched, providing a roadmap for where additional interventions are needed.

This is a fundamental empirical finding rather than an incremental insight. It establishes that knowledge tracking in streaming contexts decomposes into at least two separable sub-capabilities (update detection and state maintenance), that current models exhibit systematic biases in opposite directions on these sub-capabilities, and that the most widely-used inference-time intervention (chain-of-thought reasoning) addresses only one of them.

Innovation 4: Retrieval-Augmented Generation and Agentic Memory Systems — Purpose-Built for Dynamic Context — Do Not Solve the Tracking Problem and Sometimes Underperform Simple Concatenation

The paper includes a revealing negative result that challenges a dominant assumption in the LLM deployment community: retrieval augmentation and specialized memory architectures, which are widely believed to be the solution to long-context and dynamic-knowledge problems, provide marginal or negative benefit on OAKS, particularly under frequent knowledge updates.

Naive RAG with top-30 retrieval improves accuracy by only 0.4% on Sparse questions while decreasing accuracy by 0.8% on Frequent questions, relative to base concatenation (Table 2, comparing the "Base" and "RAG" columns across the "All" and subset breakdowns for various models; the patterns are consistent across model families). Agentic memory systems — HippoRAG-V2, MemAgent, A-Mem — built on Qwen2.5-7B all underperform naive RAG in aggregate accuracy (Table 4), with only MemAgent showing competitive performance on Moderate and Frequent subsets (33.6% and 29.1% vs. 32.5% and 27.5% for RAG, respectively).

The authors identify two specific failure mechanisms that explain this (Section 5.1, Appendix D.2): (1) retrieval becomes difficult when many semantically related chunks exist with different values for the same fact — the retriever returns all of them, and the model struggles to determine which is the most recent; and (2) even with successful retrieval, models suffer from the "lost in the middle" effect (Liu et al., 2024), where processing large numbers of retrieved chunks degrades their ability to locate and reason over the specific evidence needed. The finding that MemAgent outperforms RAG specifically on Frequent-update questions (Table 4) hints that interval-based memory tracking — MemAgent's training objective — provides some robustness to the revision-density problem, but the effect is modest and does not generalize to the full dataset.

This is significant not as a method contribution but as a boundary condition on current approaches. The field's default response to "LLMs struggle with long, dynamic contexts" has been "add retrieval" or "add memory." The paper demonstrates that these approaches, as currently implemented, are insufficient for the specific challenge of tracking facts as they are revised — a problem that requires not just surfacing relevant information but actively managing temporal validity. This finding redirects research attention: rather than building more sophisticated retrieval or memory architectures, future work should focus on mechanisms that explicitly track information freshness, detect contradictions, and suppress stale knowledge. The agentic memory results also highlight a training-evaluation mismatch: systems trained to optimize final-answer correctness (like MemAgent, which receives rewards after processing all chunks) may not maintain correct intermediate states, which matters in real-time deployment even if final accuracy looks acceptable.

5. Experimental Analysis

Evaluation Methodology

Dataset. Two datasets are used: OAKS-BABI (OAKS-B), a synthetic dataset derived from BABILong containing 1,224 questions across 12 context documents with 65 chunks of 2k tokens each (128k tokens total), and OAKS-Novel (OAKS-N), a human-curated dataset from 39 full-length novels containing 870 multiple-choice questions with an average book length of 150.6k tokens (77.6 chunks on average). Both datasets feature the same set of questions asked at every time interval, with answers transitioning as new information arrives. Questions are stratified into Sparse, Moderate, and Frequent subsets based on answer-change frequency (Table 7): for OAKS-B, the boundaries are 2–3, 4–5, and 6–20 changes; for OAKS-N, 2–3, 4, and 5–19 changes. The average number of answer changes per question is 4.7 for both datasets (Section 3.1).
Base model(s). The paper evaluates 14 LLMs spanning open-source and proprietary families (Section 4, Table 11). The primary baseline family is Qwen3 (Yang et al., 2025) across multiple sizes: 4B, 8B, 30B (MoE, 3B active), 80B (MoE, 3B active), and 235B (MoE, 22B active). Additional open-source models include Qwen2.5-7B (Yang et al., 2024), GPT-OSS at 20B (3.6B active) and 120B (5.1B active) (Agarwal et al., 2025), and Gemma 3 at 4B and 27B (Team et al., 2025). Proprietary models include Gemini 2.5 Flash, Gemini 2.5 Pro (Gemini Team et al., 2025), and Gemini 3 Pro (Google, 2025). Four configurations are tested with "thinking mode" (extended chain-of-thought reasoning): Qwen3-30B-Thinking and Gemini 2.5 Flash, Pro, and Gemini 3 Pro (all with thinking enabled). The Qwen3 family serves as the primary baseline to enable clean scaling analysis within a single architecture family (Section 4).
Metrics. The primary metric is interval-level accuracy: at each time interval $t$ , the model's prediction $p_{j,t}$ is compared to the ground-truth answer $a_{j,t}$ using normalized exact-match string comparison (OAKS-B) or option-letter matching (OAKS-N). Scores are averaged across intervals to obtain a question-level accuracy, then averaged across all questions. Formally: $\text{Accuracy} = \frac{1}{Q} \sum_{j=1}^{Q} \frac{1}{C} \sum_{i=1}^{C} \mathbb{1}[p_{i,j} = a_{i,j}]$ (Appendix B.2.2, Equation 1). Three additional behavioral metrics decompose the error rate: Acquisition Latency (AL) measures how many intervals the model takes to first produce the correct answer after a state change, normalized by total intervals; Distraction Susceptibility (DS) counts incorrect predictions occurring after the first correct prediction within a phase; Phase Miss rate (PM) measures the fraction of intervals belonging to phases where the model never produced a single correct answer. Together, $1 - \text{ACC} = \text{AL} + \text{DS} + \text{PM}$ , providing an exhaustive decomposition of all errors (Appendix B.2.2, Equations 2–4).
Baselines. Four context construction strategies are compared (Section 4, Appendix C.1.3): Base — all chunks up to $t$ are concatenated in order, with older chunks truncated only when the model's context limit is exceeded (Table 11); RAG — top-30 most relevant chunks are retrieved using Qwen3-Embedding-0.6B (Zhang et al., 2025) via cosine similarity, restricted to chunks from previous intervals, and prepended to the prompt; Rolling Window (RW) — only the most recent 30 chunks are retained; RAG+RW — 15 retrieved chunks and 15 most recent chunks are combined. Additionally, three agentic memory systems are evaluated on OAKS-B using Qwen2.5-7B-Instruct as the base model: HippoRAG-V2 (Gutiérrez et al., 2025), a graph-based retrieval framework using Personalized PageRank; MemAgent (Yu et al., 2025), trained with GRPO to incrementally update memory at each timestamp; and A-Mem (Xu et al., 2025b), using a Zettelkasten-inspired interconnected knowledge network (Appendix C.1.2).
Generation budget / compute accounting. There is no variable generation budget in this benchmark — each model produces exactly one prediction per question per interval. The "budget" is the total number of inferences: $D \times C \times Q$ where $D$ is documents, $C$ is chunks per document, and $Q$ is questions per document. This yields approximately 78k inferences for OAKS-B and 67k inferences for OAKS-N per model (Appendix B.2.3). The cost is substantial: a single Qwen3-30B run required approximately 125 GPU hours on Nvidia A100 GPUs. Due to this cost, results are reported from single comprehensive runs rather than multiple trials with different seeds.
Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. The authors note that "the sheer volume of evaluation points across thousands of unique context-question pairs provides a high degree of consistency" (Appendix B.2.3), but this is an assertion rather than a statistically validated claim. The dataset is stratified by answer-change frequency (Sparse, Moderate, Frequent) to enable analysis of how update frequency affects performance, but this stratification is a fixed property of the dataset rather than a validation fold.

Main Quantitative Results

Overall Accuracy Across Models and Datasets

The headline finding from Table 2 is that OAKS remains challenging for all evaluated systems, with substantial room for improvement. Across all open-source models, average accuracy is 33.0% on OAKS-B and 52.9% on OAKS-N. Proprietary models (Gemini) average 60.9% on OAKS-B and 72.6% on OAKS-N — better but far from ceiling, especially on the synthetic OAKS-B dataset. Even the strongest model, Gemini 3 Pro, achieves only 66.3% on OAKS-B and 75.5% on OAKS-N (Base setting, Table 2).

The ranking is consistent: performance scales with model size within the same family, and proprietary models outperform open-source models of comparable or larger scale. Within the Qwen3 family, accuracy on OAKS-B (Base) rises from 26.4% for the 4B model to 46.8% for the 235B model. On OAKS-N, the corresponding range is 51.3% to 64.7%. Qwen3-30B (MoE, 3B active parameters) achieves 35.8% on OAKS-B, which is comparable to Gemma 3-27B at 37.8% — suggesting that the MoE architecture does not fundamentally impair tracking ability despite using far fewer active parameters. Qwen2.5-7B scores 24.7% on OAKS-B, while Qwen3-8B (the generational successor at similar scale) scores 33.1%, confirming that base model improvements translate to tracking ability. GPT-OSS models underperform Qwen3 models of comparable size: GPT-OSS-120B (5.1B active) achieves 37.5% on OAKS-B versus Qwen3-30B (3B active) at 35.8%, but with less active-parameter efficiency.

Questions with frequent updates are substantially harder. As shown in Table 2, aggregating across models, performance on OAKS-B drops from 42.2% on Sparse questions to 40.6% on Moderate and 33.3% on Frequent. On OAKS-N, the corresponding degradation is 65.4% → 57.2% → 53.0%. This is a consistent pattern across nearly every model: the Frequent subset is 8–12 percentage points harder than the Sparse subset on OAKS-B and 10–15 points harder on OAKS-N. For Gemini 3 Pro, the Sparse-to-Frequent gap on OAKS-B is 70.3% → 55.6% (a 14.7-point drop), confirming that even the strongest model is severely affected by update frequency.

RAG Provides Limited and Inconsistent Benefits

The comparison between Base and RAG settings in Table 2 reveals a nuanced picture: naive RAG does not reliably improve performance and sometimes degrades it, particularly on questions with frequent updates. Averaged across models, RAG improves accuracy by 0.4% on Sparse questions but decreases it by 0.04% on Moderate and 0.8% on Frequent subsets (Section 5.1). The pattern is inconsistent across models: Qwen3-30B benefits from RAG on OAKS-B (35.8% → 37.8% overall) but loses ground on OAKS-N (62.8% → 61.0%). Gemini 3 Pro experiences a substantial RAG degradation on OAKS-B (66.3% → 51.3% overall) but only a mild one on OAKS-N (75.5% → 74.0%). This model-specific sensitivity suggests that the interaction between retrieval quality, base model capability, and task format (open-ended vs. multiple-choice) is complex and not well captured by simple retrieval strategies.

Table 12 breaks down the results for Qwen3-30B across Base, RAG, Rolling Window (RW), and RAG+RW settings on both datasets. On OAKS-B, RAG achieves the best overall performance (37.8%, +2.0% over Base), with gains concentrated on the Frequent subset (32.0% vs. 29.4%). The combined RAG+RW approach (15 retrieved + 15 recent chunks) scores 36.7%, slightly below pure RAG. On OAKS-N, the pattern reverses: RAG degrades performance (61.0%, −1.8% vs. Base) while RAG+RW achieves the best result (64.6%, +1.8% vs. Base). The authors hypothesize that novels' narrative-heavy structure makes temporal coherence (provided by the RW component) more important than relevance-based retrieval alone (Appendix D.2).

Thinking Mode Consistently Improves Performance, Especially on Complex Reasoning Questions

Table 3 shows the comparison between standard and thinking mode for Qwen3-30B and Gemini 2.5 (Flash and Pro) on OAKS-B, broken down by question type. Thinking mode provides consistent gains across all models and question types, but the magnitude varies dramatically.

For Gemini 2.5 Pro, enabling thinking raises overall accuracy from 42.9% to 60.3% (+17.4 points). The largest gains come on comparison questions (50.2% → 71.7%, +21.5 points) and bridge questions (29.8% → 51.0%, +21.2 points), which require multi-hop reasoning. Tracking questions, which involve only a single fact but frequent updates, show a much smaller gain (52.4% → 55.3%, +2.9 points). Counting questions sit in between (45.8% → 57.5%, +11.7 points). For Qwen3-30B, the pattern is similar but with smaller absolute gains: overall 35.8% → 43.6% (+7.8 points), with bridge questions showing the largest improvement (24.6% → 37.3%, +12.7 points) and tracking questions showing the smallest (27.1% → 43.8%, +16.7 points — note this is actually a large gain for Qwen3, suggesting the base model is particularly weak at tracking).

These results confirm that thinking mode's benefits are largest when task complexity is high — particularly when questions require simultaneous tracking of multiple independent states (bridge questions) or comparative reasoning over distributed evidence (comparison questions). Single-state tracking with frequent updates benefits less, which aligns with the finding in Section 6.1 that thinking mode primarily helps with transition detection (deciding when to update) rather than state maintenance (keeping the answer stable once found).

However, on OAKS-N, thinking mode produces the opposite result. Table 13 shows that for both Qwen3-30B and Gemini 2.5 Flash with RAG, enabling thinking degrades accuracy: Qwen3-30B drops from 62.8% to 61.8%, and Gemini 2.5 Flash drops from 67.3% to 65.6%. The authors hypothesize that "inference-time scaling is most effective for tasks with high structural complexity, such as those requiring simultaneous tracking of multiple states as seen in OAKS-B, rather than the nuanced linguistic extraction required by OAKS-N" (Appendix D.1). This suggests that thinking mode's internal reasoning process, while helpful for explicit multi-evidence integration, may interfere with the implicit, intuition-like processing needed to track states woven into natural narrative.

Agentic Memory Systems Underperform Simple Baselines in Aggregate, with Partial Advantages on Frequent Updates

Table 4 presents the accuracy of agentic memory systems (HippoRAG-V2, MemAgent, A-Mem) compared to Base, RAG, RW, and RAG+RW settings, all using Qwen2.5-7B-Instruct as the base model, evaluated on OAKS-B.

In aggregate, no agentic memory system outperforms naive RAG. RAG (top-30) achieves 32.4% overall accuracy, while MemAgent reaches 31.3%, A-Mem reaches 30.3%, and HippoRAG-V2 reaches only 20.8% — worse than even the simple Base setting (24.7%). The RW strategy alone (27.6%) also underperforms RAG, and the RAG+RW combination (31.5%) trails pure RAG by 0.9 points.

However, the Moderate and Frequent subsets tell a different story. MemAgent achieves 33.6% on Moderate and 29.1% on Frequent, outperforming RAG (32.5% and 27.5% respectively). A-Mem similarly outperforms RAG on Moderate (33.3% vs. 32.5%) but underperforms on Frequent (25.6% vs. 27.5%). HippoRAG-V2 underperforms RAG across all subsets. The authors attribute MemAgent's relative strength to its interval-based memory tracking training objective, which aligns with OAKS's evaluation protocol — even though the training rewards were computed after processing all chunks rather than at each interval (Section 5.3). This partial advantage on higher-frequency updates suggests that purpose-built memory architectures can help with revision tracking, but current implementations are not yet robust enough to outperform simple retrieval baselines in aggregate, and their benefits are sensitive to training-evaluation alignment.

Behavioral Analysis Reveals Distinct, Model-Specific Error Biases

Table 5 (Section 6.1) analyzes the knowledge tracking behavior of representative models on OAKS-B by categorizing every prediction at every interval into one of eight behavioral archetypes, based on whether the ground truth changed (Phase Transition vs. No Transition), whether the model's prediction changed relative to its previous prediction (Change vs. Stay), and whether the resulting answer was correct (✓ vs. ✗). The eight archetypes are: Adaptability (GT Changed, Model Changed, Correct), Maladaptation (GT Changed, Model Changed, Incorrect), Prescience (GT Changed, Model Stayed, Correct), Stubbornness (GT Changed, Model Stayed, Incorrect), Lag (GT Stayed, Model Changed, Correct), Volatility (GT Stayed, Model Changed, Incorrect), Stability (GT Stayed, Model Stayed, Correct), and Obstinacy (GT Stayed, Model Stayed, Incorrect). Rates are averaged within each GT scenario (Change or Stay) to sum to 100%, since Stay intervals constitute 94% of all intervals.

Correct transitions are easier than correct answers. Aggregating across models, correct transition behaviors (Adaptability + Stability) occur at 31.2% on average, while incorrect transition behaviors occur at 18.8%. However, correct answers (Adaptability + Prescience + Lag + Stability) sum to only 22.8%, while incorrect answers sum to 27.2%. This indicates that models more often correctly detect whether a transition should occur than correctly determine what the new answer should be.

Models exhibit systematic biases toward over-updating or under-updating. GPT-OSS and Qwen3 non-thinking variants show higher overall change rates (Change behaviors average 63.2% vs. Stay at 36.8%), while Gemini 2.5 and Gemma 3 show the opposite pattern (Stay at 55.8% vs. Change at 44.2%). This translates to different dominant error modes: Volatility is the primary error for GPT-OSS (46.5% under No GT Transition conditions) and Qwen3-30B non-thinking (36.7%), while Obstinacy dominates for Gemini 2.5 non-thinking (40.7%) and Gemma 3-27B (35.6%). The behavioral bias is model-specific and stable — it characterizes the model's tendency, not just task difficulty.

Thinking mode shifts behavior toward increased responsiveness and increased stability simultaneously. Comparing Qwen3-30B non-thinking vs. thinking: Adaptability increases (34.3% → 39.6%), Stability increases (26.3% → 30.4%), and Obstinacy drops (28.1% → 18.9%). For Gemini 2.5 Flash, the shift is even more dramatic: Adaptability rises from 36.3% to 47.5%, Stability rises from 35.0% to 43.3%, while Obstinacy plummets from 40.7% to 16.8%. Thinking mode simultaneously makes models more sensitive to true transitions and more robust against spurious ones. However, Volatility does not decrease — for Qwen3 it rises slightly (36.7% → 37.7%), and for Gemini it rises substantially (16.9% → 27.5%). This confirms that thinking mode improves transition detection but does not solve the distraction problem.

Intra-Phase Analysis Quantifies the Trade-off Between Phase Capture and Stability

Table 6 (Section 6.2) decomposes accuracy into the three behavioral metrics (AL, DS, PM) for five model configurations on OAKS-B, enabling direct comparison of how models achieve (or fail to achieve) their accuracy scores.

Gemma 3-27B and GPT-OSS-120B achieve nearly identical accuracy (37.8% vs. 37.5%), but their error profiles are fundamentally different. Gemma 3 has a high Phase Miss rate (30.3%) versus relatively low Distraction Susceptibility (26.5%) and Acquisition Latency (5.4%). This means Gemma 3 often fails to find the correct state at all for entire phases, but when it does find it, it maintains it reasonably well. GPT-OSS has a high Distraction Susceptibility (38.8%) and low Phase Miss rate (15.1%), meaning it finds the correct state at least once in most phases but frequently loses it afterward — consistent with its "over-updater" profile from Table 5.

Thinking mode improves accuracy by reducing Phase Miss rate, not by reducing distraction. For Qwen3-30B, enabling thinking raises accuracy from 35.8% to 43.6%. The dominant source of improvement is Phase Miss rate dropping from 23.2% to 10.4% — the model becomes much better at capturing each phase at least once. However, Distraction Susceptibility increases from 34.0% to 37.0%, and Acquisition Latency increases from 6.9% to 9.0%. The model gets the right answer more often but takes slightly longer to get there and gets distracted slightly more often once it does. For Gemini 2.5 Flash, thinking raises accuracy from 43.2% to 56.2%, with Phase Miss rate dropping from 22.3% to 7.0% and Distraction Susceptibility rising from 28.3% to 31.8%. The same trade-off pattern holds.

Failure Modes Vary Systematically by Question Type

Figure 2 (Section 6.3) breaks down accuracy and the three behavioral metrics across OAKS-B question types. Bridge questions, which require simultaneous tracking of multiple states, exhibit the highest Distraction Susceptibility (approximately 38%) — consistent with the hypothesis that monitoring multiple evolving facts increases susceptibility to contextual interference. Tracking questions, despite involving only a single piece of evidence, show the highest Phase Miss rate — consistent with their high update frequency (8.8 changes per question on average vs. 3.7–5.7 for other types; noted in Appendix B.1.2). Comparison questions show relatively high accuracy (approximately 58%) and lower DS and PM, which the authors attribute to the fact that "candidate answers are embedded inside the question, similar to multiple choice questions, reducing the search space" (Section 6.3) — effectively, the question format itself provides scaffolding that constrains the model's output space.

Accuracy Degrades at Later Intervals and Under Longer Contexts

Figure 3 (Section 6.4) plots accuracy as a function of the time interval (chunk index) for OAKS-B, OAKS-N, and OAKS-B broken out by question type. Accuracy degrades monotonically with interval number on OAKS-B, from approximately 45% at early intervals to approximately 25% at the final intervals (interval 65). The degradation is steepest for bridge and tracking questions, which approach 15–20% at the latest intervals. On OAKS-N, the degradation is more gradual and less severe: accuracy drops from roughly 65% at early intervals to roughly 55% at later intervals. The authors attribute this difference to the fact that OAKS-B evidence typically appears only once — if missed, the error persists — while OAKS-N narrative evidence is often "revisited across intervals, partially mitigating error accumulation" (Section 6.4).

The correlation between OAKS performance and general long-context ability (measured by LongBench-v2) is moderate but not strong (Appendix E.3). Across five models of similar size, the Pearson correlation between LongBench-v2 and OAKS-B is 0.69, dropping to 0.45 for the Frequent subset. For OAKS-N, the correlations are 0.34 and 0.30 respectively. These numbers indicate that long-context understanding is a prerequisite for OAKS but explains less than half of the variance, confirming that the benchmark captures capabilities beyond general long-context processing — specifically, online adaptation and temporal state tracking.

Ablation Studies and Robustness Checks

Number of retrieved chunks in RAG: Figure 5 (Appendix D.2) reports Pass@k for the Qwen3-Embedding-0.6B retriever as a function of $k$ (number of retrieved chunks). Pass@k approaches near-perfect recall as $k$ reaches 60, but Figures 6 and 7 show that downstream model accuracy on OAKS-B and OAKS-N peaks around $k = 30$ and degrades with more chunks, consistent with the "lost in the middle" phenomenon (Liu et al., 2024). The chosen default of $k = 30$ is empirically justified by this sweep. The RAG+RW ablation (Table 12) further validates that the combination of 15 retrieved + 15 recent chunks outperforms pure RAG on OAKS-N (64.6% vs. 61.0%) but not on OAKS-B (36.7% vs. 37.8%).

RAG vs. Rolling Window vs. combined strategies: Table 12 provides a clean comparison of all four context construction strategies for Qwen3-30B on both datasets. The key finding is that no single strategy dominates across datasets, implying that the optimal context construction strategy depends on task characteristics (narrative coherence, evidence distribution, temporal structure). This is a robustness concern: the benchmark's difficulty ranking across strategies is not invariant, meaning that results from one strategy cannot be straightforwardly extrapolated to another.

Oracle vs. non-oracle evidence matching: Appendix E.2 examines the relationship between evidence correctness and answer correctness on OAKS-N. Using Gemini 2.5 Pro as a judge to assess whether the model's predicted reasoning aligns with the annotated evidence, the analysis finds that 47.3% of errors involve both incorrect evidence and incorrect answers, 19.7% involve correct evidence but incorrect answers (the model found the right evidence but selected the wrong option), and 14.2% involve incorrect evidence but correct answers (the model arrived at the right answer through wrong reasoning, often via parametric knowledge). This decomposition reveals that answer selection errors persist even when evidence retrieval succeeds, and that parametric knowledge contamination is a non-trivial confound on OAKS-N.

Correlation with long-context understanding: Table 14 shows LongBench-v2 performance for five models of similar scale (Qwen2.5-7B: 30.0; Gemma 3-4B: 29.4; GPT-OSS-20B: 14.8; Gemma 3-27B: 33.6; Qwen3-30B: 32.6). The Pearson correlation with OAKS-B accuracy is 0.69, and with OAKS-N accuracy is 0.34. For Frequent subsets only, these drop to 0.45 and 0.30 respectively (Appendix E.3). This confirms that while OAKS partially depends on long-context processing ability, it captures a distinct capability — specifically, online adaptation to evolving knowledge — that is separable from general long-context understanding.

Critical Assessment

Claim: OAKS is difficult across all models, with substantial room for improvement.

Genuinely supported. The evidence is unambiguous: the strongest model (Gemini 3 Pro) achieves 66.3% on OAKS-B and 75.5% on OAKS-N (Table 2), meaning it fails on roughly one-third to one-quarter of all interval predictions. Open-source models fare dramatically worse, with the best open-source configuration (Qwen3-235B with RAG on OAKS-N) reaching only 66.0%. These are not ceiling effects — they represent genuine failures to track state transitions. However, the difficulty is dataset-dependent: OAKS-B (synthetic) is substantially harder than OAKS-N (natural, multiple-choice). This could be because the multiple-choice format reduces the problem space (random baseline ~18.6%), because OAKS-B facts are embedded in irrelevant literary distractor text while OAKS-N facts are integral to the narrative, or because OAKS-B questions require more explicit multi-evidence integration (counting, bridge, comparison types) while OAKS-N questions test state changes within a more naturalistic flow. The paper does not isolate these factors, so the claim "OAKS is difficult" is supported, but why it is difficult — and whether the difficulty comes from the same source across datasets — is underexplored.

Claim: Performance degrades under frequent knowledge updates and at later intervals.

Strongly supported. The Frequent-vs-Sparse gap is consistent across every model in Table 2 (8–15 percentage points on OAKS-B, 10–15 points on OAKS-N). The interval-level degradation in Figure 3 is monotonic and substantial on OAKS-B (approximately 45% → 25%). These are clean, replicated patterns that do not depend on specific model choices.

However, there is a confound that the paper acknowledges but does not fully address: as intervals increase, context length also increases. The degradation at later intervals could be caused by (a) the increasing difficulty of tracking states over longer horizons (the paper's claimed mechanism), (b) standard long-context degradation (models perform worse on longer inputs regardless of temporal dynamics), or (c) both. The correlation analysis with LongBench-v2 (moderate correlation, r=0.69 on OAKS-B) partially separates these — if degradation were purely a long-context effect, the correlation would be much higher — but does not isolate the temporal tracking component cleanly. A controlled experiment that varies context length while holding the number of state transitions constant (or vice versa) would strengthen the claim that temporal dynamics specifically cause the degradation, but such an experiment is not reported.

Claim: Models exhibit distinct, model-specific error biases (over-updating vs. under-updating).

Supported, but with caveats about small sample sizes. Tables 5 and 15 report behavior frequencies for 14 model configurations on OAKS-B (and Table 16 for OAKS-N). The patterns — GPT-OSS over-updates, Gemini under-updates, thinking mode shifts the bias — are qualitatively consistent and interpretable. However, the paper reports results from a single run per model (Appendix B.2.3), meaning there are no error bars or confidence intervals on the behavioral frequency estimates. Given that behavioral frequencies are computed from a finite number of phase transitions (an average of 4.7 transitions per question across 1,224 questions on OAKS-B yields approximately 5,750 total transition events), sampling variance could be non-negligible for individual model-bin combinations. The claim that models have stable behavioral biases would be strengthened by test-retest reliability analysis or cross-validation across different seeds.

Additionally, the behavioral categories in Table 5 are descriptive labels (Adaptability, Volatility, etc.) applied to mechanically defined conditions — there is no validation that these categories correspond to the cognitive constructs the labels imply. For example, "Volatility" simply means the model changed its prediction when the ground truth did not change, and the prediction was wrong. Whether this actually reflects distraction by surrounding context (as the paper argues in Section 6.2) or some other mechanism (miscalibration, boundary effects from truncation, format inconsistency) is not empirically verified. The labels provide a useful taxonomy for comparison, but they should not be mistaken for causal explanations of model behavior.

Claim: Thinking mode improves accuracy by reducing Phase Miss rate, not by reducing distraction.

Well-supported and genuinely informative. Table 6 provides the key evidence: for both Qwen3-30B and Gemini 2.5 Flash, thinking mode dramatically reduces PM (23.2% → 10.4% and 22.3% → 7.0% respectively) while DS slightly increases (34.0% → 37.0% and 28.3% → 31.8%). This is a clean, quantitative decomposition that reveals a specific mechanism: thinking helps the model find the right answer at least once per phase but does not help it maintain that answer stably. The result is robust across two model families. The limitation is that "thinking mode" is a black-box intervention — the paper does not analyze what the thinking traces actually contain, so the causal pathway (e.g., thinking helps by enabling explicit comparison of current state to stored state, or simply by allocating more computation to the final answer) remains speculative.

Claim: RAG and agentic memory systems do not solve the tracking problem.

Supported, with important nuance. Table 4 clearly shows that no agentic memory system outperforms naive RAG in aggregate accuracy, and HippoRAG-V2 performs catastrophically (20.8%). Table 2 shows that RAG provides marginal and inconsistent benefits over Base across models and datasets. However, the claim is conditional on the specific implementations tested: Qwen3-Embedding-0.6B for retrieval, specific hyperparameter choices (top-30 retrieval, 2k-token chunks), and specific agentic memory systems trained on different objectives. The paper cannot claim that all conceivable RAG or memory approaches would fail — only that several representative, off-the-shelf approaches do. This is a valuable negative result, but its generalizability depends on how representative the tested systems are of the broader design space. The finding that MemAgent's interval-based training provides partial advantages on Frequent subsets (Table 4) actually suggests that better-aligned training objectives could improve performance, which is an optimistic signal for future work rather than a pessimistic one.

Missing experiments that would strengthen the paper:

Statistical reliability: test-retest analysis with different seeds to quantify variance in accuracy and behavioral metrics. Given the single-run reporting, it is impossible to determine whether differences between models (e.g., Qwen3-80B at 41.1% vs. Qwen3-235B at 46.8% on OAKS-B Base) are statistically reliable or within noise.
Context-length controlled comparisons: evaluate models on contexts with identical temporal dynamics but varying total lengths to isolate degradation due to horizon effects from degradation due to absolute context length.
Thinking trace analysis: for models with thinking mode, qualitative or quantitative analysis of the reasoning traces to understand how thinking enables better phase capture — is it explicit contradiction detection, recency-weighted retrieval, something else?
Fine-tuning baselines: the paper evaluates zero-shot inference-time adaptation only (deliberately, as a scope choice), but including even a simple fine-tuning baseline (e.g., train a model to output "the answer has changed to X" when encountering revision evidence) would establish a floor for what task-specific training can achieve.
Cross-family transfer: the behavioral bias analysis (Table 5) shows that different model families have different error profiles. It would be informative to know whether these biases transfer across families when models are fine-tuned on the same data, or whether they are inherent to pretraining.
Adversarial distractor sensitivity: controlled insertion of "distractor facts" — statements that are semantically related to the tracked fact but temporally irrelevant — to directly test the Distraction Susceptibility metric's interpretation as measuring contextual interference.

Overall, the experimental analysis provides strong evidence for the benchmark's difficulty and for the existence of distinct, model-specific failure modes. The behavioral metrics (AL, DS, PM) represent a genuine advance in diagnostic evaluation. The main weaknesses are the lack of statistical reliability quantification, the absence of controlled experiments that isolate the specific mechanisms causing degradation (temporal dynamics vs. context length vs. distractor interference), and the black-box nature of the thinking mode analysis. The paper succeeds in establishing OAKS as a challenging diagnostic tool; it is less successful in explaining why specific failures occur, which limits the prescriptive value of the findings for model improvement.

6. Limitations and Trade-offs

The Difficulty Estimation Protocol Is Prohibitively Expensive and Not Accounted for in Headline Performance

The assumption or constraint. The core evaluation protocol requires models to make predictions at every time interval (every 2k-token chunk) for every question. This yields approximately 78k inferences for OAKS-B and 67k inferences for OAKS-N per model, per context construction strategy (Appendix B.2.3). The authors explicitly acknowledge this cost:

"Evaluation on OAKS entails extensive inference due to the incrementally accumulating nature of the context. While we have included 14 representative models in this study, due to limited computational costs and API expenses, we could not further test on broader range of architectures." (Limitations)

"the total inference count is calculated as D × C × Q, where D represents the number of documents(books), C the number of chunks per document, and Q the number of questions per interval. This results in approximately 78k inferences for OAKS-BABI and 67k for OAKS-Novel per model." (Appendix B.2.3)

A single Qwen3-30B run required approximately 125 A100 GPU hours (Appendix B.2.3). For proprietary models like Gemini 3 Pro, the cost is not just GPU time but API expenditure — and the paper evaluates these models across Base, RAG, and sometimes thinking-mode configurations, multiplying the cost further.

The consequence. The computational cost fundamentally constrains the benchmark's utility in several ways that are not reflected in the headline claims. First, reproducibility: independent researchers without access to large GPU clusters or substantial API budgets cannot fully reproduce the paper's main results (evaluating all 14 models across all strategies). This makes the benchmark more of a measurement report than a community tool — unlike benchmarks where a single model evaluation costs a few GPU-hours, OAKS creates a barrier to entry that will limit adoption. Second, statistical reliability: the paper reports results from single comprehensive runs with no error bars or confidence intervals, explicitly citing the cost:

"Given the substantial computational cost, we report results from a single comprehensive run for each model rather than multiple trials with different seeds." (Appendix B.2.3)

The authors argue that "the sheer volume of evaluation points across thousands of unique context-question pairs provides a high degree of consistency," but this is an assertion, not a validation. Without multi-seed analysis, it is impossible to determine whether the reported differences between models — e.g., Qwen3-235B at 46.8% vs. GPT-OSS-120B at 37.5% on OAKS-B Base (Table 2) — are statistically reliable or within sampling noise from a single decoding trajectory. Third, accessibility bias: the benchmark effectively selects for research groups with large compute budgets, which may skew the field toward incremental improvements from well-resourced labs while excluding potentially innovative approaches from groups that cannot afford the evaluation cost.

What evidence exists in the paper. The cost is quantified indirectly via inference counts (Appendix B.2.3) and GPU-hour estimates. The single-run limitation is explicitly stated. The paper does not report any analysis of variance across seeds, nor does it propose a more efficient evaluation protocol (subsampling intervals, using a held-out validation subset, amortizing difficulty estimation across models).

Mitigation status. The paper does not attempt to reduce evaluation cost. The authors suggest in Limitations that future work could "extend this benchmark to a more diverse set of models to further generalize our findings," implying that the current scope is cost-limited rather than by design. No proposal is made for efficient evaluation (e.g., evaluating at a subset of intervals, using a validation set to estimate full-benchmark performance, or releasing a "lite" version). This is a deliberate trade-off — the interval-level granularity is the benchmark's core contribution — but it comes at a steep practical cost that the paper does not adequately address for downstream users.

The Benchmarks Capture In-Context Adaptation Only, Excluding Parametric Updating — Which May Be the More Natural Solution

The assumption or constraint. OAKS evaluates models exclusively in an inference-time adaptation setting: models receive accumulating context in their prompt and must track state changes within that context. No parameter updates occur. The authors are explicit about this scoping decision:

"Our current analysis focuses on inference-time adaptation via incremental context accumulation. However, OAKS also serves as a valuable testbed for parametric online learning, and future work could explore how models update their internal weights to assimilate evolving knowledge." (Limitations)

This is a legitimate scope choice — isolating a specific capability — but it carries an implicit assumption that testing in-context tracking is the right first step, and that parametric adaptation can be studied separately later. This assumption matters because in many real-world scenarios, the natural engineering approach to tracking evolving knowledge is not to feed the entire history into the context window at each step but to update a compressed state representation (in weights or in an external memory that is not simply concatenated context). By excluding parametric updating, the benchmark may be testing the worst-case strategy for the problem and concluding that models struggle, when a more natural strategy (fine-tuning on the stream) might succeed.

The consequence. The benchmark's difficulty level — even state-of-the-art models failing on one-third of intervals — may be partly an artifact of the inference-time-only constraint rather than a fundamental limitation of language models' ability to track evolving facts. If models could update their parameters as new information arrives (e.g., through test-time fine-tuning, LoRA adaptation, or memory-augmented architectures that write to persistent storage), they might perform substantially better, and the failure modes identified in Section 6 (over-updating, under-updating, distraction) might look different or disappear. The paper's behavioral analysis (Section 6.1, Table 5) attributes errors to specific tracking behaviors — Volatility, Obstinacy, Maladaptation — but if these behaviors are caused by the context-processing mechanism (attention over accumulating text) rather than by a fundamental inability to track states, then the conclusions about model capabilities are conditional on the evaluation protocol, not general.

This limitation is particularly consequential for the paper's practical implications. The authors motivate OAKS by referencing real-world scenarios like conversational assistants and embodied agents (Section 1), but in those settings, practical systems do update their state — either through explicit memory modules, database writes, or parameter-efficient fine-tuning. By testing the in-context-only strategy, the paper measures how well models perform the task in a suboptimal configuration, and the headline finding ("models struggle with OAKS") may overstate the practical difficulty of the underlying state-tracking problem.

What evidence exists in the paper. The paper provides partial evidence on this question through the agentic memory experiments (Section 5.3, Table 4). HippoRAG-V2, MemAgent, and A-Mem are tested specifically because they do maintain persistent, structured memory across intervals — they represent a middle ground between pure in-context adaptation and full parametric updating. The finding that these systems underperform naive RAG in aggregate (MemAgent: 31.3% vs. RAG: 32.4% on OAKS-B, Table 4) provides some evidence that even structured memory approaches struggle with the benchmark — the difficulty is not purely an artifact of in-context-only evaluation. However, MemAgent's partial advantage on Moderate and Frequent subsets (33.6% and 29.1% vs. 32.5% and 27.5% for RAG) and the authors' note that its training objective is misaligned with interval-level evaluation suggest that better memory architectures could improve substantially.

Mitigation status. The paper explicitly identifies this as future work and frames OAKS as a testbed for parametric approaches, not a claim that in-context adaptation is the only relevant paradigm. The agentic memory experiments provide a partial robustness check. However, the paper does not report a simple fine-tuning baseline (e.g., fine-tune a small model on OAKS-style tracking data and evaluate its interval-level accuracy), which would establish a floor for what task-specific training can achieve. Without such a baseline, readers cannot determine how much of the benchmark's difficulty is inherent to the state-tracking problem versus specific to the inference-time-only protocol.

Dataset Construction Relies on Ambiguous Human Annotation and LLM-Based Generation, with No Inter-Annotator Agreement Reported

The assumption or constraint. OAKS-N is constructed through a human-in-the-loop pipeline where initial questions are generated by Gemini 2.5 Pro, then "rigorously curated by human experts" (Section 3.1). The authors report that only 55% of initial questions were retained as high-quality, with the rest discarded or reformulated. For the retained questions, annotators labeled correct answers at every chunk for every question, copied supporting evidence, and revised answer options. The authors then "conducted a thorough verification of all annotated files to ensure consistency and correctness" (Appendix B.1.3).

However, the paper does not report any quantitative measure of annotation quality or reliability. Specifically:

No inter-annotator agreement (IAA): the paper does not mention whether multiple annotators labeled the same question-chunk pairs independently, what agreement metric was used, or what the agreement score was. The authors note that "the validity of the question or answer options and the correctness of the selected answers were jointly reviewed and discussed by at least two annotators or authors" (Appendix B.1.3), but joint review with discussion is not the same as independent annotation with IAA measurement — it establishes that disagreements were resolved, not how frequent disagreements were or how difficult the annotation task was.
No quantification of ambiguity: the dataset design strives for exactly one correct answer per chunk, but novel narratives contain inherent ambiguity (implied states, gradual transitions, character motivations that shift subtly). The paper does not report what fraction of questions or chunks required discussion to reach consensus, how difficult annotators found specific types of state transitions, or whether certain novels produced systematically more disagreements than others.
LLM-generated initial pool quality: 55% of initial questions were retained, meaning 45% were discarded — nearly half. The paper characterizes the discarded questions as those that "merely stitched together information from multiple chunks" or had unsupported answer options, but does not report how this 55% retention rate varied across books, question types, or annotators. A 45% discard rate from an LLM-based initial pool raises the question of whether even the retained 55% contains subtle artifacts from the generation process that survived curation.

The consequence. Without IAA, readers cannot assess the difficulty or reliability of the annotation task. If the task is inherently ambiguous — if reasonable annotators frequently disagree on which answer option is correct at a given chunk — then the benchmark's ground truth is noisy, and model accuracy is bounded by annotation reliability as well as model capability. The paper's core statistical decomposition (AL, DS, PM) treats the ground truth as deterministic — each interval has exactly one correct answer — but if the ground truth itself contains annotation errors or ambiguity, then some fraction of measured "acquisition latency" or "distraction susceptibility" may actually be annotation noise rather than model failure.

This is particularly concerning for OAKS-N because the annotation task is genuinely difficult: annotators must determine, for each chunk of a 150k-token novel, which of 5–15 answer options is correct for a question like "What is Victor Frankenstein's primary goal or motivation at this point in the story?" (Table 10). Character motivations are inherently subjective and may not have a clear transition point — Victor Frankenstein does not wake up one morning and decide "today I switch from ambition to horror." The paper acknowledges this difficulty implicitly through the reformulation criterion: questions about state "evolvement" were revised to ask about "status at specific point in the story" (Appendix B.1.3), but this does not eliminate ambiguity — it merely shifts it to the more tractable question of "what does the text explicitly state," which is still subject to interpretation about which text is most recent and dispositive.

What evidence exists in the paper. The paper's discussion of annotation quality is purely qualitative and procedural: the authors describe the filtering criteria, the verification steps, and the annotator qualifications (native English speakers, familiar with the books). No IAA score, no Cohen's kappa, no Krippendorff's alpha, and no annotation difficulty analysis are reported. For OAKS-B, the annotation is algorithmic (timeline generation from deterministic state logs; Algorithm 1), so reliability is not a concern — the ground truth is mechanically correct. For OAKS-N, however, the human annotation component is central to the dataset's validity, and the absence of reliability metrics is a significant evidence gap.

Mitigation status. The authors invested substantial resources in annotation quality ($17,400, 18 annotators, verification by multiple authors) and describe a rigorous process. Joint review by at least two people per annotation is a meaningful quality control step. However, joint review with consensus-seeking discussion is not the same as independent annotation with reliability measurement — it prevents egregious errors but does not quantify the inherent difficulty of the task or the residual ambiguity in the ground truth. The paper effectively treats the consensus annotations as ground truth without measuring how much uncertainty was resolved through discussion rather than through unambiguous textual evidence. This is a standard limitation in dataset papers but a consequential one for a benchmark that derives its value from fine-grained, interval-level diagnostic metrics.

The Benchmark Is Narrowly Scoped to Fictional Narrative Tracking and Synthetic Fact Injection; Generalization to Other Domains Is Unverified

The assumption or constraint. OAKS evaluates knowledge tracking in two specific contexts: synthetic bAbI facts embedded in literary distractor text (OAKS-B) and character/plot state tracking in full-length novels (OAKS-N). Both are narrative contexts — stories with characters, locations, objects, and temporally ordered events. The authors motivate OAKS by referencing a broader range of applications: conversational assistants (Maharana et al., 2024; Wu et al., 2025), embodied agents (Majumder et al., 2023; Kim et al., 2024a), and real-time understanding (Lin et al., 2024). However, no experiments are conducted on non-narrative domains.

The paper does not claim that OAKS generalizes to all dynamic-knowledge settings, but the framing — "ONLINE ADAPTATION TO CONTINUAL KNOWLEDGE STREAMS" — implies a generality that the dataset construction does not support. The tasks tested are: tracking character locations and possessions in synthetic stories; tracking character opinions, goals, and modes of transportation in novels. These are instances of a specific problem: tracking entity states in temporally ordered, causally connected narratives. They do not represent: tracking factual knowledge from news streams (where updates are corrections rather than narrative developments), tracking structured data updates (database records, API responses), tracking multi-party dialogue state (where turn-taking and speaker identity matter), or tracking environment state from sensor observations (where noise and partial observability are central challenges).

The consequence. A model that performs well on OAKS may not perform well on other dynamic-knowledge tasks, and a model that performs poorly on OAKS may not be bad at all forms of state tracking. The behavioral patterns identified in Section 6 — over-updating due to Volatility, under-updating due to Obstinacy — are observed in a specific textual genre (narrative prose with causal coherence). It is unclear whether these patterns would replicate in, say, a stream of news headlines where updates are independent (one article contradicts another with no narrative connective tissue), or in dialogue where the model is an active participant influencing the state, or in code execution traces where states are discrete and logically defined.

The OAKS-B results are particularly limited in generalizability. The bAbI facts are synthetically simple — locations, possessions, transfers — embedded in PG-19 literary text. A model that learns to filter out the literary distractor text and attend only to the formulaic bAbI sentences solves a narrower problem than tracking states in realistic, heterogeneous text. The authors note that they modified location and character names to "randomized, non-existent strings" to prevent parametric knowledge confounds (Appendix B.1.2), but this also removes a key challenge of real-world tracking: entity disambiguation (is "Mary" in chapter 1 the same "Mary" as in chapter 5?) which is trivially solved here by unique randomized names.

What evidence exists in the paper. The paper provides no cross-domain evaluation — no experiments on news, dialogue, code, or structured data. The correlation analysis with LongBench-v2 (Appendix E.3) provides partial evidence that OAKS captures something beyond general long-context understanding (correlation 0.69 on OAKS-B, 0.34 on OAKS-N, dropping further for Frequent subsets), but this only rules out the hypothesis that OAKS is entirely redundant with existing long-context benchmarks. It does not validate that OAKS performance transfers to non-narrative dynamic-knowledge settings.

Mitigation status. The paper does not claim cross-domain generalization and does not attempt to address this limitation. The authors frame OAKS as a first benchmark in a new paradigm — "to the best of our knowledge, the first benchmark to unify these two paradigms" (Section 1) — which implies that domain expansion is future work. The scope is acknowledged implicitly through the dataset descriptions (OAKS-B: "synthetic dataset derived from the BABILong benchmark"; OAKS-N: "sourced from full-length literary novels"), but the title and abstract ("Large language models operating in dynamic real-world contexts") imply broader applicability than the experiments support. A practitioner deciding whether to use OAKS to evaluate their system should know that the benchmark's domain is strictly fictional narrative and synthetic fact tracking, and that performance on OAKS may not predict performance on their specific dynamic-knowledge task.

The Behavioral Analysis Identifies What Models Do Wrong but Not Why — the Mechanisms Underlying Tracking Failures Remain Unexplored

The assumption or constraint. The paper's behavioral metrics (AL, DS, PM) and transition archetypes (Adaptability, Volatility, Obstinacy, etc.) are descriptive taxonomies of model output, not mechanistic explanations of model internals. The paper classifies every prediction into a behavioral category based on the relationship between ground-truth state, model prediction, and previous prediction — but this classification reveals what the model did, not why it did it. The authors often use causal-sounding language to interpret these behaviors:

"Volatility is primarily driven by interference from surrounding contextual information" (Section 6.1)

"this degradation arises because frequent answer changes lead to more dynamic knowledge states, requiring models to repeatedly update multiple facts while retaining previously valid information, which exacerbates both tracking and retention difficulties" (Section 5.1)

These are plausible hypotheses, but the paper provides no direct evidence for them. No experiments manipulate the amount of "surrounding contextual information" to test whether Volatility increases; no ablation varies the number of simultaneously tracked states to test whether distraction susceptibility scales with state count. The behavioral categories are defined mechanically (did the prediction change? was it correct?), not through controlled causal interventions.

The consequence. The diagnostic metrics are valuable for characterizing failure modes across models, but they have limited prescriptive power for fixing those failures. If a model exhibits high Distraction Susceptibility, is the solution to improve attention mechanisms (so the model can better ignore irrelevant context), to improve state representation (so the model stores tracked facts more robustly), to add a confidence threshold (so the model doesn't change predictions unless it is highly certain), or something else? The metrics do not answer this question because they do not isolate the mechanism causing the distraction. Similarly, if thinking mode reduces Phase Miss rate but increases Distraction Susceptibility (Table 6: Qwen3-30B PM drops 23.2% → 10.4%, DS rises 34.0% → 37.0%), the paper can describe this trade-off but cannot explain the causal pathway — does thinking help because it enables explicit temporal reasoning (comparing "before" and "after" states), or because it simply allocates more computation to the final answer, or because the thinking traces contain meta-cognitive checks that the non-thinking model lacks? Without answering this, the practical takeaway ("use thinking mode") is a heuristic, not a principled recommendation.

The evidence-level analysis in Appendix E.2 provides a partial exception: by using an LLM judge to assess whether model predictions align with annotated evidence, the paper takes a step toward mechanistic explanation. The finding that 19.7% of errors involve correct evidence but incorrect answers (the model found the right text but chose the wrong option) suggests a reasoning failure rather than a retrieval failure. Similarly, the 14.2% of correct answers with incorrect evidence suggests parametric knowledge contamination. These are more mechanistic than the behavioral categories, but they rely on an LLM judge (Gemini 2.5 Pro) whose own reliability is not validated, and they are reported for OAKS-N only, not across models or question types.

What evidence exists in the paper. The paper provides extensive behavioral classification but minimal mechanistic investigation. The thinking vs. non-thinking comparison (Table 5, Table 6) is the closest the paper comes to a causal experiment — it shows that an intervention (enabling thinking mode) shifts the behavioral profile — but it does not analyze how thinking achieves this shift. The paper does not report analysis of thinking traces (what do models actually write in their chain-of-thought when they correctly detect a transition vs. when they miss one?), nor does it run controlled experiments that manipulate specific features of the input (context length, distractor density, state change frequency) and measure the effect on behavioral metrics. The question-type analysis (Figure 2) shows that bridge questions cause higher DS and tracking questions cause higher PM, which is suggestive of mechanisms (simultaneous tracking → distraction; frequent updates → phase misses), but these are correlations, not causal demonstrations.

Mitigation status. The paper acknowledges this limitation implicitly through the framing of the work as an evaluation benchmark — the goal is to measure behavior, not to explain it. The authors are transparent that the behavioral categories are descriptive:

"For descriptive purposes, we map these combinations to specific behavioral archetypes (e.g., Adaptability)" (Section 6.1)

But the paper's interpretive language often goes beyond description to causal attribution (e.g., "Volatility" implies the model is distracted, not just that it changed a prediction incorrectly). The distinction matters because the paper is positioned to guide future research — but if the diagnostic categories are misunderstood as causal explanations, researchers may invest effort in solving the wrong problems (e.g., building "distraction filters" when the real issue is miscalibrated confidence). This is a trade-off inherent to any descriptive benchmark, not a flaw unique to OAKS, but it is particularly consequential here because the paper's main contribution is diagnostic, and the gap between description and explanation limits the diagnostic value.

Performance Degradation at Later Intervals Is Confounded with Context Length — the Benchmark Cannot Distinguish Temporal Tracking Difficulty from Standard Long-Context Degradation

The assumption or constraint. The paper reports that accuracy degrades at later time intervals (Section 6.4, Figure 3) and attributes this to compounding state-tracking errors:

"This degradation is more pronounced in OAKS-B, where the supporting evidence typically appears only once; if the model fails to capture it when it first appears, the error persists and accumulates in subsequent intervals." (Section 6.4)

However, as intervals increase, two things change simultaneously: (1) the model must track states over a longer temporal horizon with more intervening updates, and (2) the absolute context length grows (more chunks are accumulated into $S_t$ ). These are confounded in the OAKS evaluation protocol: interval 65 always has 65 chunks of context (~128k tokens for OAKS-B), while interval 10 has only 10 chunks. The degradation at later intervals could be caused by the increasing temporal tracking difficulty (the paper's claimed mechanism), by standard long-context processing degradation (models perform worse on longer inputs regardless of temporal dynamics, a well-documented phenomenon; Liu et al., 2024), or by a combination of both.

The consequence. The paper's central claim — that OAKS measures online adaptation to evolving knowledge, distinct from long-context understanding — is partially undermined by this confound. If performance at later intervals degrades primarily because of context-length effects (models attend less effectively to relevant information when it is buried in 128k tokens of text), then the benchmark is measuring long-context retrieval difficulty as much as state-tracking ability. This would not invalidate the benchmark (state tracking in long contexts is a real problem), but it would weaken the paper's claim that OAKS captures a distinct capability separable from general long-context processing.

The confound also complicates the interpretation of the behavioral metrics. High Distraction Susceptibility at later intervals (where contexts are longer) could reflect the model losing track of the correct state because it is genuinely distracted by similar but temporally irrelevant information — or it could reflect standard attention degradation over long sequences, where the model's representation of all information degrades with distance, making any retrieval noisier. Without a control condition that varies context length independently of temporal horizon, these alternatives cannot be distinguished.

What evidence exists in the paper. The paper provides one piece of partial disambiguation: the correlation analysis with LongBench-v2 (Appendix E.3). Across five models, OAKS-B accuracy correlates with LongBench-v2 at r = 0.69, which drops to r = 0.45 for the Frequent subset. If OAKS were purely measuring long-context ability, the correlation would be much higher, and it would not drop for high-frequency-update questions (since those occur at the same context lengths as low-frequency questions). The fact that the correlation decreases when state-tracking demands increase (Frequent subset) suggests that OAKS captures something beyond long-context ability — the temporal tracking component becomes more important, and LongBench-v2 (which is a static long-context benchmark) captures less of it.

However, this is indirect evidence. A direct disambiguation would require a control experiment: evaluating models on contexts of varying length but identical temporal dynamics, or contexts of identical length but varying temporal dynamics. The paper does not report such an experiment. The comparison between OAKS-N and OAKS-B degradation patterns provides another piece of indirect evidence: OAKS-N degrades more gradually (Figure 3) despite having longer absolute context lengths (average 150.6k tokens vs. 128k), suggesting that the nature of the text (narrative revisits evidence vs. evidence appears once) matters more than absolute length. But again, this is correlational.

Mitigation status. The paper does not attempt to disentangle temporal horizon from context length experimentally. The limitation is partially acknowledged implicitly through the OAKS-B vs. OAKS-N comparison (Section 6.4), where the authors note that OAKS-N "show[s] more stable performance, likely because relevant information is often revisited across intervals, partially mitigating error accumulation." This recognizes that evidence revisitation — a feature of the text, not of the model's tracking ability — affects performance. But the paper does not frame this as a confound requiring experimental control, nor does it suggest future work to isolate the temporal tracking component from the long-context component. For a benchmark whose primary contribution is measuring a specific capability (online state tracking), the inability to attribute performance degradation uniquely to that capability is a meaningful limitation.

7. Implications and Future Directions

How This Work Changes the Landscape

OAKS is best understood as a diagnostic instrument, not a paradigm-shifting theoretical contribution. It does not propose a new architecture, a new training objective, or a new scaling law. What it does — and this is the source of its influence — is provide the first measurement tool that can decompose the problem of knowledge tracking into distinct, quantifiable failure modes, enabling the field to move beyond aggregate accuracy comparisons toward a structured understanding of how and when models fail when facts evolve over time.

Prior to OAKS, the community's picture of dynamic knowledge handling in LLMs was fragmented and ambiguous. On one side, continual learning benchmarks (StreamingQA, EvolvingQA) reported that models could adapt to new information reasonably well, with accuracy declining gradually as more facts accumulated. On the other side, long-context benchmarks (BABILong, LongBench v2) showed that models could answer questions about information embedded in very long documents. Neither paradigm asked the question that OAKS makes central: does the model maintain the correct answer continuously throughout the stream, or does it oscillate between correct and incorrect states, getting the final answer right by chance or by re-reading? The paper's interval-level annotation protocol — asking the same question at every chunk — reveals that models frequently do oscillate. The behavioral analysis in Table 5 shows that even when models correctly detect that a transition has occurred (Adaptability: 38.1% average), they also frequently change their predictions when nothing has changed (Volatility: 32.1% average under No GT Transition). These two patterns — appropriate updating and spurious updating — coexist in the same model on the same dataset, and aggregate accuracy alone cannot distinguish between a model that updates perfectly and a model that updates constantly, getting it right only by accident.

This reframing matters because it changes what "improvement" means. If all you have is aggregate accuracy, a 5-point accuracy gain could come from better transition detection, better state maintenance, or simply from a model that guesses the most common answer more often. OAKS tells you which component improved. The decomposition in Table 6 shows that thinking mode on Qwen3-30B improves accuracy by 7.8 points (35.8% → 43.6%), and the source of that gain is almost entirely Phase Miss reduction (23.2% → 10.4%) while Distraction Susceptibility actually increases (34.0% → 37.0%). This is actionable information: if you want to improve this model further, you now know that state maintenance under accumulating context is your bottleneck, not transition detection. Prior to OAKS, you would not have known this.

The paper also reconciles a latent tension in the self-correction and iterative refinement literature. Studies like Huang et al. (2023) found that "LLMs cannot self-correct reasoning," while others observed that chain-of-thought and self-refinement improve performance on complex tasks. The OAKS behavioral analysis suggests a resolution: models can correct themselves when they detect a state change (Adaptability is the most frequent behavior under GT Phase Transitions, at 38.1%), but they also over-correct by changing predictions when nothing has changed (Volatility at 32.1% under No GT Transition). The net effect depends on the ratio of true transitions to stable intervals in the evaluation set. A benchmark where most intervals involve state changes would show self-correction working; a benchmark where most intervals are stable would show self-correction failing because Volatility dominates. This explains the contradictory findings without requiring either side to be wrong — they were testing on different distributions of transition frequency, and OAKS makes that distribution explicit and controllable.

The research directions that become more attractive as a result of this work are those that treat knowledge tracking as a temporal reasoning problem rather than a retrieval or capacity problem. The paper's finding that agentic memory systems (HippoRAG-V2, MemAgent, A-Mem) underperform simple RAG in aggregate (Table 4) — despite being purpose-built for dynamic contexts — suggests that better retrieval or better memory organization is not the bottleneck. The bottleneck, as revealed by the behavioral metrics, is deciding whether and when to update, and maintaining a correct state stably once acquired. This shifts research attention away from retrieval architectures (graph-based RAG, recursive summarization, vector indexing improvements) and toward mechanisms for temporal reasoning: automatic contradiction detection, freshness-weighted attention, explicit state-persistence induction, and confidence calibration for prediction changes. The forgetting problem that continual learning has studied for years is not the dominant failure mode on OAKS — the Distraction Susceptibility results show that models forget acquired states within the same context window, not across parameter updates, which is a different and more tractable problem.

Directions that become less attractive include scaling model size as a primary solution. Table 2 shows that accuracy scales monotonically with model size within the Qwen3 family — 4B: 26.4%, 8B: 33.1%, 30B: 35.8%, 80B: 41.1%, 235B: 46.8% on OAKS-B Base. But the behavioral analysis (Tables 5, 15) reveals that larger models do not eliminate the behavioral biases; they just shift the accuracy ceiling upward while preserving the same error profile. The Gemini models, which are presumably the largest tested, still exhibit extreme Obstinacy (40.7% under No GT Transition for Gemini 2.5 Flash non-thinking) or Volatility (46.5% for GPT-OSS-120B). Scaling reduces the frequency of errors but does not change their qualitative nature. This suggests that architectural or training innovations — not just parameter count — are needed to address the behavioral biases OAKS identifies.

Follow-Up Research This Work Enables

Mechanisms for explicit temporal freshness tracking. The paper's behavioral analysis reveals that the dominant error under stable ground-truth conditions is Volatility — the model changes its prediction even though nothing has changed (32.1% average across models in Table 5). This suggests that current LLMs lack an explicit mechanism for representing whether information is still valid or whether an update has occurred. A natural follow-up would be to augment the attention mechanism or the prompt structure with explicit timestamps and train the model to use temporal metadata for retrieval: at each chunk boundary, the model receives a structured representation of "facts that have been established" and "facts that have been updated," and is finetuned with a loss that penalizes changing a prediction when no update has occurred. A concrete experiment: take a Qwen3-8B model, generate synthetic training data in the OAKS-B format with explicit [STATE: location=kitchen, since=chunk_3, confirmed=true] annotations, fine-tune with a composite loss (next-token prediction + state-consistency regularization), and evaluate on OAKS-B. The prediction is that Distraction Susceptibility would drop significantly while Acquisition Latency remains unchanged, since the freshness annotations directly address the state-maintenance problem but not the transition-detection problem.

Recency-weighted retrieval that accounts for semantic overlap across revisions. The RAG results (Table 12, Appendix D.2) reveal a specific failure mode: when many semantically related chunks exist with different values for the same fact, retrieval returns all of them, and the model cannot determine which is most recent. A targeted follow-up would design a retrieval mechanism that (a) clusters retrieved chunks by the entity-fact pair they describe, (b) within each cluster, identifies the most recent chunk by index, and (c) presents only the most recent chunk from each cluster to the model, with an explicit recency tag. This is a retrieval-time intervention, not a model-training intervention, and could be tested by modifying the RAG pipeline used in the paper: replace the top-k semantic retrieval with a two-stage process (semantic retrieval to find all relevant chunks → temporal filtering to keep only the most recent per entity-fact) and re-evaluate on OAKS-B with Qwen3-30B. The key metric would be whether this reduces the Volatility rate under No GT Transition conditions, since the model would no longer be presented with contradictory information from outdated chunks.

Thinking-trace analysis to isolate the mechanisms by which extended reasoning improves phase capture. Table 6 shows that thinking mode dramatically reduces Phase Miss rate (Qwen3-30B: 23.2% → 10.4%; Gemini 2.5 Flash: 22.3% → 7.0%) while slightly increasing Distraction Susceptibility and Acquisition Latency. The paper does not analyze what the models write in their thinking traces that produces this shift. A natural follow-up would collect the full thinking traces from Qwen3-30B-Thinking on OAKS-B (which are already generated during the paper's evaluation but not analyzed), annotate them for the presence of specific reasoning patterns (explicit before-after comparison, recency check, mention of chunk indices, explicit contradiction detection, backtracking when uncertain), and correlate the presence of these patterns with successful Phase Capture vs. Phase Miss outcomes. This would transform the black-box "thinking helps" finding into a mechanistic account of which reasoning strategies are effective for state tracking. If, for example, explicit recency checking ("the most recent mention says X, therefore the answer is X") is the dominant effective strategy, that suggests a targeted training intervention: fine-tune models on data where the correct answer requires explicit recency-weighted retrieval, with reasoning traces that demonstrate the strategy.

Cross-domain stress-test to determine whether OAKS-tracking ability generalizes beyond narrative contexts. The paper's domains are strictly fictional narrative and synthetic fact injection. A strong follow-up would construct an analogous interval-level annotation protocol for a structurally different domain: news streams (where updates are corrections rather than narrative developments), dialogue state (where turn-taking and speaker identity add complexity), or code execution traces (where states are logically defined and deterministic). The specific contribution would be to determine whether the behavioral biases identified in Section 6 — over-updating vs. under-updating, the Phase-Miss-vs-Distraction trade-off — are domain-general properties of LLM state tracking or artifacts of narrative text processing. A concrete experiment: adapt the StreamingQA dataset (Liska et al., 2022) to OAKS-style interval-level annotation by annotating answers at every article timestamp for a subset of questions, then evaluate the same five models from Table 15 on this adapted benchmark. Compare the behavioral matrices between the news domain and OAKS-B. If the same models exhibit the same over-update/under-update biases across domains, it suggests a fundamental property of the model's tracking architecture. If the biases flip (a model that over-updates in narrative under-updates in news), it suggests that tracking behavior is heavily influenced by domain-specific textual cues.

Parametric online learning as a contrasting paradigm to in-context tracking. The paper deliberately excludes parameter updating, but this scoping choice leaves open the question of whether in-context tracking is even the right mechanism for the problem OAKS poses. A direct follow-up would train a small model (e.g., Qwen2.5-7B) on an OAKS-B-style stream using a test-time fine-tuning protocol: at each chunk, compute a loss on the questions that can be answered from the current chunk (using the ground-truth annotations as targets), update parameters via a single gradient step or LoRA update, then evaluate on all questions at that interval. Compare the resulting accuracy and behavioral metrics to the in-context-only baseline (Table 2, Qwen2.5-7B Base: 24.7%). The key question is whether parametric updating eliminates the Volatility problem — a parameter-updated model should, in principle, overwrite stale information in its weights rather than carrying it forward in the context, which would reduce spurious prediction changes. If parametric updating substantially improves accuracy (say, to 50%+) and reduces Volatility, it suggests that in-context tracking is a fundamentally suboptimal mechanism for this task, and that the paper's behavioral analysis is primarily characterizing the failure modes of the in-context strategy rather than the inherent difficulty of state tracking.

Confidence-calibrated prediction-change thresholds to combat both over-updating and under-updating. The behavioral analysis in Section 6.1 reveals that different models have different biases: Gemini under-updates (high Obstinacy), GPT-OSS over-updates (high Volatility). Both could be addressed by a simple inference-time intervention: require the model to output both a prediction and a confidence score (or extract token-level log-probabilities for the predicted answer), then only change the prediction from the previous interval if the confidence in the new prediction exceeds a threshold. On OAKS, where the model makes predictions at every interval, the previous prediction is available (unlike in standard single-answer QA). A concrete experiment: take GPT-OSS-120B and Gemini 2.5 Flash non-thinking (the extremes of over-updating and under-updating respectively), run them on OAKS-B with a range of confidence thresholds, and plot accuracy and behavioral metrics as a function of threshold. The prediction is that a moderate threshold would reduce Volatility in GPT-OSS (improving accuracy on stable intervals) while a lower threshold would reduce Obstinacy in Gemini (improving accuracy on transition intervals). The optimal threshold would be model-specific, but the principle — using prediction-change frequency as a tunable parameter — would generalize and could be implemented immediately without any retraining.

Practical Applications and Downstream Use Cases

Streaming conversational assistants with state-tracking awareness. In a long-term dialogue where a user gradually reveals preferences, constraints, and corrections (e.g., "I'm looking for a restaurant," "I'm vegetarian," "Actually, I eat fish"), an assistant that fails to track which dietary constraint is current will make incorrect recommendations. The OAKS behavioral metrics provide a framework for continuously monitoring an assistant's state-tracking performance in deployment: measure, across recent dialogue turns, the rate at which the assistant changes its inferred user state (analogous to the Change rate in Table 5), and flag conversations where the change rate is anomalously high (possible over-updating due to context distraction) or anomalously low (possible under-updating where corrections are ignored). The specific numbers from the paper — even the strongest model, Gemini 3 Pro, fails on ~25% of interval predictions on OAKS-N — suggest that state-tracking errors in conversational assistants are not occasional edge cases but frequent enough to warrant explicit monitoring. A practical system could use a lightweight OAKS-style evaluation on a held-out set of dialogue state-tracking questions to estimate the assistant's Volatility and Obstinacy rates and adjust its prompting strategy accordingly (e.g., adding explicit "do not change your answer unless the user explicitly contradicts it" instructions for over-updating models, or "prioritize the most recent information" for under-updating models).

Batch inference for narrative understanding at scale. Organizations that process large volumes of narrative text — publishers generating chapter-by-chapter summaries, film studios tracking character arcs across scripts, legal firms analyzing witness testimony that evolves across depositions — face the same state-tracking challenge that OAKS measures. The benchmark's key practical finding is that naive strategies (RAG, rolling windows, agentic memory) do not reliably solve the problem, but that thinking mode with explicit reasoning provides the largest gains on complex multi-hop tracking questions (Table 3: bridge questions improve by 15–21 points with thinking enabled). For practitioners with the budget to use thinking-mode models (Gemini 2.5 Pro with thinking achieves 60.3% on OAKS-B vs. 42.9% without), the recommendation is clear: enable thinking when the task requires tracking entities whose properties change over time and reasoning over those changes, but not for simple retrieval tasks where thinking may not help (and on OAKS-N, where thinking mode slightly degrades performance; Table 13). The 4× inference cost increase from thinking mode (thinking tokens + answer tokens) must be weighed against the accuracy gain, and OAKS provides a framework for making that trade-off quantitatively.

Self-improving agent memory through interval-level feedback. The paper finds that MemAgent, which is trained with rewards computed after processing all chunks, underperforms naive RAG in aggregate but shows relative strength on Frequent-update subsets (Table 4: MemAgent achieves 29.1% on Frequent vs. RAG's 27.5%). This suggests that interval-aware training — where the model receives rewards at each chunk for maintaining correct state, not just at the end — could substantially improve memory-system performance. A practical agent architecture could adopt OAKS's interval-level annotation protocol as a training signal: during training, after each chunk of a streaming context, compute the agent's accuracy on a set of held-out state-tracking questions and use that as an intermediate reward. The MemAgent result provides a lower bound on what interval-aware training can achieve (since MemAgent was trained with end-of-stream rewards only and still partially helps); the prediction is that interval-aware training would close the gap with or exceed RAG on the Frequent subset, since the agent would learn to prioritize maintaining correct intermediate states rather than just optimizing the final answer. This is directly actionable for teams building memory-augmented LLM agents: change the reward function to include per-timestep state-tracking accuracy, using OAKS-style synthetic data for training.

Model selection for temporal reasoning tasks. The paper's comprehensive evaluation of 14 models across multiple architectures, scales, and inference strategies provides the first systematic comparison of temporal state-tracking ability across the current LLM landscape. For practitioners selecting a model for a task that involves tracking evolving facts — a legal document review system that ingests exhibits sequentially, a supply-chain monitoring dashboard that receives status updates, a medical record summarizer that processes lab results chronologically — Table 2 provides actionable guidance: Qwen3-235B with Base context achieves 64.7% on OAKS-N (the more naturalistic dataset), while Gemini 3 Pro achieves 75.5%. If the application can tolerate proprietary API costs, Gemini 3 Pro provides a ~10-point accuracy advantage. If the application requires open-source deployment, Qwen3-235B is the strongest option but requires substantial GPU resources (the paper capped its context at 133k tokens due to memory constraints). For latency-sensitive applications where thinking mode is too slow, the non-thinking baseline for Gemini 2.5 Flash achieves 43.2% on OAKS-B (Table 3) — substantially lower, confirming that the thinking-mode advantage is real and large for this task class. These numbers, while specific to the OAKS datasets, provide a more relevant signal for temporal tracking applications than generic benchmarks like MMLU or HumanEval, which do not test dynamic knowledge handling at all.