Retrieval-Infused Reasoning Sandbox: A Benchmark for Decoupling Retrieval and Reasoning Capabilities

ArXiv: 2601.21937

🎯 Pitch

Giving LLMs access to documents can actually make them perform worse than having no documents at all—a new controlled benchmark reveals that frontier models suffer from mode-switch fragility, where providing additional evidence derails reasoning rather than enhancing it. By disentangling retrieval from reasoning across four carefully controlled information regimes, DeR² identifies that the core bottleneck is not merely missing concepts but a failure to execute identified concepts as procedures in the presence of distractors, with Full-set accuracy of just 51.3% compared to 75.4% when oracle concepts are provided.

---

1. Executive Summary

This paper introduces DeR², a controlled diagnostic benchmark that decouples retrieval from reasoning in scientific problem-solving by evaluating models across four information-access regimes—Instruction-only (parametric knowledge), Concepts (oracle concept sets), Related-only (only relevant documents), and Full-set (relevant documents plus topically related distractors)—yielding interpretable regime-to-regime gaps that operationalize retrieval loss versus reasoning loss and enable fine-grained error attribution rather than binary correct/incorrect scoring. Experiments across state-of-the-art models (GPT-5.1, Gemini-3-Pro, Claude-Opus-4.1, DeepSeek-V3.1, and others) on problems drawn from 2023–2025 theoretical scientific papers reveal that Concepts-only accuracy averages 75.4% while Full-set drops to 51.3%—a gap that decomposes into document-to-concept extraction loss and noise-induced selection failure—and uncover a counterintuitive mode-switch fragility where some models perform worse with full documents than with no documents at all (Gemini-3-Pro drops from 64.2% under Instruction-only to 53.7% under Full-set), establishing that providing additional evidence can disrupt the reasoning controller rather than simply diluting signal, and that test-time document integration is not a simple retrieval-plus-reasoning composition but requires explicit evidence-grounded mode control and executable concept utilization that current architectures largely lack.

2. Context and Motivation

The Core Problem: We Cannot Tell Whether LLMs Actually Reason Over Novel Information

The fundamental question this paper challenges is deceptively simple: when a language model answers a question by consulting documents, is it genuinely reasoning over the evidence, or is it doing something else entirely? The question matters because the prevailing narrative around frontier models—reinforced by strong performance on benchmarks like AIME, SWE-bench, and various RAG evaluations—is that they are increasingly capable of "deep research": autonomously navigating information sources, synthesizing evidence across multiple documents, and producing well-supported conclusions. But the paper argues that existing evaluation practice provides almost no ability to distinguish genuine evidence-grounded reasoning from several alternative explanations that would undermine that narrative.

Consider three scenarios that current benchmarks cannot disentangle:

1. Parametric recall disguised as retrieval-infused reasoning. The model might already know the answer from pretraining and simply use the retrieved documents as decorative context rather than as the engine of its reasoning. If a benchmark draws questions from widely disseminated scientific material, a model with broad parametric knowledge could achieve correct answers without ever consulting the provided evidence. The evaluation would credit the model for retrieval-infused reasoning that never actually occurred.

2. Retrieval pipeline failures masked as reasoning errors. In an end-to-end RAG setup, the model receives output from a retrieval pipeline (text chunks from a vector database, web search results, or an agent's collected sources). If the answer is wrong, the binary correct/incorrect score provides no signal about why—was the right document not retrieved? Was it retrieved but the model failed to extract the relevant concept? Was the concept extracted but the model couldn't apply it correctly? Without knowing where the pipeline broke, developers cannot direct their optimization efforts, researchers cannot attribute capability gaps to specific components, and benchmark results provide limited guidance for model selection.

3. Correct answers from wrong reasoning paths. The model might produce the right answer through a reasoning chain that has nothing to do with the provided evidence—for example, by pattern-matching the question format to a known answer template, or by applying a generic heuristic that coincidentally yields the correct output for this instance. Standard accuracy metrics would count this as a success, even though the model completely failed at the intended task of evidence-conditioned reasoning.

This confounding is not hypothetical. The paper explicitly states that current deep-research evaluations suffer from "end-to-end pipeline confounding with weak diagnosis" (Section 1)—the variance in performance is often dominated by toolchain choices (which retriever, how many chunks, what reranking model) rather than by the model's underlying research reasoning capability, and the binary outcome provides no signal about which capability failed.

Why This Is Important: The Gap Between Evaluation and Deployment

The problem matters both practically and theoretically.

Practical importance. Organizations deploying LLMs for research-intensive applications—literature review synthesis, evidence-based policy analysis, scientific hypothesis generation, legal document review—need to know whether their chosen model can genuinely integrate novel information into its reasoning process. If a model's strong benchmark scores are largely attributable to parametric memorization or clever test-set heuristics, it will fail catastrophically when deployed on genuinely novel material (as the paper demonstrates with the Instruction-only vs. Full-set inversion). A benchmark that cannot diagnose why a model fails leaves practitioners making deployment decisions based on a contaminated signal. The paper notes that current benchmarks provide "limited guidance for model selection, debugging, or targeted training" (Section 1)—a direct practical consequence of the diagnostic gap.

Theoretical importance. The question of whether LLMs can "reason" is one of the most contested in the field. Much of the evidence on both sides comes from benchmarks that confound reasoning with retrieval, memorization, or prompt engineering. By isolating document-grounded reasoning from these confounds, DeR² provides a cleaner signal about a specific, well-defined capability: given novel concepts that the model demonstrably does not know parametrically, can it (a) extract those concepts from noisy documents, and (b) compose them into a correct multi-step derivation? This is a more targeted and interpretable capability than "can the model answer hard questions," and it speaks directly to the architecture and training questions that the research community is debating—whether current transformers have genuine compositional reasoning abilities or are sophisticated pattern matchers.

Real-world urgency. The paper emphasizes that agents—autonomous systems that navigate the internet, acquire evidence, and synthesize conclusions—are "increasingly used to extend models beyond parametric knowledge" (Section 1). These systems are being deployed in high-stakes domains (medical literature review, legal research, scientific discovery) under the assumption that they can reliably incorporate novel information. If that assumption is false—if models exhibit mode-switch fragility where providing documents actually degrades performance, or structural concept misuse where they name correct concepts but fail to execute them procedurally—then current agent designs may be unreliable in ways that are invisible to existing evaluation.

Where Existing Approaches Fall Short

The paper identifies three systematic limitations across current evaluation paradigms.

1. End-to-End Pipeline Confounding in RAG and Agent Benchmarks

Most retrieval-augmented generation benchmarks (KILT, FreshQA, and various agent-based deep research evaluations like BrowseComp and HLE) score the full stack: retrieval → reranking → summarization/compression → context stitching → reasoning. This is a system-level evaluation, and the paper argues it is fundamentally inadequate for understanding model capability:

"many benchmarks score the full stack—retrieval, reranking, summarization/compression, context stitching, and reasoning—so variance is often dominated by toolchain choices rather than the model's latent research reasoning capability" (Section 1)

To make this concrete: if Model A achieves 70% accuracy with Retriever X and Model B achieves 65% with Retriever Y, the scores tell you about the (model, retriever) pair, not about either component in isolation. You cannot conclude that Model A is "better at reasoning" because the difference might be entirely driven by retrieval quality. More critically, if Model A scores 65% on some benchmark, you cannot distinguish between the case where the model perfectly reasons over 65% of the evidence versus the case where it reasons flawlessly over 100% of the evidence but the retriever only finds the right documents 65% of the time. These have entirely different implications for how to improve the system.

The paper further argues that even within a fixed retrieval setup, the binary correct/incorrect outcome obscures the failure mode:

"most setups expose only a binary 'correct/incorrect' outcome, which makes it difficult to attribute errors to distinct failure types such as missing the key concept, identifying the concept but failing to apply it, or being diverted by topically related noise" (Section 1)

This is the diagnostic gap in its starkest form: you know the system failed, but you have no idea where or why, which means you have no principled path to improvement.

2. Parametric Leakage and Memorization

Many benchmarks draw questions from material that was likely in the model's pretraining data. If a model can answer a question from parametric knowledge alone—without consulting any provided documents—then correctness on that question tells you nothing about the model's retrieval-infused reasoning capability. It tells you about memorization, not about genuine evidence integration.

The paper is explicit about the severity of this problem:

"if tasks can be solved via memorized facts, broad encyclopedic knowledge, or easily guessable numeric forms, then correctness does not reliably indicate that the model consulted the crucial evidence and reasoned with it, contaminating the evaluation signal" (Section 1)

This is not a hypothetical concern. The paper's own experiments reveal that some models achieve substantial Instruction-only accuracy (Gemini-3-Pro scores 64.2% with no documents at all), meaning a large fraction of questions in existing benchmarks might be answerable from parametric knowledge—and that fraction varies dramatically across models, making cross-model comparisons particularly unreliable under standard evaluation protocols.

The problem is compounded by what the paper calls "easily guessable numeric forms." Consider a benchmark question that asks for a numerical answer, and the model's parametric knowledge contains a distribution over plausible numeric outputs for that type of question. Even if the model doesn't "know" the answer, it might guess correctly by sampling from the right distribution—a phenomenon that is impossible to distinguish from genuine evidence-based reasoning using binary accuracy alone.

3. Web Volatility and Irreproducibility

Many deep-research benchmarks (BrowseComp, agent-based evaluations) depend on open-web retrieval. This introduces a fundamental reproducibility problem:

"benchmarks that depend on open-web retrieval inherit non-stationarity across time, region, indexing updates, and page instability (movement, deletion, or content changes), producing high variance across repeated runs and hindering controlled comparisons and ablations" (Section 1)

This means that the same model, evaluated on the same benchmark at different times or from different geographic locations, might receive different search results, see different page versions, or encounter different link rot patterns—leading to different scores that reflect infrastructure drift rather than genuine capability differences. For researchers trying to compare model architectures or training recipes, this introduces noise that can easily swamp the signal they are trying to measure.

The problem is particularly acute for benchmarks that target "cutting-edge" or "recent" knowledge, because these are precisely the materials most likely to change, move, or disappear. The very quality that makes them good tests of retrieval-infused reasoning (their novelty relative to training data) makes them unstable as evaluation artifacts.

4. Missing Diagnostic Granularity Across All Existing Paradigms

The paper argues that existing benchmarks, even when they avoid the specific confounds above, share a deeper structural limitation: they treat "retrieval-infused reasoning" as an atomic capability rather than a composition of separable sub-capabilities. Specifically, the paper identifies that what looks like a single "answer the question from documents" task actually decomposes into at least three distinct skills:

• Evidence selection and denoising: Given a set of documents, some relevant and some not, can the model identify which documents (and which passages within documents) contain solution-relevant information?
• Document-to-concept extraction: Given the right documents, can the model extract the operative concepts, theorems, or formalisms—not just summarize surface content, but identify the procedural knowledge that must be applied?
• Concept-level composition and scheduling: Given the right concepts (even in oracle form), can the model coordinate them into a correct multi-step derivation, maintaining constraints, tracking dependencies, and applying them in the right order?

No existing benchmark cleanly separates these, which means that when a model fails (or succeeds), you cannot attribute the outcome to any specific sub-capability. The paper positions DeR² as filling precisely this gap by constructing four evaluation regimes that provide controlled variation along each of these dimensions independently.

How DeR² Positions Itself

The paper does not introduce a new model architecture, a new training procedure, or a new retrieval algorithm. It introduces a new evaluation methodology—a benchmark that is designed from first principles to isolate document-grounded reasoning from its confounds, rather than accepting the confounds as inevitable properties of "realistic" evaluation.

Concretely, DeR² makes the following design commitments that directly address the identified gaps:

Against end-to-end confounding (Gap 1): Rather than scoring a pipeline, DeR² provides the same question under four input configurations—Instruction-only (no documents), Concepts (oracle concept list, no documents to search), Related-only (only the documents that contain the necessary concepts), and Full-set (relevant documents plus topically related distractors). Performance differences between these configurations are interpretable: the Concepts-only → Related-only gap isolates document-to-concept extraction capability, the Related-only → Full-set gap isolates noise robustness and evidence selection, and the Concepts-only → Instruction-only gap measures parametric knowledge contamination. This is a controlled experiment design applied to evaluation, and it transforms the binary correct/incorrect output into an error attribution signal.

Against parametric leakage (Gap 2): Every question in DeR² must pass a two-phase validation protocol: the question must be unsolvable without documents (the model must fail in three Instruction-only attempts) but solvable with oracle concepts (the model must succeed at least once in three Concepts-only attempts). This jointly enforces that (a) the question is genuinely novel relative to the model's parametric knowledge, and (b) the question is not impossibly hard even with the right information. The paper uses offline models (e.g., DeepSeek-R1-0528 or the Doubao web client with networking disabled) for this validation, ensuring that the difficulty calibration is grounded in actual model behavior rather than annotator intuition.

Against web volatility (Gap 3): DeR² replaces open-web dependence with a frozen, per-instance document library drawn from 2023–2025 theoretical papers. Each instance has a fixed set of documents (mean: 6.5) that includes both necessary evidence and carefully curated distractor documents. Because the document set is static and bundled with the benchmark, every model evaluation sees exactly the same evidence, enabling controlled comparisons across models, across time, and across runs. The papers are drawn from theoretical (not applied) scientific work to ensure that answering requires concept-driven logical derivation rather than recall of experimental measurements or simulation results.

Against diagnostic opacity (Gap 4): Each instance provides not just an answer but also expert-annotated concept sets and validated chain-of-thought rationales. This enables process-level evaluation: rather than only checking whether the final answer matches, the benchmark's design supports analyzing whether the model's reasoning trace mentions the correct concepts, applies them in the right order, avoids hallucinated steps, and correctly executes procedural operations (algorithm instantiation, theorem application, constraint satisfaction). The paper operationalizes this through the error attribution analysis in Table 3, which classifies failures into missing core concepts, concept misuse, reasoning-process errors, and numeric/formalization errors.

The Scientific Novelty Gap

The paper explicitly targets a capability that it argues existing benchmarks do not measure: frontier scientific reasoning over genuinely novel concepts. The questions are drawn from 2023–2025 theoretical papers—material that postdates the training cutoff of most evaluated models. This is not a minor detail; it is central to the benchmark's validity. If a question is drawn from material published in 2024 and the model's training data ends in 2023, then any correct answer cannot be explained by parametric memorization. The document set becomes the only source of the necessary concepts, which is precisely the capability that the benchmark aims to measure.

The paper distinguishes this from mathematical reasoning benchmarks (GSM8K, MATH) that test multi-step reasoning over concepts the model already knows parametrically, and from knowledge-intensive QA benchmarks (Natural Questions, KILT) that emphasize fact lookup rather than multi-step synthesis. DeR² targets the intersection: reasoning that requires integrating multiple novel concepts from scientific documents into a coherent derivation—the kind of task that a human researcher performs when reading a new paper and understanding its argument well enough to answer a non-trivial question about it.

The benchmark is called a "sandbox" (rather than simply a "benchmark") because it provides controlled variation along specific dimensions (noise level, concept count, information access regime) that can be manipulated to stress particular capabilities. This distinguishes it from benchmarks that aim for "realism" by replicating end-to-end workflows; DeR² trades superficial realism for diagnostic precision, operating at a level of abstraction where the failure signal is clean and attributable.

3. Technical Approach

3.1 Reader Orientation

DeR² is a controlled evaluation benchmark — not a model, not a training method — designed as a diagnostic instrument that tests whether language models can genuinely reason over novel scientific information. The system solves the problem of attribution opacity in current RAG evaluations: instead of outputting a single binary score, it outputs a set of regime-to-regime performance gaps that decompose end-to-end failure into three separable components — evidence selection/denoising, document-to-concept extraction, and concept-level composition/scheduling — enabling researchers to pinpoint where a model's retrieval-infused reasoning breaks down rather than merely that it broke down.

3.2 Big-Picture Architecture (Diagram in Words)

The DeR² system has four major components:

1. Instance Construction Pipeline (§2.3) — A human-expert-driven process that produces benchmark instances, each containing a frontier scientific question (Instruction), its solution concepts (Concepts), a validated reasoning trace (CoT), a verifiable answer (Answer), and a frozen document library (Doc Set) with both relevant and distractor documents. This pipeline includes a two-phase difficulty calibration protocol that ensures questions are parametrically unsolvable but document-solvable.

2. Four Evaluation Regimes (§3.1) — A controlled input configuration system that presents the same question under four information-access conditions: Instruction-only (no evidence), Concepts (oracle concept list), Related-only (only relevant documents), and Full-set (relevant documents plus distractors). Each regime strips away or adds a specific capability requirement, and the performance differences between regimes are causally interpretable.

3. Evaluation Protocol (§3.2) — A standardized inference setup that applies the same sampling configuration (temperature = 1, nucleus sampling $p = 0.7$, two runs per model-setting pair) and a deterministic text-truncation rule (30,000-character budget with head-tail preservation) to ensure fair comparison across models with varying context-length capabilities.

4. Diagnostic Attribution Framework (§3.3, Table 3) — A structured error analysis that goes beyond answer accuracy to classify failures into four categories (missing core concept, concept misuse, reasoning-process error, numeric/formalization error) and measure concept-level precision/recall, enabling process-level evaluation of evidence usage.

Information flows as follows: an annotator selects a 2023–2025 theoretical source paper → extracts Instruction, Answer, Concepts, and CoT → validates parametric unsolvability and document-solvability through offline model testing → constructs Document Set with relevant and distractor documents → reviewer audits the instance → instance enters the benchmark → evaluated models receive the instance under each of the four regimes in separate runs → an auto-evaluator grades the final answer against the ground truth → the error attribution system analyzes reasoning traces to classify failure modes.

3.3 Roadmap for the Deep Dive

• First, the two-phase difficulty calibration protocol, since it is the mechanism that guarantees the benchmark's core validity claim: that questions are genuinely novel (not answerable from parametric knowledge) yet tractable (solvable with the right evidence). Without understanding this protocol, the entire regime-gap interpretation collapses.

• Second, the instance structure — the five-field tuple (Instruction, Concepts, CoT, Answer, Doc Set) — because it specifies exactly what information exists for each question and defines the ontology that the four evaluation regimes manipulate.

• Third, the document set construction, since it operationalizes the benchmark's noise model (topically adjacent distractors) and its reproducibility guarantee (frozen, per-instance libraries), and because the distinction between "relevant" and "noise" documents is what enables the Related-only → Full-set gap to measure denoising capability.

• Fourth, the four evaluation regimes as a controlled experiment design, explaining what each regime isolates, what the transitions between regimes measure, and why this particular set of four provides the diagnostic coverage it claims.

• Fifth, the evaluation protocol details — the truncation rule, the sampling configuration, the two-run averaging, and the auto-evaluation pipeline — because these engineering choices affect cross-model comparability and reproducibility.

• Sixth, the diagnostic attribution methodology — concept-level metrics (precision, recall, accuracy), the four-category error typology, and how these are computed using the doubao-seed-1.6-251015 model as a structured evaluator — because this is what transforms the benchmark from a set of accuracy numbers into a diagnostic instrument.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and evaluation methodology paper whose core technical contribution is a protocol for creating instances that cleanly separate retrieval capability from reasoning capability, and an evaluation framework that uses controlled input variation to produce interpretable performance gaps.

---

The Two-Phase Difficulty Calibration Protocol

The most critical technical mechanism in DeR² is the difficulty calibration protocol, because every subsequent interpretation of regime gaps depends on the guarantee that questions are parametrically unsolvable but document-solvable. If this guarantee fails — if a question turns out to be answerable from parametric knowledge, or if it is so hard that even oracle concepts do not enable solution — then the diagnostic regime gaps lose their causal interpretation.

The protocol operates in two sequential phases, both using offline models (the paper names DeepSeek-R1-0528 and the Doubao web client with networking disabled) to ensure that calibration reflects actual model behavior rather than annotator intuition.

Phase 1: Parametric failure validation (Instruction-only). The annotator presents the Instruction — just the question text, with no concepts and no documents — to the calibration model. The model must attempt to answer the question. This is repeated three times. The acceptance criterion is strict:

"All three attempts must be incorrect."

This means the question must be genuinely beyond the model's parametric reach. If the model succeeds even once in three Instruction-only attempts, the question is rejected (or revised — see below) because it means the model can answer the question from its pretrained knowledge, which would contaminate the benchmark's signal. Three attempts are used rather than one to reduce the false-negative rate: a model might get lucky once with a guess, but three consecutive failures provides reasonable confidence that the question is parametrically out of reach.

Why this criterion matters: It is the operational definition of "novelty" in this benchmark. The paper does not rely on metadata about training data cutoffs or heuristic judgments about whether a concept "should" be in pretraining data. Instead, it uses an empirical test: if the model consistently fails without evidence, then the question is treated as novel for that model. This is a pragmatic choice that acknowledges that different models have different parametric knowledge boundaries, and that what counts as "novel" is model-relative. The calibration is performed on specific offline models, and the resulting instances are then used to evaluate other models — this means the novelty guarantee is approximate for the evaluated models, since a model with broader parametric knowledge might still find some questions answerable. The paper addresses this implicitly by reporting Instruction-only accuracy for all evaluated models (Table 2), which serves as a post-hoc check on whether parametric leakage is occurring in practice.

Phase 2: Document-solvability validation (Concepts-only). Once a question passes Phase 1, the annotator presents the Instruction together with the oracle Concepts list to the same calibration model. The model receives the question text and the complete set of concepts, theorems, and formalisms needed to solve it. This is again run three times. The acceptance criterion is:

"Across three attempts, it must be correct at least once and incorrect at least once."

This is a more nuanced criterion than simple majority correctness. The requirement that the model must succeed at least once ensures the question is tractable — it is not impossibly hard or ill-posed, and the Concepts list genuinely contains the information needed to derive the answer. The requirement that the model must fail at least once (i.e., not be correct all three times) ensures the question is not trivially easy once concepts are provided — there is genuine reasoning difficulty beyond concept identification. This prevents degenerate instances where the Concepts list effectively contains the answer in restated form.

Why "at least once correct" rather than "majority correct": The paper is testing whether the question is solvable in principle given the concepts, not whether it is reliably solvable by the calibration model. A single success across three attempts demonstrates that the Concepts list is sufficient for a correct derivation. The calibration model might fail on the other two attempts due to sampling variance or reasoning errors, but those failures do not invalidate the instance — they confirm that there is non-trivial reasoning required.

What happens when calibration fails. If the model succeeds in all three Concepts-only trials (too easy), the paper describes two remedies:

• Option A (deep validation): Replace the Concepts input with the full Document Set (Related + Noise documents) and retest. If the model remains always-correct (succeeds in all three trials with the full document set), the item is discarded — it is too easy even with retrieval noise. If the model fails at least twice, the item can be retained (the documents introduce enough difficulty to make the question non-trivial). The paper notes that this option is "discouraged when the doc set exceeds three PDFs due to context limitations in common LLM interfaces" — a practical constraint reflecting that the calibration model's context window may not accommodate large document sets.

• Option B (iterative editing): The annotator revises the Instruction or Concepts to increase difficulty — by adding or removing concepts, narrowing the question scope, or reformulating the task to require more precise application of the concepts. The calibration protocol is then re-run on the revised instance. This creates an iterative loop between annotation and calibration, which the paper describes as "iteratively edit the Instruction and/or Concepts... to increase the difficulty until the calibration criteria are satisfied."

Why this two-phase design: The joint criteria (parametric failure AND document solvability) operationalize the paper's central validity claim. A question that passes Phase 1 but fails Phase 2 (model never gets it right even with concepts) is rejected as impossibly hard or ill-formed — it would produce zero accuracy across all regimes including Concepts-only, providing no diagnostic signal. A question that passes Phase 2 but fails Phase 1 (model can answer without concepts) is rejected as parametrically contaminated — it would produce high Instruction-only accuracy, making the regime gaps uninterpretable. The intersection of the two criteria selects for questions that are genuinely at the frontier of the model's capability: unfamiliar enough to require evidence, but tractable enough to be solvable with that evidence.

Implementation detail — model selection for calibration. The paper uses "offline models" for calibration, specifically naming DeepSeek-R1-0528 and the Doubao web client with networking disabled. The choice of offline models is deliberate: by using models without web access, the calibration isolates the effect of the provided Concepts from any external knowledge the model might retrieve. If the calibration model had web access, a Phase 1 failure might simply mean the model's search capability is weak, not that the question is genuinely novel — the model might succeed with better search. The offline constraint ensures that Phase 1 failures reflect genuine parametric knowledge gaps.

Limitation of the calibration approach. The calibration is performed on specific models (DeepSeek-R1-0528, Doubao) but the benchmark is used to evaluate other models (GPT-5.1, Gemini-3-Pro, Claude, etc.). A question that is parametrically unsolvable for the calibration model might be parametrically solvable for a model with broader or more recent training data. The paper does not re-calibrate for each evaluated model. This means the Instruction-only accuracy reported in Table 2 serves as a post-hoc validation: if an evaluated model scores high under Instruction-only on a question that was calibrated to be parametrically unsolvable, it indicates either parametric leakage in that model or calibration model mismatch. The paper reports these numbers transparently, allowing readers to assess the severity of this issue for each model (e.g., Gemini-3-Pro's 64.2% Instruction-only accuracy suggests substantial parametric knowledge of the benchmark's domain).

---

The Five-Field Instance Structure

Each DeR² instance is a tuple of five components, each serving a specific role in the benchmark's diagnostic architecture.

Instruction. This is the question text — a "high-level academic question derived from cutting-edge scientific papers" (Section 2.2). The instruction is designed to be self-contained (understandable without the source paper) and appropriately scoped: "neither overly broad (admitting multiple valid answers) nor overly narrow (only true within the specific paper instance)." This scoping constraint is critical because the benchmark requires a single verifiable answer. If the instruction admits multiple valid answers, the auto-evaluator cannot reliably grade correctness, and the regime-gap interpretation breaks down (did the model get a different answer because it reasoned incorrectly, or because its answer is equally valid under a different interpretation?).

The paper provides a concrete example in Appendix B.1: "Determine the time complexity for the proposed algorithm to provably learn a multi-head attention layer F from random labeled examples...". This instruction is specific enough to have a unique answer (a time complexity expression), draws on a specific theoretical paper, and requires understanding the paper's algorithm to derive the answer — it cannot be answered from general machine learning knowledge.

Concepts. This is the complete set of "concepts, theorems, or formulas necessary to solve the instruction" (Section 2.2). The paper specifies that concepts should be "generalizable scientific knowledge" rather than "overly specific experimental data." This distinction matters: the benchmark tests whether models can operationalize theoretical constructs (theorems, formalisms, algorithmic procedures), not whether they can recall experimental measurements. A concept like "the lifted formulation characterizes Schubert cell membership using bilinear equations" is in-scope; a concept like "the measurement was 3.7 ± 0.2 eV" is out-of-scope because it tests recall rather than reasoning.

The Concepts list is "necessary and sufficient" — it contains all the knowledge required to derive the answer, and it contains nothing irrelevant. This makes Concepts-only performance an interpretable upper bound on concept-level reasoning: if a model fails in Concepts-only, the failure cannot be attributed to missing concepts or distracting information; it must be a failure of composition, scheduling, or execution of the provided concepts. The paper acknowledges that Concepts-only is not a true upper bound in practice (Table 3 shows precision/recall below 1.0 even when concepts are provided), indicating that models can fail to comprehend or utilize even explicitly provided concepts — what the paper calls "structural concept misuse."

CoT (Chain of Thought). This is the expert-annotated reasoning trace that connects the Concepts to the Answer: "the reasoning process that connects the concepts to the final answer, illustrating how the model should use the provided knowledge step by step" (Section 2.2). The CoT serves multiple functions: (1) it is the "reference solution" that the reviewer uses to verify that the Concepts list is sufficient and correctly applied; (2) it provides the ground truth for the error attribution analysis in Table 3, where the doubao-seed-1.6-251015 evaluator compares the model's predicted reasoning against the reference CoT to diagnose failure modes; (3) it establishes the expected reasoning depth (number of steps), which is used as a stratification variable in the controlled factor analysis (Figure 5a shows the distribution of reasoning step counts).

The CoT is concept-grounded by construction: "every knowledge item used in the CoT is included in the Concepts list" (Step 2 of data collection, Section 2.3). This prevents the reference solution from relying on implicit assumptions or background knowledge not captured in the Concepts list, which would make Concepts-only performance an invalid upper bound. If the CoT used a concept not in the Concepts list, then a model could fail under Concepts-only simply because it was missing that concept, not because it failed at reasoning.

Answer. This is "the answer to the instruction, which could be a product, scalar value, or conclusion derived from the concepts and reasoning" (Section 2.2). The paper prioritizes "a short artifact, scalar, or concise conclusion" to enable reliable automated grading. When the answer is necessarily long, the annotator must "provide a rigorous checklist of key scoring items to make verification objective." This checklist approach addresses a common weakness in QA benchmarks where long-form answers are graded by human judgment (expensive and noisy) or by LLM-as-judge (which introduces its own biases). By decomposing long answers into verifiable scoring items, the benchmark maintains objective grading even for complex conclusions.

The paper also specifies that "no document contains the Answer verbatim or in an equivalent directly-recoverable form" (Step 4 of data collection). This prevents a trivial retrieval strategy: the model cannot simply locate a sentence in the document set that states the answer and copy it. The answer must be derived from the concepts, not extracted. This is what distinguishes DeR² from extractive QA benchmarks and makes it a test of reasoning rather than information retrieval.

Doc Set. This is the frozen document library provided to the model in the Related-only and Full-set regimes. Each instance has a mean of 6.5 documents, drawn from the references of the source paper. The document set contains two categories:

• Related documents: At least one document per Concept, where each document contains the necessary concept. A single related document may cover multiple concepts. These are the documents that the model needs to find and extract information from to solve the question.

• Noise documents: Documents that are "topically adjacent to the Instruction but do not contribute solution-relevant concepts" (Step 4). These are not random irrelevant documents (which would be trivially easy to filter); they are carefully selected to be on-topic — they discuss related scientific questions, use similar terminology, or address adjacent problems — but do not contain any of the specific concepts needed to solve the instruction.

The distinction between "topically adjacent" and "irrelevant" is central to the benchmark's noise model. If noise documents were clearly off-topic (e.g., a biology paper in a physics question), filtering them would be a trivial keyword-matching task. The paper's noise documents are designed to be plausibly relevant — they would appear in a literature search on the topic, they use overlapping vocabulary, and a superficial reading might suggest they contain useful information. This forces the model to perform genuine conceptual filtering: distinguishing documents that are about the same topic from documents that contain the specific concepts needed to answer this particular question.

The Doc Set is "frozen" — it is a static collection bundled with the benchmark instance, not retrieved dynamically from the web. This is the mechanism that addresses web volatility (Gap 3 from Section 1): every model evaluation sees exactly the same documents, enabling reproducible comparisons across models, across time, and across evaluation runs.

---

Document Set Construction and Noise Model

The document set construction process (Step 4 of data collection) transforms a calibrated quadruple (Instruction, Answer, Concepts, CoT) into a complete instance by adding the document library.

Source of documents. The annotator starts from the source paper's references. This is a deliberate choice: the references of the source paper are the natural set of documents that a researcher would encounter when investigating the topic. They are topically coherent (they all relate to the same research area) but vary in their relevance to the specific question. This creates a realistic retrieval scenario where the model must distinguish degrees of relevance within a coherent document collection.

Related document selection. For each Concept in the Concepts list, the annotator locates at least one document in the references that contains that concept. The paper specifies that "a single related document may cover multiple Concepts." This means the number of related documents may be less than the number of concepts — a single document might contain several of the required concepts simultaneously. The annotator must verify that each related document genuinely contains the concept it is supposed to provide: "Related documents contain the needed Concepts" (Step 4). This verification is important because a related document that fails to contain its assigned concept would create an unsolvable instance — the model cannot extract a concept that is not present in the document set.

Noise document selection. The annotator curates several documents from the references that are topically related to the Instruction but do not contain any of the solution-relevant concepts. The verification is strict: "Noise documents do not contain any solution Concepts while remaining on-topic" (Step 4). This dual constraint — topically related but conceptually irrelevant — is what makes noise documents challenging. A noise document about a closely related theoretical approach that uses different formalisms would satisfy this constraint: it is clearly on-topic (it discusses the same research problem), but it does not provide the specific concepts needed to solve this particular question.

The answer leakage constraint. The paper enforces a critical constraint: "no document contains the Answer verbatim or in an equivalent directly-recoverable form." This prevents what would otherwise be a trivial solution strategy: if the answer appears verbatim in a document, the task reduces to extractive question answering — find the sentence that matches the answer format and copy it. By requiring that the answer must be derived from concepts rather than located in text, the benchmark ensures that the Full-set and Related-only regimes genuinely test reasoning over extracted concepts, not pattern matching against answer strings.

Why this noise model matters for the diagnostic architecture. The Full-set regime provides the model with all documents — both Related and Noise. Performance in this regime depends on two separable capabilities: (a) identifying which documents contain the needed concepts (evidence selection, denoising), and (b) extracting those concepts and composing them into a solution (document-to-concept extraction and concept-level reasoning). By comparing Related-only performance (where evidence selection is unnecessary — all provided documents are relevant) to Full-set performance, the benchmark isolates the cost of noise: any performance drop must be attributable to failures in evidence selection or noise-induced reasoning disruption, since the relevant documents are present in both conditions. This is the Related-only → Full-set gap, which the paper calls "noise-induced loss" (Section 3.3).

---

The Four Evaluation Regimes as Controlled Experiment Design

This is the paper's central methodological innovation. The four evaluation regimes are not arbitrary information-access levels; they are a factorial design that systematically varies two dimensions: (1) whether the model must retrieve concepts from documents versus receiving them directly, and (2) whether the model must filter relevant information from noise versus receiving only clean evidence.

Instruction-only regime. The model receives only the question text. No documents, no concepts. This regime measures parametric knowledge: can the model answer the question from what it already knows?

• This is the baseline for detecting parametric contamination. If a model achieves high accuracy under Instruction-only, then either the calibration failed (the question was not genuinely novel for this model) or the model has broader parametric knowledge than the calibration model.
• The paper explicitly uses Instruction-only to detect mode-switch fragility: if a model performs worse under Full-set than under Instruction-only (as Gemini-3-Pro does, dropping from 64.2% to 53.7%), this indicates that adding documents disrupted a previously viable parametric reasoning path. The model had a working solution strategy based on its parametric knowledge, but the presence of documents caused it to abandon that strategy in favor of an evidence-grounded approach that it could not execute correctly.

Concepts-only regime. The model receives the question text plus the complete, oracle Concepts list — all the theorems, formalisms, and conceptual knowledge needed to derive the answer, presented directly without any documents to search. This regime measures concept-level reasoning: given all the right concepts, can the model compose and schedule them into a correct multi-step derivation?

• This is the paper's intended upper bound on reasoning capability. If a model fails under Concepts-only, the failure cannot be attributed to retrieval (there is nothing to retrieve — the concepts are provided directly), to noise (there are no distractors), or to missing concepts (all necessary concepts are present). The failure must be a reasoning failure: the model could not coordinate the provided concepts correctly.
• The paper finds that Concepts-only is not a true upper bound in practice — even with oracle concepts, models fail at non-trivial rates (average accuracy: 75.4%). Table 3 shows that under Concepts-only, precision and recall for concept usage are below 1.0, indicating that models sometimes fail to comprehend the provided concepts or apply them correctly. This is the paper's "structural concept misuse" finding: models can name the right concepts but cannot execute them as procedures.

Related-only regime. The model receives the question text plus only the relevant documents — those that contain the necessary concepts. There are no noise documents. This regime measures document-to-concept extraction combined with reasoning: can the model (a) locate the relevant information within documents, (b) extract the operative concepts from that information, and (c) compose those concepts into a correct solution?

• The transition from Concepts-only to Related-only isolates the cost of document-to-concept extraction: any performance drop must be attributable to failures in locating, extracting, or operationalizing concepts from document text, since the same concepts are available in both regimes. The paper calls this "document-to-concept loss" (Section 3.3).
• This gap can be further decomposed: does the model fail because it cannot find the right passages (evidence localization)? Because it finds the right passages but cannot extract the formal concept from the surrounding exposition (concept extraction)? Or because it extracts a surface-level definition but cannot operationalize it as an executable procedure (concept operationalization)? The CoT-based error attribution in Table 3 provides partial answers by classifying whether the model's reasoning trace mentions the correct concepts (addressing extraction) and whether it applies them correctly (addressing operationalization).

Full-set regime. The model receives the question text plus all documents — both relevant and noise. This regime measures denoising, evidence selection, and reasoning under realistic conditions: can the model identify which documents are relevant, extract the right concepts from them, ignore the distractors, and compose a correct solution?

• The transition from Related-only to Full-set isolates the cost of noise and evidence selection: any performance drop must be attributable to failures in distinguishing relevant from irrelevant documents (evidence selection), being misled by distractors (noise-induced reasoning disruption), or suffering from the increased context length (which could affect attention quality or working memory).
• The paper finds that this gap is substantial and non-linear: "As the number of noise documents increases, Full-set performance exhibits a gradual but non-linear decline" (Section 3.3). This non-linearity is important because it suggests distractors do not simply "dilute" the signal proportionally — they can cause qualitative changes in the model's reasoning trajectory, what the paper calls "irreversible trajectory drift" (Section 5, Figure 6).

What the regime gaps operationalize. The paper defines three interpretable metrics derived from regime comparisons:

1. Knowledge loss = Score(Concepts-only) − Score(Instruction-only). This measures how much performance improves when parametric knowledge is supplemented with oracle concepts. A large positive value indicates that the model's parametric knowledge is insufficient but it can effectively use provided concepts. A small or negative value (unlikely by construction due to the calibration protocol) would indicate that providing concepts does not help.

2. Retrieval loss = Score(Concepts-only) − Score(Full-set). This is the paper's headline retrieval loss metric (RLoss, defined in §3.2), measuring total degradation from having to retrieve concepts from noisy documents versus receiving them directly. This is the most diagnostically useful single number: a large retrieval loss indicates that the model struggles with the retrieval-and-reasoning pipeline even though it can reason well with oracle concepts.

3. Noise-induced loss = Score(Related-only) − Score(Full-set). This isolates the specific contribution of distractors: even when all relevant documents are present, does the addition of topically related but irrelevant documents degrade performance? A large noise-induced loss indicates fragility to evidence selection or noise-induced reasoning disruption.

Why this particular set of four regimes. The paper could have chosen different designs (e.g., varying the number of noise documents continuously, or including a "random documents" condition to test topical vs. random noise). The four-regime design is a deliberate trade-off between diagnostic coverage and evaluation cost. Four regimes provide clean identification of the three key transitions (parametric → conceptual, conceptual → document-based, clean documents → noisy documents) while keeping the total evaluation budget manageable (each model runs each instance four times). Adding more regimes would provide finer-grained decomposition but at linearly increasing evaluation cost. The paper uses the controlled factor analyses (Figure 6) to provide more continuous variation along specific dimensions (noise count, concept count) without requiring additional regimes.

---

The Evaluation Protocol: Standardization and Fairness

The evaluation protocol (§3.2) defines the exact conditions under which models are tested, ensuring that differences in scores reflect genuine capability differences rather than engineering artifacts.

Input format standardization. The paper uses a fixed prompt template (provided in Appendix A) that is identical across all models and all regimes. The prompt never instructs the model to quote documents verbatim; instead, it asks for "concise, verifiable final answers consistent with the benchmark's answer format." This suppresses a common failure mode where models produce verbose explanations that are difficult to auto-grade, and ensures that the model's output is directly comparable to the ground-truth Answer format.

Long-text truncation rule. Because some evaluated models "cannot reliably ingest the full document set (e.g., due to context-length limits that can lead to runtime errors or empty generations)" (Section 3.2), the paper applies a deterministic truncation utility to ensure every model can process every instance. The rule is:

• Input: a string content and a maximum character budget max_chars (default: 30,000).
• If max_chars is None, negative, or the input length is below the limit, return content unchanged.
• Otherwise, set half = max_chars // 2, keep the first half characters and the last half characters, and insert a fixed marker line between them: [... Observation truncated due to length ...] with newlines on both sides.

Why this particular truncation strategy. The head-tail preservation approach (keeping both the beginning and end of documents) is motivated by the observation that scientific papers often have important information in both their introduction/abstract (beginning) and their conclusions/results (end), with methodological details in the middle. Truncating only the end (common in naive implementations) would lose the conclusions. Truncating only the beginning would lose the problem setup. The head-tail strategy preserves both, at the cost of losing the middle. The fixed marker line makes the truncation explicit to the model, which might help it reason about the incompleteness of the provided text.

The choice of 30,000 characters as the default budget reflects a practical compromise: it accommodates most models' context windows while being large enough to include substantial document content. The paper does not report the distribution of document lengths, so it is unclear what fraction of documents are actually truncated by this rule.

Sampling configuration. All models use a single shared configuration:

• Temperature = 1
• Nucleus sampling with $p = 0.7$
• Two runs per model-setting pair, with the average score reported

The use of temperature = 1 (rather than greedy decoding) is noteworthy. It introduces sampling variance into the evaluation, which is why the paper runs each setting twice and averages. The motivation is likely to capture the model's distribution over answers rather than its single most likely output — if a model can produce the correct answer with non-trivial probability, temperature-1 sampling will surface that capability across multiple runs, whereas greedy decoding might consistently produce an incorrect high-probability mode. The $p = 0.7$ nucleus sampling parameter truncates the low-probability tail, preventing the model from sampling extremely unlikely tokens that would produce nonsensical outputs.

Auto-evaluation. The paper uses the doubao-seed-1-6-251015 model as an automated evaluator, with a fixed evaluation prompt (Appendix A). The evaluator compares the model's predicted answer against the ground-truth answer, applying "task-specific normalization rules, including numeric tolerance, symbolic equivalence, or checklist-based matching for structured answers" (Section 3.2). For answers that are long or structured, the evaluator uses the checklist provided by the annotator (mentioned in the Answer construction step: "a rigorous checklist of key scoring items") to make binary correct/incorrect judgments objective.

Why automated evaluation with a specific model. Using an LLM as an evaluator is a common practice in recent benchmarks, but it introduces a potential confound: the evaluator model's own biases and errors become part of the measurement. The paper does not report inter-evaluator agreement (e.g., comparing doubao-seed-1-6-251015 against human judgment or against another evaluator model), so the reliability of the automated grading is unknown. The use of checklists for long answers partially mitigates this by decomposing the evaluation into verifiable sub-judgments, but the evaluator model must still correctly apply the checklist.

Two-run averaging. Each model-setting pair is run twice, and the paper reports the average score. This reduces variance from sampling noise (with temperature = 1, the same model can produce different answers on different runs). Two runs is the minimum for any variance reduction; more runs would provide more stable estimates but at increased evaluation cost. The paper does not report the variance between runs, so the reader cannot assess whether the differences between models (which are often small — e.g., GPT-5.1 scores 57.0 vs. DeepSeek-V3.1 at 50.2 under Full-set, a 6.8 percentage point difference) are statistically reliable given the two-sample variance.

---

The Diagnostic Attribution Framework

Beyond answer-level accuracy, the paper provides a structured analysis of why models fail, operationalized through the error attribution methodology in Table 3.

Concept-level metrics. For each model under each regime, the paper computes three concept-level metrics:

• Precision: Of the concepts the model claimed to use (predicted concepts), what fraction were actually in the ground-truth Concepts list? Precision measures whether the model is hallucinating concepts or introducing irrelevant information.

• Recall: Of the concepts in the ground-truth Concepts list, what fraction did the model actually use? Recall measures whether the model is missing necessary concepts — it might produce a plausible-sounding answer that ignores a critical concept.

• Concept accuracy: An aggregate metric derived from per-instance precision and recall, measuring overall concept usage correctness.

These metrics are computed using a two-stage model-assisted protocol:

1. Concept extraction: The doubao-seed-1.6-251015 model extracts the set of concepts that the evaluated model explicitly applied, by analyzing the model's predicted chain-of-thought (pred_cot). This is a structured information extraction task: given the reasoning trace, identify which scientific concepts, theorems, or formalisms were invoked.

2. Concept alignment: The same doubao-seed-1.6-251015 model is given the predicted concepts and the ground-truth Concepts list, and prompted to align them — determining which predicted concepts correspond to which ground-truth concepts (true positives), which predicted concepts have no ground-truth counterpart (false positives), and which ground-truth concepts were not predicted (false negatives).

The paper reports these metrics in Table 3. Notably, even under Concepts-only (where the full Concepts list is provided), precision and recall are well below 1.0 for all models. For example, GPT-5.1 achieves precision 0.754 and recall 0.754 under Concepts-only. This means that even when explicitly given the correct concepts, the model (a) sometimes fails to use all of them (recall < 1), and (b) sometimes introduces concepts not in the provided list (precision < 1 — though this is less common, with false positives being rarer than false negatives). This finding challenges the assumption that providing oracle concepts eliminates concept-related errors; models can fail to comprehend or operationalize even explicitly provided concepts, which the paper categorizes as "structural concept misuse."

Error type classification. For a deeper diagnosis of failure modes, the paper samples 50 incorrect instances per model per regime and classifies each into one or more of four error categories:

1. Missing core concept (MC): The model's reasoning trace fails to include a key concept required for the correct answer. This is the most common error type across all regimes, accounting for roughly 40 errors per 50 sampled instances in Full-set (Table 3). It indicates that the model did not retrieve or did not recognize the relevance of a critical piece of knowledge.

2. Incorrect understanding or misuse of a core concept (UC): The model mentions a concept but applies it incorrectly — misinterpreting its meaning, using it in an inappropriate context, or drawing wrong conclusions from it. This is less common than missing concepts but diagnostically important because it indicates a failure of operationalization rather than retrieval: the model found the concept but could not execute it correctly.

3. Reasoning-process error (R): Flaws in the logical flow of reasoning — incorrect deductions, invalid inferences, circular arguments, or broken causal chains — even when the concepts are correctly identified and understood. These errors are common across all regimes, indicating that multi-step reasoning remains a bottleneck independent of concept availability.

4. Numeric or formalization error (NF): Mistakes in calculations, symbolic manipulation, or formal notation — arithmetic errors, algebraic mistakes, incorrect variable substitutions. These are relatively rare (typically 0–5 per 50 instances), suggesting that when models get the reasoning right, they generally execute the mathematical mechanics correctly.

The classification is performed by the doubao-seed-1.6-251015 model, prompted with four inputs: the predicted concepts, predicted chain-of-thought, ground-truth concepts, and reference chain-of-thought. The prompt instructs the evaluator to diagnose the "root causes of failure," and each instance can receive multiple error labels (e.g., a model might both miss a concept AND make a reasoning error in the same response).

Why this four-category taxonomy. The taxonomy decomposes "wrong answer" into four pathways that suggest different interventions. Missing core concepts (MC) suggests improvements to retrieval or evidence selection. Concept misuse (UC) suggests better training on concept operationalization — teaching models not just to recognize definitions but to execute procedures. Reasoning-process errors (R) suggest improvements to multi-step reasoning architectures (chain-of-thought, self-consistency, verifier-guided search). Numeric errors (NF) suggest improvements to tool use (calculator integration, symbolic math engines). By reporting these category frequencies, Table 3 tells a practitioner where to invest effort for each model, rather than merely reporting that the model is wrong.

The sampling methodology for error attribution. The error analysis is performed on a subset: "for each evaluated model and each of the four evaluation settings, we uniformly sample 50 instances where the model's final answer is incorrect." This sample size of 50 per setting per model means that the error category frequencies in Table 3 are estimates with sampling variance — a category reported as 42/50 might be 38–46/50 in a different sample. The paper does not report confidence intervals or standard errors for these counts.

More importantly, the sampling is conditioned on incorrect answers — it only examines failures, not successes. This means the error distribution in Table 3 describes how models fail, not how often they fail. A model with high overall accuracy will have fewer total failures, so the 50 sampled failures represent a larger fraction of its error population than for a low-accuracy model. This conditioning choice makes error categories comparable across models (all are based on 50 failures) but prevents estimating the absolute frequency of each error type across all instances.

The cross-regime patterns in error types. Table 3 reveals systematic shifts in error profiles across regimes:

• In Instruction-only, missing core concepts (MC) is the dominant error (averaging 39.4/50), which is expected — the model lacks the concepts entirely.
• In Concepts-only, MC drops sharply (to 9.2/50) but does not reach zero — models still sometimes fail to use provided concepts. Reasoning errors (R) drop less dramatically (8.4/50 vs. 38.8/50 in Instruction-only), indicating that reasoning difficulty persists even when concepts are available.
• In Full-set and Related-only, MC rises again (to ~34–40/50), indicating that concept extraction from documents is a major bottleneck — models often fail to recover the necessary concepts from document text even when they are present.

The paper interprets the similarity of MC counts between Instruction-only and Full-set as evidence that "the gap in the number of missing core concepts between Instruction-only and the knowledge-providing settings (Full-set and Related-only) is not large, with all hovering around 40, reflecting a limitation in the model's ability to retrieve the correct knowledge points, or the model sometimes does not even know which concepts to extract to answer the current question" (Section 3.3). In other words, providing documents does not substantially reduce concept omission compared to providing nothing — the model fails to extract and use the available evidence effectively.

4. Key Insights and Innovations

Innovation 1: Regime-to-Regime Performance Gaps as a Causal Diagnostic Language for Retrieval-Infused Reasoning

The paper's most fundamental intellectual contribution is not a new benchmark per se, but the introduction of a controlled experiment methodology for evaluating retrieval-infused reasoning that replaces binary correct/incorrect scoring with an interpretable set of performance gaps. Prior benchmarks — whether closed-book QA (GPQA, MATH), end-to-end RAG pipelines (KILT, FreshQA), or agent-based deep research evaluations (BrowseComp, HLE) — all share a common structural limitation: they produce a single aggregate accuracy score that conflates retrieval quality, evidence selection, noise filtering, concept extraction, and multi-step reasoning into one number. When a model fails, you know it failed, but you cannot determine which capability failed.

DeR² fundamentally reframes the evaluation problem by treating information access as an independent variable. The four evaluation regimes (Instruction-only, Concepts-only, Related-only, Full-set) are not merely different difficulty levels — they are an experimental design that systematically varies whether the model must (a) retrieve concepts from documents or receive them directly, and (b) filter relevant information from noise or receive only clean evidence. The performance differences between these regimes are causally interpretable: the Concepts-only → Related-only gap isolates document-to-concept extraction failure, the Related-only → Full-set gap isolates noise-induced evidence selection failure, and the Concepts-only → Instruction-only gap quantifies parametric knowledge contamination. This transforms evaluation from "model A scores 65%, model B scores 70%" into a diagnostic statement like "model A's 20-point drop from Concepts-only to Full-set indicates a retrieval bottleneck, while model B's 5-point drop indicates strong evidence selection but weak concept coordination (evidenced by its lower Concepts-only score)."

What makes this genuinely novel — rather than simply a well-designed benchmark — is that it provides a vocabulary for failure attribution that the field previously lacked. Before DeR², researchers and practitioners described retrieval-infused reasoning failures in vague terms ("the model couldn't handle the documents," "retrieval quality was poor"). The regime-gap decomposition gives precise operational definitions: retrieval loss is Score(Concepts-only) − Score(Full-set), noise-induced loss is Score(Related-only) − Score(Full-set), knowledge loss is Score(Concepts-only) − Score(Instruction-only). These are not just metrics — they are diagnostic probes that point to specific architectural or training deficiencies. A model with high retrieval loss but low noise-induced loss needs better document-to-concept extraction (perhaps better attention over long contexts, or training on concept extraction from scientific text). A model with high noise-induced loss needs better evidence selection or denoising (perhaps training on distractor-rich corpora, or architectural mechanisms for suppressing irrelevant context). This diagnostic precision directly addresses the paper's critique that current benchmarks provide "limited guidance for model selection, debugging, or targeted training" (Section 1).

The intellectual lineage here is instructive. The paper draws on the logic of controlled experiments from the behavioral sciences, where varying a single independent variable while holding others constant enables causal attribution. Prior RAG evaluation treated "retrieval quality" and "reasoning quality" as latent variables that could only be inferred indirectly (e.g., by measuring how accuracy changes when you swap retrievers). DeR² makes these variables manifest through its input configuration design — you directly observe what happens when you eliminate retrieval by providing oracle concepts, and you directly observe what happens when you eliminate noise by providing only relevant documents. This methodological move is a genuine departure from the field's default approach of accepting end-to-end confounding as inevitable in "realistic" evaluation. The paper demonstrates (Figure 2) that you can achieve diagnostic precision without sacrificing the core difficulties of deep research (multi-step synthesis, denoising, evidence-based conclusion making) by carefully constructing the input conditions rather than by sacrificing control.

Innovation 2: Mode-Switch Fragility as an Empirical Phenomenon — Providing Documents Can Degrade Performance

Perhaps the most counterintuitive finding in the paper is the mode-switch fragility documented in Table 2: several state-of-the-art models perform worse when provided with relevant documents (Full-set) than when given no documents at all (Instruction-only). Gemini-3-Pro drops from 64.2% to 53.7%; Claude-Opus-4.1 drops from 49.3% to 40.0%; GPT-5.1 drops from 59.8% to 57.0%; DeepSeek-V3.2-Exp-Thinking drops from 53.1% to 47.6% (as noted in the text). This is not a marginal effect — it is a substantial degradation that inverts the expected relationship between information access and performance.

The intellectual significance of this finding extends well beyond the specific numbers. It forces a fundamental reconceptualization of what happens when an LLM receives external context. The dominant assumption in the RAG and agent literature — implicit in the entire architecture of retrieval-augmented systems — is that providing relevant documents is at worst neutral: it might not help if the model already knows the answer, but it should not actively hurt. Mode-switch fragility demonstrates that this assumption is false. The model's reasoning controller — whatever internal mechanism governs whether to rely on parametric knowledge versus to ground reasoning in provided evidence — can be disrupted by the presence of documents, causing the model to abandon a viable parametric reasoning path in favor of an evidence-grounded approach that it executes incorrectly.

The paper categorizes this as a "reasoning mode switching failure" (Section 3.3) and describes it as evidence that "switch control between internal-knowledge reasoning and context-grounded reasoning is itself a bottleneck." This is a new failure mode that was invisible in prior evaluations. End-to-end RAG benchmarks cannot detect it because they do not measure Instruction-only performance — without the counterfactual of what the model would have answered without documents, you cannot observe the performance inversion. This means that prior work may have systematically overestimated the benefit of retrieval for certain models and tasks, because the baseline for comparison was implicit (assumed zero without documents) rather than measured.

The theoretical implications are significant. Mode-switch fragility suggests that the model's behavior is not a smooth function of evidence quality — it involves a discrete regime change where the presence of documents triggers a different reasoning strategy. This aligns with mechanistic interpretability findings about how transformers can exhibit phase transitions in their internal computation based on input features, but DeR² provides the first clear behavioral evidence of this phenomenon in the context of retrieval-infused reasoning. It also connects to broader questions about LLM "faithfulness" — whether models genuinely condition their answers on provided evidence or merely produce plausible-sounding text that may or may not be evidence-grounded. Mode-switch fragility implies that the faithfulness mechanism is fragile: providing documents can disrupt rather than strengthen evidence grounding.

This finding is not just a negative result — it is a diagnostic discovery that reframes the research agenda. Rather than asking "how can we improve retrieval quality?" (the dominant question in RAG research), DeR² demonstrates that an equally important question is "how can we ensure that providing documents does not disrupt the model's reasoning controller?" This suggests research directions that are qualitatively different from improving retrievers or chunking strategies — directions like training models to explicitly recognize and maintain their reasoning mode, developing architectures that smoothly interpolate between parametric and evidence-grounded computation, or designing prompts that stabilize mode-switch behavior.

The paper supports this finding with both quantitative evidence (the accuracy inversion in Table 2 across multiple models) and qualitative evidence (the CoT analysis in Table 3, which shows that in Full-set, models frequently exhibit reasoning mode switching failures where they cite superficially relevant evidence while executing a generic template solution — evidence is present but not integrated into the causal chain of inference). The consistency of the pattern across models (GPT-5.1, Gemini-3-Pro, Claude-Opus-4.1, DeepSeek-V3.2-Exp-Thinking all show the inversion) argues against model-specific quirks and suggests a fundamental architectural limitation.

Innovation 3: Structural Concept Misuse — Models Name the Right Concepts but Cannot Execute Them Procedurally

The third major insight from DeR² is the discovery that concept identification is not concept operationalization. The paper finds that even under Concepts-only — where the model receives the complete, correct set of concepts needed to solve the problem — accuracy averages only 75.4% (Table 2), and concept-level precision and recall are well below 1.0 (Table 3: GPT-5.1 achieves only 0.754 precision and 0.754 recall). This means that roughly one quarter of the time, even when explicitly told which concepts, theorems, and formalisms to use, state-of-the-art models fail to produce the correct answer.

This finding matters because it challenges a widespread implicit assumption in the RAG and agent literature: that the hard part of retrieval-infused reasoning is finding the right information, and that once the information is located, reasoning over it is relatively straightforward. The paper calls this assumption into question with a precise empirical demonstration. The Concepts-only condition eliminates all retrieval difficulty — there are no documents to search, no passages to locate, no noise to filter, no extraction to perform. The concepts are presented directly, in clean form, explicitly labeled as the solution-relevant knowledge. And yet models still fail at a non-trivial rate. These failures cannot be attributed to retrieval quality, evidence selection, or noise filtering. They must be reasoning failures — specifically, failures to take abstract theoretical knowledge (concepts, theorems, formalisms) and operationalize it as an executable computational procedure.

The paper distinguishes this failure mode from simpler "hallucination" or "knowledge gap" explanations. In the Concepts-only setting, the model is not hallucinating facts it doesn't know — the facts are provided. It is failing at procedural execution: taking a theorem statement and instantiating its variables correctly, applying a multi-step algorithm in the right order, tracking constraints and dependencies across steps, or maintaining intermediate invariants. The paper terms this "structural concept misuse": "even when models identify and restate the correct concepts, they often fail to execute them as procedures (e.g., applying rules step-by-step or instantiating abstractions), instead reverting to generic heuristics that appear plausible but produce incorrect conclusions" (Section 1).

Table 3 provides quantitative evidence for this claim. Under Concepts-only, the error category distribution shifts dramatically compared to Instruction-only: missing core concepts drops from ~39/50 to ~9/50, but reasoning-process errors only drop from ~39/50 to ~8/50 — proportionally, reasoning errors become more dominant among the remaining failures. The model gets the concepts (MC drops sharply) but still makes reasoning errors (R remains elevated relative to the reduced total failure count). The concept misuse category (UC) also persists at ~4/50 even under Concepts-only, indicating that some failures involve explicitly misunderstanding or misapplying the provided concepts.

The qualitative case studies in Appendix B reinforce this finding. In Case B.2 (DeepSeek-V3.1 in Concepts-only), the model correctly names "Lifted Formulation" and "Schubert Variety," but then fabricates an incorrect heuristic ("total number of equations = sum of descent set sizes") rather than executing the actual constructive procedure from the provided concepts. The paper's diagnosis captures this precisely: "The model lacked knowledge of the specific construction... and fabricated an incorrect heuristic rule." This is not concept omission — the concepts are present in the model's output. It is concept operationalization failure: the model cannot translate the abstract concept definitions into the specific computational steps required to solve this instance.

The intellectual payoff of this finding is that it identifies a bottleneck that is distinct from retrieval quality. The field has invested heavily in improving retrieval (better embeddings, better reranking, better chunking, hybrid search, agentic retrieval), implicitly assuming that better retrieval will translate to better end-to-end performance. DeR²'s Concepts-only results suggest that even with perfect retrieval (oracle concept provision), current models would leave roughly 25% of frontier scientific reasoning problems unsolved due to procedural execution failures. This implies that retrieval improvements alone cannot close the performance gap — progress requires advances in how models operationalize abstract knowledge into executable procedures, which is a fundamentally different research challenge than improving search quality.

Innovation 4: Adversarial Distractors as a First-Class Variable That Triggers Non-Linear Reasoning Disruption

The fourth innovation is the operationalization of adversarial noise as a controlled variable in evaluation, and the empirical demonstration that distractors do not simply dilute signal proportionally but can cause qualitative trajectory changes in multi-step reasoning.

Prior benchmarks that include distractors (e.g., multi-hop QA with distractor paragraphs, long-context retrieval tasks with irrelevant documents) typically treat noise as a quantity — more irrelevant documents make the task harder by increasing the search space. DeR² treats noise as a qualitative variable by carefully constructing distractors that are "topically adjacent to the Instruction but do not contribute solution-relevant concepts" (Step 4, Section 2.3). This is a crucial distinction. A noise document about a different scientific field entirely would be trivially filterable by surface-level topic matching; the model would not even need to read it. A noise document about the same scientific problem but using different formalisms or addressing a different aspect of the problem is genuinely challenging — it occupies the same semantic neighborhood, uses overlapping terminology, and would plausibly appear in a literature search. Filtering it requires conceptual discrimination, not just topic detection.

The empirical payoff of this design choice is the finding in Section 3.3 and Figure 6 that noise effects are non-linear:

"As the number of noise documents increases, Full-set performance exhibits a gradual but non-linear decline, consistent with the hypothesis that distractors do not simply 'dilute' signal, but can redirect early-step abstraction and cause irreversible trajectory drift"

This is a stronger claim than "more noise makes the task harder." It suggests that distractors can change what the model does rather than merely making correct retrieval less likely. In a linear signal-dilution model, each noise document reduces the probability of finding the right passage proportionally, and the performance decline is smooth. In a trajectory-drift model, a distractor that is encountered early in the reasoning process can bias the model's initial abstraction (what kind of problem is this? what framework should I use?), and that bias propagates through the multi-step derivation, leading to errors that are not recoverable even if the model later encounters the correct evidence. This is analogous to the difference between measurement noise (which reduces precision but doesn't change the target) and confounding (which systematically biases the estimate in a particular direction).

The existence of trajectory drift is supported by the qualitative error analysis. The paper describes failures where "the model cites superficially relevant evidence while executing a generic template solution, indicating evidence is present but not integrated into the causal chain of inference" (Section 3.3). This pattern is consistent with the model latching onto a distractor's framing of the problem and then executing a reasoning template that fits that framing, even when the correct evidence is also present in the context. The distractor doesn't just clutter the context — it hijacks the reasoning trajectory.

This finding has significant practical implications for agent design. Current agent architectures typically collect documents (through search or retrieval) and then present them all to the model for synthesis. If distractors can cause irreversible trajectory drift, then the order and selection of documents presented to the model matters enormously — presenting a misleading document before a relevant one could cause the model to commit to an incorrect reasoning path that it cannot recover from, even when the correct evidence is later in the context. This suggests that agent architectures might need explicit mechanisms for document triage (filtering before reasoning) or iterative hypothesis revision (ability to backtrack when later evidence contradicts earlier assumptions), rather than naive "collect and present" strategies.

The intellectual distinction from prior work is sharp. Most RAG evaluations treat distractors as a nuisance parameter to be minimized (through better retrieval), not as a diagnostic probe to be systematically varied. DeR² makes distractors a first-class experimental variable, varying their number and measuring their effect on performance while controlling for other factors (concept count, reasoning depth). This transforms distractors from a source of unwanted variance into a measurement instrument: the Related-only → Full-set gap becomes a direct measure of a model's robustness to adversarial noise, and the non-linearity of the noise-response curve becomes a diagnostic signature of trajectory drift. This is a conceptual advance in evaluation methodology — it treats the noise structure of the benchmark as part of the measurement apparatus, not as an imperfection to be tolerated.

5. Experimental Analysis

Evaluation Methodology

• Dataset. DeR² consists of expert-annotated instances drawn from 2023–2025 theoretical scientific papers, each containing an Instruction, oracle Concepts list, validated Chain-of-Thought, Answer, and a frozen Document Set (mean: 6.5 documents) with both relevant and topically adjacent distractor documents. The exact number of total benchmark instances is not specified in the paper, but the data collection involved 81 PhD annotators across various scientific disciplines, with each annotator producing instances within their specialized academic field. Figure 4 shows the distribution across problem domains and answer types, confirming disciplinary breadth and structural diversity of target answers.

• Base model(s). The paper evaluates a diverse set of commercial and open models: GPT-5.1, Gemini-3-Pro, Gemini-2.5-Pro, Claude-Opus-4.1, Claude-Sonnet-4.5, DeepSeek-V3.1, DeepSeek-V3.2, Doubao-1.6, Doubao-1.8, Qwen-3-max, Moonshot-Kimi-k2-thinking, and GLM-4.6 (Section 3.2). These span multiple vendors, scales, and training paradigms. The paper states the aim is "to evaluate the retrieval-enhanced reasoning capabilities of different models in a controlled sandbox environment, rather than optimizing for any specific vendor stack." No single model is treated as the primary subject; instead, cross-model variation is the object of study. For difficulty calibration during instance construction, the paper uses offline models (DeepSeek-R1-0528 and Doubao web client with networking disabled), but these are not the evaluation subjects.

• Metrics. The primary metric is answer-level accuracy — the fraction of instances for which the model's predicted final answer matches the ground-truth Answer, graded by doubao-seed-1-6-251015 using task-specific normalization rules (numeric tolerance, symbolic equivalence, checklist-based matching for structured answers) per the evaluation prompt in Appendix A (Section 3.2). Each model-setting pair is run twice and the average score is reported. Additionally, the paper computes concept-level precision and recall (Table 3): precision is the fraction of predicted concepts that match ground-truth Concepts, and recall is the fraction of ground-truth Concepts that appear in the model's predicted reasoning trace, both extracted and aligned by doubao-seed-1.6-251015. A derived Retrieval Loss metric is defined as Score(Concepts-only) − Score(Full-set), representing total degradation from having to retrieve concepts from noisy documents versus receiving them directly (Section 3.2). The paper also reports error type distributions across four categories (Missing Core Concept, Concept Misuse, Reasoning-Process Error, Numeric/Formalization Error) based on analysis of 50 sampled incorrect instances per model per regime.

• Baselines. The paper does not compare against external prior benchmarks or baseline methods in the traditional sense. Rather, the four evaluation regimes serve as internal baselines that define a performance spectrum: Instruction-only establishes parametric knowledge floor, Concepts-only establishes an empirical upper bound on concept-level reasoning, Related-only provides the document-to-concept extraction baseline without noise, and Full-set provides the noisy retrieval-conditioned baseline. The regime-to-regime gaps (Concepts-only → Related-only, Related-only → Full-set, Concepts-only → Full-set) are the primary comparative measures, not comparisons to prior benchmark scores. The paper does compare its approach conceptually against existing benchmarks (KILT, FreshQA, BrowseComp, HLE, GPQA, MATH, GSM8K) in Section 4, but these are not used as quantitative baselines in the experiments.

• Generation budget / compute accounting. The paper does not use "generations" or FLOPs as a compute budget — this is a benchmark evaluation, not a scaling study. All models are run with the same sampling configuration: temperature = 1, nucleus sampling with $p = 0.7$, two runs per model-setting pair (Section 3.2). There is no variable compute budget being allocated or compared. A deterministic long-text truncation rule is applied with a 30,000-character default budget to ensure models with varying context-length limits can all process the document sets, using a head-tail preservation strategy: keep the first max_chars // 2 characters and the last max_chars // 2 characters, inserting a marker [... Observation truncated due to length ...] between them. This truncation applies uniformly when needed, not as a variable experimental condition.

• Cross-validation / statistical protocol. There is no cross-validation or train/test split reported — DeR² is a fixed evaluation benchmark, not a training dataset, so there are no hyperparameters being selected or models being fit to the data. The paper does not report confidence intervals, standard deviations, or statistical significance tests for the accuracy scores, either across instances or across the two runs per model-setting pair. The two-run averaging reduces sampling variance from temperature-1 decoding but the magnitude of between-run variance is not reported. The error attribution analysis in Table 3 is based on a uniform sample of 50 incorrect instances per model per regime — the paper does not report sampling error for the category frequencies, so the reported counts (e.g., "MC:42") should be understood as point estimates with unknown variance. The difficulty calibration protocol during instance construction (Section 2.3) requires three Instruction-only attempts all incorrect and three Concepts-only attempts with at least one correct and at least one incorrect, but this is a per-instance acceptance criterion, not a statistical procedure applied to evaluation results.

Main Quantitative Results

DeR²'s experimental contribution is not a single headline number but a structured comparison across models, regimes, and difficulty factors. The results are presented in three layers: overall accuracy across regimes (Table 2), controlled factor analyses (Figure 6), and CoT-based error attribution (Table 3). I walk through each in turn.

Overall Accuracy and Regime Gaps (Table 2)

Table 2 reports answer-level accuracy for all twelve evaluated models under all four regimes. I focus on the five models that receive the most detailed analysis throughout the paper: GPT-5.1, Gemini-3-Pro, Claude-Opus-4.1, DeepSeek-V3.1, and Doubao-1.6-1015. The paper also includes aggregate averages across a broader set of evaluated models.

Cross-regime aggregate pattern. The paper reports (Section 3.3) that across models: Concepts-only averages 75.39%, Related-only averages 62.89%, and Full-set averages 51.25%. The ordering Concepts-only >> Related-only >> Full-set is consistent, confirming that document-to-concept extraction and noise filtering are major sources of performance degradation. Instruction-only averages 55.89% across models, which is notable: it is higher than the Full-set average (51.25%), indicating the mode-switch fragility phenomenon is systemic across the model population, not isolated to specific models. The paper states this explicitly: "we also observe a counter-intuitive but recurrent phenomenon where Instruction-only (avg. 55.89) can match or even exceed Full-set (avg. 51.21) for most models" (Section 3.3).

Per-model Full-set performance (the most realistic regime). GPT-5.1 leads at 57.0%, followed by Gemini-3-Pro at 53.7% and Gemini-2.5-Pro at 51.7% (Table 2). DeepSeek-V3.1 achieves 50.2%, Doubao-1.6-1015 achieves 43.3%, and Claude-Opus-4.1 achieves 40.0%. The spread from 57.0% (best) to 40.0% (worst among these five) represents a 17 percentage point gap — substantial headroom even among top-tier models. The paper does not normalize for model scale or training compute, so these differences may reflect architecture, training data recency, scale, or a combination of factors.

Mode-switch fragility — the Instruction-only vs. Full-set inversion. Table 2 reveals a striking pattern for several models:

• Gemini-3-Pro: Instruction-only 64.2% → Full-set 53.7% (a 10.5 percentage point drop)
• Claude-Opus-4.1-thinking: Instruction-only 49.3% → Full-set 40.0% (a 9.3 point drop)
• OpenAI-GPT-5.1-high: Instruction-only 59.8% → Full-set 57.0% (a 2.8 point drop)
• DeepSeek-V3.2-Exp-Thinking: Instruction-only 53.1% → Full-set 47.6% (a 5.5 point drop)

Not all models show this inversion. DeepSeek-V3.1 improves from 52.2% (Instruction-only) to 50.2% (Full-set) — a small drop, but notably, even here the model does not substantially benefit from having documents. Qwen-3-max shows a similar pattern. The broader point is that providing documents rarely produces large improvements over parametric knowledge alone. The expected benefit of retrieval-augmented reasoning — that adding evidence should substantially boost accuracy beyond what the model knows parametrically — is largely absent from these results. The Concepts-only scores (averaging 75.4%) confirm that the information exists to solve these problems; the models simply cannot access it effectively through document-based regimes.

The retrieval loss decomposition. Using the Retrieval Loss metric (Score(Concepts-only) − Score(Full-set)):

• GPT-5.1: 81.4% − 57.0% = 24.4 percentage point retrieval loss
• Gemini-3-Pro: 80.9% − 53.7% = 27.2 point loss
• Claude-Opus-4.1: 72.4% − 40.0% = 32.4 point loss
• DeepSeek-V3.1: 69.8% − 50.2% = 19.6 point loss
• Doubao-1.6-1015: 70.8% − 43.3% = 27.5 point loss

These are large gaps — all models lose 20-32 percentage points when they must retrieve concepts from documents rather than receiving them directly. The paper interprets this as evidence that "concept extraction from documents remains a major source of loss" (Section 3.3). Claude-Opus-4.1 shows the largest gap (32.4 points), suggesting a particular weakness in document-to-concept extraction relative to its concept-level reasoning capability (72.4% Concepts-only is competitive). GPT-5.1 has the highest absolute Concepts-only score (81.4%) and the second-highest Full-set score (57.0%), suggesting relatively better end-to-end capability even though its retrieval loss is still large (24.4 points).

Document-to-concept loss vs. noise-induced loss. The paper decomposes the total retrieval loss into two components (Section 3.3):

• Document-to-concept loss = Score(Concepts-only) − Score(Related-only)
• Noise-induced loss = Score(Related-only) − Score(Full-set)

From Table 2, computing these for the five focus models:

The decomposition reveals different bottleneck profiles for different models. DeepSeek-V3.1 has relatively small document-to-concept loss (7.7 points) but substantial noise-induced loss (11.9 points), suggesting its primary weakness is evidence selection under noise rather than concept extraction from clean documents — it can extract concepts when given clean evidence but struggles to identify which documents to extract from when distractors are present. Gemini-3-Pro shows a more balanced profile (12.6 doc→concept, 14.6 noise). Claude-Opus-4.1 has the largest document-to-concept loss (20.4 points), which the paper's concept-level metrics in Table 3 corroborate: under Related-only, Claude-Opus-4.1 achieves only 0.241 precision and 0.212 recall, substantially lower than GPT-5.1 (0.329 precision, 0.339 recall) or Gemini-3-Pro (0.349 precision, 0.332 recall). This granularity in diagnosing where each model fails — rather than merely reporting that Claude scores lower — is precisely the diagnostic value DeR² claims to provide.

Concepts-only is not a true ceiling. Table 2 shows that even under oracle concept provision, no model reaches 100% accuracy. GPT-5.1 achieves 81.4%, Gemini-3-Pro 80.9%, Claude-Opus-4.1 72.4%, DeepSeek-V3.1 69.8%, and Doubao-1.6-1015 70.8%. The paper emphasizes this: "Concepts-only is not an upper bound, revealing bottlenecks in multi-concept coordination and long-range dependency tracking" (Section 5). The concept-level precision and recall metrics in Table 3 confirm that even under Concepts-only, models do not achieve perfect concept usage — GPT-5.1 achieves only 0.754 precision and 0.754 recall, meaning approximately 25% of the oracle concepts are either not used or are supplemented with non-oracle concepts in the model's reasoning.

Controlled Factor Analyses (Figure 6)

The paper stratifies instances by structural properties of the dataset to surface which factors drive performance degradation.

Noise document count (Figure 6a). Under Full-set, model performance is plotted as a function of the number of noise documents. The paper reports a "gradual but non-linear decline" (Section 3.3). The non-linearity is the key empirical claim: if noise merely diluted signal proportionally, the decline would be smooth and roughly linear in the probability of selecting the right document. The observed non-linearity suggests that additional noise documents do more than increase the search space — they can cause qualitative changes in the model's reasoning trajectory. The paper describes this as noise that can "redirect early-step abstraction and cause irreversible trajectory drift" (Section 3.3). The figure shows this pattern across multiple models, with the decline becoming steeper at higher noise counts, consistent with a threshold effect where the model's ability to maintain a correct evidence-grounded reasoning path collapses once the noise level exceeds some critical value.

Required concept count (Figure 6b). Accuracy decreases as the number of required concepts grows, even under Concepts-only. This is a crucial finding: "providing all necessary concepts does not eliminate errors: models still struggle with concept-level working memory, dependency tracking, and coordinating multiple constraints" (Section 3.3). Under Concepts-only, where retrieval is eliminated, the performance decline with concept count must reflect fundamental reasoning limitations — multi-concept coordination, dependency management, and maintaining multiple constraints simultaneously. The paper notes that the effect is "amplified in Concepts-only and Full-set, where concept extraction errors and spurious concept injection further enlarge the hypothesis space and increase coordination burden." This suggests that concept count affects performance through two mechanisms: an intrinsic reasoning difficulty (harder to coordinate more concepts even when given exactly) and an extraction difficulty (harder to find and correctly identify more concepts from documents). The Concepts-only curve isolates the intrinsic component.

Reasoning depth and answer type. The paper reports (Section 3.3) that problems are bucketed by reasoning depth (number of steps in the reference CoT) and by answer type (formula derivation, numeric, boolean, conclusion-style). The key finding: "Deeper reasoning typically yields larger gaps between Concepts-only and document-based settings, reflecting compounding failures in multi-step grounding and intermediate-state maintenance." This is a compounding error effect — if each reasoning step has some probability of failure due to imperfect evidence grounding, longer chains accumulate more opportunities for derailment. Additionally, "formula-derivation items benefit the most from explicit concepts, suggesting that document-based settings often fail at operationalizing constructive or procedural concepts (e.g., algorithm instantiation, theorem application) even when high-level definitions are recognized." This connects to the structural concept misuse finding: procedural knowledge — how to execute a theorem or algorithm — is harder to extract from documents and apply correctly than declarative knowledge (what the theorem states).

Chain-of-Thought and Error Attribution (Table 3)

Table 3 provides the deepest diagnostic layer, analyzing why models fail rather than just that they fail. I examine the five-model average pattern across regimes and then highlight model-specific deviations.

Instruction-only error profile. For the five focus models (averaging across Claude-Opus-4.1, DeepSeek-V3.1, Doubao-1.6-1015, Gemini-3-Pro, GPT-5.1):

• Missing Core Concept (MC): 39.4/50. This is the dominant error, as expected — without documents or concepts, the model lacks the necessary knowledge.
• Reasoning-Process Error (R): 38.8/50. Nearly as frequent as MC, indicating that even when models attempt to reason with parametric knowledge, their reasoning chains are frequently flawed.
• Concept Misuse (UC): 2.6/50. Relatively rare — without concepts to misuse, this error is uncommon.
• Numeric/Formalization (NF): 3.2/50. Rare.

The high R count alongside high MC suggests that Instruction-only failures are a mix of "I don't know the concept" and "even with what I know, I reason incorrectly." This is consistent with the difficulty calibration protocol, which required parametric failure — the questions are designed to be hard without evidence.

Concepts-only error profile. The shift is dramatic:

• MC drops to 9.2/50 — providing oracle concepts largely eliminates concept omission, but does not eliminate it entirely. Some models still fail to use a provided concept roughly 18% of the time.
• R drops to 8.4/50 — reasoning errors are substantially reduced but not eliminated. Even with all necessary concepts explicitly provided, models still make logical errors in roughly 17% of sampled failures.
• UC rises relatively — 3.8/50. While the absolute count is small, it represents concept misuse when concepts are provided, which is diagnostically significant. It means the model sometimes explicitly misapplies a concept it was given.
• NF remains negligible (0.4/50).

The key insight: providing concepts shifts the error profile from "primarily missing knowledge" to "primarily reasoning and coordination failures." The paper states: "the remaining failures are predominantly procedural, indicating that simply retrieving the correct concepts does not guarantee the correct execution of multi-step reasoning" (Section 3.3).

Full-set and Related-only error profiles. In these document-based regimes:

• MC rises again to 40.2/50 (Full-set average) and 34.0/50 (Related-only average). This is the paper's most troubling finding: the MC count in Full-set (40.2) is nearly identical to Instruction-only (39.4). The paper interprets this directly: "the gap in the number of missing core concepts between Instruction-only and the knowledge-providing settings is not large, with all hovering around 40, reflecting a limitation in the model's ability to retrieve the correct knowledge points, or the model sometimes does not even know which concepts to extract to answer the current question" (Section 3.3). Providing documents barely reduces the rate of concept omission compared to providing nothing — the model fails to extract and use the available evidence.
• R in Full-set averages 40.2/50 — essentially tied with MC. This means that even when models do access some correct concepts from documents, they still make reasoning errors at high rates.
• UC rises to 7.0/50 (Full-set) — higher than in Concepts-only, reflecting that when models extract concepts from documents (rather than receiving clean oracle versions), they are more likely to misinterpret or misapply them. The extraction process may introduce distortions or the model may latch onto surface-level descriptions rather than operational definitions.
• NF remains low (4.0/50 Full-set, 2.8/50 Related-only).

Model-specific deviations. The paper highlights Gemini-3-Pro's profile as notably different:

"Gemini-3-Pro only made 29 errors in the Related-only setting and delivered its optimal performance in Instruction-only (with merely 34 mistakes). This indicates that Gemini may possess more endogenous knowledge, enabling it to retrieve and comprehend concepts effectively even with limited background information."

Gemini-3-Pro's MC count of 29 in Related-only (vs. the cross-model average of 34) and its high Instruction-only accuracy (64.2%) suggest broader parametric coverage of the benchmark's scientific domains. However, its mode-switch fragility (dropping to 53.7% in Full-set) indicates that this parametric strength does not translate to robust evidence-grounded reasoning.

Concept-level precision and recall under Concepts-only. Table 3 reports that even under Concepts-only, precision and recall are well below 1.0. GPT-5.1 achieves 0.754 precision and 0.754 recall. This means that when explicitly given the correct concepts, GPT-5.1 fails to use roughly 25% of them (recall = 0.754) and introduces concepts not in the provided list roughly 25% of the time (precision = 0.754). Gemini-3-Pro performs better on these metrics (0.740 precision, 0.730 recall) but still shows substantial concept omission and hallucination. The paper states: "the model's precision and recall both fail to reach 1, indicating that the model cannot comprehend the provided unseen concepts, even when usage instructions for those concepts are supplied" (Section 3.3). This is direct evidence for the structural concept misuse claim: concept provision does not guarantee concept utilization.

---

Ablation Studies and Robustness Checks

DeR² is a benchmark, not a method with hyperparameters, so traditional ablation studies do not apply in the same way. However, the paper includes several analyses that serve an ablation-like function by testing the robustness of its findings to different conditions or by isolating the effect of specific design choices.

Controlled variation of noise document count (Figure 6a): This serves as an ablation of distractor quantity. By varying the number of noise documents and measuring the effect on Full-set accuracy, the paper tests whether the performance degradation is simply proportional to noise volume (linear signal dilution) or involves qualitative changes in reasoning behavior (non-linear trajectory disruption). The finding of non-linearity ("gradual but non-linear decline") supports the trajectory-drift hypothesis over the simpler signal-dilution hypothesis. This is analogous to an ablation study that varies a hyperparameter and measures the dose-response curve — the shape of the curve provides evidence about the underlying mechanism.

Controlled variation of required concept count (Figure 6b): This ablates reasoning complexity while controlling for evidence access regime. By measuring accuracy under Concepts-only as concept count increases, the paper isolates the intrinsic difficulty of multi-concept coordination (since retrieval and noise are eliminated). The finding that accuracy decreases with concept count even under Concepts-only demonstrates that concept coordination itself is a bottleneck independent of retrieval quality. This is a robustness check on the paper's claim that concept-level reasoning is a distinct failure source — if Concepts-only accuracy were flat across concept counts, the failure would be purely a retrieval problem, but the decline confirms that reasoning complexity matters even with perfect information access.

Oracle vs. predicted concepts (implicit in the regime design): The four-regime structure itself serves as an ablation of information quality. Concepts-only provides oracle-clean concepts (hand-selected by expert annotators, verified to be necessary and sufficient). Related-only provides the same concepts but embedded in documents — the information is present but must be extracted. Full-set adds noise. The performance differences across these regimes ablate the effect of (a) concept cleanliness (oracle list vs. document-embedded), (b) extraction difficulty (clean documents vs. noisy documents), and (c) evidence selection (all relevant vs. mixed). The systematic gaps confirm that each of these factors contributes independently to the overall performance degradation.

Long-text truncation robustness (Section 3.2): The deterministic head-tail truncation rule (30,000 characters, preserving first half and last half with a marker) is applied uniformly to ensure models with varying context-length limits can participate. This is an engineering choice rather than a controlled ablation, but it does test whether the benchmark's findings are artifacts of context-length limitations. The fact that large context-length gaps across models produce performance differences that are not simply explained by context capacity (e.g., models with larger context windows do not uniformly outperform those with smaller ones, though the paper does not report context-length-specific analysis) suggests that the findings are not driven by truncation artifacts. However, the paper does not explicitly ablate the truncation threshold — it does not report, for example, whether increasing the budget to 60,000 characters changes results, which would be a direct test of truncation sensitivity.

Two-run averaging and sampling robustness (Section 3.2): Each model-setting pair is run twice with temperature = 1 and nucleus sampling ($p = 0.7$), and the average is reported. This reduces variance from sampling stochasticity but does not fully characterize it — the paper does not report between-run variance, standard deviations, or confidence intervals. As a robustness measure, two runs is minimal; additional runs would provide stronger evidence that the reported differences between models (often 5-10 percentage points) exceed sampling noise. The absence of variance reporting is a notable gap — a 6.8 percentage point difference between GPT-5.1 (57.0%) and DeepSeek-V3.1 (50.2%) under Full-set could be meaningful or could be within the sampling noise range depending on the between-run variance, which the reader cannot assess.

Model diversity as a robustness check on findings: The paper evaluates 12 models spanning multiple vendors and training paradigms. The consistency of key findings across this diverse set — mode-switch fragility across GPT-5.1, Gemini-3-Pro, Claude-Opus-4.1, and DeepSeek-V3.2; the Concepts-only >> Related-only >> Full-set ordering; the non-linear noise effect — serves as a robustness check that these phenomena are not model-specific quirks. This is the strength of a benchmark evaluation: by testing many models under identical conditions, patterns that replicate across architectures carry more weight than single-model findings would.

Error attribution sampling methodology (Table 3): The CoT-based error analysis samples 50 incorrect instances per model per regime. The paper does not ablate this sample size (e.g., testing whether 25 vs. 50 vs. 100 produces consistent error category distributions), nor does it report inter-annotator agreement for the doubao-seed-1.6-251015 evaluator that performs the error classification. The reliability of the error type distributions therefore depends on the unvalidated accuracy of the auto-evaluator's judgments. This is a genuine methodological limitation — the error attribution analysis provides rich qualitative insight, but its quantitative precision is unknown.

Missing ablations that would strengthen the paper:

• Truncation threshold variation: Testing whether results are stable when the 30,000-character budget is increased or decreased would address concerns about context-length artifacts.
• Human evaluation of auto-grader: Comparing doubao-seed-1-6-251015's answer grading against human expert judgment on a subset would calibrate the primary metric's reliability.
• Error attribution evaluator calibration: Similarly, validating the error category classifications against human expert diagnosis would strengthen confidence in the CoT-based analysis.
• Sampling temperature ablation: Testing whether temperature = 0 (greedy) produces different patterns than temperature = 1 would characterize the role of decoding stochasticity in the findings.
• Prompt sensitivity analysis: Testing whether small prompt variations change the mode-switch fragility pattern would assess whether the phenomenon is robust or prompt-dependent.

Critical Assessment

Does the benchmark genuinely measure "document-grounded reasoning over novel scientific information"?

Partially, with a significant caveat about what "novel" means. The difficulty calibration protocol (Section 2.3) enforces that questions are parametrically unsolvable for the calibration models (DeepSeek-R1-0528, Doubao). The empirical outcome — that even the best models achieve only 57% Full-set accuracy (GPT-5.1) with substantial Instruction-only performance (59.8%) — is consistent with the intended difficulty. The Concepts-only upper bound of 75-81% confirms that the problems are solvable in principle.

However, the benchmark does not re-calibrate novelty for each evaluated model. A question that is parametrically unsolvable for DeepSeek-R1-0528 might be parametrically solvable for Gemini-3-Pro, which scores 64.2% under Instruction-only — higher than any model's Full-set score except GPT-5.1 (57.0%). This means that for Gemini-3-Pro, a substantial fraction of the benchmark may be testing parametric recall rather than novel evidence integration. The paper acknowledges this implicitly by reporting Instruction-only accuracy transparently, but does not discuss how this parametric contamination might affect the interpretation of regime gaps for high-Instruction-only models. For models with high Instruction-only accuracy, the Concepts-only → Full-set gap partially reflects a shift from parametric reasoning to evidence-grounded reasoning (which may fail), rather than purely measuring retrieval loss from a zero-knowledge baseline. This is not a fatal flaw — the Instruction-only scores allow readers to assess this for themselves — but it means the benchmark does not provide a "pure" measurement of retrieval-infused reasoning independent of parametric knowledge variation across models.

A stronger design (though more expensive) would calibrate novelty per-model: run the Instruction-only test on each evaluated model, and exclude or flag instances where the model succeeds parametrically. The paper does not do this, likely for cost reasons (evaluating 12 models × 3 calibration runs per instance would be prohibitive).

Does the mode-switch fragility finding hold up?

Yes, but the mechanism remains speculative. The paper demonstrates that several models perform worse with Full-set documents than with no documents at all (Table 2). The numbers are clear and the pattern is consistent across models: Gemini-3-Pro drops 10.5 points, Claude-Opus-4.1-thinking drops 9.3 points, GPT-5.1 drops 2.8 points. This is a genuine empirical finding.

The paper's interpretation — that the presence of documents disrupts a "reasoning mode switch" between parametric and evidence-grounded processing — is plausible but not directly tested. Alternative explanations exist that the paper does not rule out:

1. Context-length degradation: The Full-set condition presents substantially more text than Instruction-only. Models with imperfect long-context utilization might perform worse simply because their attention quality degrades with longer inputs, not because of a discrete mode switch. The paper does not control for context length independent of information content — a "long irrelevant text" condition would help rule this out.

2. Prompt format effects: The Full-set prompt includes document texts formatted differently from the Instruction-only prompt. Models sensitive to prompt formatting might show performance differences that reflect prompt engineering rather than reasoning mode disruption. The paper uses a fixed prompt template (Appendix A) but does not ablate prompt variations to test robustness.

3. Forced evidence reliance: When documents are present, the model may attempt to ground its answer in them (as instructed), but if its grounding ability is weak, it produces worse answers than if it had relied on its (partially correct) parametric knowledge. This is still a "mode switch" in the behavioral sense, but it might reflect rational meta-cognition (the model correctly identifies that it should use evidence but executes poorly) rather than a pathological disruption. The paper's framing leans toward the pathological interpretation without strongly distinguishing these possibilities.

The paper's qualitative CoT analysis (Table 3, Appendix B) provides suggestive evidence for the mode-switch interpretation — cases where the model abandons a viable parametric approach and produces a flawed evidence-grounded approach — but the analysis is based on a 50-instance sample per model, and the paper does not quantify what fraction of Full-set failures exhibit this specific pattern versus other failure modes.

Does the structural concept misuse finding hold up?

Yes, and it is the paper's most robust and important empirical contribution. The evidence is straightforward and multiple lines of analysis converge on the same conclusion:

• Concepts-only accuracy averages 75.4% — meaning ~25% of problems are failed even with oracle concepts provided (Table 2).
• Concept precision and recall under Concepts-only are substantially below 1.0 — GPT-5.1 achieves only 0.754 on both metrics (Table 3).
• The error profile shifts from MC-dominated (Instruction-only) to R-dominated relative to the reduced total (Concepts-only), confirming that reasoning failures persist when concept knowledge is satisfied (Table 3).
• Qualitative cases (Appendix B.2) show explicit examples where the model names the correct concept but fabricates an incorrect heuristic for applying it.

This finding is robust because it does not depend on the benchmark's novelty calibration, the noise model, or the truncation rule — it only requires that the Concepts list is genuinely sufficient to solve the problem, which the annotator validation and CoT auditing enforce. If the concepts are sufficient and the model fails, the failure must be in reasoning or procedural execution. There is no plausible confound that would produce the precision/recall below 1.0 other than the model's inability to operationalize provided concepts.

The paper might have strengthened this finding by analyzing which types of concepts are most frequently misused — the distinction between declarative concepts (definitions, theorem statements) and procedural concepts (algorithms, construction methods, instantiation steps) is mentioned (Section 3.3: "formula-derivation items benefit the most from explicit concepts") but not systematically quantified across the error analysis. This would provide more targeted guidance for model improvement.

Does the noise non-linearity finding hold up?

Qualitatively yes, quantitatively under-supported. The paper reports a "gradual but non-linear decline" with increasing noise documents (Figure 6a) and interprets this as evidence of trajectory drift rather than signal dilution. The figure likely shows the pattern, but the paper provides no quantitative characterization of the non-linearity — no curve fitting, no test of linear vs. non-linear models, no statistical comparison of slopes at different noise levels. The claim of non-linearity is therefore based on visual inspection of Figure 6, which is suggestive but not rigorous.

The trajectory-drift mechanism is also not directly tested. The paper could strengthen this claim by analyzing CoT traces at low-noise vs. high-noise levels: if trajectory drift is occurring, high-noise CoTs should show early-step divergence from the reference reasoning path (e.g., selecting the wrong framework or problem formulation in the first few steps), while low-noise CoT errors might show later-step failures. The existing error attribution (Table 3) is aggregated across all noise levels, so it cannot distinguish these patterns.

What is genuinely missing from the evaluation?

Statistical rigor throughout. The paper reports point estimates (accuracy percentages, error category counts) without any measure of uncertainty — no confidence intervals, no standard deviations, no significance tests for between-model differences. The two-run averaging and the 50-instance error sampling produce estimates with unknown precision. For a benchmark paper whose primary output is comparative model rankings, this is a significant omission. A 3-5 percentage point difference between two models under Full-set could be statistically indistinguishable from zero depending on the between-instance and between-run variance. The paper provides no way to assess this.

Human baseline. The paper evaluates 12 AI models but provides no human performance baseline on the same benchmark. This is important context for interpreting the difficulty level and the gap between state-of-the-art models and expert human performance on genuine scientific reasoning tasks. The PhD annotators who created the instances demonstrated the ability to solve these problems (they wrote the CoT and Answer), but we do not know whether typical PhD students in the relevant fields, given the same document sets and the same time constraints, would outperform the models. A human baseline would help calibrate whether 57% Full-set accuracy is impressive or disappointing.

Instance count transparency. The paper does not report the total number of benchmark instances. Table 2 reports percentages without denominators, Figure 5 reports distributions of concept counts and document counts but not the total N, and the data collection description mentions 81 annotators but not instances per annotator. This makes it difficult to assess the precision of the reported accuracies — a 57% score on 100 instances has much wider confidence intervals than on 1,000 instances. For a benchmark paper, the total instance count is an essential specification, and its absence is a notable gap.

Evaluator model reliability. The paper uses doubao-seed-1-6-251015 for both answer grading and error attribution, with fixed prompts provided in Appendix A. There is no reported validation of this evaluator against human judgment — no inter-annotator agreement, no calibration study. Given that the error attribution analysis (Table 3) requires nuanced judgments about whether a concept was "missing," "misused," or involved a "reasoning-process error," the evaluator's accuracy on these classifications is critical to the paper's diagnostic claims. An unvalidated evaluator is a significant source of potential measurement error that the paper does not address.

Generalizability beyond the benchmark's domain. All instances are drawn from 2023–2025 theoretical scientific papers, with annotators restricted to their PhD specialties. The findings — mode-switch fragility, structural concept misuse, noise non-linearity — are demonstrated on this specific domain of frontier theoretical science. Whether these phenomena generalize to other retrieval-infused reasoning domains (legal document analysis, medical literature review, code documentation, historical research) is an open question. The paper does not claim generalizability (Section 5 acknowledges the theoretical science focus), but readers should be cautious about extrapolating the specific failure mode frequencies to other domains.

6. Limitations and Trade-offs

6.1 No Instance Count, Variance, or Statistical Significance Reported Anywhere

The assumption or constraint. The paper reports all results — accuracy percentages in Table 2, error category counts in Table 3, distributions in Figure 5, and factor analyses in Figure 6 — as point estimates without any measure of uncertainty. The total number of benchmark instances is never disclosed: Table 2 reports percentages without denominators, Figure 5 reports distributions of concept counts and document counts without a total N, and the data collection description mentions 81 annotators but not instances per annotator. The paper does not report confidence intervals, standard deviations, standard errors, or statistical significance tests for any comparison between models or between regimes.

The consequence. The entire diagnostic framework — regime gap decomposition, cross-model ranking, error attribution — rests on numerical differences whose reliability is unknown. A 6.8 percentage point gap between GPT-5.1 (57.0%) and DeepSeek-V3.1 (50.2%) under Full-set (Table 2) might be statistically meaningful, or it might be indistinguishable from noise depending on the number of instances and the between-instance variance. The two-run averaging with temperature-1 sampling reduces decoding stochasticity but does not address the dominant source of variance: instance-level difficulty heterogeneity. If the benchmark contains, say, 200 instances, a 5-point gap corresponds to roughly 10 instances — whether those 10 instances reflect a systematic capability difference or random sampling variation is unknowable from the reported data. This fundamentally undermines the paper's ability to support claims like "model A has a retrieval bottleneck while model B has a reasoning bottleneck," because the gap magnitudes that define those diagnoses may not be statistically reliable.

What evidence exists in the paper. The absence is the evidence. Nowhere in Sections 3, 5, or the appendices does the paper provide instance counts, standard deviations, standard errors, confidence intervals, or any form of statistical test. The two-run averaging (Section 3.2) is described as producing an "average score" but no between-run variance is reported. The error attribution analysis (Table 3) samples 50 incorrect instances per model per regime, but no sampling error is reported for the category frequencies — a count of "MC:42" could be 35-49 in a different sample. The paper's most important quantitative claims (mode-switch fragility magnitudes, retrieval loss values, concept precision/recall) are all reported as scalar point estimates with no uncertainty quantification.

Mitigation status. Not addressed. The paper does not acknowledge this as a limitation. Section 3.2 describes the evaluation protocol without mentioning statistical rigor or variance estimation as a consideration. This is a significant omission for a benchmark paper, where the primary contribution is a measurement instrument — the instrument's precision is an essential specification that DeR² does not provide.

---

6.2 Difficulty Estimation Cost Is Not Accounted For — The Benchmark Cannot Detect Parametric Contamination Per-Model

The assumption or constraint. The two-phase difficulty calibration protocol (Section 2.3) enforces parametric unsolvability using specific calibration models (DeepSeek-R1-0528 and the Doubao web client with networking disabled). A question that is parametrically unsolvable for the calibration model may be parametrically solvable for an evaluated model with broader or more recent training data. The paper does not re-calibrate novelty for each evaluated model, nor does it exclude or flag instances where an evaluated model succeeds under Instruction-only. The Instruction-only scores in Table 2 are reported transparently, but the regime-gap interpretation treats all instances uniformly regardless of per-model Instruction-only success.

The consequence. For models with high Instruction-only accuracy, the diagnostic regime gaps lose their intended interpretation. Gemini-3-Pro scores 64.2% under Instruction-only (Table 2) — higher than any model's Full-set score except GPT-5.1 (57.0%). For the instances where Gemini succeeds parametrically, the Concepts-only → Full-set gap does not measure "retrieval loss from a zero-knowledge baseline" — it measures a more complex phenomenon where the model shifts from a working parametric strategy to a (possibly failing) evidence-grounded strategy. The decomposition into "knowledge loss" vs. "retrieval loss" vs. "noise-induced loss" assumes that Instruction-only represents a no-knowledge floor, but for high-Instruction-only models, that floor is substantially above zero, and the gaps partially reflect mode-switch dynamics rather than pure retrieval difficulty. This means cross-model comparisons of retrieval loss are confounded: a model with a small Concepts-only → Full-set gap might be genuinely good at retrieval, or it might simply have low parametric knowledge (so Instruction-only is low, Concepts-only is moderate, and the gap is compressed because the model struggles with concept-level reasoning anyway). These are observationally equivalent under the reported metrics.

What evidence exists in the paper. Table 2 itself provides the evidence. Gemini-3-Pro's Instruction-only (64.2%) exceeds its own Full-set (53.7%) and several other models' Concepts-only scores. GPT-5.1's Instruction-only (59.8%) similarly exceeds its Full-set (57.0%). These numbers directly demonstrate that parametric knowledge is substantial and variable across models, violating the assumption that Instruction-only represents a common, near-zero baseline. The paper does not stratify its regime-gap analysis by whether the evaluated model answered correctly under Instruction-only, which would be the natural way to isolate retrieval loss on genuinely novel instances.

Mitigation status. Partially addressed through transparency but not through methodology. The paper reports Instruction-only scores for all models (Table 2), allowing readers to assess parametric contamination qualitatively. However, the paper does not use these scores to adjust its diagnostic framework — the retrieval loss metric and the regime-gap decomposition are computed identically for all models regardless of their Instruction-only performance. Section 3.3 notes the mode-switch fragility phenomenon as a finding rather than as a confound that complicates the gap interpretation. The paper acknowledges in Section 3.1 that "the Instruction-only → Concepts gap operationalizes knowledge loss" but does not discuss how high Instruction-only accuracy undermines this operationalization.

---

6.3 Single-Domain Evaluation: All Instances Are Theoretical Scientific Papers from a Narrow Time Window

The assumption or constraint. Every DeR² instance is drawn from 2023–2025 theoretical scientific papers, with annotators restricted to their PhD specialties in "theoretical foundational disciplines, theoretical engineering disciplines, and theory-driven interdisciplinary fields" (Section 2.3). Applied papers involving experiments, social surveys, or numerical simulations are explicitly excluded. The benchmark therefore measures retrieval-infused reasoning over a very specific type of content: abstract, formal, concept-dense theoretical material in frontier academic science. The questions require symbolic manipulation, theorem application, and formal derivation — the answer types are products, scalars, formulas, or concise conclusions (Figure 4).

The consequence. None of the paper's headline findings — mode-switch fragility, structural concept misuse, noise non-linearity, the specific magnitudes of retrieval loss (20-32 percentage points), or the cross-model rankings — can be assumed to generalize to other retrieval-infused reasoning domains without independent validation. A practitioner deploying RAG for legal document review, medical literature synthesis, code documentation search, or historical research has no evidence from this paper about whether their model will exhibit similar failure modes, or whether the diagnostic regime-gap approach would reveal the same bottlenecks. The theoretical science domain has distinctive properties that may not transfer: concepts are highly formalized (theorems have precise statements, algorithms have explicit steps), answers are objectively verifiable, documents are structured (abstract, introduction, methods, results), and the reasoning is deductive rather than probabilistic or interpretive. In domains where evidence is ambiguous, concepts are contested, answers are subjective, or documents are unstructured, the failure mode taxonomy and the regime-gap decomposition may apply differently or not at all.

What evidence exists in the paper. The paper provides no cross-domain evaluation — no experiments on legal, medical, historical, or code-related retrieval-infused reasoning tasks. Section 4 mentions existing benchmarks (KILT, FreshQA, GPQA, MATH) but only to argue that they lack DeR²'s diagnostic properties, not to validate DeR²'s findings across domains. Figure 4 shows disciplinary diversity within theoretical science (math, physics, computer science, etc.) but all within the same fundamental category of formal, derivation-based reasoning.

Mitigation status. The paper does not claim generalizability — the scope is explicitly "retrieval-infused reasoning in scientific problem solving" (Section 2.1) and "frontier scientific literature" (abstract). The limitation is one of scope, not of misrepresentation. However, the paper's framing in Sections 1 and 5 uses broad language about "deep research," "retrieval-infused reasoning," and "document-grounded synthesis" without always qualifying that the evidence comes exclusively from theoretical science. This could mislead readers into overgeneralizing the findings to domains where the failure mode profile is likely different.

---

6.4 Unvalidated Auto-Evaluator for Both Answer Grading and Error Attribution

The assumption or constraint. The paper relies on a single model — doubao-seed-1-6-251015 — for two critical measurement functions: (1) grading whether a model's predicted answer matches the ground-truth Answer (producing the accuracy scores in Table 2), and (2) diagnosing failure modes by classifying reasoning traces into missing concepts, concept misuse, reasoning errors, and numeric errors (producing the error distributions in Table 3). The answer grading uses task-specific normalization rules (numeric tolerance, symbolic equivalence, checklist-based matching), and the error attribution uses a structured prompt that provides the predicted concepts, predicted CoT, ground-truth concepts, and reference CoT (Section 3.3, Appendix A). The paper provides no validation of this evaluator against human expert judgment — no inter-annotator agreement study, no calibration on a subset of instances, no comparison against alternative evaluator models.

The consequence. Both the primary accuracy metric and the diagnostic error analysis are subject to unknown and potentially systematic measurement error from the evaluator model. For answer grading: if the evaluator model incorrectly judges a correct answer as incorrect (or vice versa), the accuracy scores in Table 2 are biased by an unknown amount. For checklist-based grading of long answers, the evaluator must parse the model's output, apply the checklist criteria, and make binary judgments — a task that may itself require domain expertise and reasoning capability that the evaluator model may lack. For error attribution: classifying whether a model's reasoning trace "misses a core concept" or "misuses a concept" requires understanding the scientific content, comparing the predicted reasoning to the reference CoT, and making nuanced judgments about conceptual equivalence and correctness. If the evaluator model has its own domain knowledge gaps or reasoning biases, the error type distributions in Table 3 may reflect evaluator errors rather than genuine failure mode frequencies.

What evidence exists in the paper. The paper provides no evidence about evaluator reliability. Appendix A contains the evaluation prompts but no calibration results. The paper does not report having human experts audit a sample of the evaluator's judgments for either answer grading or error attribution. There is no discussion of potential evaluator bias, no comparison against alternative evaluator models, and no sensitivity analysis showing whether findings change if a different evaluator model is used.

Mitigation status. Not addressed. The paper treats the doubao-seed-1-6-251015 evaluator as a transparent measurement instrument rather than as a source of potential measurement error requiring validation. This is methodologically concerning because the evaluator is itself an LLM being asked to perform tasks (scientific answer verification, reasoning error diagnosis) that are closely related to the capabilities the benchmark is designed to measure — if current models have systematic weaknesses in scientific reasoning (as the benchmark's own results demonstrate), those weaknesses may also affect the evaluator's judgments.

---

6.5 The Benchmark Provides No Human Performance Baseline

The assumption or constraint. The paper evaluates 12 state-of-the-art AI models but provides no measurement of human expert performance on the same tasks under the same conditions. The PhD annotators who created the instances demonstrated the ability to solve these problems (they authored the Concepts, CoT, and Answer), but this occurred under different conditions — they had access to the full source paper, unlimited time, and the task of constructing the solution rather than producing it given only the document set. The paper does not report what accuracy human experts (e.g., PhD students in the relevant fields who did not participate in annotation) achieve when given the same Instruction and Document Set under time-constrained conditions comparable to model evaluation.

The consequence. The paper's central findings — "substantial headroom" (Section 5), "consistent non-trivial failure modes" (Section 5) — are framed relative to an implicit ceiling of 100% accuracy. GPT-5.1's 57.0% Full-set accuracy is described as leaving "significant headroom" (abstract), but without a human baseline, we cannot determine whether this headroom is a sign of model deficiency or a reflection of genuine task difficulty that would challenge human experts equally. If expert humans achieve, say, 65% on the same Full-set task, then GPT-5.1's 57% indicates modest rather than substantial headroom, and the benchmark is primarily measuring inherent task difficulty rather than model-specific retrieval-infused reasoning gaps. Conversely, if humans achieve 90%, then the 57% figure genuinely reflects model shortcomings. The interpretation of every quantitative claim in the paper depends on this calibration, and it is absent.

This also affects the interpretation of the Concepts-only upper bound (75.4% average). Is this bound low because models are poor at multi-concept coordination (as the paper claims), or because even with oracle concepts, some of these frontier scientific problems are intrinsically difficult and would challenge human experts? If humans also achieve ~75% with oracle concepts, the "structural concept misuse" finding would require reinterpretation — the failures would reflect task difficulty shared across humans and models rather than a model-specific procedural execution weakness.

What evidence exists in the paper. None. The paper does not report any human evaluation on DeR², does not discuss the absence of a human baseline as a limitation, and does not provide any estimate of expected human performance. The annotation process (Section 2.3) demonstrates that PhD students can construct correct solutions given the full source paper, but this is a different task (solution construction with unlimited access to the original paper and unlimited time) that does not calibrate the difficulty of the benchmark's actual evaluation conditions (solving from a curated document set, presumably under practical time constraints).

Mitigation status. Not addressed. The paper treats model performance as inherently interpretable without human calibration. This is a common practice in benchmark papers but is particularly consequential here because the benchmark targets frontier scientific material that is genuinely difficult — the headline accuracy numbers might reflect task difficulty more than model deficiency, and the paper provides no way to distinguish these explanations.

---

6.6 Serial Evaluation Cost and Practical Deployability of the Four-Regime Diagnostic Framework

The assumption or constraint. The paper presents the four-regime evaluation design as a diagnostic methodology that practitioners can use for "principled model selection and diagnosis" (Section 5). However, the paper does not discuss the practical cost of running this evaluation. Every model must be evaluated under four separate conditions (Instruction-only, Concepts-only, Related-only, Full-set), each with two runs for averaging (Section 3.2). For Full-set and Related-only, the model must process document libraries averaging 6.5 documents per instance, with a 30,000-character budget applied when context limits are exceeded. The paper does not estimate the total inference cost, API calls, or wall-clock time required to evaluate a single model on the benchmark, nor does it provide guidance on whether the full four-regime evaluation is necessary for practical model selection or whether a subset of regimes (e.g., Concepts-only + Full-set) provides sufficient diagnostic signal.

The consequence. The benchmark's practical utility for model selection and debugging — which the paper explicitly claims as a contribution ("we release DeR² to facilitate principled model selection and diagnosis") — depends on whether the diagnostic value justifies the evaluation cost. If running all four regimes on all instances costs hundreds or thousands of API calls per model, practitioners may be unable or unwilling to use the benchmark for routine model comparison, limiting its adoption. Furthermore, the cost scales with the number of evaluated models: comparing five candidate models requires roughly 5 × 4 × 2 = 40 runs per instance, plus the auto-evaluator calls for answer grading and (optionally) error attribution. The paper provides no cost analysis and no recommendation for a minimal diagnostic subset that preserves most of the regime-gap information at reduced cost.

The frozen document library design (Section 2.3) eliminates web retrieval costs, but the inference cost of processing multi-document contexts for every instance under multiple regimes may still be substantial. For models with API-based access (GPT-5.1, Gemini-3-Pro, Claude-Opus-4.1), this translates directly to financial cost. For open-weight models, it translates to GPU-hours. The paper evaluates 12 models (Table 2), presumably incurring substantial total cost, but does not report this cost or discuss its implications for benchmark accessibility.

What evidence exists in the paper. The paper provides no cost analysis. The total number of instances is not reported (see Limitation 6.1), making it impossible for a reader to estimate the evaluation budget. The paper reports two-run averaging (Section 3.2) and mentions that the error attribution samples 50 incorrect instances per model per regime (Section 3.3), but these are procedural descriptions, not cost estimates. The practical overhead of running the full diagnostic framework — in API calls, tokens processed, or wall-clock time — is entirely unaddressed.

Mitigation status. Not addressed. The paper frames DeR² as a practical tool for model selection without discussing the cost of its use. This is a notable gap because the benchmark's diagnostic value proposition (regime-gap decomposition) requires running multiple regimes — if a practitioner only ran Full-set, they would get a single uninterpretable accuracy number (the very problem DeR² is designed to solve), but running all four regimes multiplies the evaluation cost. The paper does not explore whether two regimes (e.g., Concepts-only + Full-set) capture most of the diagnostic signal, or whether a lightweight difficulty pre-screen could route instances to different regimes adaptively to reduce total cost.

7. Implications and Future Directions

How This Work Changes the Landscape

DeR² does not introduce a new model, a new training paradigm, or a new retrieval algorithm. It introduces something arguably more consequential for the near-term health of the field: a diagnostic language for evaluating retrieval-infused reasoning that replaces binary "correct/incorrect" scoring with interpretable performance gaps. This is not a paradigm shift in the sense that transformers are being replaced by a new architecture. It is, however, a methodological reframing that changes what counts as a valid evaluation of document-grounded reasoning — and by doing so, it changes what research questions the field can ask and answer.

The magnitude of this shift is easiest to see by contrasting what we knew before DeR² with what we know after it. Before: if a model scored 65% on an end-to-end RAG benchmark, we knew it failed 35% of the time. We could speculate about whether the failures were retrieval-driven or reasoning-driven, but we had no systematic way to determine which, and therefore no systematic way to direct improvement efforts. After: the same model's performance can be decomposed into Concepts-only → Related-only loss (document-to-concept extraction failure) and Related-only → Full-set loss (noise-induced selection failure), with each gap quantified in percentage points and further characterized by concept-level precision/recall and error type distributions. The difference is not incremental — it transforms evaluation from a single coarse measurement into a structured diagnosis. This is analogous to what happened in medical testing when single biomarker assays were replaced by metabolic panels that measure multiple indicators simultaneously. The individual measurements existed before; what changed was the ability to see them in relation to each other and to attribute symptoms to specific underlying causes.

The paper resolves a tension that has been simmering in the RAG and agent literature without being clearly articulated. On one hand, the field has invested enormous effort into improving retrieval quality — better embeddings, hybrid search, reranking models, agentic retrieval strategies — under the implicit assumption that retrieval is the bottleneck and that better retrieval translates to better end-to-end performance. On the other hand, a parallel literature on "LLMs cannot self-correct reasoning" (Huang et al., 2023) and on reward hacking in RLHF has demonstrated that models struggle with multi-step reasoning even when given correct information. DeR² provides the evidence that reconciles these perspectives: retrieval is a major bottleneck (the Concepts-only → Full-set gap of 20–32 percentage points is substantial), but it is not the only bottleneck — even with oracle concepts provided directly, models fail roughly 25% of the time (Concepts-only average: 75.4%), and these failures are predominantly reasoning and coordination failures rather than concept omissions (Table 3). The implication is that improving retrieval alone, without also improving concept operationalization and multi-step coordination, will hit a performance ceiling of approximately 75% on this class of problems — and that ceiling is set by reasoning capability, not retrieval quality. This reframes the RAG research agenda: retrieval improvements are necessary but insufficient; reasoning improvements are equally necessary.

More subtly, the paper's discovery of mode-switch fragility changes how the field should think about the very act of providing documents to a model. The dominant assumption in RAG — implicit in every system that prepends retrieved passages to a prompt — is that adding relevant context is at worst neutral: it might not help if the model already knows the answer, but it should not actively hurt. DeR² demonstrates that this assumption is false for several state-of-the-art models (Gemini-3-Pro drops 10.5 points from Instruction-only to Full-set; Claude-Opus-4.1 drops 9.3 points). This is not a marginal effect — it is a substantial performance inversion that means providing documents can make the model worse than it was without them. This finding, if it replicates in other domains, has direct implications for every RAG deployment in production today: those systems may be degrading performance on some fraction of queries without anyone knowing, because the Instruction-only counterfactual is never measured in standard deployment monitoring. The paper thus creates a new imperative: any serious evaluation of a retrieval-augmented system must include an Instruction-only baseline to detect mode-switch degradation, not just a retrieval-absent baseline to measure retrieval benefit.

The paper also redirects research attention in a specific way. Prior to DeR², the research frontier on deep search and agentic RAG was focused on better tool use, better search strategies, and better context management — the assumption being that if you can get the right documents into the context window, the model will reason over them correctly. DeR²'s regime-gap decomposition demonstrates that document-to-concept extraction (the Concepts-only → Related-only gap) and noise-induced disruption (the Related-only → Full-set gap) are both large and independent bottlenecks that cannot be solved by better retrieval alone. This makes research on concept operationalization — teaching models to execute theorems as procedures rather than merely recognizing their definitions, to instantiate abstractions with correct variable bindings, and to maintain constraint dependencies across multi-step derivations — newly attractive as a research direction with a clear evaluation target (the Concepts-only accuracy and precision/recall metrics). Conversely, research on search algorithm sophistication (e.g., more complex tree search or iterative retrieval strategies) becomes less attractive as a primary focus, because the paper shows that even the simplest possible "retrieval" (oracle concept provision) leaves substantial headroom — the bottleneck is not in the cleverness of the search but in the model's ability to operationalize what it finds.

Follow-Up Research This Work Enables

Per-instance parametric contamination filtering to purify the regime-gap interpretation. The paper acknowledges that its difficulty calibration is performed on specific offline models (DeepSeek-R1-0528, Doubao), not on each evaluated model, which means an instance that is parametrically unsolvable for the calibration model may be parametrically solvable for a model with broader training data (as demonstrated by Gemini-3-Pro's 64.2% Instruction-only accuracy in Table 2). A direct follow-up would re-evaluate each model on each instance under Instruction-only first, then exclude or separately analyze instances where the model succeeds parametrically. For each model, the regime-gap decomposition would be computed only on instances where that specific model failed under Instruction-only, producing a purified retrieval loss metric that genuinely measures retrieval-infused reasoning from a zero-knowledge baseline. The prediction: for high-Instruction-only models like Gemini-3-Pro, the purified Concepts-only → Full-set gap would be larger than the raw gap reported in Table 2 (because instances where the model already knows the answer parametrically artificially compress the gap — the model's Full-set score on these instances is high regardless of retrieval quality). This analysis would also quantify what fraction of each model's apparent "retrieval-infused reasoning" performance is actually parametric recall in disguise — a number that would be highly informative for model selection in deployment contexts where genuinely novel material is expected.

Validating the auto-evaluator against human expert judgment for both grading and error attribution. The paper relies on doubao-seed-1-6-251015 for answer grading and error type classification without any reported validation against human judgment. A critical follow-up study would sample 100–200 instances, have 2–3 domain-expert PhD students (ideally drawn from the same annotator pool) independently grade the answers and classify errors using the same taxonomy, and compute inter-annotator agreement (Cohen's kappa) between the auto-evaluator and each human, as well as human-human agreement as a baseline. The key measurements: (1) for answer grading, what is the evaluator's false-positive rate (incorrectly accepting wrong answers) and false-negative rate (incorrectly rejecting correct answers)? If these rates are non-trivial (e.g., >5%), the accuracy scores in Table 2 would need correction. (2) For error attribution, can the evaluator reliably distinguish "missing core concept" from "concept misuse" from "reasoning-process error"? The paper's taxonomy requires nuanced judgments — a model might paraphrase a concept imprecisely (is that "misuse" or "missing"?) or might apply a correct concept in a flawed reasoning chain (is that "reasoning error" or "concept misuse"?). Without calibration, the error distribution numbers in Table 3 are suggestive but not trustworthy. If the evaluator proves unreliable at fine-grained classification, the error taxonomy might need simplification (e.g., binary "concept error vs. reasoning error" rather than four categories) or the evaluator might need replacement with a fine-tuned specialist model or a human-in-the-loop protocol.

Scaling the revision or tree-search approaches from the prior paper onto DeR² instances to test whether test-time compute amplifies concept operationalization. The prior paper (on compute-optimal test-time scaling, covered in Sections 1–6 of this analysis) demonstrated that sequential revisions and PRM-guided search can improve performance by amplifying existing capability at inference time. A natural integration would apply those techniques to DeR²'s Concepts-only regime specifically: if models fail ~25% of the time even with oracle concepts due to procedural execution failures, can iterative revision (the model re-attempts the derivation with its previous failure as context) or beam search over reasoning paths (using a process reward model trained on DeR²'s annotated CoTs) close the Concepts-only gap? The prediction: revisions should help on instances where the model makes local reasoning errors (R-type failures in Table 3), since these involve fixable mistakes in an otherwise correct framework, but should not help on concept misuse failures (UC-type), where the model fundamentally misunderstands a concept and revisions may amplify the misunderstanding. Running this experiment would test the prior paper's finding that revisions help on "easy" problems (where the model's initial attempt is roughly correct) — DeR²'s Concepts-only instances that fail due to reasoning errors would be the "easy" subset within the already-challenging benchmark. A positive result (substantial accuracy gain from revisions on the Concepts-only regime) would suggest that test-time compute can partially compensate for the concept operationalization bottleneck.

Cross-domain replication of the mode-switch fragility phenomenon. The paper demonstrates mode-switch fragility on theoretical scientific reasoning, but the mechanism remains speculative — is it caused by context-length degradation, prompt format sensitivity, a genuine mode-switch disruption, or something else? A follow-up study would replicate the four-regime design in a substantially different domain — for example, legal document analysis (reading case law to answer questions about legal precedent), medical literature review (synthesizing clinical trial results to answer treatment efficacy questions), or code documentation (reading library documentation to answer API usage questions). The key measurement: does the Instruction-only ≥ Full-set inversion appear in these domains, and with the same models? If mode-switch fragility is domain-specific (appearing only in theoretical science, where documents use highly formal abstract language), the mechanism likely involves difficulty mapping between document-exposition and procedural-execution formats. If it is domain-general, the mechanism is more likely a fundamental architectural limitation in how transformers switch between parametric and context-grounded computation. The study should also include an ablation where the Full-set documents are replaced with an equal-length irrelevant text (e.g., a random Wikipedia article) to test whether the performance drop is simply a context-length effect rather than a content-specific mode-switch disruption — if long irrelevant text causes the same drop, the phenomenon is about attention degradation, not reasoning mode switching. If only topically adjacent distractors cause the drop, the paper's trajectory-drift interpretation is supported.

Training a lightweight difficulty classifier from the PRM's score distribution to make DeR²-style diagnostic evaluation cheap enough for routine deployment. One of the prior paper's key contributions was showing that a process reward model's average score distribution can estimate question difficulty without ground-truth labels, enabling compute-optimal strategy selection. DeR²'s diagnostic framework faces an analogous cost problem: running all four regimes is expensive. A follow-up would train a classifier (perhaps a lightweight fine-tuned model, or even a linear probe on the base model's embeddings) that takes only the Instruction text as input and predicts (a) whether the model will succeed under Instruction-only (parametric contamination flag), (b) whether Concepts-only will succeed (concept coordination difficulty estimate), and (c) the expected document-to-concept loss (retrieval difficulty estimate). Training data: run a few models on a subset of DeR² instances to get ground-truth regime outcomes, then train the classifier to predict those outcomes from the Instruction alone. If successful, this would enable a two-stage evaluation: the classifier estimates instance difficulty profiles cheaply, and only a subset of instances (e.g., those where the classifier is uncertain or where specific regime gaps need measurement) are run through the full four-regime pipeline. This would make DeR²-style diagnostics practical for routine model comparison without multiplying evaluation cost by 4×.

Quantifying concept operationalization difficulty as a function of concept type. The paper's structural concept misuse finding — that models name correct concepts but fail to execute them procedurally — is the most robust result, but it is currently an aggregate observation. A fine-grained follow-up would categorize DeR²'s Concepts by type: declarative concepts (definitions, theorem statements, existence claims — "X is defined as Y"), procedural concepts (algorithms, construction methods, step-by-step instantiations — "to compute X, apply Y to Z"), and constraint concepts (conditions that must hold, invariants that must be maintained — "under the assumption that A, B must satisfy C"). The study would then measure per-category precision, recall, and accuracy under Concepts-only and Related-only. The prediction: procedural concepts should show the largest drop from Concepts-only to Related-only, because their operational details (loop structures, variable bindings, edge cases) are harder to extract from natural-language document exposition than declarative concept statements. Constraint concepts should show high failure rates even under Concepts-only because they require tracking dependencies across multiple reasoning steps. If these predictions hold, the results would guide targeted training interventions: procedural concept failure suggests training on code-execution-like tasks where natural language descriptions of algorithms are translated to executable implementations; constraint failure suggests training on tasks that explicitly require multi-step dependency tracking (e.g., formal verification, constraint satisfaction problems).

Practical Applications and Downstream Use Cases

Gated deployment with automatic mode-switch detection for production RAG systems. Any organization running a retrieval-augmented system in production — customer support with knowledge bases, legal document review, scientific literature search — should, based on this paper's findings, implement an Instruction-only baseline check as part of their monitoring pipeline. The implementation is straightforward: for a sample of production queries, run the model both with and without retrieved documents and compare accuracy. If the with-documents accuracy is lower than the without-documents accuracy for any query subset (as it is for Gemini-3-Pro, which drops from 64.2% to 53.7% on DeR²), that subset should be flagged and either routed to a different model, handled with a different prompting strategy (e.g., explicit instruction to prioritize parametric knowledge over documents), or escalated to human review. The detection threshold can be tuned based on the cost of errors: in high-stakes domains (medical, legal), even a 2-3% mode-switch degradation may justify gating; in lower-stakes domains, a 10% threshold might be acceptable. The key operational insight from DeR² is that mode-switch fragility is invisible in standard RAG monitoring (which only measures with-documents accuracy) — it requires measuring the counterfactual, which most production systems currently do not do.

Targeted fine-tuning for concept operationalization using DeR²'s Concepts-only failures as training data. The paper demonstrates that ~25% of failures under Concepts-only are not due to missing concepts but to inability to execute those concepts procedurally. This failure subset provides a natural curriculum for fine-tuning: collect the Concepts-only failure instances (where the model had all necessary concepts but produced an incorrect answer), pair them with the reference CoT as the target, and fine-tune the model to learn the correct procedural execution. This is a more targeted intervention than generic instruction tuning or retrieval-augmented training — it specifically targets the gap between "I know what this theorem says" and "I know how to apply it to this specific instance." If successful (as measured by improved Concepts-only accuracy on a held-out set), the fine-tuned model would show a reduced Concepts-only → Full-set gap even without retrieval improvements, because it would lose fewer points to procedural failures once it extracts the concepts from documents. The training data is free (generated by the benchmark evaluation itself — any model evaluation on DeR² produces an automatically labeled set of Concepts-only failures) and the intervention is surgical (only the specific capability that DeR² identifies as deficient is being trained).

Benchmark-based model selection with diagnostic profiles rather than aggregate scores. An organization choosing between, say, GPT-5.1 and Gemini-3-Pro for a scientific literature synthesis task currently has to rely on aggregate benchmark scores (which conflate retrieval, reasoning, and parametric knowledge) or on expensive domain-specific human evaluations. DeR² provides a richer basis for decision: the choice depends on the deployment's specific requirements. If the deployment involves genuinely novel papers (published after the model's training cutoff), Gemini-3-Pro's high Instruction-only accuracy (64.2%) is a liability — it may produce correct answers from parametric knowledge that happens to be wrong for the specific paper, while missing the novel evidence. GPT-5.1, with lower Instruction-only (59.8%) but higher Full-set (57.0%) and better concept precision/recall under documents (Table 3), may be more reliable for truly novel material because it depends less on parametric recall. If the deployment involves material within the model's training distribution (e.g., well-known textbook problems), Gemini-3-Pro's parametric strength becomes an asset. This type of regime-aware model selection — where the choice depends on the structure of the performance profile, not just the aggregate number — is precisely the diagnostic value Proposition that DeR² enables and that existing benchmarks cannot support. The decision rule is: for novelty-heavy deployments, prefer models with low Instruction-only accuracy (they won't be fooled by parametric knowledge) and high Concepts-only accuracy (they can reason well with provided information); for familiarity-heavy deployments, Instruction-only accuracy becomes a virtue rather than a confound.