Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs

ArXiv: 2511.05933

🎯 Pitch

RL-enhanced reasoning models don’t just think better—they navigate their own memorized hierarchies with surprising precision, beating supervised models by 24 points on medical code recall. Structured prompting closes most of this gap, revealing that the knowledge was always there, but SFT models lacked the procedural skill to find it. Internal activations confirm that RL reshapes how queries are processed rather than rewriting the stored facts themselves.

---

1. Executive Summary

This paper studies how reinforcement learning (RL) affects structured knowledge recall in LLMs, evaluating DeepSeek-V3/R1 and Qwen2.5 families on MedConceptsQA and IPC patent classification benchmarks. The core finding is that RL-enhanced models outperform instruction-tuned counterparts by 24 percentage points on hierarchical knowledge tasks not through acquiring new facts, but through improved hierarchical navigation—the procedural skill of systematically traversing knowledge structures (e.g., decomposing ICD-9-CM code 57.95 by first identifying its chapter, then its subcategory, then eliminating options). Structured prompting recovers most of this gap, reducing it from 24pp to 7pp, while layer-wise activation analysis reveals that question representations diverge substantially between models (cosine similarity dropping to 0.65–0.73) even as factual answer representations remain nearly identical (0.85–0.92), establishing that RL transforms query processing while preserving knowledge content—a separation that holds on memory-light tasks but reveals a persistent path-matching advantage for RL models (9pp gap) on deep-retrieval tasks requiring five or more hierarchical traversals.

2. Context and Motivation

The Core Problem: Does RL Really Sacrifice Knowledge for Reasoning?

The fundamental question this paper tackles is deceptively simple: when we apply reinforcement learning to improve an LLM's reasoning abilities, does the model actually lose access to its stored factual knowledge? This question matters because it gets at a tension that has shaped how the field thinks about the relationship between different kinds of model capabilities.

The paper frames this around a widely-held belief in the community: that there exists an "alignment tax" (Lin et al., 2024)—a trade-off where optimizing a model for reasoning, instruction-following, or helpfulness degrades its ability to recall memorized facts. This belief has substantial empirical backing. Studies have shown that RLHF-trained models perform worse on factual benchmarks than their base counterparts (Yuan et al., 2024; Gekhman et al., 2024), that models trained on new knowledge frequently hallucinate rather than accurately store information (Ovadia et al., 2024), and that instruction tuning primarily adjusts stylistic features rather than injecting new knowledge (Ghosh et al., 2024). The emerging picture has been one of a zero-sum game: reasoning improvements come at the expense of memorization abilities.

The authors challenge this narrative directly. Their counter-observation is striking: on tasks requiring recall from structured, hierarchical knowledge bases (specifically medical code lookups and patent classification), RL-enhanced reasoning models outperform their base and instruction-tuned counterparts by substantial margins—24 percentage points in the case of DeepSeek-R1 versus DeepSeek-V3 on MedConceptsQA. This is not a marginal difference that could be dismissed as noise or calibration; it is a large, systematic gap that runs counter to the alignment tax hypothesis. If RL uniformly degrades memorization, why would reasoning models be better at recalling that ICD-9-CM code 57.95 refers to "Replacement of indwelling urinary catheter" rather than "Other procedures for relief of elevated intraocular pressure"?

Why This Matters: Separating Knowledge from Access

The finding has both theoretical and practical significance that extends well beyond this specific benchmark.

Theoretical significance. If the alignment tax narrative is incomplete—if RL doesn't actually degrade knowledge but rather transforms how knowledge is accessed—then our understanding of what post-training does to model internals needs fundamental revision. The distinction between having knowledge and being able to retrieve it is subtle but crucial. A model might encode the fact that code 57.95 relates to urinary procedures somewhere in its parameters, yet fail to surface that fact when queried directly because it lacks the procedural scaffolding to navigate the hierarchical structure of the ICD-9-CM taxonomy. If RL primarily improves this navigational ability, then the apparent "forgetting" documented in prior work might actually reflect a retrieval failure rather than knowledge erasure. This would unify the seemingly contradictory findings: base models might have better direct recall (the kind tested by standard factual benchmarks asking "What is the capital of France?"), while RL-enhanced models have better structured recall (the kind needed to answer "Given these hierarchical constraints, which entity matches?").

Practical significance. If the paper's hypothesis is correct—that RL enhances navigation of existing knowledge rather than adding new facts—this has direct implications for how we build and deploy LLMs. It suggests that we might separate the concerns of knowledge acquisition (handled during pretraining on vast corpora) from knowledge organization (handled during post-training through RL). This separation could lead to more efficient training paradigms: rather than trying to cram more facts into models through larger pretraining runs or knowledge-intensive fine-tuning, we might invest in better post-training methods that teach models how to systematically explore the knowledge they already possess. The paper's structured prompting experiments—which recover 68% of the RL performance gain without any parameter updates—provide concrete evidence that this separation is achievable in practice.

There is also a direct connection to deployment strategy. If structured prompting can approximate RL-level performance on knowledge-intensive tasks, organizations might choose to invest in prompt engineering rather than expensive RL training pipelines for certain applications. Conversely, if the paper's finding that RL provides unique benefits on deep-retrieval tasks (the 9pp path-matching gap on tasks requiring 5+ hierarchical traversals) generalizes, then RL remains essential for complex knowledge navigation even when simple prompting suffices for shallow recall.

Why Prior Approaches Are Insufficient

The paper identifies three specific limitations in how the field has studied the relationship between RL and knowledge.

Narrow focus on flat factual recall. Existing work on factual degradation (the alignment tax literature) has overwhelmingly focused on direct, single-hop factual questions: "What is the capital of France?", "Who wrote Pride and Prejudice?", and similar probes that require the model to retrieve a single fact without navigating any hierarchical structure. The paper argues—and demonstrates empirically—that this focus misses an entire category of knowledge retrieval. Medical coding, patent classification, taxonomic reasoning, and many real-world knowledge tasks require structured traversal: you cannot simply recall that code 57.95 maps to a urinary procedure by matching on surface features; you need to understand that Volume 3, Chapter 11 covers procedures on the urinary system, that codes 57.0–57.99 are bladder procedures, and that within that range, 57.95 is specifically about catheter replacement. Prior benchmarks were ill-suited to detect whether RL improves or degrades this kind of retrieval because they simply didn't test it.

The paper cites Gekhman et al. (2024) and Yuan et al. (2024) as examples of work documenting factual degradation, but notes that these studies use datasets like MMLU, TriviaQA, and Natural Questions—all of which primarily test flat factual recall. The gap this paper fills is the systematic study of RL's effect on hierarchical knowledge retrieval, which has received almost no attention despite its practical relevance in domains like medicine, law, engineering, and patent law.

Conflating knowledge acquisition with knowledge access. Much prior work treats "the model knows X" as a binary property: either the model can produce X when queried, or it doesn't know X. The paper argues this conflation is misleading. Drawing on distinctions from Ovadia et al. (2024)—who showed that fine-tuned models often hallucinate by recombining existing knowledge rather than storing new facts—and Ruis et al. (2024)—who found that models rely on procedural knowledge extracted from reasoning-similar documents rather than memorizing new facts—the paper positions itself as investigating access mechanisms specifically.

This distinction has methodological implications. If a model fails to produce the correct answer, you cannot conclude it lacks the knowledge. It might have the knowledge encoded in its parameters but lack the procedural skill to navigate to it. The paper's central methodological innovation is designing experiments that disentangle these two explanations. If structured prompting can surface correct answers that direct prompting cannot, then the knowledge exists but the access mechanism is deficient. If layer-wise analysis shows that answer representations (how the model encodes factual content) remain stable between base and RL models while question representations (how the model processes queries) diverge, then RL is primarily transforming access, not content. These are not inferences the prior literature could make because it lacked the experimental apparatus to separate storage from retrieval.

No mechanistic account of how RL changes knowledge access. Even work that acknowledges RL might not uniformly destroy knowledge (e.g., Kirk et al., 2023; Li et al., 2025) has not provided a mechanistic explanation for how RL transforms retrieval. The paper positions its layer-wise activation analysis as filling this gap. By showing that declarative statements ("Code 57.95 refers to urinary catheter replacement") maintain high cosine similarity (0.85–0.92) between base and RL models across most layers, while interrogative queries ("What is code 57.95?") diverge substantially (dropping to 0.65–0.73), the paper provides direct evidence that RL modifies query processing circuitry while leaving factual representations largely intact. This is a concrete mechanistic claim—not just a behavioral observation—that prior work had not advanced.

Conflicting Evidence the Paper Seeks to Reconcile

The paper is motivated by a genuine tension in the literature that has not been adequately resolved.

On one side, a substantial body of work documents that post-training harms factual accuracy. Lin et al. (2024) coined the term "alignment tax" and showed systematic degradation on factual benchmarks as RLHF reward strength increases. Achiam et al. (2023) reported in the GPT-4 technical report that RLHF "does not improve exam performance (without active effort, it actually degrades it)." Phan et al. (2025) showed that reasoning-focused RL leads to regression in general capabilities, with models exhibiting increased hallucinations despite improved reasoning. The evidence for a trade-off is substantial and comes from multiple independent research groups.

On the other side, a growing body of work suggests that RL enhances rather than degrades certain cognitive capabilities. Guo et al. (2025b) demonstrated that DeepSeek-R1 develops self-reflection, verification, and dynamic strategy adaptation through RL alone, with pass@1 on AIME 2024 jumping from 15.6% to 71.0%. Zelikman et al. (2024) introduced Quiet-STaR, showing that training models to generate internal rationales improves downstream reasoning by teaching systematic exploration of solution spaces. Shinn et al. (2023) demonstrated that RL helps models learn to refine their search through problem spaces. These findings suggest RL builds valuable cognitive skills—but do they come at the cost of knowledge, as the alignment tax literature would predict?

The paper's central move is to resolve this tension by changing the question. Rather than asking "Does RL help or hurt?"—a question that has produced contradictory answers—it asks "Does RL help or hurt what kind of knowledge retrieval?" The answer, as the paper's results demonstrate, is that RL hurts flat factual recall (consistent with the alignment tax literature) but helps structured hierarchical retrieval (consistent with the reasoning enhancement literature). This is not a compromise position; it is a reframing that makes sense of both bodies of evidence by introducing a previously overlooked variable: the structural organization of the knowledge being retrieved.

How This Paper Positions Itself

The paper positions itself at the intersection of three research threads—the alignment tax literature, the RL-for-reasoning literature, and work on hierarchical reasoning—while carving out distinct claims that none of these traditions have advanced.

Relative to the alignment tax literature, the paper argues not that prior findings are wrong but that they are incomplete. The alignment tax is real—RL does degrade certain types of factual recall. But this degradation pattern does not extend to all forms of parametric knowledge retrieval. By introducing hierarchical structure as a key variable, the paper extends rather than refutes prior work. The findings about structured prompting recovering RL-level performance suggest that at least some of the apparent "forgetting" in aligned models is actually a retrieval deficit—knowledge is present but inaccessible without appropriate navigational scaffolding.

Relative to the RL-for-reasoning literature, the paper shifts attention from acquiring new reasoning capabilities to improving access to existing knowledge. Prior work emphasizing RL's benefits for reasoning (Guo et al., 2025b; Zelikman et al., 2024; Jin et al., 2025) has focused on mathematical, algorithmic, and logical reasoning tasks—domains where the model is applying procedural skills to derive answers through multi-step computation. The paper makes the novel claim that these same navigational improvements extend to retrieval from structured factual hierarchies, a domain where the answer is stored, not computed. This expands the scope of what RL's benefits are understood to include.

Crucially, the paper does not claim RL teaches models new navigation strategies from scratch. The structured prompting experiments—where a human-designed prompt template that explicitly guides hierarchical traversal recovers most of the RL performance gap—suggest that the knowledge of how to navigate hierarchies is already present in instruction-tuned models. RL's role is to make this knowledge procedural and automatic: to bake the multi-step traversal into the model's internal processing so it happens without explicit prompting. This is a more specific claim than "RL improves reasoning." It says RL transforms a capability that models possess but cannot reliably deploy into one they can access consistently.

Relative to hierarchical reasoning work, the paper provides an alternative interpretation. Wang et al. (2025a) present the Hierarchical Reasoning Model with dedicated modules for high-level planning and low-level computation. Yang et al. (2025b) show that hierarchical RL on template sequences outperforms training on detailed reasoning traces. Liao et al. (2025) demonstrate 30+ percentage point improvements through graph-driven structured medical reasoning. These works typically attribute gains to enhanced reasoning capabilities. The paper's alternative account—that these improvements may reflect better navigation of knowledge already encoded during pretraining—is testable and distinguishes this work from prior approaches. The paper's evidence (structured prompting closing the gap, layer-wise divergence patterns) is designed to support this navigational account specifically, not just to document that RL helps.

3. Technical Approach

3.1 Reader Orientation

This paper builds a systematic framework for investigating how reinforcement learning (RL) changes how language models retrieve structured knowledge, rather than just measuring whether performance improves. The system is not a new model architecture or training algorithm—it is an experimental apparatus combining behavioral probing (structured prompting, stratified retrieval tasks) with mechanistic analysis (layer-wise activation comparison) to test whether RL's benefits on hierarchical knowledge tasks come from acquiring new facts or from improving the procedural skill of navigating knowledge structures already encoded during pretraining.

The problem it solves is the confound between knowledge storage and knowledge access. When an RL-enhanced model outperforms its base counterpart on a medical code lookup, we cannot tell from accuracy alone whether the improvement reflects (a) new factual content injected by RL, (b) better retrieval of content that was already present but inaccessible, or (c) some mixture of both. The paper's experimental design disentangles these explanations by asking three targeted questions: can explicit prompting recover the RL performance advantage (suggesting the knowledge was always there), do RL models navigate deeper hierarchies better than prompted base models (suggesting RL builds genuine navigational skill), and do internal representations of facts remain stable while query representations diverge (providing mechanistic evidence for the navigation hypothesis)?

3.2 Big-Picture Architecture (Diagram in Words)

The experimental framework has five major components, organized as a pipeline from hypothesis to evidence:

1. Task Construction (Datasets) — Two knowledge-intensive benchmarks (MedConceptsQA for medical codes, IPC for patent classification) are selected or extended to test hierarchical retrieval, with IPC further stratified by retrieval depth to probe complexity-dependent effects.

2. Prompting Strategies (Behavioral Probes) — Three zero-shot prompt templates (Direct QA, Chain-of-Thought, Structured) form a gradient of navigational guidance, from no assistance to explicit step-by-step hierarchical traversal instructions. These templates are applied identically across all models to test whether externalized navigation can substitute for RL-internalized navigation.

3. Model Matrix (Controlled Comparisons) — A set of model pairs (base vs. instruct vs. reasoning vs. distilled) spanning the Qwen2.5, DeepSeek, and Mistral families, enabling paired comparisons that isolate the effect of post-training type while controlling for architecture and scale.

4. Path Matching Metric (Deep Retrieval Assessment) — For the expanded IPC dataset, a composite score combining F1 over hierarchical ancestors and longest common subsequence ratio measures not just final-answer correctness but the quality of the traversal path itself, distinguishing models that guess correctly from those that navigate correctly.

5. Representation Analysis (Mechanistic Probe) — Layer-wise hidden state extraction from contrastive question-answer pairs, analyzed via inter-model cosine similarity (do base and RL models represent facts similarly? queries similarly?) and intra-model similarity trajectories (how does information transform from question to answer within each model?).

Information flows as follows: a question enters the system → a prompt template (one of three) wraps the question for behavioral evaluation OR the question (and its ground-truth answer) are processed separately for representation extraction → model inference produces either a text response (scored for accuracy and path matching) or hidden state vectors at each layer (compared via cosine similarity) → results are stratified by retrieval complexity and vocabulary to identify condition-dependent patterns.

3.3 Roadmap for the Deep Dive

• First, the three datasets—MedConceptsQA, the base IPC dataset, and the expanded multi-level IPC retrieval dataset—because they operationalize the paper's central distinction between flat and hierarchical retrieval, and the complexity stratification of IPC is what enables the key finding that RL models excel specifically on deep traversals.

• Second, the three prompting strategies (Direct QA, Chain-of-Thought, Structured), because they are the primary behavioral intervention that tests whether knowledge exists in instruction-tuned models but is inaccessible without navigational scaffolding. Understanding exactly what each prompt instructs the model to do is essential for interpreting the gap-closing results.

• Third, the model selection and evaluation protocol, including the full matrix of model pairs (base/instruct/reasoning/distilled across three families), sampling hyperparameters, and majority voting scheme, because the controlled paired comparisons are what enable causal claims about post-training type.

• Fourth, the path matching score metric, because it is the paper's methodological innovation for measuring traversal quality (not just answer correctness), and its two-component design (F1 for ancestor recall, CSS for structural coherence) requires careful explanation.

• Fifth, the layer-wise representation analysis, including probe construction, extraction methodology, and the inter-model vs. intra-model comparison framework, because this is the mechanistic evidence that anchors the paper's central claim about navigation vs. content.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that reinforcement learning improves LLM performance on hierarchical knowledge tasks not by expanding what the model knows, but by transforming how the model navigates what it already knows. The experimental apparatus is designed to produce converging evidence from behavior (prompting recovers RL gains), process (RL models show superior traversal path quality), and representations (queries diverge while facts remain stable).

---

Datasets: Operationalizing Hierarchical vs. Flat Retrieval

The paper uses two primary datasets and one extended variant to create a spectrum of retrieval difficulty, from simple factual lookup to deep hierarchical traversal.

MedConceptsQA. This is an existing multiple-choice question answering dataset focused on biomedical and clinical concepts. As the paper describes it (Section 2.1), the questions "are designed to test factual recall of medical terminology, concept definitions, and their relationships, without reasoning over patient cases or performing calculations." The dataset spans five medical vocabularies: ATC (Anatomical Therapeutic Chemical classification), ICD9CM (International Classification of Diseases, 9th Revision, Clinical Modification), ICD10CM (10th Revision, Clinical Modification), ICD9PROC (ICD-9 Procedure Codes), and ICD10PROC (ICD-10 Procedure Codes). Each question presents a medical code and asks the model to identify its correct description from four options, as illustrated in Figure 1: "What is medical code 57.95 in ICD-9-CM?" with options including "Other procedures for relief of elevated intraocular pressure," "Replacement of indwelling urinary catheter," and two distractors.

What makes MedConceptsQA suitable for testing hierarchical retrieval—rather than flat factual recall—is the inherent taxonomic structure of the medical code systems themselves. ICD-9-CM codes are organized hierarchically: Volume 3, Chapter 11 covers procedures on the urinary system, codes 57.0–57.99 are specifically bladder procedures, and within that range, 57.95 refers to catheter replacement. A model attempting to answer by direct pattern matching (e.g., associating the digits "57.95" with memorized surface-level co-occurrences) will fail when the correct answer shares surface features with distractors, as in the Figure 1 example where "Other procedures for relief of elevated intraocular pressure" is a genuine medical code description but belongs to a completely different anatomical system. The task demands understanding the code's position in the taxonomy, not just retrieving an isolated fact.

The paper does not report the total size of MedConceptsQA or its train/test split, but the representation analysis (Section 2.3) states that for probing, they "randomly sample 100 question-answer pairs from each vocabulary" for a total of 500 probes across the five vocabularies.

International Patent Classification (IPC) Codes. This dataset (Section 2.1) consists of queries mapped to patent classification codes, requiring the model to "identify the correct category for a given technical description, relying on recalling standardized knowledge of patent domains rather than multi-step reasoning." Like medical codes, patent classification codes are hierarchically organized: the International Patent Classification system uses a nested structure where broader categories contain increasingly specific subcategories (e.g., H04B covers "transmission" broadly, H04B 1/00 covers "details of transmission systems," and H04B 1/707 covers specific spread spectrum techniques). The task probes whether models can navigate this hierarchy to match technical descriptions to their correct categories.

The paper does not report the size or construction details of the base IPC dataset explicitly in the main text, but the extended version is described in detail (Section 2.2, Table 1).

IPC Multi-Level Retrieval Dataset (Extended). This is a paper-specific extension of the IPC dataset, constructed to enable complexity-stratified analysis. The task type is "Nearest Common Ancestor": given two patent classification codes at leaf nodes of the hierarchy, identify their nearest common ancestor node from a set of four options. Table 1 provides concrete examples that make the task design clear.

For a Memory-Light example (fewer than 3 traversals required): the two query nodes are H04B 1/7075 and H04B 1/7083. To find their nearest common ancestor, the model must trace each node up the hierarchy. H04B 1/7075 → H04B 1/7073 (one step up). H04B 1/7083 → H04B 1/7073 (also one step up). The nearest common ancestor is therefore H04B 1/7073. The total number of unique ancestor nodes recalled (excluding the initial query nodes) is small: only H04B 1/7073 appears in the traversal, so the recall count is one. The paper categorizes tasks with a total ancestor recall count strictly less than 3 as Memory-Light.

For a Memory-Heavy example (5 or more traversals required): the two query nodes are A01B 3/421 and A01B 15/06. To find their nearest common ancestor, the model must trace: A01B 3/421 → A01B 3/42 → A01B 3/40 → A01B 3/36 → A01B 3/00 → A01B (five steps). And separately: A01B 15/06 → A01B 15/04 → A01B 15/02 → A01B 15/00 → A01B (four steps). The nearest common ancestor is A01B, which appears at the end of both chains. The total number of unique ancestor nodes that must be recalled across both paths (excluding the initial query nodes) is A01B 3/42, A01B 3/40, A01B 3/36, A01B 3/00, A01B 15/04, A01B 15/02, A01B 15/00, and the common ancestor A01B itself—well over 5 distinct nodes. The paper categorizes tasks requiring recalling 5 or more ancestor nodes as Memory-Heavy.

Operationalizing retrieval complexity. The paper defines retrieval complexity (Section 2.2, Table 1) as "the total number of ancestor nodes that must be recalled along both hierarchical paths (excluding the initial query nodes) to reach the nearest common ancestor." This is a structural property of the question, not a model-dependent measure. A question about two codes that are "siblings" sharing an immediate parent requires recalling very few ancestor nodes (just the parent); a question about codes from different major branches of the taxonomy requires traversing many levels.

This stratification is what enables the paper's key causal claim about navigation depth. If RL models simply had better memorization, they would outperform base models by a similar margin on both Memory-Light and Memory-Heavy tasks. If RL models have better hierarchical traversal—the ability to systematically trace paths through a taxonomy—then their advantage should grow with retrieval depth, because deeper traversals demand more steps of the navigational skill that RL enhances.

Why these datasets? The paper's choice of medical coding and patent classification is strategic, not arbitrary. Both domains have publicly documented, structurally complex taxonomies where the correct answer is determined by hierarchical position, not by surface-level word matching. This property makes them ideal for studying hierarchical navigation because you can construct questions where direct recall of isolated facts is insufficient—the model must trace the taxonomic structure. Additionally, both domains are practically important (medical coding errors have real consequences; patent classification is essential for prior art search), so the findings have applied relevance beyond the academic question about RL mechanisms.

---

Prompting Strategies: A Gradient of Navigational Guidance

The paper uses three zero-shot prompt templates (Section 2.1, full text in Appendix B.1) to create a spectrum of external navigational assistance, from none (Direct QA) to explicit step-by-step hierarchical traversal instructions (Structured). The logic is straightforward: if RL's benefit comes from internalizing navigation strategies, then providing those strategies externally through a prompt should recover the performance advantage in models that possess the underlying knowledge but lack the internal navigation skill.

Direct Question-Answering (QA) Prompting (Template 1). This baseline template instructs the model to provide only the answer letter without explanation: "Answer only A,B,C,D according to the answer to this multiple choice question. [... Insert Question Text Here ...] Answer (only the letter of your choice (A, B, C, or D)):". This is minimal prompting, providing no guidance about how to approach the problem. The model must rely entirely on whatever internal processing it performs by default—for a base or instruction-tuned model, this is typically direct pattern completion; for a reasoning model, this might still trigger internal chain-of-thought depending on the model's training. This template establishes the lower bound of performance under no navigational assistance and provides the raw gap that the other templates attempt to close.

Standard Chain-of-Thought (CoT) Prompting (Template 2). This template requests both an answer and an explanation: "You are a medical research assistant. Read the following multiple-choice question carefully. Your task is to: 1. Answer each question with one of A/B/C/D... 2. For my convenience, please give me a list of ANSWERs... with additional explanation for each answer in the format 'Explanation: ...'." The prompt formats the expected output as "Answer: <A/B/C/D>" followed by "Explanation: <your explanation here>". This is more guidance than Direct QA—it encourages the model to produce an explanation—but it imposes no structure on that explanation. The model might explain its answer by any reasoning process it chooses: direct recall, hierarchical breakdown, elimination, or even confabulation. This template tests whether the mere act of explaining (which is known to improve reasoning accuracy in some contexts) closes the gap, without providing specific navigational strategy instructions.

Structured Prompting (Template 3). This is the paper's primary behavioral intervention. The template explicitly guides the model through hierarchical traversal and systematic elimination, directly testing the hypothesis that the necessary knowledge exists in instruction-tuned models but requires structured access. The full template reads (Appendix B.1):

"You are a medical classification expert. For each option, first recall the general category and structure breakdown of the medical code, then explain why it might be wrong. Finally pick the correct one. [... Insert Question Text Here ...] Steps to follow: 1. Recall the general category and structural break down of the code. 2. Evaluate each option (A–D) briefly. 3. Choose the best option and justify. Answer format: Step 1: ... Step 2A: ... Step 2B: ... Step 2C: ... Step 2D: ... Final Answer: [A/B/C/D] because ..."

The template enforces two specific navigational behaviors. First, structural recall: before evaluating any options, the model must retrieve the taxonomic position of the queried code—its chapter, category, and subcategory within the ICD or IPC hierarchy. This step externalizes the hierarchical navigation that the paper hypothesizes RL models perform internally (as illustrated in Figure 1, where DeepSeek-R1 spontaneously produces "57.95 is ICD-9-CM Volume 3, Chapter 11, codes 57.0-57.99" and "This is a subset of 57.0... procedures relating to clearance of bladder"). Second, stepwise elimination: the model must evaluate each option individually, explaining why it might be wrong before selecting the correct one. This prevents the model from fixating on a single plausible-sounding option (the failure mode shown for DeepSeek-V3 in Figure 1, where it directly recalls "Other procedures for relief of elevated intraocular pressure" without checking whether that description belongs in the urological chapter).

The structured prompt essentially serves as a cognitive prosthesis: it provides external scaffolding for the hierarchical traversal and elimination steps that reasoning models have learned to perform internally through RL. If this external scaffolding closes the performance gap, the knowledge must have been present in the instruction-tuned model all along—it was just inaccessible without navigational guidance. If the gap persists despite structured prompting, then RL must have contributed something beyond navigation (new knowledge, better calibration, or some other capability) that prompting cannot replicate.

Why these three templates? The gradient from Template 1 (no guidance) to Template 3 (explicit structural guidance) enables the paper to make a more precise claim than "prompting helps." The comparison between Template 1 and Template 2 isolates the benefit of generic explanation (CoT effects). The comparison between Template 2 and Template 3 isolates the benefit of specific structural guidance—the hierarchical navigation and elimination that the paper claims is the mechanism of RL's advantage. If Template 3 substantially outperforms Template 2, but Template 2 only modestly outperforms Template 1, then the benefit is not from explanation per se but from the specific navigational strategy being externalized. The results in Tables 2 and 3 support exactly this pattern: for DeepSeek-V3 on MedConceptsQA, the majority vote accuracy moves from 0.541 (Template 1) to 0.632 (Template 2, a +0.091 gain from generic CoT) to 0.717 (Template 3, a further +0.085 gain from structural guidance specifically).

---

Model Selection and Evaluation Protocol

The paper constructs a matrix of model comparisons to isolate the effect of post-training type while controlling for architecture family and parameter scale (Section 2.1).

Model groups. The evaluation spans four categories:

1. Instruction-tuned models paired with their base versions: Qwen2.5 at four scales (7B, 14B, 32B, 72B parameters), Mistral-Small-3.1-24B-Instruct (paired with its base), and DeepSeek-V3 (671B parameters, paired with DeepSeek-R1 as its reasoning counterpart). The paper also includes Qwen3-235B-A22B (a mixture-of-experts model) in both instruct and reasoning modes.

2. Reasoning models paired with their instruction-tuned counterparts: QwQ-32B (the reasoning-enhanced version of Qwen2.5-32B), DeepSeek-R1 (the reasoning-enhanced version of DeepSeek-V3, trained via large-scale RL as described in Guo et al., 2025a), Magistral (the reasoning-enhanced version of Mistral-Small-3.1-24B), and the reasoning mode of Qwen3-235B-A22B.

3. Distilled models paired against the full DeepSeek-R1: Qwen2.5-Math-7B, Qwen2.5-32B, and Llama3.3-70B, all distilled from DeepSeek-R1. These are smaller models trained to mimic R1's outputs on reasoning tasks, and the paper uses them to test whether distillation captures the navigational capabilities of RL or only surface-level patterns.

4. Base models (pretrained only, no instruction tuning or RL) paired with their instruct versions: the base versions of all Qwen2.5 scales, to isolate the effect of instruction tuning separately from the effect of reasoning-focused RL.

Why this model matrix? The paired comparisons serve different analytical purposes. Comparing instruct to base isolates the effect of SFT; comparing reasoning to instruct isolates the additional effect of RL beyond SFT; comparing distilled to the full reasoning model isolates the effect of distillation versus genuine RL training. The multi-scale Qwen2.5 family (7B through 72B) allows the paper to check whether the navigation-vs-content pattern holds across model sizes or whether it emerges only at larger scales. The inclusion of three model families (Qwen, DeepSeek, Mistral) provides evidence about whether the findings are architecture-specific or general.

Sampling protocol. All models are sampled using "a temperature of 0.8 and top-p of 0.7 across three independent runs" (Section 2.1). Temperature 0.8 is moderately high—it introduces stochasticity to capture model uncertainty and enable majority voting, but is not so high as to degrade output quality severely. Top-p 0.7 is a standard nucleus sampling parameter that truncates the low-probability tail of the token distribution. Three independent runs means each question is answered three times with different random seeds, producing three potentially different answers per model-template combination.

Evaluation metrics. Performance is reported as two complementary metrics (Tables 2–4):

• Majority vote accuracy (Maj. Vote Acc.): For each question, the most frequent answer among the three runs is selected. If two or more runs agree on an answer, that answer is taken; if all three disagree, no clear majority exists (the paper does not specify how ties are handled, but standard practice is to select randomly among the tied answers or mark the question as incorrect). Majority voting reduces the impact of single-run stochastic errors and provides a more robust estimate of the model's consistent knowledge.

• Mean accuracy (Mean Acc.): The average accuracy across the three individual runs, reported with standard deviation as a subscript in parentheses (e.g., "0.371(.012)" means mean accuracy 0.371 with standard deviation 0.012 across the three runs). This captures both the model's average performance level and its run-to-run consistency.

The paper reports both metrics in all result tables, with the gap between reasoning and instruct models computed for both Maj. Vote Acc. (shaded red in tables) and Mean Acc. (shaded green). Presenting both metrics allows the reader to distinguish cases where the reasoning model is genuinely more accurate from cases where it is simply more consistent (high Maj. Vote Acc. but similar Mean Acc. would indicate that the reasoning model produces the same answer more reliably across runs, even if that answer is not always more likely to be correct).

Figure 2 analysis: response consistency. Beyond aggregate accuracy, the paper analyzes how structured prompting changes the distribution of response consistency (Figure 2). For DeepSeek-V3 and R1 on MedConceptsQA, each question is categorized based on how many of the three runs produced the correct answer: "All Incorrect" (0/3), "Majority Incorrect" (1/3), "Majority Correct" (2/3), or "All Correct" (3/3). This four-category breakdown reveals how prompting improves performance: if structured prompting shifts questions from "All Incorrect" to "Majority Correct," it is rescuing questions where the model previously had no reliable access to the answer. If it shifts questions from "Majority Correct" to "All Correct," it is improving consistency on questions the model already sometimes got right.

The paper reports that structured prompting causes "significant sample migration" in DeepSeek-V3—questions move from the incorrect categories to the correct categories—while R1's distribution remains "static," already concentrated in the "All Correct" and "Majority Correct" categories. This pattern suggests that R1 has already internalized the benefit that structured prompting provides externally to V3, which is consistent with the navigation hypothesis: RL bakes the hierarchical traversal into the model's internal processing, so the external prompt adds nothing new, while V3 benefits substantially from the external scaffold.

---

Path Matching Score: Measuring Traversal Quality, Not Just Answer Correctness

A core methodological challenge the paper faces is distinguishing between a model that navigates the hierarchy correctly to reach the right answer and a model that guesses the right answer through some other mechanism (memorization of surface patterns, lucky elimination, or random chance). The expanded IPC dataset's "Nearest Common Ancestor" task makes this distinction measurable because the model must not only select the correct answer letter but must also produce an explanation that includes the hierarchical path it traversed.

The paper introduces the Path Matching Score (Section 2.2) to quantify the quality of the predicted hierarchical path, independent of whether the final answer is correct. The score combines two established metrics via harmonic mean:

F1-Score over hierarchical ancestors. The paper defines:

$\text{F1} = 2 \times \frac{P \times R}{P + R}$

where $P$ is precision (the fraction of predicted ancestor nodes that appear in the ground-truth path) and $R$ is recall (the fraction of ground-truth ancestor nodes that appear in the predicted path).

What it computes: For a given question, the model produces a predicted set of hierarchical ancestor nodes (e.g., "H04B 1/7073, H04B 1/707, H04B 1/00, H04B" as the ancestors for code H04B 1/7075). Precision asks: of these predicted ancestors, how many are actual ancestors according to the IPC taxonomy? Recall asks: of all actual ancestors, how many did the model identify? The F1-score is the harmonic mean of these two, ranging from 0 (no overlap between predicted and true ancestors) to 1 (perfect match). F1 penalizes both over-generation (predicting ancestor nodes that don't exist in the true path, reducing precision) and under-generation (missing true ancestor nodes, reducing recall).

Common Subsequence Score (CSS). The paper defines:

$\text{CSS} = \frac{|\text{LCS}(\text{predicted}, \text{ground truth})|}{|\text{ground truth ancestors}|}$

where $\text{LCS}(\text{predicted}, \text{ground truth})$ is the Longest Common Subsequence between the predicted ancestor sequence and the ground-truth ancestor sequence, $|\text{LCS}(\cdot)|$ is its length (number of nodes in the common subsequence), and $|\text{ground truth ancestors}|$ is the total number of ancestor nodes in the correct path.

What it computes: Unlike the set-based F1 score (which treats ancestors as an unordered bag), CSS evaluates the structural integrity of the predicted path by measuring whether nodes appear in the correct order. The longest common subsequence between two sequences is the longest sequence of elements that appears in both sequences in the same relative order (though not necessarily consecutively). For example, if the ground-truth path is [A, B, C, D, E] and the predicted path is [A, C, B, E], the LCS could be [A, B, E] (length 3) because A, B, and E appear in that relative order in both sequences, even though C is out of place. The CSS is the ratio of this LCS length to the total ground-truth path length. A CSS of 1.0 means the model predicted the entire correct path in the right order; a CSS of 0.5 means at most half the correct nodes appear in the right order.

Why CSS matters in addition to F1. Consider a model that correctly identifies all ancestor nodes but lists them in reverse order (from leaf to root instead of root to leaf). The F1 score would be 1.0 (perfect precision and recall on the set of nodes), but the CSS would be low because the subsequence order is wrong. A model that navigates the hierarchy correctly should not only identify the right nodes but traverse them in the correct directional order (typically from the specific code upward toward the common ancestor). CSS penalizes models that retrieve relevant nodes but lack coherent hierarchical traversal structure.

Combined Path Matching Score. The paper combines F1 and CSS via harmonic mean:

$\text{Path Matching} = 2 \times \frac{\text{F1} \times \text{CSS}}{\text{F1} + \text{CSS}}$

What it computes: A single scalar between 0 and 1 that is high only when both set-level accuracy (F1) and structural coherence (CSS) are high. The harmonic mean is lower than the arithmetic mean and is dominated by the smaller of the two components—a model with perfect F1 but poor CSS would score near 0.5, not near 0.75.

Why harmonic mean? The harmonic mean ensures that the path matching score is conservative: a high score requires both F1 and CSS to be high. An arithmetic mean would allow a model with F1 = 1.0 and CSS = 0.0 to score 0.5, which would misleadingly suggest moderate traversal quality when in fact the model has no coherent path structure at all. The harmonic mean weights the score toward the weaker component, which is appropriate for a metric that should only award high scores when all aspects of traversal are correct.

How the metric is used. The path matching score is reported alongside standard accuracy in Table 5 for DeepSeek-R1 and DeepSeek-V3 on the Memory-Light and Memory-Heavy splits of the IPC multi-level retrieval dataset. The key comparison is not the absolute scores but the gap between R1 and V3 as retrieval complexity increases. On Memory-Light tasks (< 3 traversals), R1 achieves a path matching score of 0.681 versus V3's 0.627—a gap of roughly 0.054. On Memory-Heavy tasks (5+ traversals), R1 achieves 0.597 versus V3's 0.503—a gap of roughly 0.094. The widening gap (from 5.4 to 9.4 points) is what supports the claim that RL models have superior hierarchical navigation, specifically on deeply structured retrieval tasks where navigation skill matters most.

---

Layer-Wise Representation Analysis: Mechanistic Evidence for the Navigation Hypothesis

The behavioral experiments (prompting, path matching) can establish that RL improves hierarchical access, but they cannot directly show how—whether RL is modifying the factual content stored in the model's parameters or modifying the query-processing mechanisms that retrieve that content. The layer-wise representation analysis (Section 2.3) is designed to provide this mechanistic evidence by comparing the internal activation patterns of base and specialized models when processing questions versus answers.

Probe construction. The paper constructs contrastive probes from the MedConceptsQA dataset. For each of the five medical vocabularies (ATC, ICD9CM, ICD10CM, ICD9PROC, ICD10PROC), 100 question-answer pairs are randomly sampled, producing 500 total probes. Each probe consists of a factual question and its corresponding ground-truth answer, both formatted as natural language strings. For example, a probe for medical code 0QD20Z from ICD10PROC takes the following form (Section 2.3):

Question: "What is the description of the medical code 0QD20Z in ICD10PROC?"

Answer: "The description of the medical code 0QD20Z in ICD10PROC is extraction of right pelvic bone, open approach."

The question and answer are processed independently through each model—meaning the model receives the question string alone (not concatenated with the answer) to extract question representations, and receives the answer string alone to extract answer representations. This independent processing is crucial because it isolates query processing from answer generation: question representations capture how the model "understands" the interrogative form of the probe, while answer representations capture how the model encodes the declarative factual content independent of any retrieval demand.

Why question-answer pairs? The contrast between interrogative and declarative forms is the key to the paper's argument. If RL primarily modifies how models access knowledge (the navigation hypothesis), then representations for questions—which encode the retrieval demand—should differ between base and RL models, because RL has changed the query-processing circuitry. Representations for answers—which encode the factual content itself—should remain similar, because RL has not changed what the model knows, only how it retrieves it. If instead RL modifies knowledge content directly (the knowledge acquisition hypothesis), both question and answer representations should diverge, because the underlying factual encoding has changed.

Why independent processing? If the model received the concatenated question-answer pair, the question representation would be contaminated by attention to the forthcoming answer tokens, making it impossible to separate query processing from content knowledge. By processing questions and answers independently, the paper ensures that question representations reflect only the model's response to the query string, while answer representations reflect only the model's encoding of the factual statement. This design choice enables the clean comparison that the paper's hypothesis demands.

Representation extraction. For a model with $L$ layers and hidden dimension $d$, the paper extracts the hidden state at the final token position for each layer $\ell \in \{1, \ldots, L\}$ as the layer's representation vector $\mathbf{h}_\ell \in \mathbb{R}^d$. The paper explains: "This representation attends to all preceding tokens, thereby capturing the full input context at that layer." This means that for the question "What is the description of the medical code 0QD20Z in ICD10PROC?", the extracted representation at layer $\ell$ is the hidden state of the final token ("?") after it has attended to all preceding tokens in the question. This representation encodes the model's contextualized understanding of the entire question, not just the final token in isolation.

Using the final-token representation is standard practice in transformer interpretability because the final token's hidden state aggregates information from all preceding tokens through the causal attention mechanism—it is the model's summary representation of the input sequence. However, it is worth noting that this representation may over-emphasize information near the end of the sequence (recency bias in attention), which could affect the analysis if question and answer strings differ systematically in length or structure.

Comparison metric: cosine similarity. For each layer $\ell$, the paper computes cosine similarity between representation vectors from different sources. The general form is:

$d^{(a,b)}_{\text{cos}}(\ell) = 1 - \frac{1}{N} \sum_{i=1}^{N} \frac{\mathbf{h}^{(a)}_{\ell}(i)^\top \mathbf{h}^{(b)}_{\ell}(i)}{\|\mathbf{h}^{(a)}_{\ell}(i)\|_2 \|\mathbf{h}^{(b)}_{\ell}(i)\|_2}$

where $N = 100$ is the number of probes per vocabulary, $\mathbf{h}^{(a)}_{\ell}(i)$ is the layer-$\ell$ hidden representation for probe $i$ from source $a$, $\mathbf{h}^{(b)}_{\ell}(i)$ is the corresponding representation from source $b$, $\|\cdot\|_2$ is the Euclidean norm, and $\top$ denotes vector transpose. The sources $a$ and $b$ are drawn from the set $S = \{Q_{\text{base}}, A_{\text{base}}, Q_{\text{specialized}}, A_{\text{specialized}}\}$.

What it computes: For a given layer $\ell$, the cosine similarity measures the directional alignment between two representation vectors, normalized by their magnitudes. The dot product $\mathbf{h}^{(a)}_\ell(i)^\top \mathbf{h}^{(b)}_\ell(i)$ is large when the vectors point in similar directions (regardless of their lengths). Dividing by the product of their norms $\|\mathbf{h}^{(a)}_\ell(i)\|_2 \|\mathbf{h}^{(b)}_\ell(i)\|_2$ normalizes this to the range $[-1, 1]$, where 1 means identical direction, 0 means orthogonal, and -1 means opposite direction. The cosine distance $d = 1 - \text{cosine similarity}$ converts this to a distance metric where 0 means identical and larger values mean greater divergence. The paper averages this distance across $N$ probes to produce a single scalar per layer per comparison pair.

Why cosine similarity rather than Euclidean distance? Cosine similarity ignores vector magnitude and focuses on directional alignment. This is important because different models or different layers within a model may operate at different activation scales—a base model and an RL model might produce representations with very different norms even if they encode the same information directionally. Cosine similarity abstracts away from scale differences to isolate whether the models represent information in similar directions in the high-dimensional hidden space. If the models differ primarily in activation magnitude (e.g., RL makes the model more confident, producing larger-magnitude representations), cosine similarity will remain high. If the models differ in what information they encode (the direction of the representation vector), cosine similarity will drop.

Inter-model (Q-Q / A-A) analysis. This comparison measures representational similarity between the base model and a specialized model (instruction-tuned, reasoning, or distilled) for the same input type. The paper computes two comparisons per specialized model:

• Q-base vs Q-specialized: For each probe, compare the question representation from the base model to the question representation from the specialized model. High similarity means the models process queries similarly; low similarity means they diverge in how they understand the interrogative form.

• A-base vs A-specialized: For each probe, compare the answer representation from the base model to the answer representation from the specialized model. High similarity means the models encode factual content similarly; low similarity means the factual representations themselves have changed.

The paper's central finding (Figure 3, bottom row) is that "declarative statements maintain high cosine similarity (0.85-0.92) between base and RL models throughout most layers, while interrogative queries diverge substantially (similarity dropping to 0.65-0.73 in middle layers)." This asymmetry—answers remain similar while questions diverge—is direct evidence for the navigation hypothesis: RL modifies how the model processes retrieval demands (queries) while leaving the factual content (statements) largely unchanged.

Intra-model (Q-A) comparison. This analysis measures how representations transform from question to answer within a single model. For each model (base, instruct, reasoning, distilled), the paper computes the cosine similarity between question representations and answer representations layer by layer:

• Q-model vs A-model: For each probe, compare the question representation from a given model to the answer representation from the same model. This traces how the model's internal activations evolve from encoding the retrieval demand to encoding the factual content.

The paper finds (Figure 3, top row) that "within each model, representations for questions and answers are initially highly similar, but this similarity decreases in later layers, suggesting that the representations accumulate increasingly distinct features." This pattern is consistent with a model that begins by encoding the query and then progressively transforms that encoding into a representation of the answer content. The decreasing similarity over layers reflects the model's internal process of moving from "what is being asked" to "what the answer is."

Why intra-model analysis? This comparison provides a baseline for interpreting the inter-model results. If question and answer representations are already highly dissimilar within a model (as they inevitably must be, since they encode different information), then a finding that inter-model question representations diverge more than inter-model answer representations needs to be interpreted against this baseline. The intra-model Q-A similarity trajectory establishes the "natural" divergence between query and content within a single model, providing context for the magnitude of inter-model divergence.

Why five vocabularies? The paper reports layer-wise analysis separately for each of the five MedConceptsQA vocabularies (ATC, ICD9CM, ICD10CM, ICD9PROC, ICD10PROC). Results for ICD9PROC appear in the main paper (Figure 3); the remaining four vocabularies appear in Appendix C (Figures 8 and 11). Analyzing by vocabulary provides a replication check: if the Q-Q divergence > A-A divergence pattern appears consistently across all five medical coding systems, it is robust to the specific taxonomic structure and surface features of the codes. The paper also compares two model families (Qwen2.5-32B variants and Mistral-Small-24B variants) to check robustness across architectures.

Layer-wise granularity. By computing similarity at every layer rather than just at the final layer, the paper can identify where in the model the divergence between base and specialized models emerges. The finding that question representations diverge most in middle layers (where similarity drops to 0.65–0.73) while remaining more similar in early layers (where the raw token embeddings are processed similarly) and late layers (where the model converges toward answer-like representations) suggests that RL specifically targets the intermediate processing stages where the model transforms a query into a retrieval strategy.

Model families analyzed. The main paper analyzes the Qwen2.5-32B family: Qwen2.5-32B (base), Qwen2.5-32B-Instruct (instruction-tuned), DeepSeek-R1-Distill-Qwen-32B (distilled from R1), and QwQ-32B (reasoning-enhanced). This covers all three post-training types (SFT, distillation, RL) at a consistent 32B parameter scale. The appendix extends the analysis to the Mistral-Small-24B family (base, instruct, and reasoning variants) to demonstrate cross-architecture robustness. The paper notes that the 32B scale was chosen because it "spans multiple enhancement methods while remaining computationally tractable for single-GPU inference" (Section 2.3).

CoT step-wise probing (Appendix C.2). In addition to the question-answer pairwise analysis, the paper conducts a more fine-grained analysis of how representations evolve during chain-of-thought reasoning. For a given question (e.g., "What is the description of the medical code 743.63 in ICD9CM?"), the paper constructs a series of incrementally expanding CoT prompts:

• Step 1: "hmm let me think. 001-999.99 refers to diseases and injuries"
• Step 2: "...and 740-759.99 refers to congenital anomalies"
• ...continuing through the hierarchical breakdown...
• Final step: "...and 743.63 refers to other specified congenital anomalies of eyelid"

The number of CoT steps varies by vocabulary: "we predefine all CoT sequences to be 5 steps long, with the exception of ICD10PROC, which uses 6 steps due to its more deeply embedded code structure." For each prompt in this series, activations are extracted from each layer and grouped by vocabulary. The paper then computes both cosine similarity and L2 distance between base and specialized model representations at each CoT step, producing the layer-wise divergence trajectories shown in Figures 6 and 8.

The L2 distance metric used in this supplementary analysis is:

$d^{(a,b)}_{\text{L2}}(\ell) = \frac{1}{N} \sum_{i=1}^{N} \|\mathbf{h}^{(a)}_{\ell}(i) - \mathbf{h}^{(b)}_{\ell}(i)\|_2$

What this CoT probing adds: The question-answer pairwise analysis compares a single interrogative representation to a single declarative representation. The CoT step-wise analysis shows how representations evolve during the traversal process itself. If RL specifically improves hierarchical navigation, the divergence between base and RL model representations should grow as the CoT prompt progresses through deeper levels of the hierarchy—that is, the models should process the early CoT steps similarly but diverge when the navigational demand increases. The paper's figures support this pattern qualitatively, though no quantitative metric for divergence growth across CoT steps is reported.

Summary of the representation analysis pipeline. The full pipeline proceeds as follows: (1) sample 100 question-answer pairs per vocabulary from MedConceptsQA, (2) format questions interrogatively and answers declaratively, (3) process each string independently through each model, (4) extract the final-token hidden state at every layer, (5) compute cosine similarity (and optionally L2 distance) between representation pairs (Q vs Q across models, A vs A across models, Q vs A within models, and for the CoT analysis, representations at each step across models), (6) average across the 100 probes per vocabulary, producing layer-wise similarity curves, (7) compare the Q-Q divergence to the A-A divergence—the central claim is that the former substantially exceeds the latter, especially in middle layers.

4. Key Insights and Innovations

Innovation 1: Reframing the "Alignment Tax" as a Retrieval Deficit, Not Knowledge Erasure

The paper's most fundamental conceptual move is not a new method but a reframing of what the alignment tax actually is. Prior work—Lin et al. (2024), Yuan et al. (2024), Gekhman et al. (2024), and the GPT-4 technical report (Achiam et al., 2023)—had converged on a narrative that RLHF and reasoning-focused RL degrade factual knowledge. The implicit model was that post-training overwrites or corrupts parametric memories: the model literally knows less after alignment than before. This framing treats knowledge as a monolithic resource that post-training either preserves or destroys, leading to a zero-sum trade-off narrative: better reasoning, worse facts.

The paper challenges this framing at its foundation by introducing a distinction the field had overlooked: knowledge storage versus knowledge access. The experiment that makes this reframing convincing is the structured prompting result (Table 2, Figure 1). If RL truly destroyed knowledge, then no amount of prompting should recover it—external scaffolding cannot restore facts that have been erased from the model's parameters. Yet structured prompting reduces the 24pp gap between DeepSeek-V3 and DeepSeek-R1 on MedConceptsQA to 7pp, a 68% gap reduction achieved without any parameter updates. This is direct evidence that the knowledge was present in V3 all along; it was simply inaccessible through the model's default retrieval strategy. The alignment tax, in this reframing, is primarily a navigational deficit: aligned models lose not the facts themselves but the procedural skill of systematically locating them within their parameter space.

This is a fundamental, not incremental, shift because it changes the diagnosis and therefore the treatment. If the problem is knowledge erasure, the solution is to preserve more facts during post-training (e.g., through knowledge distillation, replay, or regularization). If the problem is access, the solution is to improve retrieval mechanisms—through better prompting (as the paper demonstrates), through RL that explicitly trains navigational strategies, or through architectures that separate content storage from content access. The paper does not just report a surprising empirical result; it provides a new conceptual vocabulary for discussing the relationship between post-training and knowledge.

The reframing also reconciles contradictory prior findings. The alignment tax literature (showing factual degradation) tested primarily flat, single-hop factual recall—tasks where the retrieval demand is minimal and the model's default direct-recall strategy suffices. The reasoning-enhancement literature (showing RL benefits) tested multi-step reasoning tasks where structured traversal is essential. The paper's insight is that these are not contradictory findings about RL's effects; they are findings about different retrieval demands. RL degrades the kind of shallow, pattern-matching access that flat factual benchmarks test, while improving the kind of structured, hierarchical access that reasoning benchmarks demand. The apparent paradox resolves when you stop treating "knowledge" as a unitary quantity and start distinguishing how it is accessed.

Innovation 2: Structured Prompting as a Diagnostic Tool for Disentangling Knowledge Storage from Knowledge Access

The paper's second conceptual contribution is methodological: it demonstrates that structured prompting can serve as a causal probe for whether knowledge exists in a model's parameters, independent of whether the model can spontaneously retrieve it. This is not just a performance optimization—it is a diagnostic instrument that the paper uses to test a specific mechanistic hypothesis.

The logic is straightforward but powerful. If you can write a prompt template that explicitly instructs the model to perform hierarchical traversal and stepwise elimination—and this template, applied to an instruction-tuned model, recovers performance close to an RL-enhanced model—then the knowledge necessary for the task must have been present in the instruction-tuned model. You have not added new facts through the prompt; you have only changed the retrieval procedure. This transforms prompting from an engineering technique ("how do we get the best answer?") into an experimental apparatus ("is the answer latent in this model?").

What makes this distinct from standard prompt engineering work (Wei et al., 2022; Zhou et al., 2023; Khattab et al., 2023) is its diagnostic intent. Prior work on chain-of-thought prompting asked "can prompting improve performance?"—an engineering question. This paper asks "does the performance recovery from structured prompting tell us something about what changed during RL?"—a scientific question. The gradient of three prompt templates (Direct QA → CoT → Structured) is designed not to maximize accuracy but to isolate which component of RL's benefit is recoverable through external guidance. The finding that Template 2 (generic CoT) provides a modest gain (+9pp for DeepSeek-V3 on MedConceptsQA) while Template 3 (structured navigational guidance) provides an additional large gain (+8.5pp beyond CoT) suggests that the navigational component specifically—not just "explaining your reasoning"—is what RL automates internally.

The diagnostic value of this approach extends beyond this paper's findings. The structured prompting methodology could be applied to any domain where one suspects that performance differences between models reflect access deficits rather than knowledge gaps: code generation from API documentation, legal reasoning from statutory texts, scientific inference from domain taxonomies. If a prompt template that externalizes the relevant structure substantially closes the gap to a stronger model, the knowledge exists but the access mechanism is deficient—a finding with different practical implications than discovering the knowledge is absent.

This is a fundamental methodological contribution, not an incremental one, because it provides a general template for answering the question "does this model know X?"—a question that has been notoriously difficult to answer given that models can fail to produce correct answers for reasons ranging from knowledge absence to retrieval failure to decoding stochasticity. The paper shows that with the right prompt scaffolding, you can distinguish these explanations, at least for domains with well-defined hierarchical structure.

Innovation 3: The Q-Q vs. A-A Divergence Asymmetry as Direct Mechanistic Evidence for the Navigation Hypothesis

The paper's third contribution is an interpretable representational signature that cleanly separates navigation from content. The finding that declarative answer representations maintain 0.85–0.92 cosine similarity between base and RL models while interrogative question representations diverge to 0.65–0.73 in middle layers (Figure 3, bottom row) is evidence of a specific type: if RL had modified factual knowledge, both question and answer representations should diverge from the base model, because the underlying facts being queried and stated would have changed. The asymmetry—answers remain stable while queries diverge—is inconsistent with the knowledge modification hypothesis and uniquely consistent with the navigation hypothesis.

What makes this finding intellectually distinctive is not the use of representational analysis per se—layer-wise similarity comparisons are standard in mechanistic interpretability (Mukherjee et al., 2025; Skean et al., 2025). It is the contrastive design of the probe set. By constructing matched question-answer pairs where the same factual content appears once as an interrogative demand and once as a declarative statement, the paper creates a controlled comparison where the informational content is identical but the processing demand differs. If RL primarily changes how models process questions (the access mechanism), the Q-Q similarity between base and specialized models should drop. If RL primarily changes what the model knows about the answer (the factual content), the A-A similarity should drop. The observed pattern—Q-Q divergence exceeding A-A divergence by roughly 0.15–0.20 in middle layers—is a quantitative signature of the navigation hypothesis that no prior work had isolated.

This is significant beyond the paper's specific finding because it demonstrates that representational analysis can be diagnostic rather than merely descriptive. Most work on layer-wise similarity between models simply reports that representations differ (or are similar) without a clear theoretical prediction about which representations should differ if a specific hypothesis is true. The paper's contrastive probe design generates such a prediction, and the observed asymmetry confirms it. This methodology could be applied to other questions about post-training: does RL change how models represent goals versus world states? Does instruction tuning change how models represent user intent versus factual content? The general principle—design paired probes where the same content appears in different processing contexts to isolate where divergence occurs—is portable.

The additional finding from the CoT step-wise analysis (Appendix C.2, Figures 6 and 8) strengthens this argument by showing that the divergence between base and specialized models grows as the CoT prompt progresses through deeper levels of the hierarchy. If RL merely changed surface-level features of question processing, the divergence should appear early (in the initial encoding of the query) and remain static. The progressive divergence through the CoT chain suggests that RL modifies the dynamic process of hierarchical traversal—exactly what the navigation hypothesis predicts.

This is a fundamental rather than incremental contribution because it provides the first internal, mechanistic evidence—beyond behavioral patterns—that RL's effect on knowledge retrieval operates through transforming access mechanisms rather than modifying content. It gives the navigation hypothesis a neural correlate, making it testable in new contexts and model families.

Innovation 4: Retrieval Complexity as the Key Variable Determining When RL Adds Unique Value

The paper's fourth insight is that the benefit of RL over prompting is not uniform but depends systematically on retrieval depth. This emerges from the stratified IPC analysis (Table 5), where the gap between DeepSeek-R1 and DeepSeek-V3 in path matching score widens from roughly 5.4 points on Memory-Light tasks (< 3 hierarchical traversals) to roughly 9.4 points on Memory-Heavy tasks (5+ traversals). On answer accuracy alone, both models perform identically on Memory-Heavy tasks (67.7%), which might lead one to conclude that RL provides no benefit. The path matching score reveals that this conclusion would be wrong: R1 achieves the same accuracy as V3 through correctly traversing the hierarchy, while V3 achieves it through some other mechanism—possibly memorization of surface patterns, lucky guessing, or partial traversal with correct final-answer selection. The deeper the traversal, the larger R1's advantage in how it arrives at the answer.

This finding refines the paper's central claim in an important way. The structured prompting results (Innovation 1) show that on shallow retrieval tasks, external scaffolding can substitute for RL-internalized navigation. The stratified IPC results show that on deep retrieval tasks, this substitution is incomplete—RL models possess a genuine navigational capability that prompting cannot fully replicate. The paper doesn't claim that RL adds nothing; it claims that RL adds a specific capability (deep hierarchical traversal) that scales with retrieval complexity, while on simpler tasks the benefit is primarily about making navigation automatic rather than externally guided.

The intellectual contribution here is the concept of retrieval complexity as a moderating variable for the effectiveness of post-training methods. The field has typically evaluated models on benchmarks without stratifying by the structural properties of the retrieval demands. This paper demonstrates that the answer to "does RL help?" depends on how many steps of hierarchical traversal the task requires. This is a fundamentally different kind of answer than "RL helps on reasoning tasks" (the standard framing) because it makes a specific, testable prediction: the RL advantage should grow monotonically with the number of hierarchical steps required, independent of the surface format or domain of the task. This prediction could be tested on any hierarchically structured knowledge domain—biology taxonomies, legal code systems, software package hierarchies, organizational structures.

The practical implication is substantial for deployment decisions. If your task distribution consists primarily of shallow retrieval (1–3 hierarchical steps), structured prompting of an instruction-tuned model may be sufficient—saving the cost and complexity of RL training. If your task distribution includes deep retrieval (5+ steps), RL-trained models provide a genuine capability advantage that prompting cannot replicate. This is a more nuanced and actionable deployment heuristic than "use the strongest model available."

This innovation is fundamental in its conceptual framing (introducing retrieval complexity as a variable) but incremental in its empirical scope (tested on a single model pair, DeepSeek-V3/R1, on a single dataset, IPC). The paper appropriately hedges this as a "case study" (Section 2.2) and does not claim universal generalization. Future work testing this complexity-dependent gap across more model families and domains would strengthen the finding into a robust principle.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses two primary datasets. MedConceptsQA is a multiple-choice question answering dataset focused on biomedical and clinical concepts across five medical vocabularies (ATC, ICD9CM, ICD10CM, ICD9PROC, ICD10PROC). The representation analysis (Section 2.3) samples 100 question-answer pairs per vocabulary, totaling 500 probes, though the full dataset size for behavioral evaluation is not explicitly stated. International Patent Classification (IPC) consists of queries mapped to patent classification codes, testing recall of standardized patent domain knowledge. The paper further extends IPC into a Multi-Level Retrieval Dataset (Section 2.2, Table 1) stratified by retrieval complexity: Memory-Light tasks require recalling fewer than 3 ancestor nodes total across both hierarchical paths, while Memory-Heavy tasks require recalling 5 or more ancestor nodes. The tasks ask for the nearest common ancestor of two given patent classification codes.

• Base model(s). The evaluation spans three model families. The Qwen2.5 family includes base and instruction-tuned variants at four scales: 7B, 14B, 32B, and 72B parameters (Team et al., 2024), with the reasoning-enhanced QwQ-32B derived from Qwen2.5-32B. The DeepSeek family includes DeepSeek-V3 (671B parameters, instruction-tuned) and DeepSeek-R1 (reasoning-enhanced via large-scale RL, from Guo et al., 2025a). The Mistral family includes Mistral-Small-3.1-24B-Instruct and its reasoning-enhanced counterpart Magistral (Karamcheti et al., 2021). Additionally, three distilled models from DeepSeek-R1 are evaluated: Qwen2.5-Math-7B, Qwen2.5-32B, and Llama3.3-70B (Grattafiori et al., 2024). The paper selects the Qwen2.5-32B family for layer-wise representation analysis because it "spans multiple enhancement methods while remaining computationally tractable for single-GPU inference" (Section 2.3). The Qwen3-235B-A22B mixture-of-experts model (Liu et al., 2024; Yang et al., 2025a) is also included in both instruct and reasoning modes.

• Metrics. Two complementary accuracy metrics are reported throughout (Tables 2–4). Majority vote accuracy (Maj. Vote Acc.) selects the most frequent answer among three independent runs per question. Mean accuracy (Mean Acc.) averages accuracy across the three individual runs, reported with standard deviation in subscripted parentheses (e.g., "0.371(.012)" indicates mean 0.371, standard deviation 0.012). For the IPC multi-level retrieval dataset, the paper introduces the Path Matching Score (Section 2.2), a harmonic mean of two components: F1-score over hierarchical ancestors (measuring precision and recall of ancestor node identification) and Common Subsequence Score (CSS, measuring structural integrity as the ratio of the longest common subsequence between predicted and true paths to the length of the true path). The paper also reports response consistency categories (Figure 2): questions are classified as "All Incorrect" (0/3 runs correct), "Majority Incorrect" (1/3 correct), "Majority Correct" (2/3 correct), or "All Correct" (3/3 correct). For representation analysis, cosine similarity between hidden state vectors is the primary metric (Section 2.3, Equation 1), with L2 distance used supplementally in Appendix C.2 (Equation 2).

• Baselines. The paper constructs paired comparisons rather than using external baselines. For each reasoning model, the corresponding instruction-tuned model serves as the primary baseline (e.g., DeepSeek-R1 vs. DeepSeek-V3, QwQ-32B vs. Qwen2.5-32B-Instruct). For instruction-tuned models, the corresponding base (pretrained-only) model serves as the baseline (e.g., Qwen2.5-32B-Instruct vs. Qwen2.5-32B). For distilled models, the full DeepSeek-R1 reasoning model serves as the upper baseline. Across all comparisons, Direct QA prompting (Template 1: answer letter only, no explanation) serves as the minimal-intervention baseline within the prompting strategy gradient.

• Generation budget / compute accounting. The paper operates in a zero-shot prompting paradigm with no explicit test-time compute scaling (no beam search, no best-of-N beyond the three independent runs used for majority voting). The primary "budget" is the number of independent runs: each model produces exactly three responses per question per prompt template. For representation analysis, the compute cost is one forward pass per probe per model. The paper does not measure or control for total FLOPs across models—the 7B, 14B, 32B, 72B, 235B, and 671B parameter models are compared at equal prompting budgets (three runs each), not at equal compute budgets. This is an important distinction from compute-controlled studies that match FLOPs between smaller and larger models.

• Cross-validation / statistical protocol. All models are sampled from using temperature 0.8 and top-p 0.7 across three independent runs (Section 2.1). Standard deviations are reported for mean accuracy, providing a measure of run-to-run variability. Majority vote accuracy is reported alongside mean accuracy to distinguish consistent knowledge from run-to-run stochastic success. For representation analysis, 100 probes per vocabulary are randomly sampled from MedConceptsQA (Section 2.3), with cosine similarity averaged across these 100 probes per layer per comparison pair. Results are reported separately for each of the five medical vocabularies (Figures 3, 8, 11) and two model families (Qwen2.5-32B in main paper, Mistral-Small-24B in Appendix C), providing replication across vocabularies and architectures. The paper does not report formal statistical tests (confidence intervals, hypothesis tests) on the representational similarity measurements. For behavioral results, the per-question response consistency breakdown (Figure 2) provides granularity beyond aggregate accuracy, showing how questions migrate between consistency categories as prompting strategy changes.

Main Quantitative Results

Structured Prompting Closes the Performance Gap Between Instruction-Tuned and Reasoning Models

The paper's headline behavioral finding is that structured prompting substantially reduces the accuracy gap between instruction-tuned and RL-enhanced reasoning models on hierarchical knowledge tasks. For the largest model pair (DeepSeek-V3 vs. DeepSeek-R1 on MedConceptsQA, Table 2), the majority vote accuracy gap shrinks from +0.237 (QA) to +0.158 (CoT) to +0.075 (Structured) — a 68% reduction in the gap when moving from the minimal prompt to the structured prompt. The mean accuracy gap follows the same trajectory: +0.279 (QA) to +0.138 (CoT) to +0.074 (Structured).

The structured prompting effect is not uniform across model pairs. For Qwen3-235B-A22B on MedConceptsQA (Table 2), structured prompting actually reverses the gap: the instruction-tuned variant outperforms the reasoning variant by -0.051 majority vote accuracy under Structured prompting, compared to a +0.108 gap favoring the reasoning variant under CoT prompting. This reversal is consistent across both majority vote (-0.051) and mean accuracy (-0.035) under Structured prompting, though the reasoning model retains an advantage under QA and CoT templates. For Qwen2.5-32B on MedConceptsQA (Table 2), the gap narrows from +0.103 (QA) to +0.038 (CoT) to +0.036 (Structured) for majority vote accuracy — a more modest reduction than the DeepSeek pair but directionally consistent.

On the IPC dataset (Table 2), the pattern is less pronounced and more variable. For DeepSeek-V3/R1 on IPC under majority vote, the gap is +0.092 (QA), -0.031 (CoT, meaning V3 actually outperforms R1 under CoT), and +0.046 (Structured). The mean accuracy trajectory is similarly non-monotonic: +0.067 (QA), -0.015 (CoT), +0.031 (Structured). For Qwen2.5-32B on IPC, the instruct model outperforms the reasoning model in mean accuracy under all three templates, with gaps of -0.046 (QA), +0.000 (CoT), and -0.005 (Structured). For Qwen3-235B-A22B on IPC, instruct outperforms reasoning in mean accuracy under CoT (-0.010) but trails under QA (+0.046) and Structured (+0.005).

For the base-to-instruct comparisons (Table 3), the pattern is similar but the absolute gaps are generally smaller. On MedConceptsQA, structured prompting reduces the majority vote accuracy gap between base and instruct models from +0.103 to +0.078 (Qwen2.5-72B), from +0.158 to +0.065 (Qwen2.5-32B), from +0.060 to +0.034 (Qwen2.5-14B), and from +0.147 to +0.027 (Qwen2.5-7B). The gap reduction is larger for smaller models in percentage terms, though the absolute gaps are small enough at the low end that floor effects may play a role. On IPC codes (Table 3), structured prompting actually causes slight reversals for Qwen2.5-7B (gap -0.014 majority vote, -0.011 mean accuracy) and Qwen2.5-32B (gap -0.003 majority vote, +0.005 mean accuracy), where the base model slightly outperforms the instruction-tuned model under structured prompting. This pattern mirrors the instruct-vs-reasoning reversals observed in Table 2.

The distilled model results (Table 4) tell a distinctly different story. For the DeepSeek-R1 distilled models on MedConceptsQA, the gaps relative to full R1 remain large regardless of prompting strategy. Qwen2.5-Math-7B distilled shows majority vote accuracy gaps of +0.482 (QA), +0.534 (CoT), +0.510 (Structured) — all massively in favor of the full reasoning model. Qwen2.5-32B distilled shows gaps of +0.403 (QA), +0.410 (CoT), +0.345 (Structured). Llama3.3-70B distilled shows more moderate but still substantial gaps: +0.241 (QA), +0.157 (CoT), +0.182 (Structured). For all three distilled models on MedConceptsQA, structured prompting provides no systematic gap reduction — the gaps under Structured prompting are similar to or larger than the gaps under QA or CoT. This is a crucial negative result: distillation does not capture the navigational capabilities that make structured prompting effective for the full RL model. On IPC codes (Table 4), the pattern is similar though the absolute gaps are smaller: Qwen2.5-32B distilled shows gaps of +0.145 (QA), +0.162 (CoT), +0.135 (Structured), while Llama3.3-70B distilled shows gaps of +0.138 (QA), +0.061 (CoT), +0.108 (Structured).

Figure 2 provides mechanistic insight into how structured prompting improves performance for DeepSeek-V3. When transitioning from the QA baseline to the structured prompt, DeepSeek-V3 shows "significant sample migration": questions initially categorized as "All Incorrect" (0/3 runs correct) and "Majority Incorrect" (1/3 correct) shift toward "Majority Correct" (2/3 correct) and "All Correct" (3/3 correct) categories. This indicates that structured prompting is not just improving aggregate accuracy but is systematically rescuing questions where the model previously had no reliable access to the correct answer. In contrast, DeepSeek-R1's distribution across these categories remains largely static across prompt templates, suggesting it already operates near its performance ceiling and already performs the hierarchical navigation internally that the structured prompt externalizes for V3.

RL Models Demonstrate Superior Path Traversal on Deep Retrieval Tasks

The stratified IPC analysis (Table 5) reveals that the benefit of RL over prompting is not captured by final-answer accuracy alone. On Memory-Light tasks (fewer than 3 hierarchical traversals), DeepSeek-R1 achieves 44.8% accuracy versus DeepSeek-V3's 37.9% — a gap of 6.9 percentage points. On Memory-Heavy tasks (5+ traversals), both models achieve identical accuracy of 67.7%. By accuracy alone, one might conclude that RL provides no benefit on deep retrieval tasks.

The path matching score reveals the opposite. On Memory-Light tasks, R1 achieves a path matching score of 0.681 versus V3's 0.627 — a gap of 0.054. On Memory-Heavy tasks, R1 achieves 0.597 versus V3's 0.503 — a gap of 0.094. The gap widens from roughly 5.4 points to 9.4 points as retrieval complexity increases, even as the accuracy gap disappears. R1 is achieving the same accuracy as V3 on deep tasks through correctly traversing the hierarchy (as measured by the path matching score), while V3 achieves it through some other mechanism: possibly memorization of surface patterns, partial traversal with correct final-answer selection, or guessing that coincidentally matches the correct option. The key finding is that accuracy alone masks a genuine capability difference that the path matching score exposes.

This result also contextualizes the structured prompting findings from Table 2. Structured prompting recovers most of the performance gap on tasks that do not require deep hierarchical traversal (MedConceptsQA and the base IPC tasks). But on the deep-retrieval IPC tasks, the path matching gap suggests that RL provides a navigation capability that external prompting cannot fully substitute — R1's internal traversal process is more accurate than V3's prompted traversal, even when both arrive at the same final answer.

Layer-Wise Analysis Reveals Asymmetric Divergence Between Base and Specialized Models

The representation analysis (Section 3.3, Figure 3) produces the paper's central mechanistic finding. For the ICD9PROC vocabulary, inter-model comparisons between Qwen2.5-32B (base) and its three specialized variants (instruction-tuned, distilled, reasoning) show a consistent pattern: answer representations for declarative statements maintain high cosine similarity between base and specialized models throughout most layers (0.85–0.92), while question representations for interrogative queries diverge substantially, with similarity dropping to 0.65–0.73 in middle layers (Figure 3, bottom row).

This asymmetry is present across all three specialized model types but with different magnitudes:

• QwQ-32B (reasoning): The Q-Q divergence is present but modest — question representations maintain relatively high similarity with the base model's question representations compared to the other specialized variants. This is consistent with QwQ being trained via reasoning-focused RL that refines query processing without drastic representational reorganization.

• Qwen2.5-32B-Instruct (instruction-tuned): The Q-Q divergence is more pronounced than QwQ, suggesting that instruction tuning (SFT) changes query processing more substantially than reasoning-focused RL, but without providing the navigational benefits that RL confers (as shown by the behavioral results where instruct models underperform reasoning models).

• DeepSeek-R1-Distill-Qwen-32B (distilled): The Q-Q divergence is the most extreme among the three specialized variants. This represents a degradation relative to the base model's query representations without the corresponding navigational capability gain seen in the full reasoning model. This finding aligns with the behavioral result in Table 4, where distilled models fail to achieve R1-level performance and structured prompting provides minimal benefit — the distilled model's internal representations have diverged significantly from the base model, but this divergence does not translate into improved hierarchical navigation.

The intra-model Q-A similarity trajectories (Figure 3, top row) show that within each model, question and answer representations start highly similar in early layers and progressively diverge in later layers. This is the expected pattern: early layers encode surface features shared by questions and answers (e.g., the medical code string "57.95" appears in both), while later layers differentiate between the interrogative form (a retrieval demand) and the declarative form (a factual statement). The intra-model divergence provides a baseline against which to interpret the inter-model Q-Q versus A-A divergence: the finding that inter-model Q-Q similarity drops more than A-A similarity is not simply because questions and answers are naturally different; it is specifically because post-training modifies query processing more than content storage.

The supplementary results in Appendix C (Figures 8 and 11) replicate this pattern across all five MedConceptsQA vocabularies (ATC, ICD10PROC, ICD9CM, ICD10CM) for the Qwen2.5-32B family and across all five vocabularies for the Mistral-Small-24B family (comparing base, instruct, and Magistral reasoning variants). The consistent Q-Q divergence exceeding A-A divergence across vocabularies and model families strengthens the claim that this is a general signature of post-training, not an artifact of a specific vocabulary or architecture.

The CoT step-wise probing (Appendix C.2, Figures 6 and 8) extends this analysis by showing how representations diverge during the chain-of-thought process itself. As the CoT prompt progressively expands through deeper levels of the hierarchy (from the top-level category to the specific code description), the divergence between base and specialized models generally increases. This matches the prediction of the navigation hypothesis: if RL modifies how models perform hierarchical traversal, the divergence should grow as the traversal depth increases, because each additional step of the traversal invokes the modified processing circuitry.

Ablation Studies and Robustness Checks

• Prompting strategy gradient (Templates 1–3): The paper implicitly ablates the contribution of generic explanation versus specific hierarchical navigation instructions through the three-template design. Comparing Template 1 (Direct QA) to Template 2 (CoT) isolates the benefit of requesting an explanation without structural constraints. For DeepSeek-V3 on MedConceptsQA (Table 2), this yields a majority vote gain of +0.091 (from 0.541 to 0.632). Comparing Template 2 to Template 3 isolates the additional benefit of explicit hierarchical traversal and stepwise elimination instructions, yielding a further gain of +0.085 (from 0.632 to 0.717). The roughly equal gains from generic CoT and from structured navigation guidance suggest that both components contribute independently, and that the CoT benefit alone is insufficient to close the gap—the structural navigational component is essential.

• Model scale ablation within the Qwen2.5 family: The paper evaluates Qwen2.5 models at 7B, 14B, 32B, and 72B scales (Table 3), testing whether the structured prompting effect depends on model size. The base-to-instruct gap under QA prompting shrinks somewhat with scale: majority vote gaps of +0.147 (7B), +0.060 (14B), +0.158 (32B), and +0.103 (72B) — the 32B pair actually shows the largest gap, suggesting the relationship is not monotonic. Under structured prompting, the gaps reduce to +0.027 (7B), +0.034 (14B), +0.065 (32B), and +0.078 (72B). The percentage reduction in the gap is largest for the 7B model (81.6% reduction: 0.147 to 0.027) and smallest for the 72B model (24.3% reduction: 0.103 to 0.078). This suggests structured prompting is especially effective at recovering knowledge access in smaller models, possibly because larger models have partially internalized navigational strategies even without explicit RL training. However, the base-accuracy floor is lower for smaller models, which may confound the percentage reduction interpretation.

• Post-training type ablation (SFT vs. RL vs. Distillation): By comparing the Qwen2.5-32B base model against its instruct variant (SFT), its reasoning variant (QwQ-32B, RL), and its distilled variant (DeepSeek-R1-Distill-Qwen-32B), the paper ablates the effect of post-training method while holding architecture and scale constant. The behavioral results (Tables 2–4) show a clear ordering: reasoning models achieve the highest performance across prompting templates, instruction-tuned models benefit dramatically from structured prompting, and distilled models underperform despite representational divergence from the base model (Figure 3). This triangulation isolates RL as the post-training method that produces genuine navigational capability, SFT as producing knowledge that exists but is inaccessible without external scaffolding, and distillation as producing representational changes that fail to confer either robust knowledge or navigational skill.

• Model family replication: The inclusion of three model families (Qwen2.5, DeepSeek, Mistral) serves as a robustness check on the generality of the findings. The structured prompting gap reduction is observed across all three families: Qwen3-235B-A22B shows an actual reversal (-0.051 majority vote in favor of instruct under Structured on MedConceptsQA, Table 2), DeepSeek-V3 shows the largest absolute gap reduction (+0.237 to +0.075 majority vote, Table 2), and the Mistral family representation analysis in Appendix C.2 (Figure 11) replicates the Q-Q divergence exceeding A-A divergence pattern. The consistency across families with different training procedures, architectures, and scales strengthens the claim that the observed effects are not artifacts of a specific training pipeline.

• Vocabulary replication within representation analysis: The layer-wise analysis is reported separately for each of the five MedConceptsQA vocabularies (ATC, ICD9CM, ICD10CM, ICD9PROC, ICD10PROC) in Figures 3, 8, and 11. The Q-Q divergence exceeding A-A divergence pattern appears consistently across all five vocabularies. Given that these vocabularies have different hierarchical structures, code formats, and medical domains, this replication reduces the likelihood that the finding is an artifact of a specific code system's surface features.

• Metric complementarity (accuracy vs. path matching): The comparison between standard accuracy and path matching score on the stratified IPC dataset (Table 5) serves as an implicit ablation on how "performance" is measured. On Memory-Heavy tasks, accuracy alone would indicate no difference between R1 and V3 (both 67.7%), incorrectly suggesting RL provides no benefit. The path matching score reveals a 0.094 gap, showing that accuracy masks improvements in how the answer is reached. This validates the paper's claim that standard metrics are insufficient for detecting navigational improvements and that process-oriented metrics are necessary.

• CoT step-wise probing as a dynamic process check: The CoT step-wise analysis (Appendix C.2, Figures 6 and 8) ablates the static question-answer comparison by showing that divergence between base and specialized models grows as the CoT prompt progresses through deeper hierarchy levels. If the divergence were purely an artifact of surface-form differences between interrogative and declarative strings (unrelated to hierarchical depth), it would not grow with CoT depth. The progressive divergence supports the claim that RL specifically modifies the process of hierarchical traversal, not just the initial encoding of the question.

• Distilled model negative result: The finding that distilled models (Table 4) show large, persistent gaps to the full reasoning model—and that structured prompting provides minimal or inconsistent benefits—is a significant negative result. It demonstrates that distillation from R1 does not transfer the navigational capabilities that make R1 effective. Combined with the representational analysis showing that distilled models have the most extreme Q-Q divergence from the base model (Figure 3d–f), this suggests that distillation causes representational reorganization without producing functional navigational skill. The paper does not deeply investigate why, but this negative finding is important for practitioners choosing between RL training and knowledge distillation for knowledge-intensive applications.

• Response consistency analysis (Figure 2): The breakdown of questions by correctness consistency across three runs provides a more granular view than aggregate accuracy. The finding that structured prompting shifts DeepSeek-V3 questions from "All Incorrect" and "Majority Incorrect" to "Majority Correct" and "All Correct" — while R1's distribution remains largely static — demonstrates that prompting is rescuing questions where the model previously had no reliable access, not just improving calibration on questions it sometimes got right. This supports the claim that structured prompting addresses an access deficit (questions the model consistently got wrong become consistently right) rather than a consistency deficit (questions it sometimes got right become always right).

Critical Assessment

Does structured prompting genuinely demonstrate that knowledge exists in base/instruction-tuned models?

The paper's central behavioral claim is that structured prompting closes most of the performance gap between instruction-tuned and RL-enhanced models, demonstrating that the necessary knowledge is present but inaccessible without proper navigation. The evidence for this claim is strong but subject to an important interpretive caveat.

On DeepSeek-V3/R1 for MedConceptsQA (Table 2), the gap reduction from +0.237 (QA) to +0.075 (Structured) majority vote accuracy is substantial and robust. This is a 68% gap reduction achieved without any parameter updates, which strongly suggests that V3 encodes substantially more of the relevant knowledge than its default retrieval strategy can access. The response consistency migration pattern (Figure 2)—questions moving from "All Incorrect" to majority-correct categories—provides additional evidence that structured prompting is rescuing genuinely inaccessible knowledge rather than just improving calibration.

However, a 7.5 percentage point gap remains under structured prompting. The paper attributes this residual gap to RL providing "superior ability to recall correct procedural paths on deep-retrieval tasks" (Section 1). But an alternative interpretation is that the residual gap reflects knowledge that is genuinely absent from the instruction-tuned model—facts that RL somehow injected, or retrieval pathways that RL created from scratch. The paper's evidence cannot definitively rule this out. The structured prompting results show that most of the gap reflects access, but the 7.5pp residual could reflect either (a) knowledge that exists in V3 but remains inaccessible even with explicit prompting (access failure that prompting cannot fully solve) or (b) knowledge that does not exist in V3 and was acquired through RL (genuine knowledge difference). The paper favors interpretation (a) based on the stratified IPC results showing that the access gap widens with retrieval depth, but this is an inference, not a direct measurement of whether specific facts are encoded in V3's parameters.

Additionally, the reversal cases (Qwen3-235B-A22B instruct outperforming reasoning under structured prompting on MedConceptsQA, with a -0.051 majority vote gap) complicate the narrative. If RL's primary benefit is internalizing navigational strategies, why would structured prompting cause the instruct model to exceed the reasoning model? This suggests that for certain model pairs and tasks, the RL model may have over-specialized its internal navigation in ways that actually harm performance when an alternative (externally provided) navigation strategy is available. The paper does not discuss this reversal pattern in detail, but it is important evidence that the relationship between RL and navigation is not simply "RL teaches navigation, therefore RL models always navigate better."

Finally, the paper does not run a critical control experiment: applying structured prompting to the base model (not just the instruct model) and comparing to the reasoning model. If structured prompting on the base model also recovers the gap, that would strengthen the claim that knowledge exists from pretraining. If it doesn't, that would suggest instruction tuning plays a necessary intermediate role in organizing knowledge for subsequent navigational access. The base-vs-instruct structured prompting results (Table 3) partially address this by showing that structured prompting reduces but does not eliminate the base-instruct gap, but there is no direct base-vs-reasoning structured prompting comparison reported.

Do the path matching results genuinely demonstrate superior navigational capability independent of accuracy?

The claim that RL models "retain superior ability to recall correct procedural paths on deep-retrieval tasks" relies entirely on the path matching score results in Table 5. The evidence is compelling in its direction but thin in its breadth.

On the positive side, the widening gap in path matching score (0.054 on Memory-Light to 0.094 on Memory-Heavy) while accuracy remains identical (67.7% for both models on Memory-Heavy) is exactly the pattern predicted by the navigation hypothesis: RL improves traversal quality, and this improvement becomes more detectable as traversal demands increase, even when final-answer selection (which can be achieved through alternative mechanisms) is held constant. This is a clean, interpretable result.

However, the analysis has significant limitations. Only one model pair (DeepSeek-V3/R1) is evaluated on the stratified IPC dataset. The paper frames this as a "case study" (Section 2.2), acknowledging the limited scope, but the claim that RL models have superior path traversal is stated in the abstract and conclusion without the same qualification. Generalizing from one model pair to "RL models" writ large is speculative, particularly given the heterogeneous results observed across model pairs in the prompting experiments (Tables 2–3).

The path matching score itself is novel to this paper and has not been validated against human judgments of traversal quality, nor benchmarked on a dataset with known traversal difficulty. The F1 component measures set overlap between predicted and true ancestor nodes; the CSS component measures sequential coherence. It is not obvious that the harmonic mean of these two is the optimal combination—and more importantly, it is not established what magnitude of path matching score difference constitutes a "meaningful" versus "statistical but practically negligible" improvement in traversal quality. The 0.094 gap on Memory-Heavy tasks is interpretable directionally, but its practical significance is unclear without an external validity criterion.

The paper reports path matching scores at only two complexity levels (Memory-Light and Memory-Heavy). A continuous analysis showing path matching score as a function of traversal depth (e.g., scatter plot with one point per question) would provide stronger evidence for the claim that the gap monotonically increases with complexity. The current binary stratification (fewer than 3 vs. 5+ traversals) leaves open the possibility that the effect is driven by some other property of the 5+ traversal questions (e.g., they involve different IPC sections with different memorability) rather than traversal depth per se.

Finally, the paper does not report path matching scores for the structured prompting condition. The critical comparison—does structured prompting close the path matching gap between V3 and R1 on deep-retrieval tasks?—is absent. If structured prompting brings V3's path matching score close to R1's, then the navigational capability is prompt-recoverable even on deep tasks, and RL's unique contribution would be narrower than claimed. If structured prompting does not close the path matching gap (even as it may close the accuracy gap), that would provide much stronger evidence for RL conferring a genuine, non-promptable navigation advantage. This experiment is the most significant missing ablation in the paper.

Does the layer-wise analysis genuinely isolate navigation from content?

The representation analysis is the paper's most novel empirical contribution, and its design is genuinely elegant. The contrast between interrogative question representations and declarative answer representations—matched for factual content, differing only in processing demand—is a clean experimental manipulation. The finding that A-A similarity (0.85–0.92) substantially exceeds Q-Q similarity (0.65–0.73) in middle layers is predicted by the navigation hypothesis and not obviously predicted by alternative accounts.

However, several interpretive challenges arise:

The Q-Q vs. A-A asymmetry is consistent with the navigation hypothesis but does not uniquely identify it. An alternative account is that post-training modifies how models represent tasks or formats without specifically modifying hierarchical navigation. If the base model processes questions using a generic "answer this query" circuit and the specialized model processes questions using a format-specific "this is a medical multiple-choice question" circuit, the Q-Q representations would diverge even if no hierarchical traversal is involved. The A-A representations would remain similar because declarative statements don't trigger task-specific processing circuitry—they are simply factual statements to be encoded. Under this alternative, the Q-Q divergence reflects task formatting rather than hierarchical navigation, and the paper's claim that "RL primarily transforms how models traverse knowledge" would be over-specific.

The CoT step-wise probing (Appendix C.2) partially addresses this concern by showing that divergence grows with CoT depth. If the divergence were purely task-formatting, it might appear early and remain static rather than growing through the hierarchy. But the CoT analysis uses constructed prompts that include the phrase "hmm let me think" at each step, which itself may trigger different processing in base versus specialized models (base models may not have been trained to produce this style of internal monologue). The progressive divergence could reflect progressive divergence in style processing rather than hierarchical depth.

The absolute cosine similarity values are hard to interpret. A cosine similarity of 0.65–0.73 in middle layers for Q-Q comparisons represents "substantial divergence" relative to the 0.85–0.92 for A-A comparisons, but whether 0.65 represents "the models are doing fundamentally different things" versus "the models are doing similar things with modest variation" is not established. Without a baseline for what cosine similarity values are typical for unrelated representations (e.g., random vector pairs, or representations of completely different inputs), the magnitude of the divergence is hard to calibrate. The L2 distance results in Appendix C.2 provide a complementary metric but face the same calibration challenge.

The final-token representation extraction method may systematically bias the comparison. Questions and answers have different surface forms—questions end with a question mark after an interrogative structure; answers end with a period after a declarative structure. The final token's hidden state may differentially weight information near the end of the sequence (due to recency bias in attention), meaning the Q-Q comparison is comparing representations that are heavily influenced by "?" while the A-A comparison is comparing representations influenced by the final words of the declarative statement. This confound could produce an artifactual Q-Q/A-A divergence asymmetry even if the underlying processing of factual content is identical. The paper does not discuss or control for this potential confound, such as by comparing mean-pooled representations across all tokens or by analyzing representations at the position of the key factual content (e.g., the code string "57.95") rather than the final token.

The analysis aggregates across 100 probes per vocabulary, masking per-question variation. The paper reports average cosine similarity across all 100 probes per vocabulary, but individual questions within a vocabulary vary in retrieval difficulty (some codes are more deeply nested than others, some have more ambiguous descriptions). If the navigation hypothesis is correct, Q-Q divergence should be larger for harder questions (deeper nesting, more ambiguous options). Analyzing the correlation between per-question difficulty and per-question Q-Q divergence would provide stronger, more fine-grained evidence for the navigation hypothesis than the aggregate analysis. The paper does not perform this analysis.

The distilled model pattern is important but under-explained. Figure 3d–f shows that the distilled model (DeepSeek-R1-Distill-Qwen-32B) exhibits the most extreme Q-Q divergence from the base model—more than either the instruct or reasoning models. Yet the distilled model performs worst behaviorally (Table 4) and shows minimal benefit from structured prompting. The paper interprets this as evidence that distillation causes representational disruption without conferring navigational capability, but this interpretation is post-hoc. It is equally consistent with the possibility that the Q-Q divergence metric does not specifically measure navigation-related processing changes—since if it did, the model with the most divergence should (all else equal) show the most navigational capability, not the least. This pattern is actually a challenge to the paper's interpretation of what Q-Q divergence means, and it deserves more discussion than the paper provides.

Does the paper adequately rule out alternative explanations?

Several alternative explanations for the main findings are not directly tested:

Surface-form memorization rather than hierarchical navigation. The improvement from structured prompting could reflect the model following the template's format instructions ("recall the general category and structure breakdown") and producing text that looks like hierarchical navigation without actually using hierarchical structure to constrain answer selection. The paper's only evidence against this is the path matching score on the IPC dataset, which shows that R1's internal traversal is genuinely more accurate than V3's. But this evidence comes from a single model pair on a single dataset. A stronger test would be to construct adversarial distractors that share surface features with correct answers but are taxonomically distant—if structured prompting helps the model reject these distractors more effectively than direct QA, that would be evidence for genuine hierarchical constraint use rather than pattern matching within the structured output format.

Calibration and confidence rather than knowledge access. The majority voting protocol (three runs) means that some of the gap between models could reflect differences in response consistency rather than differences in knowledge. If the reasoning model produces the same answer more reliably across runs while the instruct model's answers vary more, majority vote accuracy will favor the reasoning model even if per-run accuracy is similar. The paper reports both majority vote and mean accuracy precisely to address this concern, and the gap patterns are similar across both metrics (Tables 2–4), suggesting that consistency differences are not the primary driver. However, Figure 2 shows that R1's distribution across consistency categories is "static" across prompt templates—it already achieves high consistency—while V3's distribution shifts from inconsistent to consistent with structured prompting. This suggests that part of what RL does is improve response consistency on knowledge retrieval tasks, which is conceptually related to but distinct from improving hierarchical navigation per se.

Selection bias in model families. The paper evaluates models that are publicly available and well-documented (Qwen2.5, DeepSeek-V3/R1, Mistral), but these models were selected for study because they represent prominent reasoning-enhanced model pairs. It is possible that other model families—particularly those where the reasoning enhancement was less successful or used different RL algorithms—would not show the same structured prompting closure or representational divergence patterns. The paper's findings may reflect the specific RL recipe used for DeepSeek-R1 and QwQ rather than a general property of RL. Testing on a broader set of RL-enhanced models, including those trained with different algorithms (PPO, DPO, GRPO) and different reward structures, would strengthen the generality claim.

What experiments are missing that would strengthen the paper?

Continuous traversal depth analysis for path matching. Rather than binary Memory-Light/Memory-Heavy stratification, a scatter plot of path matching score gap versus traversal depth (one point per question, perhaps 50–100 questions spanning a range of depths) would provide much stronger evidence that the RL advantage monotonically increases with depth. This would also reveal whether the effect saturates at some depth or continues to grow.

Path matching scores under structured prompting. The most important missing experiment: apply structured prompting to both V3 and R1 on the stratified IPC dataset and measure path matching scores. If structured prompting closes the path matching gap (not just the accuracy gap), the paper's claim about RL's unique contribution narrows. If it doesn't, the claim is substantially strengthened.

Adversarial distractor construction for MedConceptsQA. Modify the MedConceptsQA dataset to include a condition where incorrect options are taxonomically distant from the correct answer (easy to reject via hierarchical reasoning) versus taxonomically close (require fine-grained hierarchical discrimination). If structured prompting provides more benefit in the taxonomically-close condition, that would be evidence for genuine hierarchical navigation rather than format-following.

Token-position controls for representation analysis. Compare final-token representations to mean-pooled representations or to representations at the position of the key entity (the medical code string) to test whether the Q-Q/A-A asymmetry is robust to the token position chosen for extraction, or whether it reflects recency-bias confounds.

Per-question difficulty correlation with Q-Q divergence. Compute per-question difficulty (e.g., pass@1 rate of the base model, or structural depth of the code in the taxonomy) and correlate with per-question Q-Q cosine similarity between base and specialized models. A negative correlation (harder questions show greater Q-Q divergence) would support the navigation hypothesis. The paper's aggregate analysis cannot test this prediction.

Direct probing for factual knowledge presence. Use a linear probe trained on base model representations to decode the correct answer from the specialized model's representations at various layers. If the factual content is present in the specialized model but inaccessible through default generation, a probe should be able to decode it with high accuracy even when the model's text output is incorrect. This would provide convergent evidence with the structured prompting results for the claim that knowledge is present but access-limited.

Human evaluation of traversal quality. Validate the path matching score by having human experts rate the quality of model-generated hierarchical traversals (for a sample of IPC questions) and correlating those ratings with path matching scores. This would establish the metric's validity as a measure of "good navigation" rather than just "matching the ground truth path format."

In summary, the paper's experiments provide substantial but incomplete support for its central claims. The structured prompting results convincingly demonstrate that much of the RL advantage on MedConceptsQA reflects access rather than knowledge differences, but the residual gap and the reversal cases complicate a clean "RL only changes navigation" interpretation. The path matching results provide genuine evidence for navigational improvement on deep tasks but come from a narrow empirical base (one model pair, one dataset). The representation analysis elegantly isolates query processing from content storage, but alternative interpretations (task-formatting, recency bias, the distilled model's anomalous pattern) are not ruled out. The paper's claims would be strengthened by additional experiments that test fine-grained predictions of the navigation hypothesis (correlation with depth, adversarial distractors, structured prompting path matching) and by broader model and dataset coverage for the path matching and representation analyses.

6. Limitations and Trade-offs

6.1 Retrieval Complexity Analysis Is Limited to a Single Model Pair

The assumption or constraint. The paper's claim that RL models demonstrate superior hierarchical navigation on deep-retrieval tasks—one of the paper's four headline innovations—rests entirely on the stratified IPC analysis comparing DeepSeek-R1 and DeepSeek-V3. The paper acknowledges this explicitly: "While a broader evaluation would be ideal, we select the DeepSeek pair due to their instruction-following capabilities suitable for a reliable analysis" (Section 2.2). No other model pairs (Qwen2.5-32B vs. QwQ-32B, Qwen3-235B instruct vs. reasoning, Mistral instruct vs. Magistral) are evaluated on the stratified IPC dataset or with the path matching score.

The consequence. The finding that the path matching score gap widens from 0.054 (Memory-Light) to 0.094 (Memory-Heavy) while accuracy remains identical at 67.7% (Table 5) is a clean, interpretable result—but we cannot know whether it reflects a general property of RL-enhanced models or a peculiarity of the DeepSeek-R1 training recipe. DeepSeek-R1 was trained with a specific large-scale RL procedure (Guo et al., 2025a) that may differ substantially from the training of QwQ-32B or Magistral. If the path matching advantage does not replicate across other reasoning models, the paper's claim that "reasoning models demonstrate superior path recall accuracy" (Section 1) would be over-generalized. Conversely, if the finding does replicate, it would substantially strengthen the paper's central argument. The current single-pair evidence cannot distinguish these cases.

What evidence exists in the paper. Only Table 5, comparing DeepSeek-R1 and DeepSeek-V3 on the expanded IPC dataset at two complexity levels. No other model pairs are tested on this dataset or with this metric. The structured prompting experiments (Tables 2–4) involve multiple model families but test only final-answer accuracy on MedConceptsQA and the base IPC dataset, not path matching quality on deep-retrieval tasks. The representation analysis (Figure 3) involves the Qwen2.5-32B family and Mistral-Small-24B family (Appendix C) but does not include any task that measures traversal quality directly.

Mitigation status. The paper partially acknowledges this by framing the analysis as a "case study" (Section 2.2), but the limitation is not discussed in the limitations or conclusion sections. The abstract and introduction state the finding about superior path traversal on deep-retrieval tasks without qualification about scope. No future work is suggested to replicate the stratified analysis across model families. A practitioner deciding whether to invest in RL training for deep-retrieval applications would need to verify this finding on their own model family, since the paper provides evidence from only one pair.

---

6.2 Difficulty Estimation Cost Is Absent from the Framework

The assumption or constraint. The paper's core framework—that we can disentangle knowledge storage from knowledge access by testing whether structured prompting recovers RL-level performance—operates entirely in a zero-shot prompting paradigm. The structured prompt template is hand-crafted based on human knowledge of the medical and patent code hierarchies. The paper provides no method for automatically constructing structured prompts for new domains, measuring how much human effort is required, or estimating whether such prompts can be reliably constructed without domain expertise. This is distinct from the prompting results themselves (which demonstrate a scientific point); it is about the practical deployability of structured prompting as a solution rather than a diagnostic tool.

The consequence. The paper demonstrates that structured prompting can recover ~68% of the performance gap on MedConceptsQA and IPC—but someone who wants to apply this insight to a new domain (legal code systems, biological taxonomies, software package hierarchies) must manually design a structured prompt template that encodes the relevant hierarchical traversal strategy for that domain. The paper provides no guidance on how difficult this is, how sensitive the results are to prompt quality, or whether suboptimal structured prompts (written by non-experts) still provide benefits. The headline finding—that structured prompting nearly closes the gap—may not generalize to domains where the hierarchy is less well-understood, more complex, or poorly documented, because the human prompt designer cannot encode traversal strategies they do not themselves understand.

Additionally, the paper's three-template gradient (Direct QA → CoT → Structured) was designed by the authors with full knowledge of the paper's hypothesis and the hierarchical structure of the target domains. This introduces an experimenter-degree-of-freedom concern: the authors could iterate on the structured prompt until it maximized the gap reduction, without reporting how many prompt variants were tested or how sensitive results are to prompt phrasing. If the reported 68% gap reduction reflects optimization over multiple prompt candidates, it may overstate what a practitioner would achieve with a single attempt at structured prompt design.

What evidence exists in the paper. None directly. The structured prompt template is provided in full (Appendix B.1), demonstrating that the authors are transparent about what prompt was used. But there are no ablations on prompt variants (e.g., does "recall the general category and structure breakdown" produce different results from "identify the chapter and subchapter of this code"?), no comparison of expert-written versus novice-written structured prompts, and no measurement of how much domain expertise was required to construct the template. The paper does not claim that structured prompting is a practical deployment solution—it uses it as a diagnostic tool—but the practical implications are left entirely to the reader's extrapolation.

Mitigation status. Not addressed. The paper presents structured prompting as an experimental apparatus for testing the navigation hypothesis, not as a proposed method for deployment. But Section 5's concluding paragraph frames the findings as having implications for "practical deployment" without discussing the cost of prompt engineering. The paper could have strengthened its practical relevance by including an ablation with a deliberately suboptimal structured prompt (e.g., one that instructs hierarchical recall but in the wrong order, or one that omits the elimination step) to measure how much the specific prompt design matters. The lack of such ablations means the robustness of structured prompting to prompt quality is entirely unknown.

---

6.3 The Representation Analysis Confounds Task Formatting with Hierarchical Navigation

The assumption or constraint. The paper's representation analysis (Section 2.3, Figure 3) compares interrogative questions ("What is code 57.95?") to declarative answers ("Code 57.95 refers to..."). The finding that question representations diverge more than answer representations between base and specialized models is interpreted as evidence that "RL transforms query processing while preserving factual representations" (Section 3.3). This interpretation assumes that the Q-Q divergence specifically reflects changes in hierarchical navigation processing rather than changes in task formatting or instruction-following circuitry.

The consequence. If the Q-Q divergence reflects task formatting (the specialized model processes the question through an "instruction-following" circuit that the base model lacks) rather than hierarchical navigation specifically, then the representational evidence does not uniquely support the navigation hypothesis. It would support a weaker claim: post-training changes how models process queries, and some of this change may be navigation-related, but the representational analysis cannot distinguish navigational processing changes from general instruction-tuning effects on query encoding. This would not invalidate the paper's overall argument (the behavioral prompting and path matching results provide independent evidence for the navigation hypothesis), but it would weaken the claim that the representation analysis provides "direct mechanistic evidence" for navigation specifically, as opposed to task-format processing generally.

A related confound: the interrogative queries (ending with "?") and declarative answers (ending with ".") have systematically different surface forms at the final token position. Since the paper extracts representations at the final token position, the Q-Q comparison compares representations influenced by "?" while the A-A comparison compares representations influenced by the final words of a declarative sentence. If post-training changes how the model processes punctuation or sentence-final position differently from how it processes declarative content, the Q-Q/A-A asymmetry could partly reflect this surface-form confound rather than genuine differences in how factual content versus retrieval demands are processed.

What evidence exists in the paper. The CoT step-wise probing (Appendix C.2, Figures 6 and 8) partially addresses the task-formatting confound by showing that divergence between base and specialized models grows as the CoT prompt expands through deeper hierarchical levels. If divergence were purely task-formatting (a static "am I in instruction-following mode?" signal), it should appear early and remain constant. The progressive divergence through the hierarchy is more consistent with navigational processing changes. However, the CoT prompts include the phrase "hmm let me think" at each step—a stylistic marker that may trigger different processing in base versus instruct/RL models independently of hierarchical depth. The paper does not control for this by testing CoT prompts without the "hmm let me think" framing.

Regarding the surface-form confound, the paper provides no control analysis: no comparison using mean-pooled representations (across all tokens, reducing final-token bias), no comparison at the token position of the key entity (the code string "57.95" rather than the final punctuation), and no control experiment using declarative queries (e.g., "Identify the description of code 57.95." ending with a period rather than a question mark) to test whether the interrogative/declarative surface form drives the divergence asymmetry.

Mitigation status. Partially addressed through the CoT step-wise analysis, but the surface-form confound is not discussed or controlled. The paper would be strengthened by a token-position ablation (comparing final-token to mean-pooled or entity-position representations) and by a declarative-query control experiment. Without these, the representation analysis provides suggestive but not definitive evidence that the Q-Q divergence specifically reflects navigational processing changes.

---

6.4 The Distilled Model Exception Challenges the Navigation-Only Interpretation

The assumption or constraint. The paper's central claim is that RL improves performance on hierarchical knowledge tasks primarily by enhancing navigation of existing knowledge, not by adding new knowledge or modifying knowledge representations. Under this claim, behavioral performance should correlate with navigational capability: models that navigate better should perform better, and models whose internal query representations have been substantially modified (presumably reflecting navigation-enhancing changes) should show improved performance.

The consequence. The distilled models break this expected correlation. DeepSeek-R1-Distill-Qwen-32B shows the most extreme Q-Q divergence from the base model among all specialized variants (Figure 3d–f)—more divergence than either the instruction-tuned or reasoning models. If Q-Q divergence reflects navigation-related query processing changes, the distilled model should show the strongest navigational capability. Yet behaviorally, the distilled models show the worst performance (Table 4), with gaps to the full R1 model of +0.345 to +0.534 majority vote accuracy on MedConceptsQA, and structured prompting provides minimal or inconsistent benefits. This is a direct challenge to the interpretation that Q-Q divergence specifically measures navigation-enhancing representational change. The distilled model has undergone substantial representational reorganization (diverging from the base model's query representations more than any other variant) but this reorganization either does not confer navigational capability, or actively disrupts whatever navigational capability existed in the base or instruct model.

This pattern admits several interpretations, none of which the paper resolves. It could mean that Q-Q divergence measures representational disruption in general, not navigation specifically—distillation from a much larger reasoning model into a smaller architecture causes representational changes, but these changes are disorganized rather than functionally adaptive. It could mean that navigation requires a specific kind of representational change that the RL process produces but distillation fails to capture, and Q-Q divergence is a necessary but not sufficient condition for navigational improvement. Or it could mean that distillation degrades factual knowledge (the A-A representations for the distilled model also diverge somewhat from the base model in Figure 3, though less than Q-Q), and the behavioral deficit reflects knowledge loss rather than or in addition to navigation failure. The paper's framework does not cleanly distinguish these explanations.

What evidence exists in the paper. Table 4 shows the behavioral deficit of distilled models relative to R1. Figure 3d–f shows the representational divergence patterns for the distilled model on the ICD9PROC vocabulary (the pattern is replicated across all five vocabularies in Appendix C, Figure 8). The paper notes that "distilled models capture only surface-level improvements without acquiring robust navigation capabilities" (Section 1), but this explanation is post-hoc and does not engage with the tension that the representation analysis—if interpreted as measuring navigation—should predict the opposite.

Mitigation status. The paper acknowledges the distilled model limitation in the abstract and introduction: "distilled models capture only surface-level improvements without acquiring robust navigation capabilities—achieving intermediate performance on complex retrieval tasks. Structured prompting provides minimal gains for distilled models, and layer-wise analysis reveals greater representational changes than instruction-tuned variants, yet without improved deep-retrieval navigation." But this descriptive summary does not address the interpretive challenge that the distilled model pattern poses for the paper's framework. The paper does not suggest why distillation produces representational divergence without navigational capability, nor does it propose experiments to distinguish competing explanations. For a practitioner deciding between RL training and distillation from a reasoning model, this is a critical finding that the paper reports but does not adequately explain.

---

6.5 The Generalization Boundary to Non-Hierarchical Tasks Is Undefined

The assumption or constraint. The paper demonstrates that RL-enhanced models outperform instruction-tuned models on structured, hierarchical knowledge tasks (medical code lookup, patent classification) and that this advantage stems from improved hierarchical navigation. The paper explicitly contrasts this with prior work on flat factual recall where RL has been shown to degrade performance (the alignment tax literature). However, the paper does not define or test the boundary between these two regimes. At what point does a task become "hierarchical enough" for RL to provide a benefit rather than a cost? Is it the presence of any taxonomic structure, or is there a minimum depth/complexity threshold? Does the benefit appear for weakly hierarchical structures (e.g., two-level categorizations) or only for deeply nested ones (5+ levels)?

The consequence. Without a characterization of the generalization boundary, a practitioner cannot determine whether their specific task distribution will benefit from RL-enhanced models or suffer from the alignment tax. The paper's findings suggest that for medical coding and patent classification—both deeply hierarchical domains with well-defined taxonomies—RL provides substantial benefits recoverable through structured prompting. But what about tasks that are moderately structured: product categorization (hierarchical but often ambiguous), legal document classification (hierarchical but with overlapping categories), or biological species identification (taxonomic but with fuzzy boundaries)? The paper provides no framework for predicting when the navigational benefit will appear, beyond the general principle that "deeper traversals show larger RL advantages" (Table 5). The Memory-Light/Memory-Heavy stratification on IPC (fewer than 3 vs. 5+ traversals) is a start, but it is specific to one dataset and one structural property (ancestor count). It does not address other forms of hierarchical complexity: branching factor, depth versus breadth, ambiguity in node assignment, or cases where the taxonomy is implicit rather than explicitly documented.

The paper's opening move—contrasting hierarchical retrieval with flat factual recall—implies that the distinction matters, but the paper provides no methodology for classifying a new task into the "RL helps" versus "RL hurts" regime short of running the full experimental battery (structured prompting, path matching, representation analysis) on that specific task.

What evidence exists in the paper. Table 5 provides the only direct evidence about how the RL advantage varies with structural depth (wider path matching gap on Memory-Heavy vs. Memory-Light tasks). The MedConceptsQA results (Tables 2–3) show that structured prompting reduces the gap across all medical vocabularies, suggesting the benefit is not limited to a single hierarchy type. But neither dataset includes shallow-hierarchy or non-hierarchical control conditions that would establish the boundary. The paper cites prior work on flat factual recall degradation (Lin et al., 2024; Yuan et al., 2024; Gekhman et al., 2024) as context but does not test any flat factual recall tasks in its own experimental framework to directly compare the RL effect on hierarchical versus flat retrieval within the same models and methodology.

Mitigation status. Not addressed. The paper acknowledges in the conclusion that future work should investigate "how these findings extend to knowledge domains with different structural properties—flat versus deeply nested hierarchies, dense versus sparse connectivity" (Section 5), but this is listed as an open question rather than a limitation of the current work. The paper could have strengthened its claims by including a flat factual recall baseline (e.g., a subset of MMLU factual questions) to demonstrate that the same RL models that outperform on hierarchical tasks do in fact underperform on flat tasks—providing direct evidence for the claimed dissociation rather than relying on cross-paper comparisons with different models and methodologies.

---

6.6 The Distinction Between Consistency and Capability Is Not Disentangled

The assumption or constraint. The paper uses majority voting over three independent runs to measure accuracy, which combines two distinct signals: whether the model can produce the correct answer (capability, reflected in per-run accuracy) and whether the model reliably produces the same answer across runs (consistency). A model could improve in majority vote accuracy by becoming more consistent (producing the same answer more often, making it more likely to achieve a majority) without becoming more capable (the answer it consistently produces might be wrong). The paper's hypothesis—that RL improves hierarchical navigation—predicts a capability improvement (the model navigates more accurately and therefore produces correct answers more often). But some of the observed gap between reasoning and instruct models could reflect improved consistency rather than improved navigation per se.

The consequence. If part of the RL advantage reflects improved consistency (the model settles on an answer and sticks with it across runs) rather than improved accuracy of the underlying retrieval process, then the paper's interpretation of the behavioral results overstates the navigational benefit. The Figure 2 analysis—showing that R1's distribution across consistency categories is "static" while V3's shifts with prompting—supports the capability interpretation (structured prompting causes V3 to produce correct answers on questions where it previously produced only incorrect answers). But the fact that R1's distribution is already concentrated in the "All Correct" and "Majority Correct" categories from the baseline QA prompt means we cannot observe whether R1 would also show migration from incorrect to correct categories if tested in a regime where it has room to improve. The ceiling effect for R1 (already performing near its maximum) masks whether RL's benefit is partially driven by consistency improvements that happen to align with correctness on this particular dataset.

More importantly, the majority voting protocol (3 runs) makes it impossible to distinguish the following two scenarios: (a) the instruct model has a 60% chance of producing the correct answer on each run, while the reasoning model has an 80% chance (capability difference), versus (b) both models have a 70% chance per run, but the instruct model's runs are anti-correlated (if one run is wrong, another is likely right) while the reasoning model's runs are positively correlated (if one run is right, they tend to all be right, and vice versa). Both scenarios could produce the same majority vote patterns but have very different implications for what RL is doing. The paper's representational analysis partially addresses this by showing internal processing differences, but the behavioral metrics alone cannot cleanly separate capability from consistency.

What evidence exists in the paper. The paper reports both mean accuracy (per-run average) and majority vote accuracy in all tables (2–4). The gap patterns are directionally similar across both metrics—for DeepSeek-R1 vs. V3 on MedConceptsQA QA prompting, the mean accuracy gap is +0.279 and the majority vote gap is +0.237, both large—suggesting that consistency differences are not the primary driver. However, the standard deviations on mean accuracy are non-trivial (e.g., DeepSeek-V3 shows 0.551 ± 0.014, while DeepSeek-R1 shows 0.830 ± 0.006 under QA prompting), indicating that R1 is indeed more consistent run-to-run. The question is whether this consistency improvement is a separable effect (RL makes models more deterministic in their answer selection, independently of whether the answer is correct) or an integral part of improved navigation (better navigation leads to higher confidence, which manifests as greater consistency). The paper does not provide the per-question correlation structure that would be needed to distinguish these accounts.

Mitigation status. Partially addressed by reporting both mean and majority vote accuracy, and by the Figure 2 consistency category breakdown. But the paper does not discuss the capability-versus-consistency distinction explicitly, does not report metrics that would disentangle them (e.g., per-question probability that the model's most likely answer is correct, independent of sampling stochasticity), and does not analyze whether the consistency improvement generalizes across difficulty levels or is concentrated on easy questions where the model is already correct. A practitioner deploying these models with greedy decoding (temperature 0) would not observe consistency effects at all, making it important to know how much of the reported gap would persist under deterministic decoding—a question the paper's sampling protocol (temperature 0.8) cannot answer.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture, training algorithm, or benchmark. Its contribution is conceptual and diagnostic: it provides a framework and experimental apparatus for distinguishing whether post-training (RL, SFT, distillation) modifies what a model knows versus how a model accesses what it knows. This is not a paradigm shift—the paper does not overturn the alignment tax literature—but it is a substantive reframing that changes how the field should interpret performance differences between base, instruction-tuned, and reasoning-enhanced models on knowledge-intensive tasks.

The reframing: from "knowledge erasure" to "access deficit." The dominant narrative in the alignment tax literature (Lin et al., 2024; Yuan et al., 2024; Gekhman et al., 2024) has been that post-training—particularly RLHF and reasoning-focused RL—degrades factual knowledge. The implicit model treats parametric knowledge as a static resource that post-training either preserves or overwrites. This paper demonstrates that for hierarchical knowledge tasks, this model is incorrect. The structured prompting result—reducing the 24pp gap between DeepSeek-R1 and DeepSeek-V3 on MedConceptsQA to 7pp without any parameter updates—is direct evidence that the knowledge was present in the instruction-tuned model but inaccessible through its default retrieval strategy. The alignment tax, at least for structured knowledge, is not primarily about losing facts; it is about losing the procedural skill to navigate to those facts.

This reframing has substantial practical consequences. If the problem is knowledge erasure, the solution is preservation: you need replay, regularization, or knowledge distillation during post-training to prevent factual degradation. If the problem is access, the solution is organization: you need training procedures or prompting strategies that teach or externalize systematic retrieval. The paper's findings suggest the latter approach is viable—structured prompting recovers 68% of the performance gap for the largest model pair—meaning that some of the apparent "cost" of post-training can be mitigated without modifying the training pipeline at all. Organizations that have already deployed instruction-tuned models and observe poor performance on structured knowledge tasks may be able to recover substantial accuracy through prompt engineering, without retraining.

What the paper resolves. The paper provides a unified explanation for seemingly contradictory prior findings. The alignment tax literature (showing post-training hurts factual accuracy) primarily tested flat, single-hop factual recall—tasks where the retrieval demand is minimal and direct pattern matching suffices. The reasoning-enhancement literature (showing RL improves reasoning) tested multi-step tasks requiring structured exploration of knowledge. The paper demonstrates that the same RL procedure can produce both outcomes depending on the retrieval demands of the evaluation: degrading shallow, pattern-matching access while improving deep, structured traversal. The contradiction was never about RL being "good" or "bad" for knowledge—it was about what kind of access the evaluation tasks demanded. This resolution does not make prior findings wrong; it makes them incomplete by identifying retrieval structure as a critical moderating variable that prior work did not measure or control.

What the paper does not resolve. The generalization boundary between "RL helps" and "RL hurts" is not characterized. The paper shows benefits on deeply hierarchical domains (medical coding, patent classification) and cites prior work showing costs on flat factual recall, but provides no methodology for predicting where a new task falls on this spectrum without running the full experimental battery. The Memory-Light/Memory-Heavy stratification (Table 5) is a proof of concept for complexity-dependent analysis but comes from a single model pair and a single structural property (ancestor count). Until the retrieval-complexity-dependence is replicated across more model families and structural properties (branching factor, depth vs. breadth, taxonomy ambiguity), the paper's central reframing remains empirically suggestive but not operationally robust.

Which research directions become more attractive, and which become less so. The paper makes mechanistic studies of retrieval processes significantly more attractive by demonstrating that contrastive probe design (matched question-answer pairs with factorial manipulation of processing demand) can isolate navigation from content in internal representations. The Q-Q versus A-A divergence asymmetry (Figure 3) provides a template for asking "did post-training change access or content?" that can be applied to other post-training methods, other domains, and other model families. The paper also makes prompt optimization for knowledge access more attractive as a complement or alternative to retraining—if structured prompting can recover 68% of RL's benefit, systematic prompt search (using DSPy, GEPA, or related frameworks) might recover even more, potentially making expensive RL pipelines unnecessary for knowledge-access improvements in some deployment contexts.

Conversely, the paper makes distillation from reasoning models as a substitute for RL training less attractive. The distilled models (Table 4) show large, persistent gaps to the full R1 model, minimal benefit from structured prompting, and the most extreme representational divergence from the base model (Figure 3d–f) without corresponding navigational capability. This triple dissociation—representational change without functional improvement—suggests that distillation captures surface-level output patterns of reasoning models without transferring the internal navigational processes that make those outputs correct. For practitioners considering whether to distill from a large reasoning model versus training their own RL pipeline, this is a negative result with practical consequences: distillation may be cheaper but appears to leave the core navigational deficit unaddressed.

---

Follow-Up Research This Work Enables

Replicate the path matching score gap across reasoning model families. The paper's finding that DeepSeek-R1 achieves a path matching score of 0.597 versus DeepSeek-V3's 0.503 on Memory-Heavy IPC tasks—a 0.094 gap—while accuracy remains identical at 67.7% is the key evidence that RL provides a genuine, non-promptable navigation advantage on deep retrieval. But this evidence comes from a single model pair. A strong follow-up would replicate the stratified IPC analysis on QwQ-32B, Magistral, and the Qwen3-235B reasoning variant, computing both accuracy and path matching scores at multiple traversal depths. If the path matching gap widens with depth across all reasoning models, the navigation hypothesis generalizes and becomes a robust principle of RL's effect. If the gap appears only for DeepSeek-R1 (which used a specific large-scale RL recipe with process rewards), the effect is training-procedure-specific rather than RL-general. A negative result—other reasoning models show no path matching advantage or a much smaller one—would substantially narrow the paper's claims and redirect attention to what specific components of R1's training produce navigational improvement.

Continuous traversal depth analysis with per-question difficulty correlation. The binary Memory-Light/Memory-Heavy stratification in Table 5 is a coarse first pass. A more fine-grained analysis would construct 50–100 IPC common-ancestor questions spanning a continuous range of traversal depths (from 1 to 10+ ancestors recalled) and plot the path matching score gap between reasoning and instruct models as a function of depth, with one data point per question. This would reveal whether the gap grows monotonically, saturates at some depth, or shows threshold effects (e.g., no gap below 3 traversals, sharp gap emergence at 4–5, plateauing beyond 7). It would also enable per-question correlation between structural depth and Q-Q representational divergence (using the layer-wise probing methodology from Section 2.3 applied to IPC rather than MedConceptsQA)—a direct test of whether deeper traversals produce larger query-processing divergence between models.

Structured prompting with path matching measurement. The most critical missing experiment: apply structured prompting to DeepSeek-V3 on the stratified IPC dataset and measure the resulting path matching scores, not just final-answer accuracy. If structured prompting brings V3's path matching score close to R1's (narrowing or eliminating the 0.094 gap on Memory-Heavy tasks), then the navigational capability is fully externalizable—structured prompting can substitute for RL even on deep retrieval, and RL's unique contribution is limited to automating what prompting can achieve. If structured prompting improves V3's accuracy but leaves its path matching score far below R1's, then RL provides a genuine navigational capability that external scaffolding cannot replicate—a much stronger and more interesting claim with direct implications for whether RL training is necessary for complex knowledge access tasks. This experiment requires minimal additional infrastructure (the IPC dataset, prompt templates, and path matching scoring code already exist) and would substantially sharpen the paper's central claim.

Adversarial distractor construction to test whether structured prompting genuinely induces hierarchical constraint use. An alternative explanation for the structured prompting results is that the template's instructions ("recall the general category and structure breakdown of the medical code") cause the model to produce text that looks like hierarchical navigation without actually using hierarchical structure to constrain answer selection—the model may still be pattern-matching against surface features, now expressed in a more elaborate format. This can be tested by constructing an adversarial version of MedConceptsQA where incorrect options are systematically manipulated: in the "taxonomically-distant" condition, all distractors come from different major chapters of the ICD hierarchy than the correct answer, making hierarchical reasoning easy; in the "taxonomically-close" condition, distractors come from the same subchapter as the correct answer, requiring fine-grained hierarchical discrimination. If structured prompting provides substantially more benefit in the taxonomically-close condition (where pattern matching fails and hierarchical reasoning is necessary), that is evidence for genuine navigational constraint. If structured prompting provides similar benefit in both conditions, the model may be following the format without leveraging the structure. This experiment directly tests the mechanism the paper claims, not just the outcome.

Token-position ablation for the Q-Q versus A-A representational divergence asymmetry. The paper's representation analysis extracts hidden states at the final token position, which may introduce recency-bias confounds: questions end with "?" while answers end with the final words of a declarative sentence. A targeted follow-up would compare three extraction methods on the same MedConceptsQA probe set: (1) final-token representation (the paper's current method), (2) mean-pooled representation across all tokens, and (3) representation at the token position of the key medical code string (e.g., "57.95"). If the Q-Q divergence exceeding A-A divergence pattern is robust across all three extraction methods, the finding is not an artifact of final-token recency bias. If the asymmetry disappears or substantially attenuates under mean-pooling or entity-position extraction, the paper's representational evidence for the navigation hypothesis is weaker than claimed, and future work using this methodology should adopt extraction methods that control for surface-form confounds.

Linear probing for factual knowledge in specialized model representations. The paper claims that factual knowledge exists in instruction-tuned models but is inaccessible without proper navigation (based on the structured prompting gap closure). A convergent test would train a linear classifier on the base model's layer-wise representations to decode the correct answer from the hidden states (using the declarative answer format, which avoids the retrieval demand). Then probe the instruction-tuned and reasoning model representations with this same classifier at each layer. If the knowledge is present but access-limited, the probe should decode the correct answer from the specialized models' hidden states with high accuracy even on questions where the model's text output is incorrect. If the probe's decoding accuracy tracks the model's text-output accuracy (low when the model answers incorrectly), the knowledge may genuinely be absent or degraded rather than merely inaccessible. This experiment would provide independent, mechanistic evidence for the storage-vs-access distinction that does not rely on prompting interventions and cannot be explained by format-following artifacts. It requires only the MedConceptsQA dataset, the models already analyzed, and standard probing methodology—no new data collection.

RL objectives that explicitly optimize for hierarchical navigation. The paper demonstrates that structured navigation emerges from general reasoning-focused RL (Guo et al., 2025b) but does not test whether RL objectives designed to explicitly reward hierarchical traversal produce stronger, more robust navigational capabilities. A natural follow-up would design a training procedure where the reward signal includes a component for correct hierarchical path recall (not just final-answer correctness), using the IPC dataset where ground-truth ancestor paths are known. Train a model with this hierarchical-path reward and compare to standard outcome-only RL on both the MedConceptsQA and IPC benchmarks. If path-aware RL produces larger path matching score improvements, deeper generalization to unseen hierarchy depths, or reduced degradation on flat factual recall (because the model learns to switch between direct recall and hierarchical traversal based on query structure), this would demonstrate that navigation can be explicitly optimized rather than relying on implicit emergence—with practical implications for post-training pipeline design. The paper provides the measurement infrastructure (path matching score, stratified complexity analysis) that makes this experiment evaluable.

---

Practical Applications and Downstream Use Cases

Medical coding and clinical documentation systems. The most direct application is in clinical settings where ICD code assignment from clinical notes is a high-volume, high-stakes task. The paper's results—particularly the 24pp gap between DeepSeek-R1 and DeepSeek-V3 on MedConceptsQA under direct QA prompting—suggest that deploying a reasoning-enhanced model for medical code lookup can dramatically reduce error rates compared to instruction-tuned alternatives. The structured prompting finding (reducing the gap to 7pp) provides a lower-cost deployment path: rather than running the full 671B R1 model (expensive at inference time), a clinical coding system could use the smaller DeepSeek-V3 with a structured prompt template that encodes the ICD hierarchy traversal strategy. This template could be integrated directly into the electronic health record interface—when a clinician queries a code, the system formats the query with hierarchical decomposition instructions before sending it to the model, transparent to the user. The 68% gap recovery from structured prompting means the prompted V3 system would achieve approximately 71.7% accuracy (from Table 2, V3 Structured majority vote) versus R1's 79.2%—trading a modest accuracy reduction for substantial inference cost savings, with the tradeoff adjustable based on the clinical stakes of each query type.

Patent prior art search and classification. The IPC dataset results directly apply to patent office workflows where incoming applications must be classified into the hierarchical IPC taxonomy for prior art search. The paper's finding that reasoning models achieve a path matching score of 0.597 on deep-retrieval tasks (5+ traversals) versus 0.503 for the instruction-tuned baseline (Table 5) means that RL-enhanced models are not just picking the correct category more often—they are correctly navigating the full hierarchical path at a substantially higher rate. This matters for patent examination because correct classification requires understanding why a category applies (the hierarchical path), not just which category; an examiner using the model's output to guide their search needs to trust that the hierarchical reasoning is sound. The stratified IPC dataset, with its common-ancestor task design, could be adapted into a production classification verification tool: when a model proposes a classification, the system also outputs the predicted hierarchical path, and a human examiner reviews flagged cases where the path matching score is low relative to the confidence—indicating the model might have guessed the right answer through the wrong reasoning.

Knowledge-intensive enterprise search with structured prompt templates. Many enterprise knowledge bases are organized hierarchically: product catalogs with category/subcategory structures, legal document repositories organized by jurisdiction/statute/section, internal wikis with topic trees. The paper's core finding—that structured prompting can recover 68% of RL's performance advantage without retraining—suggests a deployment pattern where organizations integrate structured prompt templates into their internal search interfaces rather than training or fine-tuning custom models. A legal research tool, for instance, could wrap each user query in a template that instructs the model to decompose the query into the relevant jurisdiction, statute chapter, and section before answering, mirroring the "Step 1: recall general category and structure breakdown" instruction used in the paper's Template 3. This requires no model modification and no training data—only domain expertise to construct the hierarchy traversal instructions for the relevant taxonomy. The paper's results on IPC and MedConceptsQA provide two proof-of-concept domains where this approach works; the pattern should transfer to any domain where (a) the knowledge base has explicit hierarchical structure and (b) that structure can be described in a zero-shot prompt.

Model selection decisions for deployment with known task complexity profiles. The paper's complexity-stratified IPC results (Table 5) provide a concrete decision heuristic for organizations choosing between instruction-tuned and reasoning-enhanced models. If the deployment task distribution consists primarily of shallow retrieval (fewer than 3 hierarchical steps, analogous to the Memory-Light condition where the path matching gap is 0.054), a prompted instruction-tuned model may be sufficient, because structured prompting recovers most of the performance gap on simple tasks. If the task distribution includes deep retrieval (5+ steps, where the path matching gap widens to 0.094), a reasoning-enhanced model provides a genuine capability advantage that prompting cannot fully substitute. The practical implication: profile your task distribution for retrieval depth (which can be measured automatically from the taxonomy structure without running model evaluations) before deciding whether to invest in RL training or larger reasoning models. Tasks with median traversal depth below 3 likely don't require RL; tasks with substantial mass above depth 5 likely do. This is directly actionable from the paper's results without additional research, requiring only that the deploying organization characterize the hierarchical depth of queries in their target domain.