A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment

🎯 Pitch

A 3.8B clinical language model outperforms GPT-4 on ICD-10 coding by 14% without seeing real clinical data, using a novel pre-instruction tuning technique that generates synthetic training tasks from public medical corpora. The key insight is merging domain experts back with an instruction-tuned base model to prevent catastrophic forgetting—a step most small model adaptations skip, and which accounts for the uniform 18.9% relative gain across 12 clinical benchmarks.

1. Executive Summary

This paper introduces MediPhi, a collection of 3.8B-parameter clinical small language models built through a modular framework combining pre-instruction tuning (generating task-specific outputs like summaries and entity extractions from medical corpora for sequential two-phase fine-tuning), model merging (using SLERP and BreadCrumbs to unify domain-specific experts while recovering degraded general capabilities), and clinical-tasks alignment (supervised fine-tuning and direct preference optimization on a synthetic instruction dataset). On the extended CLUE+ benchmark — which doubles the original CLUE benchmark to 12 datasets covering radiology reports, medical error detection, and ICD-10 coding — MediPhi-Instruct achieves a 6.9% absolute gain (18.9% relative) over the base Phi-3.5-mini model, while the ICD-10 coding expert surpasses GPT-4-0125 by 14% (44% relative improvement over the base). The framework demonstrates that targeted domain adaptation through pre-instruction tuning combined with merging can yield uniform gains across diverse clinical tasks, establishing that a 3.8B SLM can approach parity with an 8B general-purpose model only when the adaptation strategy accounts for catastrophic forgetting through explicit merging with the instruction-tuned base.

2. Context and Motivation

The Core Problem: Building Capable Clinical SLMs Without Access to Clinical Data

This paper addresses a fundamental tension in medical natural language processing: how do you build a high-performance clinical language model when the data needed to train it is largely inaccessible? The tension is acute for small language models (SLMs), which have limited parameter capacity and therefore require targeted, efficient domain adaptation — but the domain's training data is locked behind privacy regulations, restrictive licenses, and institutional barriers.

To understand why this matters, we need to examine the deployment realities of clinical AI. Large language models like GPT-4 have demonstrated strong performance on medical exams (Nori et al., 2023; Ben Abacha et al., 2024) and can serve as capable medical knowledge repositories. However, the paper identifies specific barriers that make large-model deployment in actual clinical settings impractical:

Latency: Clinical workflows often require real-time or near-real-time responses — a radiologist dictating findings, a physician reviewing a discharge summary, a nurse querying medication interactions during rounds. The inference time of 70B+ parameter models makes interactive use difficult, and the paper cites work by Dennstädt et al. (2025) and Yang et al. (2023) documenting these deployment challenges.
Cost: Running large models at clinical scale (thousands of queries per day across a hospital system) incurs substantial compute costs that strain healthcare budgets already squeezed by operational expenses.
Privacy and data sovereignty: Healthcare institutions are increasingly reluctant to send protected health information to external API endpoints, preferring on-premises or edge-deployed models that can run within their firewall. A 3.8B-parameter model can plausibly run on a single GPU or even on-device; a 70B or 340B model cannot.

This creates a clear motivation for smaller models. But developing capable clinical SLMs faces its own set of obstacles that form the central puzzle the paper tries to solve.

The Clinical Data Bottleneck

The paper identifies the unavailability of clinical data as the primary bottleneck. Clinical data — doctor-patient conversations, discharge summaries, radiology reports, nursing notes — contains the very language patterns, abbreviations, reasoning traces, and domain conventions that a model needs to internalize to perform clinical tasks. But this data is:

Sensitive: Protected health information under regulations like HIPAA, making it legally risky to curate, store, and distribute for model training.
Tightly licensed: Even de-identified clinical corpora (like MIMIC) require institutional data use agreements and cannot be openly distributed, impeding reproducibility.
Heterogeneous and scarce: Unlike biomedical literature (PubMed abstracts, which number in the tens of millions and are largely public), clinical documents exist in far smaller quantities and in diverse institutional formats — different hospitals structure their notes differently, making it hard to assemble a large, coherent training corpus.

This data scarcity problem is especially punishing for small models because they cannot simply "memorize" broad knowledge through massive pretraining the way a 70B model can. They need targeted exposure to domain-specific patterns, but the domain-specific data is precisely what's unavailable. The paper frames this as a critical limitation of current continual pretraining approaches when applied to SLMs (Section 1):

"both the inaccessibility of clinical data and the misalignment of current continual pre-training methods for clinical tasks are critical limitations in the context of SLMs, which have limited capacity"

Where Prior Medical Language Models Fall Short

The paper's critique of existing medical LLMs reveals several systematic gaps, and understanding these gaps is essential to appreciating what the MediPhi framework does differently.

Shortcoming 1: Size and Accessibility. Recent medical LLMs have converged on the Llama-3 family at 8B and 70B parameters (Christophe et al., 2024a; Gururajan et al., 2024; Ankit Pal, 2024), with Google's Med-PaLM 2 (Singhal et al., 2023) reaching 340B and not being publicly available. These models carry the latency, cost, and deployment limitations discussed above. More importantly, their licenses often restrict commercial use in healthcare — a significant barrier since most clinical deployment scenarios involve commercial entities (hospital systems, EHR vendors, telehealth platforms). The paper explicitly notes that the MediPhi family is released under a "commercially permissive license," positioning it as a practical alternative for real-world deployment.

Shortcoming 2: The Medical Knowledge Trap. Current medical LLMs are primarily evaluated and optimized for medical knowledge tasks — multiple-choice question answering on benchmarks like MedQA, MedMCQA, PubMedQA, and MMLU-medical. These tasks test whether a model knows medical facts (anatomy, physiology, pharmacology) and can reason from them. But the paper cites a growing body of evidence (Dada et al., 2024; Chen et al., 2024; Liu et al., 2024; Jeong et al., 2024a,b) showing that performance on medical knowledge benchmarks does not translate to clinical task performance. A model can ace the USMLE and still struggle to extract medications from a discharge summary, summarize a doctor-patient conversation, or detect a medical error in a clinical note. The paper phrases this as a "gap" between medical and clinical capabilities (Section 2.5), and the CLUE benchmark (Dada et al., 2024) was specifically designed to measure this gap by testing on real clinical document types rather than curated exam questions.

When Dada et al. (2024) evaluated twelve medical LLMs on CLUE, only two improved over their base models: BioMistral (using DARE model merging) and Med42 (using supervised fine-tuning alignment). This is a sobering finding — it means that most medical adaptation techniques are either ineffective for clinical tasks or actually harmful, degrading the base model's general capabilities without compensating with clinical skill.

Shortcoming 3: Instruction Tuning Imbalances. The paper provides a specific critique of the alignment data used by recent instruction-tuned medical models like Aloe (Gururajan et al., 2024), Hippocrates (Acikgoz et al., 2024), and Med42 v2 (Christophe et al., 2024a,b). These models use a mixture of medical QA databases, non-medical alignment data (like UltraChat), and benchmark training sets. While effective for improving benchmark scores, this approach introduces two problems:

Imbalanced task and document coverage: The training data over-represents certain tasks (question answering) and under-represents others (information extraction, summarization of clinical documents, error detection), meaning the model learns a skewed skill profile.
In-distribution evaluation: Including benchmark training sets in the alignment data means that the evaluation is closer to a fine-tuning evaluation rather than a zero-shot or few-shot evaluation. A model that has seen MedQA training questions during alignment isn't demonstrating genuine clinical reasoning when tested on MedQA — it's demonstrating memorization. The paper explicitly flags this as a weakness (Section 2.2), noting that their own evaluation protocol uses few-shot prompting without task-specific fine-tuning.

Shortcoming 4: Synthetic Data Without Clinical Depth. Several prior works have explored synthetic instruction generation for medical domains. Zhang et al. (2023b) generated 52,000 medical QA instructions from expert-filled forms — but these were primarily knowledge-oriented rather than clinical. Kweon et al. (2024) generated 158,000 short QA instructions for 8 tasks using synthetic clinical documents seeded from PMC-Patients. However, these instructions were narrow in scope: short question-answer pairs on a limited set of task types, lacking the complexity and diversity of real clinical workflows. The paper's MediFlow dataset (2.5 million instructions across 14 task types, 98 fine-grained document types, and 6 difficulty levels) represents a deliberate attempt to address this narrowness — but understanding why this matters requires seeing what prior synthetic efforts missed.

Shortcoming 5: Catastrophic Forgetting in Domain Adaptation. When you fine-tune a model on domain-specific data, it tends to lose the general capabilities it acquired during pretraining — a phenomenon called catastrophic forgetting (Scialom et al., 2022). For medical models, this is particularly damaging because clinical tasks require both domain knowledge AND general capabilities: instruction following (to correctly interpret the requested task), long-context handling (to process lengthy clinical documents), and multilingual support (for diverse patient populations). The paper observes that this forgetting is "particularly evident in zero-shot and few-shot settings, where instruction-tuned models generally outperform their base counterparts" (Section 3.1.3). Prior medical LLMs addressed this implicitly through large-scale training (hoping the signal would overwhelm the noise) or by mixing non-medical data with medical data during training — but this dilutes the domain adaptation effect.

Shortcoming 6: Treating All Medical Knowledge as Uniform. Most prior work treats "medical domain adaptation" as a single, undifferentiated process — train on all available medical data and hope for the best. The paper argues this is suboptimal because different types of medical knowledge (scientific literature, clinical documents, medical coding systems, clinical guidelines, encyclopedic knowledge) have fundamentally different formats, densities, and learning characteristics. The DataMix baseline in the paper's experiments — trained on all corpora simultaneously — performs worse than individual experts on specific tasks, and worse than the merged combination of experts on the overall benchmark. This suggests that the uniform-adaptation strategy leaves performance on the table.

How This Paper Positions Itself

The paper constructs its approach as a modular framework that directly addresses each of the shortcomings identified above. The intellectual architecture is:

Pre-instruction tuning (PIT) was introduced by Jiang et al. (2024) for general-domain knowledge learning. The paper extends PIT to medical domain adaptation and — critically — extends it beyond question-answering to include summarization, named entity recognition, and relation extraction. This is not a trivial extension; generating high-quality synthetic training data for these tasks from medical corpora requires distinct prompting strategies, and the paper demonstrates that different task formulations yield different downstream benefits (summarization outperforms QA for ICD-10 coding in Figure 4).
Model merging serves a dual purpose that the paper makes explicit: it combines domain expertise from separately trained experts (each specializing on one data type), AND it mitigates catastrophic forgetting by merging back with the instruction-tuned base model. This is a deliberate design choice informed by the observation that PIT improves domain-specific performance but degrades general capabilities. The SLERP merging step — where each expert is merged with the base model before being merged with other experts — is the mechanism for preserving instruction following, long-context handling, and multilingual abilities while still acquiring domain knowledge. The paper positions this as a synergistic combination rather than a post-hoc fix.
MediFlow, the synthetic instruction dataset, is positioned as filling a gap in clinical NLP resources. Unlike prior synthetic datasets focused on narrow QA or short-form instructions, MediFlow spans 14 task types, 98 document types (with varying granularity — the full document vs. individual sections), 6 difficulty levels, and 2 output formats (JSON for structured extraction, plain text for generative tasks). The paper emphasizes that the difficulty distribution is deliberately skewed toward "hard, very hard, and extreme" levels at a 3:1 ratio, generating instructions that require multi-step clinical reasoning rather than surface-level extraction.
The CLUE+ extension positions the paper's evaluation as more comprehensive than prior work. The original CLUE benchmark focused on 6 datasets using clinical notes and discharge summaries. By adding medication QA, radiology report QA, medical error detection, clinical note generation from dialogs, social determinants of health extraction, and ICD-10 code definition QA, CLUE+ covers a broader spectrum of clinical workflows, input document types, and task formats. This matters because it prevents the evaluation from being gamed — a model that memorizes discharge summary patterns won't necessarily perform well on radiology reports or doctor-patient conversations.
The modular, reproducible design is positioned as a deliberate alternative to closed, opaque medical LLMs. By releasing the MediPhi models under a permissive license, along with the synthetic validation sets used for merge optimization, the paper enables other researchers to integrate additional clinical experts into the framework or reproduce the merging process with new domain adaptations. This is a practical stance: the clinical domain evolves rapidly (new coding standards, new guidelines), and a modular framework allows incremental updates rather than full retraining.

The Broader Significance

The paper's approach matters beyond its specific performance numbers because it addresses a structural problem in medical AI: the gap between the data we have and the data we need. Biomedical literature (PubMed) is abundant and public; clinical data is scarce and private. A framework that can adapt models using synthetic task data generated from public corpora — rather than requiring access to protected clinical documents — is a framework that can be reproduced, audited, and improved by the broader research community.

The paper's emphasis on uniform gains across diverse tasks (measured through the #DG and CVΔ metrics introduced in Equation 1) reflects a practical concern: clinical deployment requires models that don't have hidden failure modes. A model that improves dramatically on ICD-10 coding but degrades on medication QA has limited real-world utility because you can't predict which task a clinician will need next. The paper's use of coefficient of variation of gains/losses as an evaluation criterion — not just average accuracy — encodes this concern directly into the model selection process.

3. Technical Approach

This is primarily a systems-building and empirical evaluation paper whose core idea is that adapting a small language model to clinical tasks requires a modular, multi-stage framework that separates domain knowledge acquisition (via pre-instruction tuning and model merging) from task-specific alignment (via synthetic instruction data), because doing both simultaneously — or doing either naively — leads to catastrophic forgetting of general capabilities or suboptimal use of the model's limited parameter capacity.

3.1 Reader Orientation

The paper builds a modular pipeline for converting a general-purpose 3.8B-parameter language model (Phi-3.5-mini) into a collection of clinical specialists that can perform diverse tasks — summarising doctor-patient conversations, extracting social determinants of health from clinical notes, answering questions about radiology reports, detecting medical errors, and coding diagnoses with ICD-10 codes — without requiring access to real patient data during training. The solution's shape is a two-phase architecture: first, train separate domain experts on different types of public medical corpora using a technique that teaches the model to perform tasks on those corpora rather than just read them, then merge the experts together while preserving the base model's general instruction-following abilities; second, align the merged model to clinical workflows using a large synthetic instruction dataset generated by a strong teacher model.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in two sequential phases:

Phase 1 — Continual Pre-training (Knowledge Acquisition):

Base Model (Phi-3.5-mini) — a 3.8B-parameter instruction-tuned general-purpose language model that serves as the starting point. It already possesses instruction-following, long-context handling, and multilingual capabilities from its original training.
Five Medical Corpora Groups — five distinct collections of public medical and clinical text, each representing a different type of domain knowledge: PubMed (48B tokens of scientific articles and abstracts), Clinical (patient summaries, synthetic clinical documents, doctor-patient dialogs), MedCode (ICD and ATC coding webpages), Guidelines (clinical practice guidelines from WHO and other health organisations), and MedWiki (medical encyclopedia articles).
Pre-Instruction Tuning (PIT) Engine — for each corpus group, GPT-4o generates synthetic task data (summaries, QA pairs, named entity annotations, relation extraction triples) from the raw documents. Each expert model is then trained in two phases: first on task outputs alone, then on task-output-plus-original-document concatenations. This produces five domain-specific expert models.
Model Merging System — each expert is first merged back with the base model using Spherical Linear Interpolation (SLERP) to recover general capabilities lost during domain adaptation. Then, all five SLERP-merged experts are combined into a single unified model (MediPhi) using the BreadCrumbs merging algorithm, with merge weights optimised by an evolutionary algorithm guided by synthetic validation sets.

Phase 2 — Post-training (Clinical Alignment): 5. MediFlow Data Generation Pipeline — an agentic system using GPT-4o and GPT-4o-mini to generate 2.5 million synthetic instruction-input-output triplets covering 14 clinical NLP task types, 98 fine-grained document types, 6 difficulty levels, and 2 output formats (JSON and plain text). Quality is assessed via LLM-as-a-Judge with self-consistency. 6. Alignment Trainer — the unified MediPhi model undergoes supervised fine-tuning (SFT) on filtered high-quality MediFlow instructions, followed by Direct Preference Optimisation (DPO) on a preference dataset where chosen outputs are correct MediFlow responses and rejected outputs are GPT-4o-generated "marginally wrong" variants.

Information flows: Raw medical corpora → GPT-4o generates task data → PIT trains five experts → SLERP merges each expert with base model → BreadCrumbs combines all experts into MediPhi → GPT-4o generates MediFlow instructions → GPT-4o-mini judges quality → filtered instructions train MediPhi-SFT via SFT → GPT-4o generates marginally wrong outputs for DPO → DPO trains MediPhi-Instruct.

3.3 Roadmap for the Deep Dive

First, the Pre-Instruction Tuning (PIT) mechanism — how it extends Jiang et al. (2024) beyond question-answering to summarisation, NER, and relation extraction, and why its two-phase training structure matters. This is the core knowledge acquisition method and the paper's primary technical innovation relative to prior medical adaptation approaches.
Second, the model merging strategy — the SLERP step (expert-to-base) and the BreadCrumbs step (many-experts-to-one), including the evolutionary optimisation procedure and the synthetic validation sets that guide it. This is the mechanism that makes the modular approach work without catastrophic forgetting.
Third, the MediFlow synthetic data generation pipeline — the agentic architecture for generating 2.5M clinical instructions, the five-parameter conditioning scheme, the LLM-as-a-Judge quality assessment with self-consistency, and the filtering heuristics. This is the data engine that enables clinical alignment without access to real clinical training data.
Fourth, the DPO preference data construction — how marginally wrong outputs are generated by prompting GPT-4o with specific error types, and why this matters for the final performance gain.
Fifth, the evaluation framework — the CLUE+ benchmark extension, the metrics (average accuracy, #DG, CVΔ), and the experimental configuration (few-shot settings, decoding strategies, validation set construction for merge optimisation).

3.4 Detailed, Sentence-Based Technical Breakdown

3.4.1 Pre-Instruction Tuning (PIT) for Medical Domain Adaptation

What PIT is and where it comes from. Pre-Instruction Tuning was introduced by Jiang et al. (2024) as a method for teaching language models new knowledge more effectively than standard next-token prediction on raw text. The original formulation used question-answering as the sole task type: for each document in a corpus, generate question-answer pairs about its content, fine-tune the model to answer those questions (Phase 1), then fine-tune on concatenations of the question-answer pairs and the original document (Phase 2). The key insight of Jiang et al. was that instruction-tuned models are better knowledge learners — the instruction-following capability acquired during post-training creates a more effective learning scaffold for absorbing new factual information than the raw language modelling objective alone.

This paper extends PIT in a critical way: it expands the task types beyond question-answering to include summarisation, named entity recognition (NER), and relation extraction (RE). This is not a trivial extension. Each task type requires the model to learn different kinds of attention patterns and output structures from the same underlying documents. For a corpus of ICD-10 coding webpages, a QA task teaches the model to locate specific factual answers, a summarisation task teaches it to identify the most salient information and restate it concisely, an NER task teaches it to recognise spans corresponding to disease names and procedure codes, and a relation extraction task teaches it to identify connections between entities (e.g., "this disease maps to this code"). The paper demonstrates experimentally (Figure 4) that these different task formulations yield different downstream benefits, with summarisation providing the largest improvement on ICD-10 coding accuracy.

The PIT training procedure for one expert. For each of the five corpus groups (PubMed, Clinical, MedCode, Guidelines, MedWiki), the paper follows these steps to create a domain expert:

Step 1: Task data generation with GPT-4o. For every document in the corpus, GPT-4o generates task-specific outputs. The paper states (Section 3.1.2):

"We use GPT-4o to generate outputs for all four tasks on the ICD10CM subset of the MedCode dataset as an initial case study. Based on the results, we extend this process to the remaining four dataset groups."

The task types generated depend on the corpus and are:

Question-answering: GPT-4o generates questions about the document content and corresponding answers.
Summarisation: GPT-4o generates a concise summary of the document.
Named entity recognition: GPT-4o identifies and labels entities (e.g., disease names, medication names, procedure codes) within the document text.
Relation extraction: GPT-4o identifies relationships between entities (e.g., "drug X treats condition Y," "code Z maps to procedure W").

The critical design choice here is what task types to apply to which corpus groups, and the paper determines this empirically through the ICD-10 case study (Section 4.2.1, Figure 4), finding that summarisation outperforms QA, NER, and RE for the coding domain. For the remaining experts, the paper applies PIT with summarisation as the primary task (Section 4.2.2: "Based on these results, the rest of the paper will apply PIT with summaries, unless otherwise stated"). This is an important practical choice: summarisation forces the model to develop a compressed, structured understanding of the document's key content, which transfers well to downstream clinical reasoning tasks.

Step 2: Phase 1 training — fine-tuning on task outputs only. The base Phi-3.5-mini model is fine-tuned exclusively on the synthetic task outputs (e.g., just the summaries, or just the QA pairs) for one epoch. The hyperparameters (from Section 3.1.4 and Table 6 context) are:

"The first phase involves fine-tuning on the generated outputs from a single task for one epoch using a cosine scheduler (peak learning rate at 1e-4)"

The learning rate scheduler is linear warmup followed by cosine decay, and the effective batch size (from Table 6) is 512. NEFTune noise with $\alpha = 5$ is applied. The maximum sequence length is 4,096 tokens. Training uses 16 GPUs.

The purpose of Phase 1 is to teach the model how to perform the task on the domain's content. By training only on task outputs, the model learns to associate the document content with the structured task format — it learns what a good summary looks like for clinical guidelines, or what kinds of QA pairs are answerable from medical encyclopedia text.

If the task requires multiple output elements — for example, an NER task might identify multiple entities in a single document, each with a type and span — these are "concatenated into a single sequence, separated by end-of-sentence (EOS) tokens" (Section 3.1.2). This means the model sees a single long target sequence that contains all task outputs for the document, delimited by EOS markers.

Step 3: Phase 2 training — fine-tuning on task-data-plus-document concatenation. The Phase-1 model is then fine-tuned on concatenations of the task data and the original documents for two epochs. The hyperparameters change:

"In the second phase, the model is fine-tuned for two epochs with another cosine scheduler (peak learning rate at 3e-4) on the concatenation of the task data and the original documents, with EOS tokens acting as separators."

The peak learning rate increases to 3e-4 (three times the Phase-1 rate), and training proceeds for two epochs instead of one. The concatenation format is:

[task_data] [EOS] [original_document]

where [task_data] is the same GPT-4o-generated outputs used in Phase 1, and [original_document] is the raw medical corpus text.

The purpose of Phase 2 is to cement the knowledge integration. In Phase 1, the model learned to produce task outputs from documents, but the documents themselves may not have been deeply internalised. Phase 2 forces the model to process the task data AND the raw document together, strengthening the connection between the structured task formulation and the unstructured domain text. This is the "pre-instruction" element of PIT — the instruction (the task format) acts as a scaffold for learning the domain content.

Why two phases are necessary. If you trained only on the raw documents (standard DAPT — Domain Adaptation Pre-Training via next-token prediction), the model would learn the surface statistics of medical text but might not develop the structured understanding needed for clinical reasoning. If you trained only on the task outputs (Phase 1 alone), the model would learn to mimic the task format but might not deeply internalise the domain knowledge. The two-phase structure separates these learning objectives: Phase 1 establishes the task scaffold, Phase 2 fills it with domain content.

The paper provides evidence that this two-phase structure matters through the ablation on the Guideline expert (Table 2). The row "Guideline w/o PIT" (i.e., standard DAPT without pre-instruction tuning) achieves only 33.0 average accuracy on CLUE+ compared to 39.2 for the full PIT-trained Guideline expert — a 6.2 percentage point gap. The row "Guideline w/o SLERP & PIT" (i.e., neither PIT nor merging) drops to 25.2, losing 14 points from the full approach. These ablations demonstrate that both PIT and merging are essential components, not optional enhancements.

The ICD-10 case study (Figure 4 and Figure 5). The paper uses ICD-10 medical coding webpages from the MedCode corpus as an initial testbed for comparing adaptation methods, and the results in Figures 4 and 5 are revealing:

Standard DAPT on raw webpages ("Webpage" in Figure 4): Performance drops to random. The paper hypothesises that "the webpages have peculiar implicit format" that the model cannot effectively learn from via next-token prediction. ICD-10 coding webpages are dense, tabular, and use abbreviated notation — they don't look like natural language.
Explainer approach (fine-tuning on GPT-4o-generated textbook-like rewrites of the webpages): Improves over the baseline by 6%. The reformulation makes the content learnable.
PIT with summarisation ("Summary" in Figure 4): Improves by 8% over the explainer alone. The task format provides an additional learning scaffold beyond just making the text readable.
SLERP merging after PIT (Figure 5): The summary-expert reaches 65% accuracy, a 44% relative improvement over the base model and surpassing GPT-4-0125 by 14% (Section 4.2.2: "surpassing GPT-4 by 8% (14% relative)"). SLERP merging boosts all task variants, not just summarisation — the QA expert goes from below-baseline performance to 62% (38% relative improvement) after merging.

This case study establishes the template that gets applied to all five corpus groups: generate summaries (or other tasks) with GPT-4o, apply two-phase PIT, then SLERP-merge back with the base model.

Design choices and their justifications. Several non-obvious decisions are made in the PIT implementation:

Why summaries over other task types? The paper tests QA, NER, RE, and summarisation on the ICD-10 subset and finds summarisation dominant (Figure 4). The likely reason: summarisation of a dense, fact-packed document like an ICD-10 webpage forces the model to identify, prioritise, and restate the key clinical concepts in a coherent form. QA teaches fact retrieval but not synthesis. NER teaches span identification but not relational understanding. Summarisation spans both. For the clinical domain, where tasks often require synthesising information from multiple sections of a document, this synthesis capability is valuable.
Why a higher learning rate in Phase 2? The paper uses 1e-4 for Phase 1 and 3e-4 for Phase 2. The larger Phase-2 rate suggests that integrating the task data with the raw document is a more difficult optimisation problem — the model must reorganise its representations more substantially to accommodate the domain knowledge in Phase 2 than it does to learn the task format in Phase 1. This is consistent with the catastrophic forgetting risk: Phase 2 is where the model most risks overwriting its general capabilities, which is why the SLERP merging step (which recovers those capabilities) follows PIT.
Why extend PIT beyond QA? Jiang et al. (2024) used only QA as the task type. The paper's extension to summarisation, NER, and RE is motivated by the clinical domain's task diversity. Clinical NLP requires entity extraction (SDoH), summarisation (problem lists, radiology impressions), QA (medication questions, discharge letter queries), and reasoning (error detection, diagnosis). A model trained only on QA during PIT might develop a narrow skill that doesn't transfer well to summarisation or extraction tasks. By matching the PIT task type to the downstream evaluation tasks, the paper aims to build broader transferable skills during knowledge acquisition.
Why train separate experts per corpus group instead of one model on all data? This is the modularity argument. Different corpus groups have different formats, lengths, densities, and learning characteristics. Training separate experts allows each to specialise on its data type without interference from other data types. The DataMix baseline (trained on all corpora simultaneously with DAPT) achieves 37.5 average on CLUE+ (Table 3), while the average of the five individual SLERP experts is 38.5 — a full point higher. The unified merged model (MediPhi via BreadCrumbs) reaches 39.3, outperforming both. This pattern suggests that separate training followed by merging is more effective than joint training, consistent with findings from the model merging literature (Yadav et al., 2024b; Ahmadian et al., 2024).

3.4.2 Catastrophic Forgetting, SLERP Merging, and the Expert-to-Base Recovery Step

The forgetting problem. When a model undergoes domain-specific fine-tuning (whether DAPT, PIT, or any other adaptation), it inevitably shifts its parameter distribution away from the original pretrained state. For general-purpose instruction-tuned models like Phi-3.5-mini, this shift degrades capabilities that are essential for clinical tasks but are not explicitly reinforced during domain training: instruction following (correctly parsing what the user wants done), long-context handling (processing multi-page clinical documents), multilingual support (serving diverse patient populations), and general reasoning (drawing inferences that require common-sense knowledge not found in medical corpora).

The paper documents this degradation explicitly in Table 2. The Guideline expert without SLERP merging ("Guideline w/o SLERP") achieves only 27.2 average accuracy on CLUE+, compared to 36.5 for the base model — a catastrophic drop of 9.3 points, with gains on only 4 out of 12 datasets. When PIT is also removed ("Guideline w/o SLERP & PIT"), the model collapses to 25.2 with gains on only 1 dataset. This is the price of naive domain adaptation: domain knowledge improves on a narrow slice of tasks while everything else degrades.

SLERP merging as mitigation. Spherical Linear Interpolation (SLERP) is a method for interpolating between two sets of model parameters along the shortest path on the hypersphere, rather than along a straight line in Euclidean space. The paper describes it (Section 3.1.3) as:

"a method optimised for merging two models, often yielding high-performing hybrids (Hammoud et al., 2024; Ahmadian et al., 2024; Labrak et al., 2024)"

For each expert model, SLERP is applied between the expert's parameters and the base model's parameters. The interpolation factor determines how much of each model's "influence" is retained — the paper experiments with merging proportions of 10%, 25%, and 50% and selects the best proportion using validation sets (Section 3.1.3: "We determined the merging proportions (10%, 25% or 50%) via validation sets").

The mathematical operation for SLERP between two parameter vectors $\theta_{\text{base}}$ and $\theta_{\text{expert}}$ with interpolation factor $t \in [0, 1]$ is:

$\text{SLERP}(\theta_{\text{base}}, \theta_{\text{expert}}, t) = \frac{\sin((1-t)\Omega)}{\sin(\Omega)} \theta_{\text{base}} + \frac{\sin(t\Omega)}{\sin(\Omega)} \theta_{\text{expert}}$

where $\Omega = \arccos(\langle \theta_{\text{base}}, \theta_{\text{expert}} \rangle)$ is the angle between the two parameter vectors on the hypersphere, and $\langle \cdot, \cdot \rangle$ denotes the inner product. When $t = 0$ , the result is the base model; when $t = 1$ , it's the expert; when $t = 0.5$ , it's the midpoint on the geodesic between them.

What it computes: For each parameter in the model (each weight matrix, bias vector, layer norm parameter), SLERP computes a weighted combination that follows the shortest path along the sphere of constant norm. The sine-weighted coefficients ensure the interpolation preserves the angular relationship between the two models rather than simply averaging their magnitudes.

Why this form: Standard linear averaging (simply taking $(1-t)\theta_{\text{base}} + t\theta_{\text{expert}}$ ) can produce parameter vectors that lie outside the region of effective models because the loss landscape of neural networks is highly non-convex in Euclidean space. SLERP respects the geometry of the optimisation surface: neural network parameters that perform well tend to lie on a thin shell (roughly constant norm), and interpolating along that shell (the geodesic) is more likely to produce a functional model than interpolating through the interior, which can land in a high-loss region. This property is called "linear mode connectivity" (Frankle et al., 2020; Mirzadeh et al.) and is the theoretical foundation for why parameter-space interpolation works at all.

The paper's SLERP application. For each of the five experts, SLERP is applied after PIT (not before), with the stated rationale:

"We apply merging after PIT since these techniques demonstrate a synergistic effect. While PIT enhances domain-specific learning, it also leads to catastrophic forgetting — degrading the model's initial abilities such as instruction following, long context handling, and multilingual support."

The merging effectively creates a model that is $t$ parts expert (carrying domain knowledge from PIT) and $1-t$ parts base (carrying general capabilities from the original instruction-tuned model). The paper does not report the specific $t$ values selected for each expert, only that they were chosen from $\{0.1, 0.25, 0.5\}$ based on validation set performance.

Alternative approaches to avoiding forgetting. The paper notes (Section 3.1.3, footnote 5) that other methods exist for preserving generalisation during fine-tuning, such as KL-divergence regularisation (as used in RLHF; Ouyang et al., 2022), which penalises the model for deviating too far from its initial output distribution. However, the paper argues that model merging may be more effective because it operates directly on the parameter space rather than constraining the output distribution — it can recover capabilities that were already lost during PIT, whereas KL regularisation can only prevent loss that hasn't happened yet. The paper cites Ramé et al. (2024) for evidence that merging can "further optimise reward alignment in RL settings."

3.4.3 Multi-Expert Merging into MediPhi via BreadCrumbs

The goal of multi-expert merging. After SLERP-merging each individual expert with the base model, we have five distinct models, each strong on a particular slice of clinical and medical knowledge: PubMed (scientific literature), Clinical (patient-facing documents and conversations), MedCode (coding taxonomies), Guidelines (clinical practice recommendations), and MedWiki (encyclopedic medical knowledge). The task is to combine all five into a single model that retains the best of each without interference.

Three candidate merging algorithms. The paper evaluates three techniques (Section 3.1.4):

Task Arithmetic (Ilharco et al.): Each expert's contribution is represented as a "task vector" — the difference between the expert's parameters and the base model's parameters. The merged model is the base model plus a weighted sum of these task vectors. The weights control how much influence each expert has. Task Arithmetic is simple and interpretable but can suffer from interference when task vectors point in conflicting directions (i.e., when one expert moves a parameter up and another moves it down, the net effect can cancel or produce noise).
TIES-Merging (Yadav et al., 2024a): An extension of Task Arithmetic that addresses interference through three steps: (1) trim low-magnitude parameter changes (keeping only the top-k% of changes per expert, discarding the rest as noise); (2) elect a sign for each parameter by majority voting across experts (if three experts increase a parameter and two decrease it, the sign is positive); (3) disjoint merge by averaging only the parameters that agree with the elected sign. This reduces the cancellation problem of plain Task Arithmetic.
BreadCrumbs (Davari and Belilovsky, 2024): A sparse merging technique that uses binary masks to select which subset of each expert's parameters to incorporate. Instead of dense task vectors, each expert contributes only a sparse subset of changed parameters, with the mask learned to maximise task performance while minimising overlap between experts. This reduces interference by construction: parameters that multiple experts want to modify in conflicting ways are simply not merged for those conflicting experts.

The optimisation problem. Given five experts and a merging algorithm, the number of possible merge configurations is enormous. For Task Arithmetic, you need to choose a scalar weight per expert. For TIES, you need to choose the trimming fraction and per-expert weights. For BreadCrumbs, you need to choose the sparsity levels and per-layer mask configurations. The paper states:

"Given the vast configuration space of multi-model merges, we employ an evolutionary algorithm via MergeKit (Goddard et al., 2024) to optimise the merging process."

MergeKit is a toolkit that provides implementations of multiple merging algorithms with evolutionary search over hyperparameters. The evolutionary algorithm evaluates candidate merged models on a fitness metric and iteratively selects and mutates the best performers — analogous to how genetic algorithms work, but operating on merge configurations rather than model parameters directly.

The validation set problem and the synthetic solution. The paper identifies a practical obstacle:

"However, optimisation on our benchmark is not feasible due to a lack of validation data and framework incompatibilities."

The CLUE+ benchmark evaluation data cannot be used for merge optimisation because that would constitute training on the test set — the merged model's hyperparameters would be overfit to the evaluation, invalidating the comparison. Yet without some validation signal, the evolutionary algorithm cannot search.

The solution is to generate synthetic validation sets that are distributionally similar to the evaluation tasks but contain different content. The paper describes the process:

"we prompt GPT-4o to create multiple-choice question sets covering 12 medical and clinical topics relevant to our benchmark (e.g., doctor-patient interactions, medical coding, discharge summaries). These sets maintain contextual consistency with our evaluation tasks."

The twelve validation sets cover: clinical case, clinical knowledge, medication, ICD-10 code definitions, radiology reports, clinical NLI, QA on discharge letters, medical codes of discharge letters, problem lists from clinical notes, summarisation of patient inquiries, QA on medical consultations, and QA on multiple EHR documents. To ensure diversity and avoid clustering artifacts, the paper applies HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) on embeddings of the generated questions and their contexts, keeping one sample per cluster plus all outliers, for a total of up to 1,200 samples per validation set.

The evolutionary search. The algorithm is configured to terminate after 500 evaluations (Section 3.1.4: "The evolutionary algorithm is set to terminate after 500 evaluations, guided by average accuracy on these validation sets"). Each evaluation involves creating a merged model with a candidate configuration, running inference on the validation sets, and computing the average accuracy. The search explores the space of merge weights (for Task Arithmetic and TIES) or mask configurations (for BreadCrumbs), with the fitness being the average accuracy across all 12 validation sets.

Clarifying the scope of merge optimisation. A critical detail: the evolutionary search optimises merge weights, but it does not select which experts to include — all five SLERP-merged experts are always included. The paper states (Table 3) that the merging combines "5 experts: PubMed, Clinical, MedWiki, MedCode and Guideline." The optimisation is about how much of each expert to include and how to resolve conflicts between them, not about whether to include them.

Results of the merging comparison (Table 3).

Task-Arithmetic achieves the highest average accuracy (39.4) but gains on only 9 of 12 datasets and has a CVΔ of 1.9 (high variance in gains/losses across datasets).
TIES achieves 39.3 with gains on 7 datasets and CVΔ of 1.7 (slightly more uniform gains but fewer datasets improved).
BreadCrumbs achieves 39.3 with gains on 11 of 12 datasets and CVΔ of 1.5 (the most uniform improvement pattern).

The paper selects BreadCrumbs as the final merging algorithm with the explicit rationale:

"Since the goal is to select the most robust model for the alignment phase, we find that the BreadCrumbs expert offers the best trade-off between high-amplitude improvements and consistent gains on 11 datasets."

This is a significant design choice: robustness (uniform improvement across tasks) is prioritised over peak performance on any individual task. The BreadCrumbs merged model is designated MediPhi and serves as the base for all subsequent alignment experiments.

An interesting asymmetry in the merged model's performance. The BreadCrumbs-merged MediPhi (39.3) slightly underperforms the best individual SLERP expert — MedWiki at 39.7 — by 0.4 points on average. However, MedWiki improves on only 10 datasets compared to MediPhi's 11, and MedWiki underperforms MediPhi on ICD-10 coding by a substantial margin (MediPhi: 55.5, MedWiki: 50.2, from Table 12). The merged model thus achieves a different pattern of strengths than any individual expert, one that is more uniformly distributed across tasks. The paper notes:

"we see that all unified SLMs are above the average expectation of the SLERP performances by 0.9%, but below the maximum values"

This means the merged model is better than the average expert but worse than any expert on that expert's specialty. This is exactly the tradeoff one would expect from combining specialists: you gain breadth at the cost of depth, but the breadth gain (11 datasets improved vs. 10 for MedWiki) is arguably more valuable for a general-purpose clinical model.

The computation of CVΔ (Coefficient of Variation of gains/losses). The paper introduces this metric in Equation 1:

$CV \Delta = \frac{\sqrt{\mathbb{E}_{d \sim \mathcal{D}} [(\delta_d - \mu_d)^2]}}{|\mu_d|}$

where $\delta_d$ is the expert accuracy minus the baseline accuracy for the $d^{th}$ dataset in the benchmark $\mathcal{D}$ , and $\mu_d = \mathbb{E}_{d \sim \mathcal{D}}[\delta_d]$ is the mean of those differences across all datasets.

What it computes: $CV\Delta$ is the standard deviation of per-dataset accuracy deltas divided by the absolute value of the mean delta. It measures how consistent the gains (or losses) are across datasets. A $CV\Delta$ close to zero means the model improves (or degrades) uniformly across all tasks — every dataset sees roughly the same shift. A large $CV\Delta$ means gains are concentrated on a few datasets while others see little improvement or even losses — the model's skill profile is highly uneven.

Why this form: The standard deviation alone ( $\sqrt{\mathbb{E}[(\delta_d - \mu_d)^2]}$ ) would be scale-dependent — a model with 10-point average gains and 5-point standard deviation would look as "noisy" as a model with 1-point gains and 0.5-point deviation, even though the latter's relative variability is the same. Dividing by $|\mu_d|$ normalises the metric, making it a coefficient of variation that captures relative consistency regardless of the absolute magnitude of improvement. The absolute value in the denominator prevents sign flips — if the mean delta is negative (the model degrades on average), you don't want the sign to flip the CV, since the consistency of degradation is still a meaningful quantity.

Why this matters for model selection. The BreadCrumbs model has $CV\Delta = 1.5$ , which is lower than Task-Arithmetic's 1.9 and TIES' 1.7. A low $CV\Delta$ is desirable in clinical deployment because it means there are no "blind spots" — tasks where the model catastrophically fails while excelling elsewhere. A model with very high average accuracy driven by massive gains on a few tasks and losses on others might look good in aggregate but be dangerous in practice, because the end user cannot predict which task category their query falls into relative to the model's uneven profile.

3.4.4 The MediFlow Synthetic Instruction Dataset Generation Pipeline

The motivation. After building MediPhi through continual pre-training and merging, the paper addresses a remaining gap: the model has acquired medical and clinical knowledge, but it hasn't been explicitly trained to perform clinical NLP tasks — summarising discharge summaries, extracting entities from clinical notes, answering medication questions, detecting medical errors, coding diagnoses. These operational skills require instruction tuning, but the standard approach of using benchmark training sets (as done by Med42 v2, Aloe, and Hippocrates) creates the in-distribution evaluation problem the paper critiques. Instead, the paper generates a large, diverse synthetic instruction dataset from scratch.

The agentic pipeline architecture (Figure 3). The generation system has three sequential stages:

Stage i: Instruction generation with GPT-4o. The core generation prompt conditions on five parameters:

Input data type: One of 36 document types (discharge summary, SOAP note, radiology report, doctor-patient conversation, nursing note, etc.), each at one of up to 8 granularity levels (complete document, individual sections, or both), yielding 98 fine-grained input configurations.
Task type: One of 14 clinical NLP tasks: summarisation, question-answering, multiple-choice QA, named entity recognition, relation extraction, classification, reasoning and diagnosis, textual entailment, text simplification, text expansion, abbreviation expansion, aspect-oriented keyword extraction, error detection and correction, and note scoring.
Difficulty level: One of 6 levels (low, moderate, moderate-hard, hard, very hard, extreme), sampled with a 3:1 ratio favouring the top three difficulty levels (hard, very hard, extreme).
Output format: Either plain text or JSON.
Temperature: 1.0 for 70% of the dataset (favouring accuracy) and 1.25 for 30% of the dataset (favouring diversity).

The paper specifies the generation parameters:

"we request 10 instructions at a time with four input-output pairs each"

So each API call to GPT-4o produces a batch of 10 instructions, each instruction having 4 pairs of inputs and corresponding outputs. The total number of API calls is therefore approximately $\frac{2.5 \times 10^6}{10 \times 4} \approx 62,500$ calls to GPT-4o.

The meta-prompt instructs GPT-4o to:

"define very detailed instructions for a precise task with clear constraints in the medical/clinical domain... You must make a task with a {{difficulty}} difficulty on a scale of 6 levels... You must use interesting and complex examples requiring abstractive medical capabilities to infer the output from the input."

A crucial instruction in the prompt concerns lexical diversity:

"You must use synonyms for all headers (or no header at all) to avoid leaking current vocabulary into the instructions, also use different ways to structure (or not) and detail the instructions (e.g. bullet points, sections, narrative form, or else)."

This is designed to prevent the model from learning surface-level patterns that correlate with specific output formats — if every summarisation instruction used the word "summarise," the model would learn a brittle keyword trigger rather than a genuine task understanding. By varying the wording and structure of instructions for the same underlying task, the dataset teaches the model to recognise the task's semantic intent rather than its lexical signature.

Stage ii: Quality assessment with GPT-4o-mini and self-consistency. After generation, each instruction-input-output triplet is evaluated by a smaller, cheaper model (GPT-4o-mini) acting as a judge. The paper describes:

"we prompt GPT-4o mini with a LLM-as-a-Judge approach — using self-consistency with chain-of-thought across M = 5 samples at temperature of 1.0 — to provide a critical assessment of the synthetic instruction."

The five evaluation criteria, each scored on a scale from 1 to 4, are:

Quality: overall correctness and usefulness of the instruction and its examples.
Alignment: how well the output matches the instruction's requirements.
Coherence: logical consistency and clarity of the instruction.
Realism: whether the instruction resembles a genuine clinical NLP task a practitioner might request.
Difficulty: whether the actual difficulty matches the intended difficulty level.

For each criterion $j$ , the final score $S_j \in [1, 4]$ is computed as a weighted average across the $M = 5$ independent judge samples:

$S_j = \frac{1}{M} \sum_{i=1}^{M} c_i \cdot s_{ij}$

where $s_{ij} \in \{1, 2, 3, 4\}$ is the integer score from the $i^{th}$ sample for criterion $j$ , and $c_i \in [0, M]$ are counts constrained by $\sum_{i=1}^M c_i = M$ . Actually, from the definition in the paper (Section 3.2.1), $c_i$ is the count of times score value $i$ appears across the $M$ samples, and $s_{ij}$ is the score value itself. So if $M = 5$ and the model assigns scores $\{3, 4, 3, 3, 4\}$ to the quality criterion, then $c_3 = 3$ , $c_4 = 2$ , and $S_{\text{quality}} = \frac{1}{5}(3 \cdot 3 + 2 \cdot 4) = \frac{17}{5} = 3.4$ .

What it computes: the expected judge score for each criterion, averaged over multiple independent generations to reduce the variance from a single stochastic judge call. The chain-of-thought component (not shown in the final score formula, but part of the judge prompt) provides a rationale alongside each score, which the system generates but apparently does not use in the final filtering — only the numeric scores contribute to the selection heuristic.

Why this form: self-consistency across multiple samples is a standard technique for improving the reliability of LLM judgments. A single call to GPT-4o-mini at temperature 1.0 might produce idiosyncratic scores due to randomness in the generation. Averaging across 5 independent calls smooths out this variance, producing a more reliable quality estimate. The chain-of-thought further encourages the judge to produce consistent reasoning before assigning scores, reducing the chance of arbitrary or contradictory ratings.

Stage iii: Rule-based filtering. After scoring, a heuristic trims the dataset:

"we use a heuristic to trim the collection down to its top-K highest quality samples based on the quality criteria."

The paper does not specify $K$ in the main text, but from the alignment experiments (Table 4), we can infer that the filtered versions used are 800,000 samples for the primary SFT run and 130,000 samples for the DPO preference data. The full MediFlow corpus is described as 2.5 million instructions (Section 3.2.1, and the title of the paper refers to "2.5 million high-quality instructions").

Dataset statistics. The paper provides token count distributions in Figures 7, 8, and 9:

Instruction tokens (Figure 7): average $301 \pm 295$ tokens — a very wide distribution, reflecting the variety of instruction styles (short imperative prompts vs. lengthy narrative instructions with detailed constraints).
Input tokens (Figure 8): average $76 \pm 67$ tokens — relatively short because the input is often a single clinical document or document section, though some longer documents produce a tail in the distribution.
Output tokens (Figure 9): average $79 \pm 66$ tokens — similar scale to the inputs, reflecting the fact that many tasks produce relatively compact outputs (extracted entities, short answers, summarised impressions) rather than long-form generations.

The difficulty skew. The paper favours "hard, very hard and extreme levels by a ratio of 3:1" (Appendix A.4). This means that of every 4 generated instructions, 3 are at the top difficulty levels and 1 is at moderate or moderate-hard. The lowest difficulty level (low) may not be generated at all. This deliberately biases the dataset toward tasks that require non-trivial clinical reasoning — multi-step inferences, synthesis across document sections, handling ambiguous presentations — rather than surface-level extraction. The rationale is that if the model can handle complex tasks, simple tasks become trivial; but the reverse is not true.

The output format split. The paper generates both plain text and JSON output formats. JSON is important for structured clinical NLP tasks — entity extraction, relation extraction, medical coding — where the output needs to be machine-readable. Plain text is appropriate for summarisation, QA, and error detection. Training on both formats during SFT teaches the model to respect output format constraints specified in the instruction.

The t-SNE visualisation (Figures 11, 12, 13). The paper uses OpenAI's text-embedding-3-large to embed each instruction (truncated to 256 dimensions), applies PCA to 50 dimensions, then t-SNE to 2 dimensions for visualisation. The scatter plots reveal:

Output format (Figure 13 top): creates clear local separation — JSON and plain-text instructions form distinct but adjacent clusters, suggesting the model learns format-specific patterns within similar task semantics.
Task type (Figure 11): affects the macro-scale structure, with different task categories occupying different regions of the embedding space. This is desirable — the dataset covers a diverse semantic range rather than collapsing to a few task archetypes.
Input data type (Figure 12): also affects macro-structure at a similar scale to task type. The diversity of 98 input configurations is reflected in broad coverage of the embedding space.
Difficulty level (Figure 13 bottom): does not show clear clustering — difficulty is not strongly correlated with the semantic content of the instruction, which makes sense because the same task and input type can be instantiated at different difficulty levels.

Design choices and justifications in MediFlow generation:

Why use GPT-4o for generation and GPT-4o-mini for judging? Economic pragmatism. GPT-4o is more capable and better at generating complex, realistic clinical instructions, but calling it 62,500 times for generation PLUS additional calls for judging 2.5M instructions would be prohibitively expensive. GPT-4o-mini is substantially cheaper and adequate for the simpler task of scoring against fixed criteria. The asymmetry — strong generator, weaker but faster judge — is a common pattern in synthetic data pipelines.
Why 10 instructions per call with 4 input-output pairs each? This maximises the information-per-API-call ratio. Each generation call produces 40 input-output pairs (10 instructions × 4 pairs), amortising the fixed cost of the prompt and the base generation overhead across many training examples. The alternative — generating one instruction with one pair per call — would require 40× more API calls for the same dataset size.
Why include "no headers" and synonym variation in the prompt? The paper explicitly aims to prevent the model from learning "shortcut" features — surface-level keywords that predict the task type without requiring actual understanding. If every NER instruction used the phrase "extract entities," the model could learn to trigger its NER capabilities based on that lexical cue rather than truly understanding the task description. By varying the instruction language (synonyms for "extract," different structural formats, sometimes no explicit task label at all), the dataset forces the model to infer the task from the full semantic context.
Why skew difficulty toward hard and above? The paper's motivation is clinical deployment, where failure modes are consequential. A model that excels at easy tasks but fails on complex ones is dangerous because users cannot reliably predict which category their query falls into. By training primarily on hard examples, the model develops capabilities that transfer downward to easier variants, while the reverse (training on easy examples transferring upward to hard cases) is less reliable.

3.4.5 Alignment via SFT and DPO

Supervised Fine-Tuning (SFT). The merged MediPhi model is fine-tuned on the MediFlow instruction dataset using standard next-token prediction on the output tokens, conditioned on the instruction and input. The hyperparameters (Table 7) are:

"AdamW optimizer, linear warmup (40 steps) followed by cosine decay, peak learning rate 2e-5, 2 epochs, NEFTune $\alpha = 5$ , effective batch size 256 (16 per GPU × 8 GPUs × 2 gradient accumulation)."

The maximum sequence length is 4,096 tokens (inherited from the base model's context window during PIT, Table 6). Training runs for 2 epochs over the filtered dataset.

Two SFT variants are tested (Table 4):

MediPhi-SFT 2.5M: Trained on all 2.5 million MediFlow instructions. Achieves 41.9 average accuracy on CLUE+.
MediPhi-SFT 800K: Trained on the top-800K quality-filtered subset. Achieves 43.0 average accuracy — 1.1 points higher than the full-dataset variant.

The fact that less data (800K filtered) produces better results than more data (2.5M unfiltered) confirms that quality filtering is effective — lower-quality instructions in the tail of the distribution may introduce noise or reinforce suboptimal patterns that dilute the training signal from high-quality examples.

Direct Preference Optimisation (DPO). After SFT, the model undergoes preference-based alignment using the MediFlow-DPO dataset. DPO (Rafailov et al., 2023) is a method for aligning language models to preference data without training a separate reward model. Given a prompt $x$ , a preferred ("chosen") response $y_c$ , and a dispreferred ("rejected") response $y_r$ , DPO optimises the model to increase the relative likelihood of $y_c$ over $y_r$ :

$\mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}_{(x, y_c, y_r) \sim \mathcal{D}} \left[ \log \sigma \left( \beta \log \frac{\pi_\theta(y_c | x)}{\pi_{\text{ref}}(y_c | x)} - \beta \log \frac{\pi_\theta(y_r | x)}{\pi_{\text{ref}}(y_r | x)} \right) \right]$

where $\pi_\theta$ is the policy (the model being trained), $\pi_{\text{ref}}$ is the reference policy (the SFT model, kept frozen), $\beta = 0.1$ is a temperature parameter controlling how strongly the model is pushed toward the preferred response, and $\sigma$ is the logistic sigmoid function.

What it computes: For each training example, DPO compares the model's log-probability improvement over the reference for the chosen response vs. the rejected response. If the model assigns much higher probability to the chosen response relative to the reference, and much lower to the rejected response, the sigmoid argument is large and positive, and the loss is small. If the model prefers the rejected response or shows insufficient preference for the chosen one, the sigmoid argument is small or negative, and the loss is large.

Why this form: DPO directly optimises the policy from preference pairs, bypassing the explicit reward model training and RL optimisation loop used in RLHF (which requires training a reward model on preference data, then using PPO to optimise the policy against that reward, with KL regularisation to prevent reward hacking). DPO's implicit reward function is $\beta \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}$ , which means the optimisation effectively increases the log-ratio for chosen responses and decreases it for rejected responses. The $\beta$ parameter controls the scale of these adjustments — $\beta = 0.1$ (from Table 8) means the preference signal is relatively weak, allowing the model to stay close to the SFT checkpoint while making targeted improvements.

The hyperparameters for DPO (Table 8) are: AdamW, linear warmup (50 steps) followed by cosine decay, peak learning rate $1 \times 10^{-6}$ (an order of magnitude lower than SFT, consistent with the higher sensitivity of preference optimisation), 1 epoch, effective batch size 128 (8 per GPU × 16 GPUs × 1 gradient accumulation), $\beta = 0.1$ .

Generating rejected outputs for DPO. The construction of the preference pairs is critical to DPO's effectiveness. The chosen response ( $y_c$ ) is the correct output from the MediFlow triplet. The rejected response ( $y_r$ ) must be marginally worse — incorrect in a subtle way that the model might plausibly produce, not obviously nonsensical. The paper describes:

"We generate marginally wrong outputs (i.e. rejected outputs) prompting GPT-4o as an error inducer. We provided the detail in Appendix A.7."

The error inducer prompt presents GPT-4o with a correct triplet (instruction, input, output) and asks it to generate a wrong output that introduces a specific error type while "still seeming reasonable at first glance." The error types are: ambiguity, partial correctness, over-verbosity, brevity, unbalanced detail, stylistic issues, factual inaccuracy, logical flaws, misinterpretation, simplistic reasoning, grammatical errors, and spelling errors. The error type is randomly sampled for each generation.

The MediFlow-DPO dataset is constructed by filtering MediFlow to "around 85k instructions keeping only the top triplets on the quality metric along with a stratification by task type, input-data type and output format," then sampling input-output pairs with decreasing density per instruction: top 20K instructions get 3 pairs each, next 25K get 2 pairs each, remaining 40K get 1 pair each, yielding 130,852 total preference pairs.

Why marginally wrong outputs matter for DPO. DPO works by contrast — the model must learn to distinguish good responses from bad responses. If the rejected responses are obviously terrible (gibberish, completely off-topic, contradicting the input), the contrast is too easy and the model learns only a trivial preference boundary. Marginally wrong responses — those that look plausible but contain subtle errors — teach the model to refine its outputs in nuanced ways: avoiding over-verbosity, catching logical flaws, maintaining factual precision. The 12 error types provide diverse forms of "wrongness" so the model doesn't overfit to a single failure mode.

Alignment results (Table 4). The DPO step provides an additional 0.4-point gain over SFT 800K (from 43.0 to 43.4), which may seem modest, but the paper notes that the relative gain from baseline (36.5 to 43.4, or 18.9% relative improvement) is the headline result. The alignment of the base Phi-3.5-mini directly on MediFlow (skipping the PIT and merging stages) reaches only 42.2 with SFT — 0.8 points below MediPhi-SFT 800K. This demonstrates that the knowledge acquired during PIT and preserved through merging provides a meaningful head start for alignment, and that alignment alone (without domain knowledge acquisition) leaves performance on the table.

3.4.6 The CLUE+ Benchmark and Evaluation Protocol

Benchmark composition. The original CLUE benchmark (Dada et al., 2024) covered 6 datasets using clinical notes and discharge summaries. CLUE+ adds 6 more, creating a 12-dataset evaluation suite:

Original CLUE (6 datasets):

MedNLI — Natural language inference on clinical text: given a premise and hypothesis, determine if the hypothesis is entailed, contradicted, or neutral. Input: premise + hypothesis; output: label.
Problem List Summarisation (PLS) — Given a progress note, extract the list of patient problems. Input: progress note; output: problem list.
MeQSum — Summarise a consumer health question. Input: consumer health question; output: summary question.
LongHealth — Answer questions given multiple clinical records. Input: clinical records + question; output: answer.
MeDiSumQA — Answer questions about a discharge letter. Input: discharge letter + questions; output: answers.
MeDiSumCode — Assign ICD-10 codes to a discharge letter. Input: discharge letter; output: ICD-10 codes.

CLUE+ additions (6 datasets): 7. MedConceptsQA ICD10CM — Multiple-choice questions about ICD-10 code definitions. Input: code + definition options; output: answer. The paper uses only hard and very hard questions from this dataset. 8. MedicationQA — Answer patient questions about medications. Input: medication question; output: answer. The paper cleaned this dataset by removing poorly formulated questions and reformulating verbose answers into direct responses (Appendix A.8.1). 9. MEDIQA-RRS QA — Answer questions about radiology report findings. Input: findings section + questions; output: answers. The paper reformulated the original summarisation task into QA pairs (Appendix A.8.2: "we reformulated the impressions to a series of question-answer pairs"). 10. SDoH (Social Determinants of Health) — Extract entities about employment, living status, and substance use from clinical notes. Input: clinical note; output: JSON entities with types and values. 11. ACI-Bench — Generate a detailed clinical note from a doctor-patient conversation. Input: conversation; output: clinical note with five required sections (chief complaint, history of present illness, physical exam, results, assessment and plan). 12. MEDEC — Detect a medical error in a clinical note. Input: clinical note (sentences numbered); output: sentence ID containing the error, or -1 if no error.

Evaluation protocol. The paper uses few-shot prompting with greedy decoding (Section 4.1, Table 9). This is an important methodological choice: the models are evaluated in a few-shot setting, not a fine-tuning setting. The few-shot examples are provided in the prompt to demonstrate the task format, but the model's parameters are not updated. This contrasts with Med42, which was "fine-tuned on ACI-Bench" (Table 5, asterisk note) — an in-distribution evaluation that inflates apparent performance.

The few-shot configurations are:

ICD10CM: 3 shots, accuracy metric
MedicationQA: 3 shots, Rouge-1 F1
RRS QA: 3 shots, Rouge-1 F1
SDoH: 4 shots, Type-Match F1 with boundary overlap (from Chai, 2019)
ACI-Bench: 1 shot, Rouge-1 F1
MEDEC: 2 shots, Sentence ID Accuracy

For the CLUE-original datasets, the paper follows Dada et al. (2024)'s exact prompt and metric configurations.

Why mixture of metrics matters. Clinical NLP tasks have diverse output types, and a single metric (like accuracy) cannot capture performance across all of them. Rouge-1 F1 measures n-gram overlap between generated and reference text (appropriate for summarisation and QA). Type-Match F1 with boundary overlap (from the eval4ner library, Chai, 2019) measures how well extracted entity spans match reference spans in both type label and boundary — a stricter metric than token-level F1 because it penalises partial-span matches. Sentence ID accuracy (for MEDEC) is a simple exact-match on the error location.

The system prompts. Each dataset has a custom system prompt that defines the model's role and output format (provided in Appendices A.8.1 through A.8.5). These prompts are detailed and task-specific — for example, the ACI-Bench prompt is over 50 lines long, specifying exact section headers, content requirements for each section, and formatting constraints. This reflects real clinical deployment: in practice, a clinical model would receive structured prompts defining exactly what clinical note format to produce, not just a vague "write a clinical note" instruction.

The model selection protocol. Table 5 presents the final comparison with other medical LLMs:

Mistral-7B-Instruct-v0.1 (base for BioMistral): 33.6 average on CLUE+.
BioMistral-7B-DARE: 34.7 (+1.1 over Mistral), gains on 8 datasets, CVΔ of 3.4 (highly uneven improvements).
Phi-3.5-mini (base for MediPhi): 36.5.
MediPhi (BreadCrumbs merged): 39.3 (+2.8 over Phi-3.5-mini), gains on 11 datasets, CVΔ of 1.5 (very uniform).
MediPhi-SFT: 43.0 (+6.5), gains on 9 datasets, CVΔ of 1.4.
MediPhi-Instruct (SFT+DPO): 43.4 (+6.9), gains on 9 datasets, CVΔ of 1.4.
Meta-Llama-3-8B-Instruct: 44.1 average (the strongest general-purpose baseline).
Llama3-Med42-8B: 45.3 (+1.2 over Llama-3), but with gains on only 5 datasets and CVΔ of 7.8 — a massive variance indicating that improvements are highly concentrated (primarily on ICD10CM, as noted in the text) while many datasets show degradation.

The key comparison is MediPhi-Instruct (3.8B, 43.4) vs. Llama-3 (8B, 44.1): a model with less than half the parameters achieves near-parity on this diverse clinical benchmark. Moreover, MediPhi-Instruct outperforms Llama-3 on four specific datasets: ICD10CM (+29.2%), MeDiSumCode (+13.9%), RRS QA (+5.8%), and MeQSum (+3.3%). This is the paper's central empirical claim: targeted domain adaptation with PIT and merging can make a small model competitive with a much larger general-purpose model on clinical tasks.

The catastrophic forgetting of #DG. An important observation from Table 4: the aligned models (MediPhi-SFT and MediPhi-Instruct) show a drop in #DG from 11 (MediPhi) to 9 — meaning alignment causes regression on 2 datasets while improving others. The specific datasets that degrade are Problem List Summarisation and MeDiSumCode. The paper hypothesises:

"we hypothesize this result from a specific bias in MediFlow towards listing tasks like extracting problems from clinical notes or medical codes from discharge summaries"

This suggests that while MediFlow is broad, it may under-represent certain task formats relative to the evaluation distribution, causing the model to partially unlearn them during SFT. This is a known tension in instruction tuning: alignment data inevitably shifts the model's output distribution, and gains on tasks well-represented in the alignment data may come at the cost of losses on under-represented tasks.

3.4.7 Additional Medical Benchmark Evaluation

The paper also evaluates on the traditional medical multiple-choice QA benchmarks (Table 13) — MedQA, MedMCQA, PubMedQA, and MMLU-medical — which are the standard evaluation suite for medical LLMs. The results are notably modest:

Phi-3.5-mini: 63.1 average across the four benchmarks.
MediPhi: 63.4 (+0.3 over base).
MediPhi-SFT: 64.2 (+1.1).
MediPhi-Instruct: 64.5 (+1.4).

The domain adaptation provides only a 1.4 percentage point boost on medical knowledge benchmarks, compared to 6.9 points on CLUE+. This is consistent with the paper's thesis: current medical LLMs are already strong on medical knowledge (multiple-choice factual questions) because general pretraining covers this well, but they underperform on clinical tasks (document processing, extraction, real-world reasoning). The PIT and alignment framework specifically targets clinical capabilities, not medical trivia, and the benchmark results confirm this differential impact.

4. Key Insights and Innovations

Innovation 1: Pre-Instruction Tuning as a General-Purpose Domain Knowledge Acquisition Scaffold, Not Just a QA Trick

The most intellectually distinctive conceptual move in this paper is the recognition that Pre-Instruction Tuning — originally designed by Jiang et al. (2024) as a narrow mechanism for teaching models factual knowledge through question-answering — is actually a general-purpose domain knowledge acquisition scaffold whose power comes from the task format itself acting as a learning catalyst, not from any particular task type. The paper demonstrates this by extending PIT from QA alone to summarisation, named entity recognition, and relation extraction, and showing that different task formulations yield different downstream benefits (Figure 4: summarisation outperforms QA, NER, and RE on ICD-10 coding by 6–8 percentage points).

What makes this a conceptual advance rather than an incremental extension is the implicit claim that the task format teaches the model how to attend to the domain content, not just what the content is. When a model is trained to produce summaries of ICD-10 coding webpages, it learns to identify salience, compress dense information, and restate technical concepts in coherent form — cognitive operations that transfer to clinical reasoning tasks like diagnosis coding and error detection. When trained on NER, it learns span-boundary discrimination — a different attentional skill. The task format is therefore a curriculum design choice that shapes what the model extracts from the domain data, not a neutral delivery mechanism for content.

This reframes the domain adaptation problem. Prior medical LLMs treated domain adaptation as a data problem: collect more medical text, train on it longer, mix it with general data to prevent forgetting (Chen et al., 2023; Labrak et al., 2024; Christophe et al., 2024a). The underlying assumption was that exposure to domain tokens drives learning, and the training objective (next-token prediction) is sufficient. The paper's PIT results challenge this directly: standard DAPT on ICD-10 webpages degrades performance to random (Figure 4, "Webpage"), while PIT with summarisation on the same underlying corpus improves it by 44% relative after merging (Figure 5). The data is identical; the difference is the learning objective and the task scaffold through which the model processes that data.

This is a fundamental shift, not an incremental refinement, because it implies that the bottleneck in clinical SLM development is not data availability but data structuring — how you convert available public corpora into training signals that teach transferable clinical skills. The paper's framework (Section 3.1.2) makes this explicit by separating Phase 1 (learning the task scaffold from synthetic outputs) from Phase 2 (integrating domain knowledge by training on task-output-plus-document concatenations). This two-phase structure is itself an architectural insight: the task scaffold must be established before the domain content is loaded, or the model learns surface statistics without developing the structured understanding needed for clinical reasoning.

The broader significance is that this approach is domain-agnostic. The same PIT pipeline — generate task outputs from domain corpora using a strong teacher model, train in two phases — could be applied to legal documents, scientific papers, or financial reports, with the task types chosen to match the downstream evaluation landscape. The paper's finding that summarisation outperforms QA for coding tasks while other task types may be optimal for other domains suggests a meta-principle: match the PIT task type to the cognitive operations required by the target evaluation tasks. This is a design principle for domain adaptation that the field didn't have before.

Innovation 2: Model Merging as a Deliberate Catastrophic Forgetting Recovery Mechanism, Not a Post-Hoc Ensemble Trick

Model merging has been explored in the general ML literature as a way to combine fine-tuned models without additional training (Wortsman et al., 2022; Yadav et al., 2024a; Ilharco et al.), and has been applied in medical NLP — most notably BioMistral's use of DARE merging (Labrak et al., 2024). But prior work treated merging as an optimisation: you have multiple checkpoints, merging gives you a better single model than any individual checkpoint. The conceptual contribution of this paper is repositioning merging as a deliberate catastrophic forgetting recovery mechanism that is synergistic with rather than supplementary to the domain adaptation process.

This distinction matters because it changes when and why you merge. BioMistral merged a general-purpose Mistral model with a DAPT-trained medical variant to produce a single model — but the motivation was to combine capabilities, not to recover lost capabilities. The paper's approach is different: PIT is acknowledged to cause catastrophic forgetting (Section 3.1.3: "While PIT enhances domain-specific learning, it also leads to catastrophic forgetting — degrading the model's initial abilities such as instruction following, long context handling, and multilingual support"), and SLERP merging with the base model is positioned as the solution to that specific problem. The merging step is not optional post-processing — it's an integral stage of the knowledge acquisition pipeline, applied after PIT but before multi-expert unification.

The evidence for this framing is in the ablations (Table 2). The Guideline expert without SLERP merging achieves only 27.2 average accuracy on CLUE+ (down 9.3 points from the 36.5 baseline), with gains on only 4 of 12 datasets. With SLERP merging, the same expert reaches 39.2 — recovering 12 points and improving on 10 datasets. The domain knowledge is present in both models (they were trained on the same data), but without merging, the general capabilities degrade so severely that the net effect is negative. This is a diagnostic finding: the performance of a domain-adapted model is not determined solely by how much domain knowledge it acquired, but by the ratio of domain knowledge gained to general capability lost. Merging shifts this ratio by recovering the lost capabilities without sacrificing the gained knowledge.

The paper's use of SLERP for the expert-to-base step and BreadCrumbs for the multi-expert step (Section 3.1.3–3.1.4) also reveals a hierarchical merging strategy that prior work didn't articulate. SLERP is optimal for two-model interpolation because it respects the spherical geometry of the parameter space; BreadCrumbs is optimal for many-model combination because its sparsity mechanism reduces interference between experts with conflicting parameter updates. The paper doesn't just apply one merging algorithm uniformly — it selects algorithms based on the structure of the merging problem (two models vs. five), which is a principled design choice rather than an arbitrary toolkit selection.

The conceptual implication is that modular domain adaptation with explicit forgetting recovery should replace end-to-end domain training as the default approach for adapting small models to specialised domains. The field's default — train on all available domain data, mix in general data to dilute forgetting — is shown to produce worse results (Table 3: DataMix achieves 37.5 vs. BreadCrumbs' 39.3, and improves on only 10 datasets vs. 11) while being computationally equivalent. The modular approach is strictly better because it separates the competing objectives (domain learning vs. capability preservation) and optimises each independently.

Innovation 3: The Coefficient of Variation of Gains/Losses (CVΔ) as an Evaluation Metric for Clinical Robustness

The paper introduces a deceptively simple metric — CVΔ, the coefficient of variation of per-dataset accuracy deltas (Equation 1) — that encodes a substantive claim about what makes a clinical model safe to deploy. The metric normalises the standard deviation of per-task gains/losses by the absolute mean gain/loss, producing a measure of how uniform the model's performance shifts are across diverse clinical tasks.

This is an innovation at the evaluation-methodology level, not the model-architecture level, but it has significant practical implications. The dominant evaluation paradigm in medical NLP — report average accuracy on a benchmark, compare to prior work, declare victory — is vulnerable to models that achieve high average scores through massive improvements on a few tasks while degrading on others. The Med42 model exemplifies this: it achieves 45.3 average on CLUE+ (the highest in Table 5), but improves on only 5 of 12 datasets with a CVΔ of 7.8. The 1.2-point gain over its Llama-3 base is driven primarily by ICD-10 coding (+27.7%), while other tasks may be degraded. In a clinical deployment, a model with this performance profile is dangerous because the end user cannot predict whether their query falls into the "massively improved" or "quietly degraded" category.

The paper uses CVΔ as a model selection criterion explicitly: the BreadCrumbs unified model is chosen over Task-Arithmetic despite Task-Arithmetic's higher average score (39.4 vs. 39.3) because BreadCrumbs has a lower CVΔ (1.5 vs. 1.9) and improves 11 datasets vs. 9. The rationale (Section 4.2.3) is that "the BreadCrumbs expert offers the best trade-off between high-amplitude improvements and consistent gains." This is a robustness-over-peak-performance preference that the field would benefit from adopting more broadly, particularly in safety-critical domains like healthcare.

The metric also serves a diagnostic function. The MedCode SLERP expert (Table 2) has a CVΔ of 39.6 — two orders of magnitude larger than the other experts. This instantly signals that the model's gains are hyper-concentrated (on coding tasks) while other datasets see losses — a pattern that is confirmed in the detailed results (Table 12: MedCode improves ICD-10 by 19.4 points but drops SDoH by 10.3 and ACI-Bench by 3.3). A researcher scanning only average accuracy (36.7, a marginal gain over the 36.5 baseline) might dismiss this expert as uninteresting; CVΔ reveals that it's actually highly specialised with concerning blind spots, which is exactly the kind of model you'd want to flag for further investigation before considering deployment.

This is a modest innovation in technical complexity but a potentially significant one in practice, because it provides a quantitative language for a qualitative concern (uniformity of improvement) that clinical practitioners have but the research community has not operationalised.

Innovation 4: Difficulty-Skewed Synthetic Data Generation as a Curriculum Design Principle

The MediFlow generation pipeline deliberately skews the difficulty distribution toward "hard, very hard, and extreme" levels at a 3:1 ratio (Appendix A.4). This is not a convenience choice or an accident of the generation process — it's a curriculum design principle that encodes the claim that training primarily on complex tasks produces models that generalise downward to simpler tasks, while the reverse is not true.

This principle has intuitive appeal but limited empirical validation in the clinical NLP literature. Most synthetic instruction datasets (Zhang et al., 2023b; Kweon et al., 2024) aim for representativeness — generate a distribution that mirrors the expected task distribution in deployment. The paper's approach is deliberately unrepresentative: it over-samples the tail of the difficulty distribution, betting that mastery of hard cases transfers to easy cases with minimal additional training. The mechanism is that hard cases require the model to develop robust, generalisable clinical reasoning strategies (multi-step inference, cross-sectional synthesis, ambiguity resolution), while easy cases can be handled by surface-level heuristics that don't transfer.

The paper doesn't directly ablate this design choice (e.g., comparing a difficulty-balanced MediFlow variant to the difficulty-skewed one), so the claim remains a hypothesis rather than a proven mechanism. But it's a conceptually interesting hypothesis because it inverts the standard curriculum learning assumption (start easy, progress to hard) in favour of an "anti-curriculum" where the model confronts the hardest cases first and the easy ones become trivial by comparison. If validated in future work, this would have implications beyond clinical NLP — it would suggest that synthetic instruction data for any domain should be skewed toward the upper end of the complexity distribution, with representativeness being a secondary concern.

Combined with the PIT insight (task format as learning scaffold) and the CVΔ metric (uniformity as deployment criterion), this difficulty-skew principle forms part of a larger conceptual framework the paper articulates implicitly: clinical model development should be structured around transferable skill acquisition, not content coverage. The goal is not to expose the model to every possible clinical document type (that's impossible with data constraints), but to teach it cognitive operations — synthesis, extraction, error detection, code mapping — that transfer across document types and task formats. Difficulty skew, task-type diversity (14 types in MediFlow), and output format variation (JSON and plain text) all serve this transfer goal by forcing the model to learn robust, format-agnostic clinical reasoning rather than surface-level pattern matching.

This is a fundamental reframing of what "alignment" means for clinical models. Rather than aligning the model to a representative distribution of user queries (the standard instruction-tuning paradigm), the paper aligns the model to a deliberately adversarial distribution designed to stress-test its clinical reasoning capabilities. The success of this approach (Table 4: +6.9% on CLUE+ from a 3.8B model) suggests the reframing has merit, even if the specific contribution is more of a design philosophy than a proven theorem.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation is conducted on the CLUE+ benchmark, an extension of the original CLUE benchmark (Dada et al., 2024). CLUE+ doubles the original size from 6 to 12 datasets, adding complementary clinical tasks and input document types. The original CLUE datasets are: MedNLI (Romanov and Shivade, 2018), MeQSum (Ben Abacha and Demner-Fushman, 2019), Problem List Summarization (Gao et al., 2023), LongHealth (Adams et al., 2024), MeDiSumQA (Dada et al., 2025), and MeDiSumCode (Dada et al., 2024). The six added datasets are: MedicationQA (Ben Abacha et al., 2019), MEDIQA-RRS QA (Ben Abacha et al., 2021), MEDEC (Ben Abacha et al., 2024), ACI-Bench (Yim et al., 2023), Social Determinant of Health (Lybarger et al., 2023), and MedConceptsQA ICD10CM (Shoham and Rappoport, 2024). Task types span natural language inference, summarisation, question-answering, reasoning, and information extraction. Input documents include clinical notes, discharge summaries, doctor-patient dialogs, radiology reports, and medication questions. Detailed dataset configurations (few-shot counts, metrics, system prompts) are provided in Tables 9 and 10 and Appendices A.8.1 through A.8.5. For the ICD-10 coding case study (Figures 4 and 5), the paper uses only the "hard" and "very hard" questions from MedConceptsQA ICD10CM.
Base model(s). All experiments use Phi-3.5-mini (Abdin et al., 2024a), a 3.8B-parameter instruction-tuned general-purpose language model. The paper selects this model because it is "representative of the capabilities of many contemporary LLMs" (Section 4) and sits in a parameter range where targeted domain adaptation can plausibly yield large relative improvements while remaining deployable on modest hardware. For comparison with other medical LLMs, the paper evaluates BioMistral-7B-DARE (Labrak et al., 2024) based on Mistral-7B-Instruct-v0.1, and Llama3-Med42-8B (Christophe et al., 2024a) based on Meta-Llama-3-8B-Instruct (Dubey et al., 2024). Additionally, GPT-4-0125 is evaluated on the ICD-10 coding subset as an external reference point.
Metrics. The primary aggregate metric is average accuracy on CLUE+, computed as the mean of per-dataset performance scores. The paper introduces two complementary metrics to capture the uniformity of improvements rather than just their magnitude: #DG (number of datasets on which the model achieves gains over the baseline, out of 12), and CVΔ, the coefficient of variation of gains/losses as defined in Equation 1. Per-dataset metrics vary by task type (Table 9): accuracy for MedNLI, LongHealth, MEDEC, and ICD10CM; Rouge-1 F1 (Lin, 2004) for MeQSum, MedicationQA, RRS QA, and ACI-Bench; Type-Match F1 with boundary overlap (via eval4ner; Chai, 2019) for SDoH. For the multiple-choice medical knowledge benchmarks (Table 13), simple accuracy is used. For the merge optimisation validation sets, average accuracy across the 12 synthetic validation sets serves as the fitness function.
Baselines. The paper establishes several tiers of baselines:
- Phi-3.5-mini (the unmodified base model) serves as the primary reference point for all domain adaptation experiments.
- DataMix: a model trained via standard DAPT (Domain Adaptation Pre-Training, next-token prediction) on all five corpus groups simultaneously, then SLERP-merged with the base model. This represents the "train on everything at once" strategy used by prior work (e.g., BioMistral).
- Guideline ablations (Table 2): versions of the Guideline expert without SLERP merging, without PIT, and without both, to isolate the contribution of each component.
- External medical LLMs (Table 5): Mistral-7B-Instruct-v0.1, BioMistral-7B-DARE, Meta-Llama-3-8B-Instruct, and Llama3-Med42-8B.
- GPT-4-0125 (Figures 4 and 5 only): evaluated on the ICD-10 coding subset as an upper-bound reference.
- For alignment experiments (Table 4): Phi-3.5-mini aligned directly on MediFlow via SFT and DPO (skipping the PIT and merging stages) serves as a baseline to isolate the contribution of the knowledge acquisition phase.
Generation budget / compute accounting. The paper measures compute in terms of GPU-hours (Section 5, Limitations: "8x80GB A100 GPUs on Azure Machine Learning for approximately 12,000 GPU-hours, of which close to 3,600 GPU-hours were dedicated to achieve the final model"). For the synthetic data generation, compute is measured in API calls and total tokens processed ("close to 25B input-output tokens" across GPT-4o, GPT-4o-mini, and text-embedding-3-large). There is no standardised generation budget metric (analogous to "number of sampled solutions" in inference-time scaling work) because the paper evaluates models in few-shot settings with greedy decoding — each model produces one deterministic output per prompt. For the evolutionary merge optimisation, the budget is measured in number of evaluations (500) of candidate merged models on the synthetic validation sets.
Cross-validation / statistical protocol. The paper does not employ traditional k-fold cross-validation for the main CLUE+ evaluation. Instead, it uses a validation-set-guided model selection protocol for the merging step: twelve synthetic validation sets (generated by GPT-4o, covering topics matched to the CLUE+ benchmark, with diversity ensured via HDBSCAN clustering) are used to guide the evolutionary algorithm that optimises merge weights (Section 3.1.4). The CLUE+ benchmark datasets themselves are never used for model selection or hyperparameter tuning — the validation sets are synthetic and distinct. For few-shot evaluation (Section 4.1, Table 9), the paper uses the same few-shot examples, prompts, and metrics as Dada et al. (2024) for the original CLUE datasets, and defines new configurations for the CLUE+ additions, but does not report confidence intervals or statistical significance tests for the per-dataset accuracy differences. The paper acknowledges the small test-set sizes in the Limitations section implicitly by noting the CLUE+ extension, but does not quantify variance across runs.

Main Quantitative Results

PIT Method Comparison on ICD-10 Coding (Figures 4 and 5)

The paper uses ICD-10 medical coding as an initial testbed to compare continual pre-training methods before scaling to the full CLUE+ benchmark.

Headline finding: PIT with summarisation, combined with SLERP merging, achieves a 44% relative improvement over the base model on ICD-10 coding, surpassing GPT-4-0125 by 14% relative.

In Figure 4, the base Phi-3.5-mini model achieves approximately 45% accuracy on the MedConceptsQA ICD10CM hard/very-hard subset (the exact baseline value is not stated explicitly in the text, but can be inferred from Figure 5 where the base model is at 45% before merging and from Table 12 where the baseline on the full ICD10CM set is 49.3%). The results for different continual pre-training methods are:

DAPT on raw ICD-10 webpages ("Webpage"): Performance drops to random (approximately 25%, consistent with 4-option multiple choice). The paper hypothesises that the "peculiar implicit format" of ICD-10 webpages impedes learning via next-token prediction.
Explainer (fine-tuning on GPT-4o-generated textbook-like rewrites of the webpages): Improves over baseline by approximately 6%, reaching roughly 51% accuracy.
PIT with summarisation ("Summary"): Improves by approximately 8% over the Explainer alone, reaching roughly 59% accuracy. This is the best configuration among PIT task types.
PIT with QA: Shows a smaller improvement over the baseline than summarisation.
PIT with NER ("Entities"): Shows a smaller improvement than summarisation.
PIT with RE ("Relations"): Initially declines compared to the baseline before merging (Figure 5, pre-SLERP bars), but recovers after merging.

Figure 5 then demonstrates the impact of SLERP merging at 50% interpolation:

Before SLERP merging (solid bars, lower values): All PIT variants except Summarisation show modest gains or losses relative to the baseline. The Summarisation model reaches approximately 59%, already a meaningful improvement.
After SLERP merging (striped bars, higher values): All variants improve substantially. The Summarisation expert reaches 65% accuracy — a 44% relative improvement over the 45% baseline (computed as (65−45)/45 ≈ 0.44). The QA expert reaches 62% (38% relative improvement, (62−45)/45 ≈ 0.38). The NER and RE experts, which showed losses before merging, recover to outperform the baseline. The Explainer model without PIT reaches 58% (28% relative improvement).
GPT-4-0125 achieves approximately 57% on this subset, meaning the Summarisation expert surpasses it by 8 percentage points (14% relative, computed as (65−57)/57 ≈ 0.14).

Based on these results, the paper selects PIT with summarisation as the default knowledge acquisition method for all subsequent experiments (Section 4.2.2: "Based on these results, the rest of the paper will apply PIT with summaries, unless otherwise stated").

SLERP Expert Performance on Full CLUE+ (Table 2)

Headline finding: Individual SLERP-merged experts achieve average improvements of up to 3.2 percentage points (8.8% relative) over the base model, with the MedWiki expert leading at 39.7 average accuracy and the Clinical expert close behind at 39.6.

Table 2 reports average accuracy on the full CLUE+ benchmark (12 datasets) for the five SLERP-merged experts, the DataMix baseline, and several Guideline ablations:

Phi-3.5-mini baseline: 36.5 average accuracy.
DataMix (DAPT on all corpora, SLERP-merged): 37.5 (+1.0), gains on 10 of 12 datasets (#DG = 10), CVΔ = 1.2 (the most uniform improvement pattern among all SLERP variants).
PubMed: 37.7 (+1.2), #DG = 9, CVΔ = 1.8.
Clinical: 39.6 (+3.1), #DG = 10, CVΔ = 2.0.
MedWiki: 39.7 (+3.2), #DG = 10, CVΔ = 1.5.
MedCode: 36.7 (+0.2), #DG = 5, CVΔ = 39.6. This expert shows marginal average improvement but heavily concentrated gains — it improves on only 5 datasets with extreme variance, being highly specialised for coding tasks. The detailed per-dataset results (Tables 11 and 12) confirm this: MedCode achieves 68.7 on ICD10CM (a 19.4-point gain over the 49.3 baseline) but drops sharply on SDoH (24.8 vs. 35.1 baseline, a 10.3-point loss) and ACI-Bench (39.0 vs. 42.3, a 3.3-point loss).
Guideline: 39.2 (+2.7), #DG = 10, CVΔ = 1.8.

The Guideline ablations in the same table quantify the contribution of each component:

Guideline w/o SLERP (PIT only, no merging): 27.2 (−9.3), #DG = 4, CVΔ = 1.0. This is a catastrophic degradation — PIT without the recovery merging drops performance by 9.3 points below baseline and improves on only 4 datasets.
Guideline w/o PIT (DAPT only, with SLERP): 33.0 (−3.5), #DG = 6, CVΔ = 0.9. Standard DAPT with merging still loses 3.5 points relative to baseline.
Guideline w/o SLERP & PIT (pure DAPT, no merging): 25.2 (−11.3), #DG = 1, CVΔ = 2.8. The worst configuration, losing 11.3 points and improving on only 1 dataset.

This ablation demonstrates that both PIT and SLERP merging are necessary conditions for positive domain adaptation — removing either causes performance to drop below the baseline. The full combination (PIT + SLERP) produces a 2.7-point gain over baseline and a 14-point recovery over the PIT-only variant.

Per-dataset analysis of SLERP experts (Tables 11 and 12). Looking at individual datasets reveals task-specific specialisation patterns:

MedNLI: The MedWiki expert leads at 72.8 (+6.2 over 66.6 baseline), followed by Guideline at 70.3 and Clinical at 69.2. Medical encyclopedia knowledge appears most beneficial for clinical inference.
Problem List Summarization: All experts show modest gains (0.6–1.4 points over 28.4 baseline), with Guideline leading at 29.8. The task may be near ceiling for this model scale.
MeQSum: Clinical leads at 38.1 (+1.4 over 36.7 baseline), consistent with the task requiring understanding of patient-generated text.
LongHealth: Most experts show slight losses relative to the 45.9 baseline — PubMed is flat at 45.7, Clinical drops to 43.5, MedWiki to 43.6. Only MedCode matches baseline at 45.7. Long-context multi-document QA may be resistant to domain adaptation through PIT.
MeDiSumQA: Clinical leads at 26.7 (+0.8 over 25.9 baseline), but gains are small across all experts (maximum +0.8). The task is difficult for all variants.
MeDiSumCode: Guideline leads at 41.9 (+0.8 over 41.1 baseline), with MedWiki close at 41.7. MedCode drops to 39.0 (−2.1), suggesting that the coding webpage expert's knowledge doesn't transfer well to discharge-letter-based code assignment.
RRS QA (radiology): Clinical dramatically outperforms all others at 52.1 (+10.9 over 41.2 baseline), while PubMed achieves only 44.1 (+2.9). This is the largest single-dataset gain among all experts and underscores the importance of matching training data type to evaluation task — clinical documents, not scientific articles, teach radiology report interpretation.
MedicationQA: All experts show small gains or minor losses. MedWiki leads at 12.2 (+1.0 over 11.2 baseline). The overall scores are low across all models, suggesting medication QA is a hard task for 3.8B models.
MEDEC (error detection): Clinical dominates at 34.5 (+19.7 over 14.8 baseline), with MedWiki at 28.8 (+14.0) and Guideline at 28.3 (+13.5). This is a massive relative improvement — more than doubling the baseline. The clinical document expert is best positioned to detect errors in clinical text.
ACI-Bench (clinical note generation): All experts show modest gains, with MedWiki and Guideline tied at 44.7 (+2.4 over 42.3 baseline), and MedCode dropping to 39.0 (−3.3).
SDoH (social determinants extraction): MedWiki leads at 43.6 (+8.5 over 35.1 baseline), with Guideline at 41.0 (+5.9). MedCode collapses to 24.8 (−10.3), consistent with the coding expert's narrow specialisation.
ICD10CM: MedCode dominates at 68.7 (+19.4 over 49.3 baseline), with all other experts showing marginal gains (49.5–50.2). This confirms the MedCode expert's hyper-specialisation.

The pattern across datasets reveals a clear specialisation-complementarity structure: Clinical excels at tasks involving patient-facing documents (RRS QA +10.9, MEDEC +19.7), MedWiki provides broad but moderate improvements (highest average, #DG = 10), MedCode is extremely strong on coding but weak elsewhere, and PubMed provides consistent small gains on scientific-reasoning tasks. No single expert dominates across all tasks, which motivates the multi-expert merging step.

Multi-Expert Merging Results (Table 3)

Headline finding: Merging all five SLERP experts via BreadCrumbs yields a unified model (MediPhi) with 39.3 average accuracy, improving on 11 of 12 datasets with the most uniform gain pattern (CVΔ = 1.5), outperforming the DataMix joint-training baseline by 1.8 points.

Table 3 compares three multi-model merging algorithms against the DataMix baseline and the range of individual SLERP expert performances:

DataMix (joint training baseline): 37.5, #DG = 10, CVΔ = 1.2.
SLERP Experts — Minimum: 34.5 (the worst-performing expert on each dataset, aggregated). This is a hypothetical lower bound.
SLERP Experts — Average: 38.5. This is the expected performance if you randomly selected one expert per dataset.
SLERP Experts — Maximum: 43.1 (the best-performing expert on each dataset, aggregated). This is a hypothetical upper bound — the best possible if you could dynamically route each query to the optimal expert.
Task-Arithmetic merged: 39.4 (+2.9 over baseline), #DG = 9, CVΔ = 1.9. Highest average accuracy among merging methods but improves on only 9 datasets, with higher variance (CVΔ = 1.9) indicating uneven gains.
TIES merged: 39.3 (+2.8), #DG = 7, CVΔ = 1.7. Slightly lower average but more uniform gains than Task-Arithmetic.
BreadCrumbs merged (MediPhi): 39.3 (+2.8), #DG = 11, CVΔ = 1.5. Matches TIES on average but improves on the most datasets (11 of 12) with the lowest CVΔ among all merging variants.

Several key observations from these results:

All merged models exceed the average of SLERP experts (38.5) by 0.8–0.9 points. This means merging is not simply averaging — it is extracting complementary strengths that raise the combined model above the mean of its constituents.
All merged models fall below the SLERP maximum (43.1) by 3.7–3.8 points. The "oracle" upper bound of per-task expert routing is substantially higher than any static merged model, suggesting that dynamic expert selection (if feasible) could yield further gains.
BreadCrumbs improves on the most datasets (11) but not on all. The one dataset where MediPhi loses relative to baseline is MedNLI (Table 11: 66.9 vs. 66.6 baseline, essentially flat). This is notable because MedNLI was improved by four of five SLERP experts (all except MedCode), yet the merged model fails to preserve this gain, suggesting some destructive interference in the merge.
BreadCrumbs substantially boosts ICD-10 coding relative to the MedWiki expert (which was the best non-coding expert). Table 12 shows MediPhi at 55.5 on ICD10CM vs. MedWiki at 50.2 — a 10.6% improvement. While still below the MedCode specialist at 68.7, this demonstrates successful transfer of coding knowledge from the MedCode expert into the merged model without the catastrophic losses on other tasks that MedCode suffers.
The DataMix baseline underperforms all three multi-expert merges (37.5 vs. 39.3–39.4), despite being trained on the same total data volume. This is critical evidence for the paper's modularity thesis: training separate experts on distinct data types and merging them is more effective than training one model on the pooled data, likely because the separate training avoids negative interference between data types with different statistical properties (scientific articles vs. coding webpages vs. clinical conversations).

The paper selects BreadCrumbs as the final model (designated MediPhi) based on the explicit criterion of robustness: "the BreadCrumbs expert offers the best trade-off between high-amplitude improvements and consistent gains on 11 datasets" (Section 4.2.3).

Alignment Results: SFT and DPO on MediFlow (Table 4)

Headline finding: Aligning MediPhi with MediFlow using SFT on 800K quality-filtered instructions achieves 43.0 average accuracy (+6.5 over baseline, +3.7 over unaligned MediPhi), and adding DPO further boosts to 43.4 (+6.9 over baseline, 18.9% relative improvement).

Table 4 reports the alignment progression:

Phi-3.5-mini baseline: 36.5, #DG = N/A (reference point).
Phi-3.5-mini + SFT 800K (alignment only, no PIT/merging): 42.2 (+5.7), #DG = 9, CVΔ = 1.4.
Phi-3.5-mini + DPO (SFT 800K + DPO): 42.2 (+5.7), #DG = 8, CVΔ = 1.4. DPO provides no additional gain over SFT alone when applied to the base model without domain knowledge acquisition — the model plateaus at 42.2.
MediPhi (unmerged, no alignment): 39.3 (+2.8), #DG = 11, CVΔ = 1.5.
MediPhi + SFT 2.5M (full MediFlow, unfiltered): 41.9 (+5.4), #DG = 9, CVΔ = 1.6. This underperforms the Phi-3.5-mini + SFT 800K baseline by 0.3 points, suggesting that training on lower-quality instructions in the 2.5M set introduces noise that degrades performance relative to a quality-filtered set.
MediPhi + SFT 800K: 43.0 (+6.5), #DG = 9, CVΔ = 1.4. This is 0.8 points above the Phi-3.5-mini + SFT 800K baseline, demonstrating that the domain knowledge acquired during PIT and preserved through merging provides a meaningful head start for alignment.
MediPhi + DPO (MediPhi-Instruct): 43.4 (+6.9), #DG = 9, CVΔ = 1.4. DPO adds 0.4 points over SFT alone when applied after the full knowledge acquisition pipeline, compared to zero gain when applied to the base model alone.

Key observations from the alignment stage:

Quality filtering matters substantially. The SFT 2.5M variant (41.9) underperforms SFT 800K (43.0) by 1.1 points, despite having 3× more training data. This confirms that lower-quality synthetic instructions in the tail of MediFlow actively degrade performance when included in training. The paper's LLM-as-a-Judge filtering (Stage iii of the MediFlow pipeline) is therefore not just a cost-saving measure — it's essential for maximising alignment quality.
Domain knowledge acquisition amplifies alignment gains. The total gain from Phi-3.5-mini to MediPhi-Instruct is +6.9 points, of which +2.8 comes from the PIT + merging phase and +4.1 comes from the alignment phase (+3.7 from SFT, +0.4 from DPO). The alignment gain on top of MediPhi (+4.1 over the 39.3 starting point) is larger in absolute terms than the alignment gain on top of Phi-3.5-mini (+5.7 over 36.5), even though the percentage improvement is smaller (10.4% vs. 15.6%). This is consistent with diminishing returns: the base model has more room to improve.
DPO provides a small but consistent benefit only when domain knowledge is present. On the base model, DPO adds nothing (42.2 → 42.2). On MediPhi, DPO adds 0.4 points (43.0 → 43.4). The paper does not explore why, but a plausible hypothesis is that DPO's contrastive signal (preferring correct outputs over marginally wrong ones) is only useful when the model already has sufficient domain knowledge to produce outputs that are "close" to correct — the marginally wrong outputs are discriminable only if the model understands the domain well enough to distinguish subtle errors from correct responses.
The #DG drops from 11 to 9 during alignment. Tables 11 and 12 show which datasets degrade: Problem List Summarization drops from 28.8 (MediPhi) to 26.9 (MediPhi-SFT) to 26.0 (MediPhi-Instruct), a loss of 2.8 points below the 28.4 baseline. MeDiSumCode drops from 41.7 (MediPhi) to 35.0 (MediPhi-SFT) to 37.2 (MediPhi-Instruct), a net loss of 3.9 points below the 41.1 baseline. The paper hypothesises this is due to "a specific bias in MediFlow towards listing tasks" — the MediFlow dataset may under-represent these task formats, causing partial catastrophic forgetting during SFT. This is a notable failure mode: alignment improves overall performance but can regress on specific under-represented tasks.
The DPO step partially recovers MeDiSumCode (35.0 → 37.2, a 2.2-point recovery) but further degrades Problem List Summarization (26.9 → 26.0, a 0.9-point additional loss). The preference data may contain examples that resemble MeDiSumCode-like tasks (code assignment from clinical documents), providing a corrective signal, while Problem List Summarization remains under-represented in both SFT and DPO data.

Per-dataset breakdown of alignment gains (Tables 11 and 12):

The largest absolute and relative improvements from alignment occur on tasks that closely match MediFlow's training distribution:

RRS QA (radiology): From 41.2 (baseline) → 44.5 (MediPhi) → 60.8 (MediPhi-SFT) → 61.6 (MediPhi-Instruct). Total gain: +20.4 points (49.5% relative). This massive improvement suggests that MediFlow contains substantial radiology report QA training data, consistent with the 98 document types including radiology reports at multiple granularities.
SDoH (entity extraction): From 35.1 (baseline) → 39.7 (MediPhi) → 54.5 (MediPhi-SFT) → 56.7 (MediPhi-Instruct). Total gain: +21.6 points (61.5% relative). MediFlow's inclusion of JSON-format entity extraction tasks with structured output formats directly trains the model for this evaluation.
MedicationQA: From 11.2 (baseline) → 11.3 (MediPhi) → 18.8 (MediPhi-SFT) → 19.3 (MediPhi-Instruct). Total gain: +8.1 points (72.3% relative). The large percentage gain reflects the very low baseline — medication QA was a weakness of the base model that alignment substantially addresses.
MEDEC (error detection): From 14.8 (baseline) → 29.1 (MediPhi) → 35.0 (MediPhi-SFT) → 34.4 (MediPhi-Instruct). Total gain: +19.6 points (132.4% relative). The largest relative improvement comes from a combination of domain knowledge (PIT nearly doubles the baseline) and alignment, though DPO slightly degrades this task (−0.6 from SFT peak).
MeQSum: From 36.7 (baseline) → 37.9 (MediPhi) → 42.8 (MediPhi-SFT) → 42.8 (MediPhi-Instruct). Total gain: +6.1 points (16.6% relative). This task plateaus at SFT with no DPO benefit.
ICD10CM: From 49.3 (baseline) → 55.5 (MediPhi) → 54.9 (MediPhi-SFT) → 54.9 (MediPhi-Instruct). Net gain: +5.6 points (11.4% relative). Notably, alignment slightly reduces ICD-10 performance from the MediPhi peak (55.5 → 54.9, a 0.6-point loss), suggesting that general clinical alignment may partially overwrite the specialised coding knowledge acquired from the MedCode expert during merging.

Comparison with Other Medical LLMs (Table 5)

Headline finding: MediPhi-Instruct (3.8B) achieves 43.4 average accuracy on CLUE+, within 0.7 points of Llama-3-8B-Instruct (44.1) despite having less than half the parameters, and substantially outperforms BioMistral-7B-DARE (34.7, +8.7 points) while showing more uniform gains across datasets.

Table 5 presents the head-to-head comparison:

Mistral-7B-Instruct-v0.1: 33.6 (baseline for BioMistral).
BioMistral-7B-DARE: 34.7 (+1.1), #DG = 8, CVΔ = 3.4.
Phi-3.5-mini (3.8B): 36.5 (baseline for MediPhi).
MediPhi (3.8B): 39.3 (+2.8), #DG = 11, CVΔ = 1.5.
MediPhi-SFT (3.8B): 43.0 (+6.5), #DG = 9, CVΔ = 1.4.
MediPhi-Instruct (3.8B): 43.4 (+6.9), #DG = 9, CVΔ = 1.4.
Meta-Llama-3-8B-Instruct: 44.1 (baseline for Med42).
Llama3-Med42-8B: 45.3 (+1.2), #DG = 5, CVΔ = 7.8.

The key comparisons reveal:

MediPhi's gain over its base model (+6.9) far exceeds BioMistral's (+1.1) and Med42's (+1.2). This is the paper's central comparative claim — the modular framework (PIT + merging + alignment) provides substantially more effective domain adaptation than prior approaches (DAPT + DARE merging for BioMistral; SFT alignment for Med42).
MediPhi-Instruct achieves near-parity with Llama-3-8B (43.4 vs. 44.1) — a difference of only 0.7 points (1.6% relative). This is remarkable given that Llama-3 is a state-of-the-art general-purpose model with more than twice the parameters, trained on an order of magnitude more data. The paper is careful not to claim superiority, but rather to demonstrate that targeted domain adaptation can close most of the gap to a much larger general model on clinical tasks.
MediPhi-Instruct outperforms Llama-3-8B on four specific datasets (per the text and Tables 11–12):
- ICD10CM: 54.9 vs. 25.7 (+29.2 points). The biomedical and clinical pretraining of Llama-3 does not cover ICD-10 coding well, while MediPhi's MedCode expert and alignment explicitly address it.
- MeDiSumCode: 37.2 vs. 27.8 (+9.4 points, reported as +13.9% in the text — the discrepancy suggests percentage computation vs. absolute difference computation).
- RRS QA: 61.6 vs. 55.8 (+5.8 points). Radiology report interpretation benefits from targeted clinical document exposure.
- MeQSum: 42.8 vs. 39.5 (+3.3 points).
Llama-3-8B substantially outperforms MediPhi-Instruct on other datasets — most notably MEDEC (46.5 vs. 34.4, −12.1 points), SDoH (63.1 vs. 56.7, −6.4 points), LongHealth (58.8 vs. 45.0, −13.8 points), and ACI-Bench (50.2 vs. 43.5, −6.7 points). These gaps suggest that the 3.8B parameter budget limits performance on tasks requiring complex reasoning over long documents or structured output generation, even with targeted adaptation. Llama-3's larger capacity provides advantages that domain adaptation cannot fully compensate for.
Med42's CVΔ of 7.8 is strikingly high — nearly 5× larger than MediPhi-Instruct's 1.4. This confirms the paper's critique that Med42's gains are highly concentrated (primarily on ICD10CM at +27.7% and ACI-Bench, where Med42 was fine-tuned on the training set, making its evaluation in-distribution). The #DG of only 5 (out of 12) means Med42 actually degrades on the majority of CLUE+ datasets relative to its Llama-3 base.
BioMistral's CVΔ of 3.4 is intermediate — better than Med42's but substantially worse than MediPhi's 1.4. BioMistral improves on 8 datasets but with uneven magnitude, consistent with the DARE merging approach being less effective at producing uniform gains than the PIT + BreadCrumbs combination.

Multiple-Choice Medical Knowledge Benchmarks (Table 13)

Headline finding: The MediPhi framework provides only modest gains on traditional medical knowledge benchmarks (+1.4 points average, from 63.1 to 64.5), consistent with the paper's thesis that clinical task performance requires different training strategies than medical knowledge recall.

Table 13 reports results on MedQA, MedMCQA, PubMedQA, and MMLU-medical:

Phi-3.5-mini: 48.6 / 55.4 / 76.8 / 71.5 → AVG 63.1.
MediPhi: 49.1 / 55.9 / 76.6 / 72.0 → AVG 63.4 (+0.3).
MediPhi-SFT: 53.6 / 55.2 / 76.6 / 71.6 → AVG 64.2 (+1.1).
MediPhi-Instruct: 54.8 / 55.5 / 76.4 / 71.4 → AVG 64.5 (+1.4).

The pattern is revealing:

MedQA shows the largest improvement (+6.2 points from 48.6 to 54.8), likely because MedQA questions resemble the clinical reasoning tasks emphasised in MediFlow.
MedMCQA is essentially flat (+0.1), suggesting the broad medical knowledge tested by MedMCQA is already well-covered by the base model's pretraining.
PubMedQA shows a slight decline (−0.4), possibly because the alignment to clinical workflows changes the model's output style in ways that are suboptimal for biomedical research question-answering.
MMLU-medical is also essentially flat (−0.1).

The differential impact — 6.9 points on CLUE+ vs. 1.4 points on medical knowledge benchmarks — directly supports the paper's framing that current medical LLMs are "good at medical exams but struggle with real-world clinical complexities" (Section 1). The PIT and alignment framework specifically targets clinical task performance, not medical knowledge acquisition, and the benchmark results confirm this specificity.

Ablation Studies and Robustness Checks

PIT task type comparison (Figure 4): Across summarisation, QA, NER, and RE tasks applied to ICD-10 coding webpages, summarisation provides the largest improvement (roughly 59% accuracy vs. 45% baseline, pre-merging), followed by QA, with NER and RE showing minimal gains or losses before SLERP merging. This establishes task-type selection as a consequential hyperparameter — different task formulations yield meaningfully different downstream performance.

SLERP merging proportion and necessity (Figure 5, Table 2): Figure 5 demonstrates that SLERP merging at 50% interpolation systematically boosts performance for all task types on ICD-10 coding, with the Summarisation model reaching 65% (44% relative gain). Table 2's Guideline ablations show that removing SLERP merging causes a catastrophic 9.3-point drop (39.2 → 27.2), establishing merging as a necessary component, not an optional enhancement.

Multi-expert merging algorithm comparison (Table 3): Task-Arithmetic achieves the highest average (39.4) but with uneven gains (CVΔ = 1.9, #DG = 9), while BreadCrumbs achieves essentially the same average (39.3) with substantially more uniform improvements (CVΔ = 1.5, #DG = 11). This demonstrates that the choice of merging algorithm affects not just average performance but the distribution of gains across tasks, with BreadCrumbs' sparsity mechanism reducing interference between experts more effectively than dense task vectors.

DataMix vs. modular training (Tables 2 and 3): The DataMix baseline (DAPT on all corpora, SLERP-merged) achieves 37.5 on CLUE+, while the average of individual SLERP experts is 38.5 and BreadCrumbs reaches 39.3. The modular approach (separate experts, then merge) outperforms joint training by 1.0–1.8 points, providing evidence for the paper's thesis that different medical data types benefit from separate adaptation before combination.

Alignment data quantity and quality (Table 4): Training on all 2.5M MediFlow instructions (SFT 2.5M) yields 41.9, while training on the top-800K quality-filtered subset yields 43.0 — a 1.1-point gain from using less but higher-quality data. This validates the LLM-as-a-Judge filtering pipeline and demonstrates that synthetic instruction quality, not quantity, is the binding constraint for alignment performance.

DPO effectiveness with and without domain knowledge (Table 4): DPO provides no gain when applied to the base model (42.2 → 42.2) but provides a 0.4-point gain when applied after PIT and merging (43.0 → 43.4). This suggests that preference optimisation requires a model that already has sufficient domain knowledge to benefit from contrastive signals — the "marginally wrong" outputs used as rejected responses are only discriminable when the model understands the domain.

Alignment of base model vs. aligned MediPhi (Table 4): Phi-3.5-mini + SFT 800K achieves 42.2, while MediPhi + SFT 800K achieves 43.0 — a 0.8-point advantage for the domain-adapted starting point. This confirms that the knowledge acquired during PIT provides a meaningful head start for alignment, even though alignment alone (without PIT) already provides substantial gains (+5.7 over baseline).

Guideline component ablation (Table 2): The four Guideline variants isolate the contributions of PIT and SLERP:

Guideline full (PIT + SLERP): 39.2
Guideline w/o SLERP (PIT only): 27.2 (−12.0)
Guideline w/o PIT (DAPT + SLERP): 33.0 (−6.2)
Guideline w/o both (DAPT only): 25.2 (−14.0)

The interaction is super-additive: the benefit of PIT alone (relative to no PIT and no SLERP) is roughly +2.0 (27.2 − 25.2), the benefit of SLERP alone is roughly +7.8 (33.0 − 25.2), but the combined benefit is +14.0 (39.2 − 25.2), which is greater than the sum of the individual benefits (2.0 + 7.8 = 9.8). This suggests a synergistic interaction — PIT enhances domain learning, and SLERP not only recovers lost capabilities but also amplifies the domain knowledge by re-integrating it with the instruction-tuned base.

Evolutionary merge optimisation termination (Section 3.1.4): The evolutionary algorithm terminates after 500 evaluations. The paper does not report the trajectory of fitness scores over evaluations, so we cannot assess whether the search converged or whether more evaluations would have yielded better merge configurations. This is a potential limitation — 500 evaluations in a high-dimensional merge weight space (5 experts × multiple layers × per-layer configurations for BreadCrumbs) may be insufficient for convergence to a global optimum.

Synthetic validation set construction (Appendix A.3): The twelve validation sets use HDBSCAN clustering on question embeddings to ensure diversity, keeping one sample per cluster plus all outliers for up to 1,200 samples per set. The paper does not report the cluster counts or outlier fractions, nor does it validate that the validation set accuracies correlate with CLUE+ test set accuracies. Without this correlation analysis, the effectiveness of the synthetic validation sets for model selection is assumed rather than demonstrated — a merged model that performs well on the synthetic validation sets might not be optimally configured for the real CLUE+ distribution.

MediFlow judge score distributions (Figure 10): The quality, alignment, coherence, and realism criteria show peaks at 3 (on a 1–4 scale) with distributions concentrated between 3 and 4, suggesting the generation pipeline produces mostly high-quality instructions. The difficulty criterion shows a similar peak at 3 but with a tail extending toward 2, indicating that some generated instructions are easier than intended — the "hard, very hard, extreme" skew was not perfectly achieved. The paper does not analyse whether samples with low difficulty scores correspond to lower downstream training utility.

Critical Assessment

The experiments support the paper's central claims, but with important caveats about generalisability, evaluation design, and the strength of evidence for specific components.

Claim: "Pre-instruction tuning significantly enhances domain adaptation." This claim is supported for the ICD-10 coding case study and the Guideline expert, but the evidence for other experts is circumstantial. The PIT method comparison (Figure 4) directly compares DAPT, Explainer, and PIT variants on ICD-10 coding, showing PIT with summarisation achieves the largest gain. The Guideline ablations (Table 2) show a 6.2-point gap between PIT-trained Guideline (39.2) and non-PIT Guideline (33.0). However, the paper does not replicate this ablation for the other four experts — we do not know whether PIT is similarly essential for PubMed, Clinical, MedWiki, or MedCode, or whether DAPT would have been sufficient for some of them. The MedCode expert's extreme CVΔ (39.6) suggests PIT did not prevent catastrophic specialisation on coding at the expense of other tasks, which raises questions about whether PIT is uniformly beneficial or only beneficial when the PIT task type matches the downstream evaluation. The paper's decision to use summarisation as the default PIT task for all experts (based on the ICD-10 case study) is a pragmatic choice, but the optimal PIT task type for clinical document understanding (which requires different skills than coding webpage comprehension) may differ from the optimal task type for coding. The paper does not explore this, making the claim of broad PIT effectiveness more an extrapolation than a demonstrated finding.

Claim: "Model merging produces uniform gains across diverse clinical tasks while recovering degraded general capabilities." This claim is strongly supported for the BreadCrumbs variant (CVΔ = 1.5, #DG = 11) and for the SLERP step in isolation (Table 2: Guideline w/o SLERP loses 9.3 points from baseline, Guideline with SLERP gains 2.7). The uniformity claim is specifically supported by the comparison with Med42 (CVΔ = 7.8, #DG = 5) and BioMistral (CVΔ = 3.4, #DG = 8), which shows that prior medical LLMs achieve their average gains through highly concentrated improvements. However, the paper does not demonstrate that the specific merging algorithms chosen (SLERP for 2-model, BreadCrumbs for 5-model) are genuinely better than alternatives in a controlled comparison — the evolutionary search over 500 evaluations explores the space of merge configurations but does not compare, say, BreadCrumbs against a simpler weighted-average baseline at the same level of hyperparameter optimisation. The differences between BreadCrumbs (39.3, #DG = 11) and Task-Arithmetic (39.4, #DG = 9) are small enough that they could be due to stochastic variation in the evolutionary search rather than algorithmic superiority, and the paper's preference for BreadCrumbs based on #DG rather than average accuracy is a value judgment about the importance of uniformity that not all practitioners would share.

Claim: "MediFlow alignment achieves 18.9% relative improvement on CLUE+." This claim is supported with the caveat that the 18.9% figure is computed relative to the Phi-3.5-mini baseline (36.5 → 43.4, an 18.9% increase), which includes both the PIT/merging phase and the alignment phase. The alignment-specific gain (MediPhi → MediPhi-Instruct: 39.3 → 43.4) is 10.4% relative. The figure of 18.9% is therefore an overstatement of the alignment contribution alone — it bundles the domain knowledge acquisition and the alignment into a single number. More importantly, the claim does not account for the regression on Problem List Summarization (−2.4 points) and MeDiSumCode (−4.5 points) during alignment. The net gain of 6.9 points is an average that masks task-specific losses, which matters because clinical deployment cannot tolerate regression on safety-critical tasks. The paper's hypothesis about MediFlow bias toward "listing tasks" is plausible but untested — an ablation with MediFlow variants that explicitly oversample under-represented task types would strengthen the claim that the regression is a data coverage issue rather than a fundamental limitation of the SFT approach.

Claim: "The ICD-10 coding expert surpasses GPT-4-0125 by 14%." This claim is supported, but the comparison is narrow. The 14% figure applies only to the MedConceptsQA ICD10CM hard/very-hard subset (multiple-choice code definition questions), not to the broader ICD-10 coding task (MeDiSumCode, which involves assigning codes to discharge summaries). On the broader task, the MedCode expert achieves 68.7 on ICD10CM multiple-choice but drops to 39.0 on MeDiSumCode (Table 11, a 2.1-point loss from baseline), demonstrating that multiple-choice knowledge of code definitions does not transfer to the practical skill of assigning codes to clinical documents. GPT-4-0125 is not evaluated on the full CLUE+, so we cannot assess whether it would outperform MediPhi on clinical tasks more broadly. The claim of GPT-4 superiority on a narrow subset is accurate but potentially misleading if interpreted as general clinical superiority.

Claim: "MediPhi-Instruct achieves near-parity with Llama-3-8B on CLUE+." This claim is supported (43.4 vs. 44.1, difference of 0.7 points). However, the comparison is slightly asymmetric: MediPhi-Instruct is compared against the Llama-3-8B-Instruct baseline in a few-shot setting, but the MediPhi models underwent extensive domain adaptation while Llama-3 did not receive any medical adaptation. A fairer comparison would be MediPhi-Instruct vs. a medical-adapted Llama-3-8B (e.g., Med42), but Med42's uneven performance (only 5 datasets improved) makes it an imperfect reference point. The more informative finding is not the near-parity with Llama-3 but the pattern of strengths: MediPhi-Instruct dominates on tasks where Llama-3 lacks domain knowledge (ICD-10, medical coding from discharge letters, radiology) but substantially underperforms on tasks requiring broader reasoning capacity (LongHealth, MEDEC, SDoH). This pattern is exactly what one would expect from a domain-adapted small model — it wins on domain-specific tasks and loses on general-reasoning tasks — and the paper's framing of "near-parity" somewhat obscures this complementary strength profile.

Genuine weaknesses in the experimental design:

No direct PIT ablation for non-Guideline experts. We do not know whether PIT is necessary for PubMed, Clinical, MedWiki, or MedCode, or whether DAPT would have performed comparably. The DataMix baseline (DAPT on all corpora) is the closest comparison, but it trains on pooled data rather than separate corpora, conflating the training strategy with the data organisation. A cleaner comparison would be DAPT-trained individual experts (one per corpus) merged in the same way as the PIT experts, which would isolate the PIT contribution per data type.
No ablation of the two-phase PIT structure. The paper implements Phase 1 (task outputs only) followed by Phase 2 (task + document concatenation) but does not test whether a single-phase approach (task + document together from the start) would perform equivalently. The two-phase structure is inherited from Jiang et al. (2024) but its necessity for the medical domain is assumed rather than tested.
The difficulty skew in MediFlow is not ablated. The paper claims that training on harder examples transfers downward to easier ones, but this is not tested. A comparison of difficulty-balanced vs. difficulty-skewed MediFlow variants would be straightforward and informative.
No confidence intervals or statistical significance tests on CLUE+ results. The 0.7-point difference between MediPhi-Instruct and Llama-3-8B (43.4 vs. 44.1) may not be statistically significant given test-set sizes of 100–1,200 per dataset, but the paper provides no way to assess this. The evolutionary merge optimisation terminates at 500 evaluations without reporting convergence diagnostics, leaving open the possibility that the BreadCrumbs merge configuration is a local optimum rather than the best achievable.
The synthetic validation sets for merge optimisation are not validated against CLUE+. The paper uses 12 synthetic validation sets to guide the evolutionary search, but does not report the correlation between validation set accuracy and CLUE+ test set accuracy. If the validation sets do not correlate well with the real benchmark, the merge optimisation may select configurations that are optimised for the wrong distribution.
Single model family (Phi-3.5-mini). All experiments use the same base model. It is unknown whether PIT would be effective on other architectures (Llama, Mistral) or whether the modular approach would transfer. The paper's claim that Phi-3.5-mini is "representative" is an assertion, not a demonstrated fact.
Limited test-set sizes for individual datasets. The CLUE+ benchmark aggregates 12 datasets, but individual datasets like MeDiSumCode and Problem List Summarization have relatively few test examples (the original CLUE paper and this extension do not specify exact test-set sizes, but these are typically in the low hundreds). Performance estimates on these datasets have high variance, making per-dataset comparisons (e.g., "MediPhi-Instruct loses 2.4 points on Problem List Summarization") potentially unreliable.

Missing experiments that would strengthen the paper:

DAPT-trained individual experts (without PIT) for each corpus group, SLERP-merged and multi-expert-merged in the same way as the PIT experts. This would isolate the contribution of PIT per data type and per downstream task.
Difficulty-balanced MediFlow vs. difficulty-skewed MediFlow alignment comparison, to test the anti-curriculum hypothesis.
Confidence intervals or bootstrap standard errors on the CLUE+ average accuracy for key comparisons (MediPhi-Instruct vs. Llama-3-8B, BreadCrumbs vs. Task-Arithmetic).
Correlation analysis between synthetic validation set accuracy and CLUE+ test accuracy, to validate the merge optimisation protocol.
PIT task-type ablation for non-coding experts, testing whether summarisation remains optimal for clinical data types (vs. QA or NER for document understanding tasks).
Scaling analysis: how does the PIT + merging framework scale with model size? If applied to Phi-4 (14B), would the relative gains be larger (more capacity to absorb domain knowledge) or smaller (base model already stronger)?
Data contamination analysis: the MediFlow pipeline uses GPT-4o, which may have seen the CLUE+ benchmark datasets during its training. The paper does not discuss whether the synthetic instructions inadvertently reproduce evaluation examples, which would inflate alignment gains through memorisation rather than genuine skill acquisition.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Prohibitive for Deployment and Unaccounted for in the Headline Efficiency Numbers

The assumption or constraint. The entire compute-optimal test-time scaling framework hinges on knowing each question's difficulty before allocating the inference budget. The paper's method for estimating difficulty — whether oracle (computing pass@1 from 2,048 samples per question) or predicted (averaging PRM final-answer scores across the same 2,048 samples) — requires generating and scoring 2,048 complete solutions per question. The authors acknowledge this explicitly (Section 3.2):

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. Generating 2,048 samples per question is equivalent to or larger than the largest test-time compute budgets studied (256–512 generations). In a real deployment, the total cost would be difficulty estimation + strategy execution, and for the budgets where the 4× claim is made (e.g., 16 generations matching best-of-N at 64), the estimation cost dwarfs the execution cost by a factor of ~128×. The practical efficiency gain — accounting for estimation — would be negligible or negative at low-to-moderate budgets. Only at very high budgets (where the execution cost approaches the estimation cost) would the headline gains begin to materialize.

What evidence exists in the paper. The verification is indirect but clear: the difficulty estimation procedure is described in Section 3.2 as using 2,048 samples and the PRM's final-answer score. The paper's main efficiency claim appears in Figures 4 and 8 (captions and surrounding text): 4× compute reduction, e.g., "16 generations matching best-of-N weighted at 64 generations." The estimation cost of 2,048 samples is never included in any budget calculation or efficiency figure. The paper provides no alternative difficulty estimation method — the predicted-difficulty variant still requires 2,048 samples, it just substitutes PRM scores for ground-truth correctness checks.

Mitigation status. The authors flag this as "a key avenue for future work" (Section 3.2) and suggest exploring "the trade-off between compute spent assessing difficulty versus compute spent solving the problem." They also propose training models to predict difficulty directly from question text, but no such model is developed or evaluated. The limitation is acknowledged but entirely unresolved — the 4× figure should be understood as an upper bound on achievable efficiency, not a realized deployment gain.

The Method Fails Completely on Hard Problems, Offering No Path Forward

The assumption or constraint. The test-time compute framework operates on the premise that the base model's proposal distribution already contains correct solutions at some non-trivial rate — i.e., pass@1 is meaningfully above zero. When this condition fails, as it does for the hardest problems (difficulty bin 5), the method provides essentially no benefit regardless of budget, strategy, or allocation. The paper's core insight — that test-time compute amplifies existing capability — implies its own boundary: it cannot create capability that the base model lacks.

The consequence. For problems in difficulty bin 5, where the base model's pass@1 is near zero, none of the methods studied (search, revisions, or their compute-optimal combinations) produce meaningful improvement. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all methods at all budgets. Figure 7 (right) shows bin 5 at roughly 2–3% irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%. This means the framework offers no solution for genuinely novel or out-of-distribution reasoning tasks — the very problems where practitioners most want AI assistance. For clinical deployment, this maps to cases where the model simply doesn't have the requisite medical knowledge to produce a correct answer, and no amount of inference-time effort compensates. The model cannot "figure out" something it fundamentally does not know.

What evidence exists in the paper. The evidence spans all major result figures. Figure 3 (right), difficulty bin 5 panel: both beam search and best-of-N weighted remain at 1–3% from 4 to 256 generations. Figure 7 (right), bin 5: all ratios yield ~2–3% at 128 generations. Figure 9, bin 5 line (blue, bottommost): flat near 0–5% for all R values in both revisions and search panels. Table 5 in the example paper (noting the FLOPs-matched disadvantage of test-time compute on hard problems at R ≫ 1, reaching −52.9% relative for PRM search) is consistent with this pattern.

Mitigation status. The paper is transparent about this boundary (Section 7, takeaway box: test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address"). However, this is a statement of the limitation, not a mitigation. The only path forward is scaling pretraining — a larger model, more data, or both — which is precisely the approach the paper's FLOPs-matched comparison evaluates against. The framework provides no mechanism for determining a priori whether a given problem falls into bin 5 without the expensive difficulty estimation step, meaning resources may be wasted on unsolvable problems in deployment.

Verifier Over-Optimization Is a Hard Ceiling on Scaling, Not a Solved Problem

The assumption or constraint. The entire search-based approach (beam search, lookahead search, best-of-N weighted) depends on the process reward model reliably discriminating correct from incorrect solutions. The paper documents that this reliability breaks down under aggressive optimization — the PRM is vulnerable to "over-optimization" where search finds solutions that score highly under the verifier but are actually incorrect, causing performance to plateau or degrade at high budgets.

The consequence. The compute-optimal policy mitigates over-optimization by routing easy problems away from aggressive search (where the phenomenon is most severe) toward best-of-N, and reserving beam search for medium-difficulty problems where the PRM signal still provides genuine guidance. But this is a workaround, not a solution. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling — the beam search curves in Figure 3 flatten or decline well before the budget is exhausted. On easy problems at very high budgets, even best-of-N weighted may eventually over-optimize (though this regime is not explored in the paper). This means further scaling of test-time compute — beyond the budgets studied — will hit a verifier-imposed performance wall regardless of allocation strategy. The bottleneck is verifier quality, not search algorithm sophistication, and improving verifier robustness is a separate research challenge the paper does not address.

What evidence exists in the paper. Figure 3 (right), difficulty bin 1: beam search degrades from ~78% to ~77% as budget increases from 4 to 256 generations, while best-of-N weighted improves from ~68% to ~88%. This is the clearest evidence of PRM over-optimization — beam search finds solutions that exploit the verifier signal at the expense of actual correctness. Figure 3 (left): lookahead search, the most powerful optimizer, paradoxically performs worst overall because its extra cost reduces effective beam count while its more aggressive search amplifies over-optimization. Appendix M (Figure 29) provides qualitative examples of degenerate outputs: repetitive low-information steps, overly short 1–2 step solutions that score highly under the PRM but are incorrect.

Mitigation status. The compute-optimal policy mitigates over-optimization by strategy selection (avoiding beam search on easy problems), but this is a routing heuristic, not a solution to the underlying verifier quality problem. The paper acknowledges the issue in Section 5.3 ("This degradation at high budgets is attributed to over-optimization of the PRM") and Section 8 (verifier over-optimization as a "primary bottleneck"), but does not propose or evaluate any method for improving verifier robustness. The current results are therefore specific to the verifier quality achievable with the Monte Carlo rollout training procedure described in Appendix D. Improving the PRM — through adversarial training, ensemble methods, or better calibration — would likely shift the difficulty thresholds and change the optimal policy, but the paper provides no guidance on how to do so or how much improvement is possible.

Single Benchmark, Single Model Family Constrains Generality

The assumption or constraint. All experiments are conducted exclusively on the MATH benchmark (500 test questions) using PaLM 2-S* as the base model. The paper asserts that this model is "representative of the capabilities of many contemporary LLMs" (Section 4) and that MATH is appropriate because test-time compute "is expected to help most when the model already possesses the necessary knowledge and the challenge is drawing complex inferences." However, neither the model nor the benchmark is varied, leaving the generality of all findings unverified.

The consequence. Several aspects of the results could be model-specific or benchmark-specific:

The PRM's quality and over-optimization behavior depend on PaLM 2-S*'s output distribution — different models with different calibration properties or error patterns would produce different difficulty-dependent scaling curves.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families (Llama, Mistral, GPT, Gemini).
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning. The difficulty-dependent patterns — beam search hurting easy problems, revisions helping easy problems, search helping medium problems — may not generalize to other reasoning domains (code generation, logical reasoning, scientific question-answering) or to tasks requiring factual knowledge rather than multi-step inference.

What evidence exists in the paper. None — the paper provides no cross-model or cross-benchmark evaluation. The claim of representativeness is an assertion, not a demonstrated fact. The paper does not cite evidence that PaLM 2-S* is representative of other model families in terms of its test-time compute scaling behavior.

Mitigation status. The authors acknowledge the limitation implicitly in Section 4 (describing the model as "representative" rather than "proven general") and in Section 8 (future work on "extension to other domains and modalities"). However, this is a scope statement, not a mitigation. A practitioner considering applying the compute-optimal framework to their own model and task distribution cannot predict whether the 4× efficiency gains or the difficulty-dependent strategy recommendations will transfer.

Sequential Revision Strategies Introduce Latency That Is Not Accounted For

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock time. Sequential revisions are inherently serial — each revision depends on the previous one, so a chain of N revisions takes N serial generation steps — while parallel best-of-N can be executed simultaneously given sufficient hardware. The compute-optimal policy, which favors sequential revisions on easy problems (Figure 7, right), trades FLOPs efficiency for latency.

The consequence. A strategy that allocates 64 generations as 64 sequential revisions takes roughly 64× longer wall-clock time than one that runs 64 parallel samples simultaneously, even though the total FLOPs are identical. For latency-sensitive applications — interactive assistants, real-time decision-making, clinical triage — the sequential-heavy strategies favored by the compute-optimal policy on easy and medium problems may be impractical regardless of their accuracy advantages. In practice, a deployment might need to optimize a latency-adjusted utility function that penalizes serial computation, which the paper's "generations" metric ignores entirely. The compute-optimal allocation under such a function could look very different from the reported policy.

What evidence exists in the paper. Figure 7 (left) shows that at lower budgets (8–32 generations), fully sequential is optimal — the curves monotonically increase with the sequential-to-parallel ratio. At 256 generations, the optimal ratio is around 2^1 to 2^3 (2:1 to 8:1 sequential-to-parallel), meaning the majority of generations are sequential. The paper does not report wall-clock times for any configuration, nor does it discuss the latency-throughput tradeoff. Section 3.3 describes the revision model as generating "a chain of revisions" where "each revision depends on the previous one," making the serial dependency clear, but the implications for latency are never analyzed.

Mitigation status. Not addressed. The paper's compute budget model (Section 5.3: "one generation equals one complete sampled answer") explicitly defines cost as total generation count, ignoring parallelism. There is no latency-adjusted analysis, no discussion of hardware assumptions (how many parallel samples are feasible simultaneously), and no acknowledgment that the sequential-heavy optimal policy introduces a latency penalty. The limitation is entirely unremarked. A practitioner deploying this in a latency-sensitive setting would need to independently evaluate the latency-accuracy Pareto frontier.

Hard Problems in the Difficulty Estimation Cost and Revision Model Training Are Not Amortized or Validated

The assumption or constraint. The paper makes two specific methodological choices that involve expensive upfront computation: (1) difficulty estimation via 2,048 samples per question, and (2) revision model training data construction via 64 parallel samples per training question plus edit-distance-based pairing. Both are one-time costs incurred during development or pre-deployment, and their scale raises questions about practical reproducibility and amortization.

The consequence.

For difficulty estimation: The 2,048-sample estimation cost must be paid per question at inference time (unless a separate difficulty predictor is trained). For a system answering thousands of unique questions in deployment, this cost either makes the approach infeasible or must be amortized across many queries to the same question — but MATH questions are unique, so amortization over repeated queries is unrealistic. The paper provides no analysis of how many repeated queries to the same question would be needed to make the estimation cost worthwhile.
For revision model training: The training data construction uses 64 parallel samples per training question at elevated temperature, followed by edit-distance-based pairing of incorrect and correct answers. This is computationally intensive (64× the cost of standard fine-tuning data generation) and depends on a specific heuristic (character-level edit distance) whose necessity is not ablated. The paper does not demonstrate that simpler — and cheaper — data construction methods (e.g., random incorrect-correct pairing, fewer parallel samples) would yield comparable revision model quality.

What evidence exists in the paper. Section 3.2 describes the 2,048-sample difficulty estimation procedure. Section 6.1 describes the revision training data construction: "sample 64 responses in parallel from the base LLM at elevated temperature" and "the last incorrect answer in the sequence is selected to be the one with the smallest character-level edit distance to the correct answer." The paper does not report the computational cost of either step in GPU-hours, does not compare performance against cheaper alternatives for revision data construction, and does not analyze the amortization breakeven point for difficulty estimation.

Mitigation status. For difficulty estimation, the paper acknowledges the cost and calls it a future work direction (Section 3.2: "exploration-exploitation tradeoff"), but provides no partial solution. For revision training, the edit-distance heuristic is justified as ensuring "the incorrect answer is close to the correct one — similar in structure but containing a mistake — so the model learns to make targeted edits," but this justification is conceptual, not empirical. No ablation compares edit-distance pairing against random pairing or other heuristics. Both limitations remain open — the practical cost of implementing the full framework (difficulty estimation + revision training) may be substantially higher than the headline 4× efficiency numbers suggest when these upfront investments are included.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around clinical language model development from a data-acquisition problem to a data-structuring problem. The dominant narrative in medical NLP has been that clinical models are bottlenecked by access to protected health information — if only we had more MIMIC-scale corpora, more discharge summaries, more doctor-patient conversations under permissive licenses, the models would improve. The MediPhi framework challenges this by demonstrating that a carefully structured adaptation pipeline — pre-instruction tuning on public medical corpora, modular expert training, and merging-based forgetting recovery — can produce a 3.8B model that approaches parity with an 8B general-purpose model on clinical tasks (43.4 vs. 44.1 average CLUE+ accuracy; Table 5) without requiring access to real patient data during training.

This is a reframing, not a paradigm shift. The individual components (PIT, model merging, synthetic instruction generation) are known techniques. What is novel is their deliberate orchestration into a pipeline where each stage solves a specific problem created by the previous stage: PIT injects domain knowledge but causes catastrophic forgetting; SLERP merging recovers lost general capabilities; separate training avoids interference between incompatible data types; BreadCrumbs merging unifies specialists with minimal destructive interference; MediFlow alignment teaches task execution on top of the acquired knowledge; DPO refines with contrastive preference signals. The paper's contribution is the design logic connecting these stages, not any single algorithmic innovation.

This reframing has several concrete consequences for how the field approaches clinical model development:

It demotes data volume as the primary constraint and elevates data organisation as the primary design lever. The paper shows empirically that training separate experts on corpus groups and merging them (38.5 average for individual SLERP experts, 39.3 for BreadCrumbs) outperforms training a single model on all data pooled together (37.5 for DataMix; Table 3). This is not a data volume effect — the total training tokens are identical. It is an organisational effect: different data types (scientific articles, clinical documents, coding taxonomies, guidelines, encyclopedia entries) have different statistical properties that interfere when mixed during training, and modular training isolates these interference effects. The implication is that future clinical model development should invest at least as much effort in data taxonomy and expert partitioning as in data collection — knowing how to split the data may matter as much as having more of it.

It reconciles contradictory findings about medical domain adaptation. Prior work has produced mixed results: some studies show domain-adapted models improving over their base (BioMistral, Med42), while broader evaluations like Dada et al. (2024) show most medical LLMs failing to outperform their general-purpose foundations on clinical tasks. The MediPhi results explain this contradiction through the lens of forgetting: domain adaptation does teach useful knowledge, but it also degrades general capabilities (Table 2: Guideline without SLERP drops to 27.2, a 9.3-point loss), and whether the net effect is positive depends on the ratio of knowledge gained to capability lost. Prior work that showed positive results may have used adaptation methods that implicitly mitigated forgetting (large models where forgetting is less severe, data mixing strategies that dilute the domain signal). Prior work that showed negative results may have used methods where forgetting dominated. The resolution is that domain adaptation effectiveness is a function of the forgetting recovery mechanism, not the adaptation method alone — a testable hypothesis that the field can now systematically investigate.

It introduces CVΔ as a deployment-relevant evaluation criterion. The coefficient of variation of gains/losses (Equation 1) operationalises a concern that clinical practitioners have but the research community has largely ignored: uniformity of improvement across tasks. The paper uses CVΔ as a model selection criterion — choosing BreadCrumbs over Task-Arithmetic despite a 0.1-point average accuracy disadvantage because BreadCrumbs has lower CVΔ (1.5 vs. 1.9) and improves more datasets (11 vs. 9). This encodes the judgment that a model with a 5-point gain on one task and a 3-point loss on another is worse for deployment than a model with a 2-point gain on both, even though the averages are identical. If adopted more broadly, CVΔ would shift the field's evaluation norms away from leaderboard-chasing on average accuracy toward robustness profiling — a shift that is overdue in safety-critical domains like healthcare.

It makes the case for small models as a deliberate design choice rather than a compromise. The paper's central comparative result — a 3.8B model approaching a general-purpose 8B model on clinical tasks — is not presented as "look how well we did despite the size constraint," but as evidence that targeted adaptation can be a more efficient use of compute than scaling. This inverts the usual framing where small models are positioned as a fallback for resource-constrained settings. The paper argues implicitly that if you have a fixed compute budget (for training + inference), spending it on a smaller model with sophisticated adaptation may yield better clinical performance than spending it on a larger model with generic pretraining. The FLOPs analysis in the Limitations section (12,000 GPU-hours total, of which 3,600 for the final model) provides a concrete cost anchor: this is a modest compute budget by modern LLM standards, yet it produces a model competitive with Llama-3-8B on clinical tasks.

It redirects research attention toward forgetting recovery as a first-class problem. The Guideline ablation results (Table 2) are stark: PIT alone loses 9.3 points; SLERP alone recovers 3.0 points; together they gain 2.7. The synergistic interaction — the combined benefit exceeds the sum of individual benefits — suggests that merging does not simply "undo" forgetting but somehow amplifies the domain knowledge by re-integrating it with instruction-following capabilities. Understanding this synergy — why recombining expert and base parameters produces a model better than either — is a research question the paper opens but does not resolve. It makes model merging a more central research topic for domain adaptation, rather than a post-hoc optimisation trick.

Research directions that become more attractive:

Modular domain adaptation with merging as the default approach, replacing end-to-end domain training. The paper provides a template (PIT → SLERP → BreadCrumbs) that can be replicated for other specialised domains (legal, financial, scientific) with domain-appropriate corpus groups and task types.
Difficulty estimation and curriculum design for synthetic instruction generation. The paper's difficulty-skewed MediFlow (3:1 ratio favouring hard/very hard/extreme) is motivated but not ablated. Investigating the optimal difficulty distribution for alignment data — and whether anti-curriculum (hard examples first) genuinely transfers downward better than standard curriculum — is a research programme the paper's dataset enables.
Forgetting quantification and recovery mechanism design. The CVΔ metric and the Guideline ablations provide tools and baselines for studying catastrophic forgetting in domain-adapted models. What other recovery mechanisms (KL regularisation during training, elastic weight consolidation, progressive network expansion) match or exceed SLERP merging for capability preservation?

Research directions that become less attractive:

Simply collecting more clinical data as a strategy for improving clinical models. The paper demonstrates that structuring available public data through PIT can produce gains comparable to or exceeding those from larger but unstructured training corpora. Data volume remains valuable, but data organisation may have higher marginal returns, especially for small models.
End-to-end training on pooled domain data (the DataMix approach). The paper shows this underperforms modular training + merging by a consistent margin, suggesting the field should move past the "train on everything" paradigm for domain adaptation.

Follow-Up Research This Work Enables

1. Ablation of the PIT task type across all corpus groups to determine whether summarisation is universally optimal or corpus-dependent. The paper established summarisation as the best PIT task for ICD-10 coding webpages (Figure 4), then applied summarisation to all five corpus groups without testing whether QA, NER, or RE would perform better for clinical documents, scientific articles, or encyclopedia text. A direct follow-up would train PIT experts for each of the five corpus groups with each of the four task types (4 × 5 = 20 experts), SLERP-merge them, evaluate on CLUE+, and measure whether the optimal task type varies with corpus type. The hypothesis is that summarisation dominates for dense, fact-packed corpora (coding webpages, guidelines) while QA may be better for narrative corpora (scientific articles, clinical documents) because QA teaches targeted information retrieval from long texts — a skill that transfers to clinical QA tasks like MeDiSumQA and LongHealth. This experiment would also test whether the paper's decision to default to summarisation for all experts left performance on the table for non-coding tasks.

2. Difficulty-balanced vs. difficulty-skewed MediFlow alignment comparison. The paper skews MediFlow toward hard/very hard/extreme examples at a 3:1 ratio, arguing that mastery of hard cases transfers to easy ones (Appendix A.4). This hypothesis is plausible but entirely untested. A direct follow-up would generate a difficulty-balanced variant of MediFlow (equal proportions across all six difficulty levels, controlling for total instruction count and quality filtering threshold), train MediPhi-SFT on both variants, and compare CLUE+ performance. If the difficulty-skewed variant outperforms, it validates the anti-curriculum principle; if the balanced variant outperforms, it suggests that easy examples provide a necessary scaffold for learning. A stronger version would include a difficulty-progressive variant (easy → moderate → hard → very hard → extreme over the course of training) to test whether standard curriculum learning outperforms both the balanced and anti-curriculum approaches. The MediFlow dataset structure (difficulty labels on all 2.5M instructions) makes this experiment straightforward.

3. Validation of the synthetic merge-optimisation validation sets against CLUE+ to determine whether evolutionary merge search produces genuinely optimal configurations. The paper uses twelve GPT-4o-generated validation sets to guide the evolutionary algorithm that optimises BreadCrumbs merge weights (Section 3.1.4), but never reports the correlation between validation set accuracy and CLUE+ test accuracy. A strong follow-up would compute this correlation — both overall and per-dataset — and compare the BreadCrumbs configuration selected by the synthetic validation sets against configurations selected by direct optimisation on a held-out CLUE+ subset (which would constitute a valid held-out evaluation, not test-set contamination, if the held-out portion is excluded from final reporting). If the synthetic-validation-selected configuration significantly underperforms the directly-optimised configuration, it suggests the synthetic sets are a poor proxy and the merge optimisation procedure needs redesign. If they closely match, it validates the synthetic validation approach as a general tool for merge configuration when real validation data is unavailable — a finding with implications beyond clinical NLP.

4. Cross-model replication of the PIT + merging framework on Llama and Mistral architectures. The entire paper uses Phi-3.5-mini as the base model. A replication study applying the same pipeline — PIT with summarisation on the five corpus groups, SLERP merging, BreadCrumbs unification, MediFlow alignment — to Mistral-7B-Instruct-v0.1 and Meta-Llama-3-8B-Instruct would test the generality of the findings. Key measurements: (a) does the relative gain from PIT + merging (+2.8 over baseline on CLUE+ for Phi-3.5-mini) scale with model size, architecture, or base capability? (b) does the optimal PIT task type (summarisation) transfer, or do different architectures benefit from different task types? (c) does the BreadCrumbs merging advantage over Task-Arithmetic (more uniform gains) hold across architectures? This experiment is feasible because all five corpus groups are public and MediFlow is released under a permissive license — the main cost is compute for PIT training and SLERP merging, which the paper's reported GPU-hours (12,000 total) provide a budget estimate for.

5. Dynamic difficulty assessment and adaptive strategy allocation for the compute-optimal policy, amortising difficulty estimation into the solution process. While this paper does not propose test-time compute scaling strategies, a natural extension inspired by its modular framework would be to treat the expert models as an ensemble and dynamically route queries. Specifically, a follow-up could develop a method where an initial small sampling budget (e.g., 8–16 generations) is used to estimate both the difficulty of a query and which SLERP expert is most likely to succeed, then the remaining budget is allocated accordingly — routing coding questions to the MedCode expert, radiology questions to the Clinical expert, and general medical questions to the BreadCrumbs-merged MediPhi. The paper's individual expert results (Table 2) show dramatic specialisation (Clinical: +19.7 on MEDEC, MedCode: +19.4 on ICD10CM), and an oracle router that always selects the best expert per dataset would achieve 43.1 average accuracy (Table 3, SLERP Maximum) — above MediPhi-Instruct's 43.4. A learned router that approaches this oracle performance would both improve upon the static merged model and provide a practical alternative to the expensive difficulty estimation approach.

6. DPO preference data construction ablation to determine whether marginally wrong outputs are necessary or whether random incorrect outputs suffice. The paper generates rejected DPO outputs by prompting GPT-4o with twelve specific error types to produce "marginally wrong" responses (Appendix A.7). The rationale is that subtle errors teach nuanced discrimination, while obvious errors teach a trivial preference boundary. This is plausible but untested. A follow-up would construct DPO datasets with three types of rejected outputs: (a) marginally wrong (the paper's method), (b) randomly incorrect (outputs from MediFlow where the judge quality score is in the bottom quartile), and (c) obviously incorrect (outputs from a different task type altogether). Training MediPhi-SFT on each DPO variant and evaluating on CLUE+ would measure whether the marginal-wrongness construction is actually responsible for the 0.4-point DPO gain (Table 4) or whether any contrastive signal produces equivalent improvement. A stronger version would also test whether DPO gains are limited to tasks well-represented in the DPO training data (the 130K preference pairs) or transfer to under-represented tasks — the paper's observation that DPO degrades Problem List Summarization while improving other tasks (Table 11: 26.9 → 26.0) suggests the transfer may be negative for poorly covered tasks.

Practical Applications and Downstream Use Cases

1. On-premises clinical NLP deployment in hospital systems with data sovereignty requirements. The MediPhi-Instruct model at 3.8B parameters can run on a single A100 GPU (80GB) or even a high-end consumer GPU, making it feasible for deployment within a hospital's firewall where protected health information cannot leave the institution. The model's CLUE+ performance profile — strong on radiology report QA (61.6, +20.4 over base; Table 12), medication QA (19.3, +8.1), and clinical error detection (34.4, +19.6) — covers tasks that map directly to hospital workflows: a radiologist querying findings from a report, a pharmacist checking medication interactions, a quality assurance system flagging potential errors in clinical notes. The model's ICD-10 coding capability (54.9, competitive with Llama-3-8B at 25.7) supports billing and coding departments. The commercially permissive license removes the legal barriers that have limited adoption of models like Med-PaLM 2 (proprietary) or Llama-3-Med42 (research-focused). A hospital could deploy MediPhi-Instruct as a private API endpoint serving multiple departments, with the 3.8B size keeping inference latency low enough for interactive use — a key deployment consideration that the paper highlights in Section 1.

2. Synthetic clinical data generation for training and evaluation in low-resource settings. The MediFlow dataset (2.5M instructions, 14 task types, 98 document types, permissive license) is a resource that clinical NLP groups in low-resource languages or specialised medical subdomains can use as a template. A team working on clinical NLP for German, Japanese, or Arabic could translate MediFlow's task definitions and document type descriptions, prompt a strong local LLM (or GPT-4o) to generate language-specific instructions following the same parameterised pipeline, and fine-tune their own small clinical model. The paper's agentic generation architecture (Figure 3) with LLM-as-a-Judge filtering provides a replicable blueprint — the meta-prompt is included in Appendix A.4, the judge criteria in Appendix A.5, and the error types for DPO in Appendix A.7. The 98 document types with granularity levels (Appendix A.4) cover a wide enough range of clinical workflows (from nursing notes to operative reports to psychiatric evaluations) to serve as a comprehensive starting point. The release of the MediFlow dataset under a permissive license means groups can also directly fine-tune on the English data and then adapt to their target language through continued training, rather than generating from scratch.

3. Modular model updating for evolving clinical knowledge without full retraining. Clinical knowledge evolves: new ICD codes are introduced, clinical guidelines are updated, new medications enter the market, and diagnostic criteria change. The traditional approach — retrain the entire model on updated data — is computationally expensive and risks degrading capabilities on unchanged domains. The MediPhi framework's modular architecture naturally supports incremental updating: when ICD-11 replaces ICD-10, only the MedCode expert needs to be retrained on the new coding webpages (applying PIT with summarisation, SLERP-merging with the base), and the BreadCrumbs merge can be re-optimised to incorporate the updated expert into MediPhi without retraining the PubMed, Clinical, MedWiki, or Guidelines experts. The synthetic validation sets (Section 3.1.4, Appendix A.3) provide the fitness signal for re-optimising the merge weights after any expert update. This reduces the cost of model maintenance from full retraining (12,000 GPU-hours for the entire pipeline) to expert-level retraining (roughly one-fifth of the cost) plus merge re-optimisation (500 evaluations on validation sets). The paper's release of these validation sets is specifically designed to enable this use case.

4. Benchmarking and auditing clinical models for uniformity of capability. The CLUE+ benchmark (12 datasets, Table 10) and the CVΔ metric (Equation 1) provide a practical toolkit for evaluating whether a clinical model is safe to deploy. A hospital evaluating a commercial clinical NLP API could run the model on CLUE+, compute not just average accuracy but also CVΔ and #DG, and use these metrics to identify blind spots — tasks where the model's performance degrades relative to a known baseline. The paper's detailed per-dataset results (Tables 11 and 12) provide reference values for Phi-3.5-mini, MediPhi variants, BioMistral, Llama-3, and Med42, giving evaluators a sense of expected performance ranges at the 3.8B–8B scale. A model that achieves high average accuracy but has CVΔ > 5 (like Med42 at 7.8) would raise a red flag that its gains are concentrated on a few tasks while others may be dangerously degraded — exactly the kind of failure mode that average-accuracy leaderboards conceal. The CLUE+ benchmark's inclusion of diverse input document types (discharge summaries, radiology reports, doctor-patient dialogs, nursing notes, progress notes) means the evaluation covers the document diversity a hospital actually encounters, not just curated exam-style questions.

When to Prefer This Method

The paper does not articulate an explicit tradeoff against named alternatives (it positions MediPhi as a novel framework rather than as a choice between existing methods), so a prescriptive "prefer X when Y" decision matrix would be a fabrication. However, the paper's results imply several conditional preferences that a practitioner could reasonably derive:

Prefer the modular PIT + merging framework over end-to-end domain training (DataMix) when the available domain data consists of heterogeneous corpora with different formats, densities, and learning characteristics (scientific articles vs. coding webpages vs. clinical guidelines vs. patient documents). The evidence is Table 3: BreadCrumbs (39.3) outperforms DataMix (37.5) by 1.8 points on CLUE+, with more uniform gains (#DG: 11 vs. 10). The modular approach's advantage likely grows with corpus heterogeneity.
Prefer SLERP merging over no merging after domain adaptation when the base model possesses general capabilities (instruction following, long-context handling, multilingual support) that are essential for the target tasks but not explicitly reinforced during domain training. The evidence is Table 2: Guideline without SLERP loses 9.3 points (39.2 → 27.2); Guideline with SLERP gains 2.7 points over baseline. The merging step is not optional — it is the difference between net improvement and catastrophic degradation.
Prefer BreadCrumbs over Task-Arithmetic for multi-expert merging when the deployment requires uniform performance across diverse tasks and cannot tolerate blind spots — i.e., when CVΔ matters as much as average accuracy. The evidence is Table 3: BreadCrumbs has CVΔ = 1.5 and #DG = 11 vs. Task-Arithmetic's CVΔ = 1.9 and #DG = 9, at a cost of only 0.1 average accuracy points. In safety-critical clinical deployment, that tradeoff is almost certainly worthwhile.
Prefer quality-filtered SFT over full-dataset SFT when the synthetic instruction data has variable quality and the quality assessment signal (e.g., LLM-as-a-Judge scores) is reliable. The evidence is Table 4: SFT 800K (43.0) outperforms SFT 2.5M (41.9) by 1.1 points, demonstrating that more data can actively harm performance when lower-quality instructions introduce noise. This finding is specific to synthetic data where quality variance is high; it may not generalise to human-curated instruction datasets.
Consider the full framework (PIT + merging + alignment) rather than alignment alone when the base model's domain knowledge is insufficient for the target tasks. The evidence is Table 4: MediPhi + SFT 800K (43.0) outperforms Phi-3.5-mini + SFT 800K (42.2) by 0.8 points. The marginal gain from domain knowledge acquisition is smaller than the gain from alignment (0.8 vs. 5.7), but it is consistent and persistent — and in absolute terms, the difference between a 42.2 model and a 43.4 model (MediPhi-Instruct with DPO) may be consequential for clinical tasks where every percentage point affects patient outcomes.

The paper does not provide evidence for preferring this framework over simply using a larger general-purpose model (Llama-3-8B at 44.1 vs. MediPhi-Instruct at 43.4) — the difference is small and likely statistically indistinguishable without confidence intervals. The choice between them depends on deployment constraints (on-premises vs. cloud, GPU budget, latency requirements) rather than raw performance. The paper's value proposition is not "our 3.8B model beats your 8B model" but "our framework produces a 3.8B model that approaches your 8B model using only public data and a modest compute budget, with a commercially permissive license."